> ## Documentation Index
> Fetch the complete documentation index at: https://anam.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Custom TTS (client-side)
> Use your own text-to-speech provider with Anam avatars via audio passthrough mode.
**Want to use ElevenLabs Agents with Anam?** We recommend the [server-side ElevenLabs integration](https://anam.ai/cookbook/elevenlabs-server-side-agents) instead—it's simpler and has lower latency. This page covers the client-side approach for when you need direct control over the audio pipeline.
This guide shows how to use Anam's **audio passthrough** mode to pipe externally-generated speech audio into an avatar for real-time lip-sync. The example below uses [ElevenLabs Conversational AI](https://elevenlabs.io/conversational-ai) as the TTS source, but the same pattern works with **any TTS provider** (Cartesia, PlayHT, Azure Speech, Google Cloud TTS, etc.)—you just need to deliver PCM audio chunks to the Anam SDK.
Full source code for the ElevenLabs conversational agent with Anam avatar (client-side).
## How It Works
The integration uses Anam's **audio passthrough** mode, where Anam renders an avatar that lip-syncs to audio you provide—without using Anam's own AI or microphone input.
**Bring Your Own Voice**: Your TTS provider generates the speech audio. Anam renders the lip-synced avatar video.
## Quick Start
### Prerequisites
* An account with your TTS provider (ElevenLabs used in this example)
* [Anam](https://anam.ai) account with API access
* Node.js or Bun runtime
* Modern browser with WebRTC support (Chrome, Firefox, Safari, Edge)
### Installation
```bash theme={"system"}
npm install @anam-ai/js-sdk chatdio
```
`chatdio` provides microphone capture utilities used to send user audio to ElevenLabs.
### Basic Integration
Here's the core pattern for connecting an external TTS source to Anam:
```typescript theme={"system"}
import { createClient } from "@anam-ai/js-sdk";
// 1. Create Anam client with audio passthrough session
const anamClient = createClient(sessionToken, {
disableInputAudio: true, // Your TTS provider handles microphone
});
await anamClient.streamToVideoElement("video-element");
// 2. Create agent audio input stream
const audioInputStream = anamClient.createAgentAudioInputStream({
encoding: "pcm_s16le",
sampleRate: 16000,
channels: 1,
});
// 3. Connect to your TTS provider and forward audio
// (ElevenLabs WebSocket shown here as an example)
const ws = new WebSocket(`wss://api.elevenlabs.io/v1/convai/conversation?agent_id=${agentId}`);
ws.onmessage = (event) => {
const msg = JSON.parse(event.data);
if (msg.type === "audio" && msg.audio_event?.audio_base_64) {
// Forward audio chunks to Anam for lip-sync
audioInputStream.sendAudioChunk(msg.audio_event.audio_base_64);
}
if (msg.type === "agent_response_complete") {
// Signal that ElevenLabs completed this response
audioInputStream.endSequence();
}
if (msg.type === "interruption") {
// Handle barge-in: stop the avatar animation and end the audio sequence
anamClient.interruptPersona();
audioInputStream.endSequence();
}
};
```
## Full Example
### Project Structure
```
src/
├── client.ts # Main client orchestration
├── elevenlabs.ts # ElevenLabs WebSocket handling
└── routes/
└── api/
└── config.ts # Server-side session token endpoint
```
### Server: Create Anam Session
Your server creates an Anam session token with `enableAudioPassthrough: true`:
```typescript config.ts theme={"system"}
const response = await fetch("https://api.anam.ai/v1/auth/session-token", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${ANAM_API_KEY}`,
},
body: JSON.stringify({
personaConfig: {
avatarId: AVATAR_ID,
avatarModel: "cara-4",
enableAudioPassthrough: true, // Enable external audio input
},
}),
});
const { sessionToken } = await response.json();
```
### Client: ElevenLabs Module
Handle the WebSocket connection and microphone capture:
```typescript elevenlabs.ts theme={"system"}
import { MicrophoneCapture, arrayBufferToBase64 } from "chatdio";
const SAMPLE_RATE = 16000;
export interface ElevenLabsCallbacks {
onReady?: () => void;
onAudio?: (base64Audio: string) => void;
onUserTranscript?: (text: string) => void;
onAgentResponse?: (text: string) => void;
onAgentResponseComplete?: () => void;
onInterrupt?: () => void;
onError?: () => void;
onDisconnect?: () => void;
}
export async function connectElevenLabs(agentId: string, callbacks: ElevenLabsCallbacks) {
const ws = new WebSocket(`wss://api.elevenlabs.io/v1/convai/conversation?agent_id=${agentId}`);
// Set up microphone capture
const mic = new MicrophoneCapture({
sampleRate: SAMPLE_RATE,
echoCancellation: true,
noiseSuppression: true,
});
mic.on("data", (data: ArrayBuffer) => {
if (ws.readyState === WebSocket.OPEN) {
ws.send(
JSON.stringify({
user_audio_chunk: arrayBufferToBase64(data),
})
);
}
});
ws.onopen = async () => {
await mic.start();
callbacks.onReady?.();
};
ws.onmessage = (event) => {
const msg = JSON.parse(event.data);
switch (msg.type) {
case "audio":
callbacks.onAudio?.(msg.audio_event.audio_base_64);
break;
case "agent_response":
callbacks.onAgentResponse?.(msg.agent_response_event.agent_response);
break;
case "agent_response_complete":
callbacks.onAgentResponseComplete?.();
break;
case "user_transcript":
callbacks.onUserTranscript?.(msg.user_transcription_event.user_transcript);
break;
case "interruption":
callbacks.onInterrupt?.();
break;
case "ping":
ws.send(JSON.stringify({ type: "pong", event_id: msg.ping_event.event_id }));
break;
}
};
ws.onclose = () => {
mic.stop();
callbacks.onDisconnect?.();
};
}
```
### Client: Main Integration
Wire everything together:
```typescript client.ts theme={"system"}
import { createClient } from "@anam-ai/js-sdk";
import { connectElevenLabs } from "./elevenlabs";
let anamClient: ReturnType | undefined;
async function startConversation() {
// Get session config from your server
const { anamSessionToken, elevenLabsAgentId } = await fetch("/api/config").then((r) => r.json());
// Initialize Anam avatar (disable input audio since ElevenLabs handles mic)
anamClient = createClient(anamSessionToken, {
disableInputAudio: true,
});
await anamClient.streamToVideoElement("anam-video");
// Create agent audio input stream
const audioInputStream = anamClient.createAgentAudioInputStream({
encoding: "pcm_s16le",
sampleRate: 16000,
channels: 1,
});
// Connect to ElevenLabs
await connectElevenLabs(elevenLabsAgentId, {
onAudio: (audio) => {
audioInputStream.sendAudioChunk(audio);
},
onAgentResponseComplete: () => {
audioInputStream.endSequence();
},
onInterrupt: () => {
anamClient.interruptPersona();
audioInputStream.endSequence();
},
});
}
```
### Cleanup
Stop the conversation and release resources:
```typescript theme={"system"}
async function stopConversation() {
await anamClient?.stopStreaming();
}
```
## Configuration
### Environment Variables
You'll need credentials from both services:
| Service | Where to get it |
| -------------- | -------------------------------------------------------- |
| **Anam** | [lab.anam.ai](https://lab.anam.ai) → Settings → API Keys |
| **ElevenLabs** | [elevenlabs.io](https://elevenlabs.io) → Agents |
```bash .env theme={"system"}
# Anam credentials
ANAM_API_KEY=your_anam_api_key
ANAM_AVATAR_ID=your_avatar_id
# ElevenLabs credentials
ELEVENLABS_AGENT_ID=your_agent_id
```
### ElevenLabs Agent Setup
When configuring your ElevenLabs agent, set the output audio format to match Anam's expectations:
| Setting | Value |
| --------------- | ---------- |
| **Format** | PCM 16-bit |
| **Sample Rate** | 16000 Hz |
| **Channels** | Mono |
Enable the `agent_response_complete` and `interruption` events in the agent's **Advanced → Client Events** settings. The `agent_response` event is sent with the first audio chunk, so it does not indicate that the audio sequence is complete. See [ElevenLabs client events](https://elevenlabs.io/docs/eleven-agents/customization/events/client-events).
The stream configuration must match the audio you send. This example uses signed 16-bit PCM at 16 kHz with one channel.
### Choosing an Avatar
Browse ready-to-use avatars in our gallery. Copy the avatar ID directly into your config.
Create your own personalized avatar in Anam Lab with custom appearance and style.
## Audio Passthrough API
To guide avatar expression in audio passthrough sessions, send Director Notes cues over the data channel. See [Director Notes](/docs/personas/director-notes#send-cues-over-the-data-channel).
### createAgentAudioInputStream()
Creates a stream for sending audio chunks to the avatar for lip-sync. Must be called **after** `streamToVideoElement()` resolves (the session must be started first).
```typescript theme={"system"}
const audioInputStream = anamClient.createAgentAudioInputStream({
encoding: "pcm_s16le",
sampleRate: 16000,
channels: 1,
});
```
Audio encoding format. Only `pcm_s16le` (16-bit signed little-endian PCM) is supported.
Sample rate in Hz. This must match the PCM audio sent to the stream.
Number of audio channels in each chunk. Use `1` for mono or `2` for stereo.
### sendAudioChunk()
Send a PCM audio chunk for lip-sync rendering. The method accepts an `ArrayBuffer`, `Uint8Array`, or base64-encoded string. The SDK converts binary input to base64, attaches the stream configuration, and increments the sequence number for each chunk.
```typescript theme={"system"}
function forwardTtsAudio(chunk: ArrayBuffer | Uint8Array | string) {
audioInputStream.sendAudioChunk(chunk);
}
```
### endSequence()
Signal that the current audio sequence has ended. The SDK sends an end-of-sequence message and resets its sequence number to `0` for the next turn.
```typescript theme={"system"}
audioInputStream.endSequence();
```
Call this when:
* Your TTS provider confirms that it has sent the final audio chunk for the turn
* The user interrupts (barge-in)
Do not call `endSequence()` when response text first becomes available if the provider can still send audio for that response.
For diagnostics, `getSequenceNumber()` returns the number of chunks sent in the current sequence, and `getConfig()` returns the stream's audio configuration.
## Handling Interruptions
When a user speaks while the agent is talking (barge-in), your TTS provider sends an interruption event. Handle it by interrupting the avatar and ending the audio sequence:
```typescript theme={"system"}
onInterrupt: () => {
anamClient.interruptPersona();
audioInputStream.endSequence();
},
```
`interruptPersona()` stops the avatar's current speech. `endSequence()` closes the current input sequence and resets chunk numbering. Call both so the avatar stops and the next utterance starts a new sequence.
## Optimize latency
Time to first visible speech depends on your TTS provider's time to first byte (TTFB) and how quickly you deliver enough audio for Anam to begin rendering. Anam needs to receive **800 ms of audio** before it can start frame generation.
Optimize both parts of the path:
* **Reduce TTS TTFB.** Use a low-latency TTS model, reuse connections where the provider supports it, and send text to the provider as soon as it is ready.
* **Forward the first audio immediately.** Do not wait for the full response before calling `sendAudioChunk()`.
* **Send small chunks frequently and faster than real time.** The 800 ms requirement refers to audio duration, not 800 ms of wall-clock time. High-throughput delivery fills the initial buffer sooner, while sustained delivery slower than real time can cause stutter or dropped frames.
* **Minimize network distance.** Where your architecture permits, place server-side TTS and orchestration close to Anam's `eu-south` infrastructure for `eu` sessions or `us-east` infrastructure for `us` sessions. Use `eu` or `us` when [selecting a session region](/docs/personas/session/regions). In a browser-only integration, audio still travels through the user's client, so server co-location cannot remove that network leg.
For lower latency requirements, consider using Anam's [turnkey solution](https://anam.ai/docs/javascript-sdk/quickstart) which handles STT, LLM, and TTS in an optimized pipeline, or the [server-side ElevenLabs integration](https://anam.ai/cookbook/elevenlabs-server-side-agents) which reduces latency through server-to-server audio flow.
## Browser compatibility
The integration requires WebRTC support. Tested browsers:
| Browser | Support |
| ----------- | ------------ |
| Chrome 80+ | Full support |
| Firefox 75+ | Full support |
| Safari 14+ | Full support |
| Edge 80+ | Full support |
Mobile browsers are supported but may have higher latency on cellular networks.
## Billing
When using audio passthrough mode:
* **Anam**: Billed for avatar streaming time (session duration)
* **TTS Provider**: Billed separately for STT, LLM, and TTS usage
Check both [Anam pricing](https://anam.ai/pricing) and your TTS provider's pricing to understand total costs.
## When to Use This Approach
This **client-side** approach is a good fit when you:
* Need direct control over the audio pipeline in the browser
* Want to use client-side tools with your TTS provider's agent
* Have an existing client-side integration you want to add avatars to
For most new projects, we recommend the **[server-side integration](https://anam.ai/cookbook/elevenlabs-server-side-agents)** instead—it's simpler to set up and has lower latency.
## Troubleshooting
* Verify audio format matches (PCM16, 16kHz, mono)
* Check that `sendAudioChunk()` is receiving data
* Ensure the audio input stream was created successfully
* Look for errors in browser console
* Call `endSequence()` after the final audio chunk for each response
* Ensure you're handling interruptions correctly
* Check network latency to both services
* Verify your TTS provider agent is configured correctly
* Check the WebSocket connection is established
* Look for audio events in the message handler
* Confirm your agent ID is correct
* Check browser permissions for microphone access
* Ensure `echoCancellation` is enabled to prevent feedback
* Verify the microphone is sending data at 16kHz
* Verify your `ANAM_API_KEY` is valid
* Check that `enableAudioPassthrough: true` is set in the session request
* Ensure the avatar ID exists in your account
## Resources
Recommended: simpler setup with lower latency
Full source code for the client-side integration
Guide to using ElevenLabs V3 expressive voices with Anam
Browse available stock avatars
