Server-Side ElevenLabs Agents with Anam Avatars

Overview

Anam’s server-side ElevenLabs integration connects an Anam avatar to an ElevenLabs Conversational AI agent. Your server fetches a short-lived ElevenLabs signed URL and includes it when creating an Anam session token.

The browser uses the Anam JavaScript SDK to start the WebRTC session and render the avatar. Anam sends conversation audio to ElevenLabs, receives the generated speech, and keeps the avatar in sync. API keys stay on the server. Completed sessions appear in Anam Lab with a transcript and, when available, a recording. Zero Data Retention sessions store neither.

Quickstart

Follow these three steps to give your ElevenLabs agent a face:

Configure your ElevenLabs agent for the audio format and voice settings.

Set up the server to fetch a signed URL and create an Anam session token.

Start the avatar session in your browser.

Upgrade your avatar

Build on your working session with Director Notes for expressive avatar performance, session customisation for personalised conversations, and client tools for actions in your app. For connection and playback issues, see Troubleshooting.

How it works

The session is split between your Next.js server, the browser, and Anam’s engine:

Server:
  1. Fetch signed URL from ElevenLabs API
  2. Create Anam session token with Cara 4 Director Notes
     and elevenLabsAgentSettings

Client:
  3. createClient(sessionToken)
  4. streamToVideoElement("avatar-video")

Engine (automatic):
  5. Connects to ElevenLabs agent via signed URL
  6. User speech → ElevenLabs STT → LLM → TTS → Anam face rendering
  7. Supported audio tags → Director Notes cues
  8. Avatar video delivered over WebRTC
  9. Client tool calls and results forwarded over the same connection

The Next.js route supplies the ElevenLabs settings when it creates the session token. The browser receives that token and uses the Anam JavaScript SDK to start the avatar stream.

Supported features

These ElevenLabs agent features work through the server-side integration: voice intelligence (STT, LLM, TTS), expressive voices, mapped Director Notes cues, interruption handling, custom knowledge bases, server-side tools (webhooks), conversation history, and client tools.

The ElevenLabs connection and API keys remain server-side when client tools are enabled. Only the specific handler code runs in the browser.

Step 1: Configure your agent

Configure the ElevenLabs agent before adding the server route and browser client.

​Overview

​Quickstart

​Upgrade your avatar

​How it works

​Supported features