Server-Side ElevenLabs Agents with Anam Avatars

Overview

Anam's server-side ElevenLabs integration connects an Anam avatar to an ElevenLabs Conversational AI agent. Your server fetches a short-lived ElevenLabs signed URL and includes it when creating an Anam session token.

The browser uses the Anam JavaScript SDK to start the WebRTC session and render the avatar. Anam sends conversation audio to ElevenLabs, receives the generated speech, and keeps the avatar in sync. API keys stay on the server, and completed sessions include recordings and transcripts in Anam Lab.

The complete example code is available at examples/elevenlabs-server-side-agent-nextjs.

Best practices for configuring ElevenLabs agents

All of this configuration happens in the ElevenLabs agent dashboard.

LLM choice. Pick a model with low latency and fast throughput — for example we found Qwen3-30B-A3B to be great for low latency. Reasoning models are not recommended unless you're using a high-throughput provider like Groq. If slower LLM responses are a problem, set a soft timeout in the Advanced tab (e.g. 2 seconds) and configure it to either use a fixed filler phrase or generate one with a faster LLM while the main response generates.

Voices. For maximum expressivity, use V3 Conversational with Expressive mode enabled. Anam maps supported v3 tags such as [laughs], [whispers], and [gasps] to Director Notes, so they affect the avatar's performance as well as the voice. If latency is your highest concern, it's worth trying Flash or another model that is less expressive but faster.

Background noise. If background noise is being picked up, turn on the Filter Background Speech toggle in the Advanced menu. It's unclear whether this has an effect on latency.

User input audio format. In the Advanced tab, set the user input audio format to PCM 16000Hz. Other input formats are not currently supported with Anam.

Response speed. For the quickest responses, set Eagerness to Eager in the Advanced menu.

How it works

The session is split between your Next.js server, the browser, and Anam's engine:

Server:
  1. Fetch signed URL from ElevenLabs API
  2. Create Anam session token with Cara 4 Director Notes
     and elevenLabsAgentSettings

Client:
  3. createClient(sessionToken)
  4. streamToVideoElement("avatar-video")

Engine (automatic):
  5. Connects to ElevenLabs agent via signed URL
  6. User speech → ElevenLabs STT → LLM → TTS → Anam face rendering
  7. Supported v3 tags → Director Notes cues
  8. Avatar video delivered over WebRTC
  9. Client tool calls and results forwarded over the same connection

The Next.js route supplies the ElevenLabs settings when it creates the session token. The browser receives that token and uses the Anam JavaScript SDK to start the avatar stream.

Prerequisites

• Node.js 20.9+
• An Anam account and API key (sign up free at lab.anam.ai)
• A Cara 4 avatar from the Anam Lab Avatars page
• An ElevenLabs account with a Conversational AI agent configured via the dashboard

Server-side setup

You need one API route. It fetches a signed URL from ElevenLabs, then passes it to Anam when creating a session token.

API keys stay server-side. The client sends only the avatarId (which face to render) and agentId (which ElevenLabs agent to use).

Environment variables

ANAM_API_KEY=your_anam_api_key
ELEVENLABS_API_KEY=your_elevenlabs_api_key

API route

// app/api/anam-session/route.ts
import { NextResponse } from "next/server";

export async function POST(request: Request) {
  const anamApiKey = process.env.ANAM_API_KEY;
  if (!anamApiKey) {
    return NextResponse.json(
      { error: "ANAM_API_KEY must be set" },
      { status: 500 }
    );
  }

  const elevenLabsApiKey = process.env.ELEVENLABS_API_KEY;
  if (!elevenLabsApiKey) {
    return NextResponse.json(
      { error: "ELEVENLABS_API_KEY must be set" },
      { status: 500 }
    );
  }

  const body = await request.json().catch(() => ({}));
  const { avatarId, agentId } = body;

  if (!avatarId) {
    return NextResponse.json(
      { error: "avatarId is required" },
      { status: 400 }
    );
  }
  if (!agentId) {
    return NextResponse.json(
      { error: "agentId is required" },
      { status: 400 }
    );
  }

  // 1. Get a signed URL from ElevenLabs
  const elRes = await fetch(
    `https://api.elevenlabs.io/v1/convai/conversation/get-signed-url?agent_id=${agentId}`,
    {
      headers: { "xi-api-key": elevenLabsApiKey },
    }
  );

  if (!elRes.ok) {
    const text = await elRes.text();
    return NextResponse.json(
      { error: `ElevenLabs API error: ${elRes.status} ${text}` },
      { status: elRes.status }
    );
  }

  const { signed_url: signedUrl } = await elRes.json();

  // 2. Create an Anam session token with the ElevenLabs agent settings
  const anamRes = await fetch("https://api.anam.ai/v1/auth/session-token", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${anamApiKey}`,
    },
    body: JSON.stringify({
      personaConfig: {
        avatarId,
        avatarModel: "cara-4",
        directorNotes: {
          presetStyle: "warm",
          expressivity: 0.5,
        },
      },
      environment: {
        elevenLabsAgentSettings: {
          signedUrl,
          agentId,
        },
      },
    }),
  });

  if (!anamRes.ok) {
    const text = await anamRes.text();
    return NextResponse.json(
      { error: `Anam API error: ${anamRes.status} ${text}` },
      { status: anamRes.status }
    );
  }

  const data = await anamRes.json();
  return NextResponse.json({ sessionToken: data.sessionToken });
}

The environment.elevenLabsAgentSettings field configures ElevenLabs as the session's speech recognition, language model, and text-to-speech provider. avatarModel and directorNotes enable a warm Cara 4 performance at the start of the session.

Director Notes with ElevenLabs v3

Director Notes give the avatar a baseline performance style and temporary expression cues. The server route above starts with the warm preset at 0.5 expressivity. Expressivity accepts values from 0 to 1; higher values strengthen both the active style and speech-driven motion.

Select V3 Conversational in the ElevenLabs Agent Voice settings. Expressive mode is enabled automatically for that model. Then add a short rule to the agent's system prompt:

Use ElevenLabs v3 expressive tags sparingly when the delivery should change.
For testing, use tags such as [excited], [laughs], [whispers], [sighs],
[gasps], [angrily], [sarcastic], or [nervous].
Place each tag immediately before the words it should affect.
Do not explain or read the tags aloud.

The ElevenLabs Expressive mode guide explains how v3 tags affect the voice. For this integration, Anam recognizes the following provider tags and maps each group to one Director Note cue:

| Director Note cue | Recognized ElevenLabs v3 tags | |---|---| | happy | [excited], [happily], [cheerfully], [elated] | | laughter | [laugh], [laughs], [laughing], [laughs harder], [starts laughing], [chuckles], [chuckling], [giggles], [giggling] | | warm | [whispers], [whispering], [quietly] | | playful | [sarcastic], [playfully] | | curious | [curiously], [quizzical], [quizzically] | | concerned | [sigh], [sighs], [exhales] | | sad | [sadly], [sorrowful], [tearful], [crying] | | surprised | [gasp], [gasps], [startled] | | angry | [angrily], [furious], [intensely] | | distressed | [nervous], [fearful], [scared] |

This table is the complete alias list that Anam recognizes. ElevenLabs supports other audio tags, but Anam does not map them to Director Notes. A recognized tag is removed from the transcript before it reaches the browser. An unknown bracketed tag remains ordinary transcript text.

The example listens for DIRECTOR_NOTE_CUE_APPLIED so you can see whether the engine recognized and applied a tag:

const [directorNoteActivity, setDirectorNoteActivity] = useState<{
  cueTag: string;
  correlationId: string;
} | null>(null);

anamClient.addListener(AnamEvent.DIRECTOR_NOTE_CUE_APPLIED, (evt) => {
  setDirectorNoteActivity({
    cueTag: evt.cueTag,
    correlationId: evt.correlationId,
  });
});

The event contains the normalized Anam cue. For example, ElevenLabs [laughs] produces a laughter cue, while [whispers] produces warm. The Anam Director Notes guide covers baseline styles, expressivity, cue events, and Cara 4 requirements.

Per-session customisation

The Anam session token API accepts additional fields that are passed through to ElevenLabs when starting the conversation. These let you personalise each session — greet users by name, switch languages, override the system prompt, or pass data to a custom LLM backend. See the ElevenLabs personalisation overview for full details.

All of these fields are optional and go inside elevenLabsAgentSettings alongside signedUrl and agentId:

body: JSON.stringify({
  personaConfig: { avatarId },
  environment: {
    elevenLabsAgentSettings: {
      signedUrl,
      agentId,
      dynamicVariables: { ... },          // optional
      conversationConfigOverride: { ... }, // optional
      userId: "...",                       // optional
      customLlmExtraBody: { ... },        // optional
    },
  },
}),

Dynamic variables

Dynamic variables are the simplest way to personalise a conversation at runtime. Define placeholders like {{user_name}} in your agent's system prompt or first message in the ElevenLabs dashboard, then fill them in when creating the session token:

body: JSON.stringify({
  personaConfig: { avatarId },
  environment: {
    elevenLabsAgentSettings: {
      signedUrl,
      agentId,
      dynamicVariables: {
        user_name: "Alice",
        account_type: "premium",
      },
    },
  },
}),

See ElevenLabs dynamic variables docs for the full syntax.

Overrides

Overrides let you change agent behaviour per conversation — things like the first message, language, system prompt, or TTS voice. Only include the fields you want to override; everything else keeps its default from the ElevenLabs dashboard.

body: JSON.stringify({
  personaConfig: { avatarId },
  environment: {
    elevenLabsAgentSettings: {
      signedUrl,
      agentId,
      conversationConfigOverride: {
        agent: {
          prompt: {
            prompt: "You are a helpful assistant. Always respond in Spanish.",
          },
          firstMessage: "¡Hola! ¿En qué puedo ayudarte hoy?",
          language: "es",
        },
      },
    },
  },
}),

See ElevenLabs overrides docs for the full list of overridable fields.

User ID

Pass userId to identify the caller in ElevenLabs analytics and conversation history:

elevenLabsAgentSettings: {
  signedUrl,
  agentId,
  userId: "user_abc123",
},

Custom LLM extra body

If your ElevenLabs agent is configured to call a custom LLM backend, customLlmExtraBody lets you pass arbitrary parameters alongside each request:

body: JSON.stringify({
  personaConfig: { avatarId },
  environment: {
    elevenLabsAgentSettings: {
      signedUrl,
      agentId,
      customLlmExtraBody: {
        session_context: { region: "eu", tier: "enterprise" },
      },
    },
  },
}),

See ElevenLabs custom LLM docs for how to set up a custom LLM backend.

Start the avatar session

Request the session token, create an Anam client, and attach its stream to your video element:

import { createClient } from "@anam-ai/js-sdk";

const res = await fetch("/api/anam-session", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ avatarId, agentId }),
});
const { sessionToken } = await res.json();

const client = createClient(sessionToken);
await client.streamToVideoElement("avatar-video");

streamToVideoElement() starts microphone capture and renders the avatar's video and audio in the element with the matching ID.

Working component example

Below is a React component with connection lifecycle and a streaming transcript.

The MESSAGE_STREAM_EVENT_RECEIVED event fires for each chunk of text from both the user and the agent. Accumulate chunks by message ID to build the full transcript.

"use client";

import { useRef, useState, useCallback } from "react";
import { AnamEvent, createClient, type AnamClient } from "@anam-ai/js-sdk";

type Message = {
  id: string;
  role: "user" | "persona";
  content: string;
  interrupted?: boolean;
};

export default function AvatarChat({
  avatarId,
  agentId,
}: {
  avatarId: string;
  agentId: string;
}) {
  const clientRef = useRef<AnamClient | null>(null);
  const [status, setStatus] = useState<"idle" | "connecting" | "connected">("idle");
  const [messages, setMessages] = useState<Message[]>([]);

  const start = useCallback(async () => {
    setStatus("connecting");
    setMessages([]);

    const res = await fetch("/api/anam-session", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ avatarId, agentId }),
    });
    const { sessionToken } = await res.json();

    const anamClient = createClient(sessionToken);
    clientRef.current = anamClient;

    // Accumulate transcript chunks by message ID
    anamClient.addListener(
      AnamEvent.MESSAGE_STREAM_EVENT_RECEIVED,
      (evt: {
        id: string;
        content: string;
        role: string;
        interrupted: boolean;
      }) => {
        setMessages((prev) => {
          const idx = prev.findIndex((m) => m.id === evt.id);
          if (idx >= 0) {
            const next = [...prev];
            next[idx] = {
              ...next[idx],
              content: next[idx].content + evt.content,
              interrupted: evt.interrupted,
            };
            return next;
          }
          return [
            ...prev,
            {
              id: evt.id,
              role: evt.role as "user" | "persona",
              content: evt.content,
              interrupted: evt.interrupted,
            },
          ];
        });
      }
    );

    anamClient.addListener(AnamEvent.CONNECTION_CLOSED, () => {
      setStatus("idle");
    });

    await anamClient.streamToVideoElement("avatar-video");
    setStatus("connected");
  }, [avatarId, agentId]);

  const stop = useCallback(async () => {
    await clientRef.current?.stopStreaming();
    clientRef.current = null;
    setStatus("idle");
  }, []);

  return (
    <div>
      <video id="avatar-video" autoPlay playsInline />
      <button onClick={status === "connected" ? stop : start}>
        {status === "connected" ? "Stop" : "Start"}
      </button>
    </div>
  );
}

For more on event handling, message history, and connection lifecycle, see the basic Next.js app recipe.

Add a client tool

This example includes an ElevenLabs client tool named show_notification. When the agent calls it, Anam forwards the call to the JavaScript SDK in the browser. The handler displays the requested message and returns a result to the ElevenLabs conversation.

Configure the tool in ElevenLabs

In the ElevenLabs agent dashboard, open Tools and add the following tool:

| Setting | Value | |---|---| | Tool Type | Client | | Name | show_notification | | Wait for response | Enabled | | Required parameter | message (String) |

Tool and parameter names are case-sensitive. Describe the tool as displaying a message on screen when the user asks for one. The ElevenLabs client tools guide shows where to find these settings.

Register the handler

Register the handler before starting the Anam stream:

const unregisterToolHandlers = [
  anamClient.registerToolCallHandler("show_notification", {
    onStart: async ({ arguments: args }) => {
      if (typeof args.message !== "string" || !args.message.trim()) {
        throw new Error("message must be a non-empty string");
      }

      const message = args.message.trim();
      setNotification(message);
      return `Displayed the notification: ${message}`;
    },
    onFail: async ({ errorMessage }) => {
      console.error("show_notification failed:", errorMessage);
    },
  }),
];

await anamClient.streamToVideoElement("avatar-video");

Returning a string from onStart sends that value back as the tool result. With Wait for response enabled, ElevenLabs adds it to the conversation context before the agent continues. Keep the functions in unregisterToolHandlers and call them when the session ends or the component unmounts. The Anam client tools docs cover the full handler lifecycle.

The complete example includes argument validation, a notification UI, handler cleanup, and a status panel for checking tool calls.

Supported features

These ElevenLabs agent features work through the server-side integration: voice intelligence (STT, LLM, TTS), expressive V3 voices, mapped Director Notes cues, interruption handling, custom knowledge bases, server-side tools (webhooks), conversation history, and client tools.

The ElevenLabs connection and API keys remain server-side when client tools are enabled. Only the specific handler code runs in the browser.

Troubleshooting

"Failed to get ElevenLabs signed URL"

• Verify ELEVENLABS_API_KEY is set and valid
• Confirm the agentId matches an agent in your ElevenLabs dashboard
• Check that your ElevenLabs plan includes Conversational AI access

Avatar connects but no conversation starts

• The signed URL may have expired. Create the session token right before the client needs it, not minutes in advance
• Verify the ElevenLabs agent is active and not paused in the dashboard

Director Notes status stays "waiting"

• Confirm the avatar is a Cara 4 avatar and avatarModel is set to cara-4
• Select V3 Conversational in ElevenLabs and prompt the agent to emit one of the recognized tags listed above
• Check the response transcript in ElevenLabs to confirm the LLM produced the tag
• Confirm the Anam engine build includes the ElevenLabs Director Notes aliases

Client tool status stays "ready"

• Confirm the tool type is Client in ElevenLabs
• Match the registered tool and parameter names exactly; they are case-sensitive
• Enable Wait for response if the agent needs to receive the handler's returned string

"Anam API error: 400" when creating the session token

• Check that avatarId is valid. Find available avatars at lab.anam.ai/avatars
• Make sure the elevenLabsAgentSettings object includes both signedUrl and agentId

Avatar lips not syncing / no audio

• This integration handles audio format matching automatically. If you hit this, check the ElevenLabs agent configuration and make sure it's using a supported voice model

Session recordings not appearing in Anam Lab

• Recordings are generated after the session ends. Allow a few minutes for processing
• Check the session completed cleanly (the client called stopStreaming() or the connection closed normally)