Expressive audio passthrough with Director Notes cues

In audio passthrough mode Anam runs face generation only — it never sees the persona's text, so it can't infer facial expression on its own, and there's no text to carry Director Notes inline cue tags like [warm] or [surprised]. Instead, you send those same cues as director_note_cue messages over the WebRTC data channel to steer the avatar's face during a turn.

This recipe combines three things into one Python script:

Cartesia Sonic 3.5 synthesises the speech for the voice — Sonic reads the emotional subtext of the text, and [laughter] stays inline so it laughs.
Audio passthrough streams that PCM into Anam for lip-sync.
Director Notes cues steer the face, timed to the spoken word using Cartesia's word-level timestamps.

The neat part: one annotated line drives the whole performance. You write [warm] Come closer. [surprised] Wait — what was that? [laughter] Oh, it's nothing. — the tags steer the face, [laughter] makes the voice laugh too, and Sonic reads the rest from the words.

The complete code is at examples/python-director-notes-audio-passthrough.

Director Notes require a cara-4 avatar (avatar_model="cara-4") — older models ignore the cues. send_director_note_cue also only exists in anam ≥ 0.7.0a1, so this example installs a prerelease. Audio passthrough and Director Notes are both Beta.

What you'll build

A Python script that:

Accepts a typed line (optionally annotated with [tags]) and splits it into expression segments
Streams it from Cartesia Sonic 3.5 as 24 kHz PCM, with word timestamps
Forwards the audio to the avatar as it arrives (audio passthrough)
Fires Director Notes cues over the data channel as each word streams in, aligned to the spoken word

Prerequisites

Python 3.10+ and uv
An Anam API key from lab.anam.ai and a cara-4 avatar
A Cartesia API key from cartesia.ai

Project setup

git clone https://github.com/anam-org/anam-cookbook.git
cd anam-cookbook/examples/python-director-notes-audio-passthrough
uv sync --prerelease=allow
cp .env.example .env

Edit .env:

ANAM_API_KEY=your_anam_api_key
ANAM_AVATAR_ID=your_cara4_avatar_id
ANAM_AVATAR_MODEL=cara-4
CARTESIA_API_KEY=your_cartesia_api_key
CARTESIA_VOICE_ID=6ccbfb76-1fc6-48f7-b71d-91ac6298247b

How the tags drive expression

You author one line with the Anam cue tags — [happy] [warm] [playful] [curious] [supportive] [concerned] [sad] [surprised] [angry] [distressed] [laughter] — and they steer the face, sent verbatim via send_director_note_cue.

The voice needs no separate tag vocabulary: Sonic 3.5 reads the emotional subtext of the words on its own, so you just synthesise the plain text. The one exception is [laughter] — Cartesia's documented non-verbal — which is kept inline in the transcript so the voice laughs while the face does.

Configure the persona

Enable passthrough and set a baseline performance style. avatar_model="cara-4" is what makes Director Notes work.

from anam.types import PersonaConfig, DirectorNotes

persona_config = PersonaConfig(
    avatar_id=avatar_id,
    avatar_model="cara-4",
    enable_audio_passthrough=True,
    director_notes=DirectorNotes(preset_style="warm", expressivity=0.7),
)

The baseline director_notes style sets the avatar's default presence for the whole session; the cues below shift it during a specific turn on top of that. expressivity (0–1) dials how strongly cues are followed.

Parse the line into segments

Split the typed line at each known [tag]. Each segment carries the tag (for the face) and clean spoken text. [laughter] is kept inline so Cartesia's voice actually laughs.

segments = parse_tagged_line("[warm] Hi there. [curious] What's that noise? [laughter]")
# -> [Segment("warm", "Hi there."), Segment("curious", "What's that noise?"), Segment("laughter", "")]

Open the Cartesia stream

Open one websocket context with add_timestamps=True and push each segment's text. Request raw 24 kHz PCM for a good latency/quality balance; Anam suggests 24000 Hz for best performance.

with cartesia.tts.websocket_connect() as ws:
    ctx = ws.context(
        model_id="sonic-3.5",
        voice={"mode": "id", "id": voice_id},
        output_format={"container": "raw", "encoding": "pcm_s16le", "sample_rate": 24000},
        language="en",
        add_timestamps=True,
    )
    for seg in segments:
        text = seg.cartesia_text()  # "[laughter]" kept inline so the voice laughs
        if text:
            ctx.push(text)
    ctx.no_more_inputs()

Turn streaming timestamps into cues

Audio chunks and word timestamps stream back interleaved (resp.type is "chunk" or "timestamps"). We don't wait for the whole line — a small CueTimer consumes the timestamps as they arrive and emits each face cue the moment its segment's first word appears. It matches forward through the word stream, so a stray non-verbal token (Cartesia emitting laughs for your inline [laughter]) never desyncs the cues. feed() returns any cues that just started; flush() emits the rest at the end. (Full implementation in cues.py.)

timer = CueTimer(segments)
new_cues = timer.feed(wt.words, wt.start, wt.end)  # -> [(tag, at_seconds), ...]

Stream into the avatar

Drain Cartesia: forward each audio chunk to the passthrough stream as it arrives, and fire each cue the moment CueTimer produces it. The avatar starts speaking almost immediately instead of waiting for the whole line to synthesise.

from anam.types import AgentAudioInputConfig

agent = session.create_agent_audio_input_stream(
    AgentAudioInputConfig(encoding="pcm_s16le", sample_rate=24000, channels=1)
)
timer = CueTimer(segments)

for resp in ctx.receive():
    if resp.type == "chunk" and resp.audio:
        await agent.send_audio_chunk(resp.audio)                 # forward as it arrives
    elif resp.type == "timestamps" and resp.word_timestamps:
        wt = resp.word_timestamps
        for tag, at in timer.feed(wt.words, wt.start, wt.end):
            await session.send_director_note_cue(tag, at_seconds=at)

for tag, at in timer.flush():
    await session.send_director_note_cue(tag, at_seconds=at)

await agent.end_sequence()

Cues can be sent early — Anam latches each onto the persona speech turn that begins within ~1 second, and at_seconds sequences them across the turn so each lands on its word. Streaming audio faster than realtime is fine: Anam buffers it and paces playback, and because Cartesia streams faster than realtime each cue reaches Anam before its word renders. end_sequence() returns the avatar to a neutral listening pose.

Cartesia's Python client is synchronous, so in the example these awaits run in a worker thread that bridges each send back to the event loop with asyncio.run_coroutine_threadsafe(...). See main.py for the wiring.

Run it

# Interactive: type lines, the avatar speaks each one
uv run python main.py

# One-shot
uv run python main.py --text "[warm] Come closer. [surprised] Wait — what was that? [laughter] Oh, it's nothing."

The avatar appears in an OpenCV window, lip-syncing to the Cartesia audio, and its face shifts on each cue word — brightening on [happy], widening on [surprised], laughing on [laughter].

Terminology

Audio passthrough – you supply the speech audio (your own TTS); Anam only renders the face.
Director Notes – a baseline style for the session plus cues that steer emotion or delivery during a turn.

In this recipe the voice comes from Cartesia and the facial performance from Director Notes cues — both driven by the same tags, kept in sync by Cartesia's word timestamps.