Director-note cues with Pipecat, timed to word timings

Director notes let a Cara-4 avatar perform its speech — a laugh on the funny line, concern on the caveat. In a turnkey session you write inline [tag] cues in the text and Anam handles them. But in a Pipecat pipeline with audio passthrough, Anam only receives audio, not text — so it can't see your tags. You send cues yourself, over the WebRTC data channel, each stamped with the moment it should fire.

This recipe wires that up: a Pipecat agent (Deepgram STT, OpenAI LLM, Cartesia TTS, Anam avatar via AnamTransport) that reads Cartesia's word timestamps and fires each cue on the word it belongs to.

The complete code is at examples/pipecat-director-notes.

The timing problem

A cue over the data channel is a small JSON message:

{ "message_type": "director_note_cue", "cue": { "tag": "laughter" }, "at_seconds": 1.2 }

at_seconds is an offset from the start of the current speech turn. So to land a laugh on the word "hilarious", you need to know when the avatar will say "hilarious" — before it says it, with enough lead time for the message to reach Anam.

Cartesia hands you the timing

Here's the shortcut: Cartesia (Sonic) accepts inline [tag] markers in the transcript and echoes them straight back in its word timestamps, each stamped with the moment it's spoken, with respect to the TTS audio this turn. One turn is one Cartesia context, so those offsets are already turn-relative — exactly your at_seconds.

Send "Hello [laughter] everyone today." with add_timestamps on, and the timestamps come back like this:

words  = ['Hello', '[laughter]', 'everyone', 'today.']
starts = [ 0.122,  0.706,        1.219,      1.721]

The [laughter] marker comes back as its own token, timed to where it sits — 0.706, just ahead of "everyone". That start time is your at_seconds; no anchor-matching required. You asked Cartesia to speak the marked text, and it told you exactly when each mark lands. (The tag takes its own short slot in the audio; glue it to the next word — [laughter]everyone — and Cartesia folds them into one token with no gap, if you prefer.)

So there's no stripping-and-re-deriving from neighbouring words: the TTS already did the alignment. You read it off.

Reading the cue

Two small pieces. cues.py pulls the tag(s) out of a Cartesia token — pure logic, so python cues.py self-checks:

_TAG_RE = re.compile(r"\[([a-zA-Z_]+)\]")

def cue_tags(token):
    """Valid Anam tags in a Cartesia word token: '[happy]' -> ['happy']."""
    return [t for m in _TAG_RE.finditer(token) if (t := m.group(1).lower()) in CUE_TAGS]

The TTS subclass overrides one method — add_word_timestamps, which Cartesia calls with the raw word timings ahead of playback — and fires a cue for any token that carries one:

class DirectorCueCartesiaTTSService(CartesiaTTSService):
    def __init__(self, *args, on_cue, **kwargs):
        super().__init__(*args, **kwargs)
        self._on_cue = on_cue

    async def add_word_timestamps(self, word_times, context_id=None, *args, **kwargs):
        for word, at_seconds in word_times:                 # [(word, start_seconds), ...]
            for tag in cue_tags(word):
                await self._fire(tag, max(0.0, float(at_seconds)))
        await super().add_word_timestamps(word_times, context_id, *args, **kwargs)  # keep normal behaviour

on_cue is a callback, so the TTS never has to know about the transport; _fire wraps it in a try/except so a cue that arrives before the session is live can't kill the pipeline.

Authoring cues

Write the tag in square brackets just before the word it should land on. It comes back as its own token, timed to that spot:

"Hi [warm] there! [concerned] Honestly, that story was [laughter] hilarious."

The system prompt teaches the LLM the tag vocabulary and where to place cues (just before the word), so it can add its own as the conversation goes.

Wiring it up

AnamTransport.send_director_note_cue(tag, at_seconds=...) sends the cue on the active Anam session. Wire it straight into the TTS as on_cue:

transport = AnamTransport(
    api_key=os.environ["ANAM_API_KEY"],
    persona_config=PersonaConfig(
        avatar_id=os.environ["ANAM_AVATAR_ID"],
        avatar_model="cara-4",            # director notes require Cara-4
        enable_audio_passthrough=True,
    ),
    daily_room_url=os.environ["DAILY_ROOM_URL"],
)

tts = DirectorCueCartesiaTTSService(
    api_key=os.environ["CARTESIA_API_KEY"],
    on_cue=transport.send_director_note_cue,   # cue → Anam, over the data channel
    settings=CartesiaTTSService.Settings(voice="e8e5fffb-252c-436d-b842-8879b84445b6"),
)

Drop tts into the pipeline in its usual spot (... llm, tts, transport.output(), ...). Set a baseline delivery for the whole session with PersonaConfig(director_notes=DirectorNotes(preset_style="warm", expressivity=0.7)); the per-word cues shift it for the moments you mark.

Running it

cd examples/pipecat-director-notes
uv sync
cp .env.example .env   # add your keys and DAILY_ROOM_URL
uv run python bot.py

Open your DAILY_ROOM_URL in a browser to join. On connect the avatar opens with a short LLM greeting — primed to use a cue so you can see one fire — and adds more as the conversation goes (it's told the tag vocabulary in the system prompt).

Caveats

• Cara-4 only. Director notes need a Cara-4 avatar; stock avatars default to Cara-3.
• Cartesia sees the tags too. The markers go to Cartesia unchanged. An emotion tag it doesn't recognise ([warm], [concerned], …) is absorbed into the word as a timing marker with negligible audio effect; a non-verbal it does know — notably [laughter] — it will also render in the voice. Usually a plus (voice and face align), but test your tag set, and expect the markers to show up in the assistant transcript.
• A spaced tag takes its own slot. [tag] word comes back as its own token — its start is the cue's at_seconds, a beat ahead of the word, with a short slot in the audio. Glue it ([tag]word) and Cartesia folds it into the word with no gap instead; the code reads both.
• One turn, one context. at_seconds is relative to the turn's Cartesia context, which spans the whole turn (reuse_context_id_within_turn). The echoed start times are already in that frame — nothing to reset.
• Cues need a live session. send_director_note_cue raises if the Anam session isn't active yet, so the demo runs its opening LLM turn from on_avatar_connected — which fires only after the session is up (reactive turns are naturally post-connection). _fire still wraps the call in a try/except, so a cue that races a session drop is logged and skipped, never fatal.
• Provider-specific. This leans on Cartesia echoing inline markers in its timestamps (verified on sonic-3.5). A TTS that strips unknown markers, or omits them from timestamps, won't work this way — you'd fall back to matching a cue against a neighbouring word's timing.