Streaming inference

Streaming inference is the practice of producing model output incrementally as it is generated, so an avatar can start responding sooner instead of waiting for a full answer to finish.

How streaming inference works

Streaming inference means a model sends partial output as soon as it is ready. The system does not wait for the entire answer before the next stage can begin.

In real-time avatars, streaming inference helps reduce perceived latency. The LLM can stream early tokens, text-to-speech can begin turning them into audio, and the avatar can start speaking while the rest of the response is still being generated.

A concrete example: an onboarding avatar starts answering a setup question after the first phrase is ready, while the remaining explanation continues to stream behind it.

Streaming inference is most useful when the whole pipeline supports streaming. If TTS, lip sync, or networking still wait for the full response, the user will not feel the benefit.

Diagram explaining how the Anam Engine is used for AI developers

Diagram of how Anam works

What Anam ships

Anam's Cara-4 model delivers expressive real-time avatars with around 150 ms server-side avatar-generation latency once a session is running, across 70+ languages. Builders use JavaScript and Python SDKs or integrations for LiveKit, Pipecat, ElevenLabs Agents, Agora, and VideoSDK. Bring any AI stack including OpenAI, Claude, Gemini, Mistral, Groq, Deepgram, Cartesia, or custom providers. The platform supports WebRTC delivery, SOC 2 Type II, HIPAA, zero data retention, and regional data residency. Sessions stream low-latency audio and video to browsers and native apps.

Explore the real-time avatar API →

Related terms

LLM (large language model)

An LLM, or large language model, is the AI model that interprets user input, reasons over context, and generates the response an avatar or voice agent will speak back to the user.

TTS (text-to-speech)

TTS, or text-to-speech, is the system that converts an agent's text response into spoken audio, giving an avatar a voice that can be streamed and lip-synced in real time.

Sub-second latency

Sub-second latency is the end-to-end response time under one second between a user speaking and an avatar replying, covering speech recognition, model reasoning, speech synthesis, animation, and network streaming.

Concurrency

Concurrency is the number of avatar sessions or conversations a system can run at the same time, including media streams, agent loops, speech services, and backend resources without slowing the experience for users.

Frequently asked questions

What is streaming inference?

Streaming inference is when a model returns output in pieces as it generates them, instead of waiting until the full response is complete.

Why is streaming inference useful for avatars?

It lets downstream systems start earlier. The avatar can begin speech and animation sooner, which makes the conversation feel more responsive.

Does streaming inference guarantee low latency?

No. It helps, but the full pipeline still matters. Speech recognition, tools, text-to-speech, rendering, WebRTC, and playback all contribute to the user's perceived delay.

What can break streaming inference in an agent flow?

Long tool calls, slow retrieval, non-streaming TTS, overly long responses, buffering, or waiting for final model output can all erase the benefit of streaming.

Last updated: 17th July 2026 · Reviewed quarterly.

← Back to glossary

Try the real-time avatar API trusted by 8,000 builders

Read the docs

HIPAA & SOC 2 Certified