Why 300ms Matters: Latency in Voice AI That Feels Human

A patient calls your clinic at 11:47pm. They ask, "Do you have appointments available Tuesday morning?" Your AI agent begins to answer. One second passes. Two seconds. Three. They've already hung up.

In voice conversations, every 100 milliseconds of delay erodes trust. Text chat users will tolerate a two-second typing indicator. Voice callers won't. When someone speaks to you and you pause for three seconds before responding, it feels broken. Awkward. Robotic.

This is the latency problem in conversational AI, and it's exactly why most "AI phone systems" still feel like the IVR hell we've all learned to hate.

The 300ms Rule for Voice AI

Human conversation flows at roughly 300ms response time. When you finish a sentence, the other person typically begins responding within a third of a second. That's the psychological threshold where conversation feels natural.

Break that threshold—say, respond in 1,500ms instead—and the caller's brain registers confusion. "Is it still listening? Did I break it? Should I repeat myself?"

This isn't just theory. We've seen it in production call logs. Calls with average response latency above 1.2 seconds have 34% higher abandonment rates than calls under 600ms. The caller's patience has a very real, very measurable half-life.

Where Latency Hides in the Stack

Building a voice AI agent isn't a single API call. It's a chain of five sequential operations, each adding milliseconds:

• Audio capture — User finishes speaking, your system detects silence (50-200ms depending on Voice Activity Detection tuning)
• Speech-to-text — Audio converted to text (100-300ms for streaming STT like Deepgram or AssemblyAI)
• LLM inference — GPT-4o or Claude generates a response (200-800ms depending on prompt complexity and token count)
• Text-to-speech — Response text converted to audio (150-400ms for OpenAI tts-1 or ElevenLabs)
• Audio playback start — First audio chunk begins streaming to the caller (50-100ms network + buffer)

Add those up and you're looking at 550ms on a very good day, 1,800ms on a bad one. And that's assuming zero network jitter, zero retry logic, and a perfectly warm inference stack.

Every millisecond you shave from this pipeline is a millisecond closer to human.

How We Keep Softnode Agents Under 600ms

At Softnode, voice latency is a first-class metric. We don't treat it as a nice-to-have; we treat it as the core product experience. Here's how we stay fast:

1. Streaming everywhere. We don't wait for the LLM to finish the entire response before sending it to TTS. The moment the first few tokens arrive, we're already converting them to audio and streaming that audio to the caller. This cuts perceived latency by 40-60%.

2. Aggressive VAD tuning. Voice Activity Detection is a trade-off: detect silence too early and you cut off the user mid-sentence. Detect it too late and you add 400ms of dead air. We tune VAD per-language (Turkish speakers pause differently than English speakers) and per-use-case (a legal intake call tolerates longer pauses than a restaurant booking).

3. Smart model selection. Not every query needs GPT-4. For high-confidence, low-ambiguity responses ("What are your hours?" "Do you take insurance?"), we route to faster models or even cached responses. Save the big model for the complex stuff.

4. Regional inference. We run inference as close to the caller as possible. A call from Prague routes to EU-central instances; a call from Istanbul routes to the nearest Azure region. Physics matters. 80ms of transatlantic latency is 80ms you'll never get back.

Why Text Chatbots Get Away with Slow

Text-based AI widgets don't have a latency problem—yet. When a user types a message and sees a typing indicator, they'll happily wait two, even three seconds for a response. The visual feedback (the bouncing dots, the "Agent is typing...") buys you time.

Voice strips that forgiveness away. There's no "I'm thinking" animation in audio. Silence is just silence, and silence is unnerving.

This is why competitors like Intercom, Drift, and Tidio can get away with slower inference pipelines. They don't do voice. They're optimized for text chat, where latency is a second-order concern.

We're not. At Softnode, voice is the primary interface, and latency is the primary constraint.

"Every 100ms you cut from response time is a 3-5% improvement in caller satisfaction. At scale, that's the difference between a voice agent customers tolerate and one they actually prefer."

Measuring What Matters

If you're building or buying voice AI, here are the latency metrics you should demand:

• P50 response latency — Median time from caller finishing speech to agent starting response. Target: <600ms.
• P95 response latency — 95th percentile. This catches the slow tail. Target: <1,200ms.
• Time to first audio byte — How fast does the agent start speaking? Target: <400ms.
• VAD accuracy — Percentage of turns where the system correctly detected end-of-speech without cutting off the user. Target: >96%.

If your vendor can't give you these numbers, they're not measuring them. And if they're not measuring them, they're not optimizing them.

The Voice-First Advantage

Solo founders and clinic owners don't have time to babysit a chatbot that only works in text. Your customers call. They call at 1:47am. They call from the car. They call because typing on mobile is annoying and they just want an answer.

A voice agent that responds in under 600ms feels like talking to a sharp, attentive human. A voice agent that responds in 2,000ms feels like talking to a broken IVR.

That's the difference between a tool your customers love and a tool they route around.

We built Softnode to be fast by default. Because in voice AI, speed isn't a feature. It's the foundation.