Every other interface gets a cheap way to hide its delays. A webpage renders progressively, a chatbot shows a typing indicator, an app puts up a spinner and you forgive it for ten seconds, free of charge. A phone call gets a filler phrase like "let me just pull that up for you" and that's the whole toolkit.

It buys two seconds, maybe three, before it curdles into stalling. Humans run turn-taking on a clock measured in milliseconds; a pause of two hundred feels attentive, a pause of two thousand feels broken, and one filler phrase is all the cushioning the medium allows. The caller doesn't think "high P99." They think nobody's there, and then they hang up.

Right now the answer, for almost every developer, is: NOT you. Every turn of every conversation round-trips a frontier model in someone else's region, behind someone else's queue, and your product's feel on any given afternoon depends on that company's load that afternoon. You don't have a latency problem. You have a latency landlord.

Now look at where the wait and the bill actually come from. In a cascade turn, two meters dominate: the language model generating the response, and the synthesis speaking it.

And audit what they're generating. The LLM is produced, token by token, at full compute, with the entire conversation re-fed as input every single turn - confirmations, readbacks, templated responses: sentences that are 90% the same sentence, call after call.

Then the TTS renders that mostly-identical speech fresh, as if it had never been spoken before.

Yes, the savvy teams already hand-cache their greetings and filler phrases, and good for them. But a cached greeting is where the reuse ends, because the moment a sentence has a variable in it, exact-match caching is useless.

"Your appointment is Tuesday the 14th at 3:15" gets said, in shape, millions of times a day, one slot changing, everything else identical, and every single one is generated fresh and synthesized fresh, twice through the meter. That's most of what an agent actually says.

And around those two meters, the same expensive loop is also picking tools and validating inputs - decisions a lookup table could make.

A CPU from 2009 handles the structural work without noticing. We put all of it on the scarcest silicon on earth, in the one position where every millisecond is audible.

To be precise about the patient: I'm talking about voice agents on the cascade architecture. You have an orchestrator coordinating speech-to-text, a language model, and text-to-speech, which is how nearly every production voice agent runs today. There are other architectures, and choosing between them is a discussion for another day.

The moment this clicked for us wasn't a latency graph. It was watching our customers fight non-deterministic models for reliability while all of them were wrestling with the same two problems:

1. Getting model calls to behave dependably across the stack,
2. Getting the same question to produce the same answer twice.

Somewhere in the middle of that it landed: the industry's reflex is to throw a model at every problem, when for most of the work there are tools that are faster, cheaper, more reliable and already extraordinarily distributed, in a way high-end GPUs very much are not.

We over-attribute agents to models the way we once over-attributed websites to servers — the behavior looks like intelligence, so we assume it's all inference.

It isn't.

Most of it is structure, and structure runs anywhere. A voice agent shouldn't outsource all its thinking to a non-deterministic LLM. We just need to pull in some older, less sexy tooling.

I didn't come up with the underlying law and neither did anyone in voice. Computing has now had to relearn it four times.

In 1993 the web was two years old and arguing about whether HTML should have an image tag. In the very thread where Marc Andreessen proposed the tag, he was already defending what he called the single-hop principle i.e. a page should arrive in one round trip, against proposals that would piece pages together on the fly. The document web's first architecture fight was a latency fight.

In 1996, as dial-up brought the internet to consumers, Stuart Cheshire wrote the rant that named the law: "It's the latency, stupid." Bandwidth improves forever, latency has a floor set by the speed of light, and the floor doesn't negotiate. The industry was marketing fatter pipes. The pipe was never the problem.

In 2014 Guillermo Rauch dragged the law into the application web, in an essay that examined JavaScript purely through UX which he defined as minimizing the time it takes a user to get what they came for. Pre-render what can be known before the request. Act on input before the network answers. Predict what's coming; he compressed that one into two words, "negative latency." And he called out, by name, the then-fashionable tools whose whole trick was making the round trip feel shorter while leaving it in place.

He founded Vercel the next year, and the doctrine — compute the heavy things once, ahead of time, push the result next to the user, spend live computation only on what genuinely can't be known in advance — won the web so completely that we forgot it was ever contested. Even the sensitive stuff obeyed: the static shell caches everywhere precisely because your account details were never baked into it. They inject at request time into holes the precomputed layer left open.

The expensive resource sits in the hot path of every single request, the precompute doctrine is available and unclaimed, and the medium is less forgiving than the web ever was.

The web's spinner was free and patient; voice's costs money and expires in three seconds. And we've already rebuilt the school Rauch dismissed. The industry's consensus answer to voice latency is streaming. Overlap the pipeline stages so the wait feels shorter.

Streaming is genuinely useful and I'm not telling anyone to turn it off. But it polishes the round trip. It never asks why the round trip is there.

Take the agent apart and put every piece of work in one of two piles: needs the frontier, live, or just needs to have been done already.

Sort honestly and the second pile dwarfs the first. Routing and deterministic logic run on the CPU in microseconds. The state sits in memory, in place.

Speech — this is the one people resist, then can't unsee — is overwhelmingly assembled from pieces that could have been generated once, on GPUs, offline, at leisure, and spliced together at delivery the way a page assembles from static assets. The sensitive data gets injected at the splice point, never baked into the reusable layer. Same dynamic hole the web used, transposed to audio.

What's left over is reasoning that genuinely can't be precomputed or structured. It's what frontier models are for, and it's a fraction of the work, and even its output can usually be consumed off the hot path: prefetched, masked behind speech, arriving while the conversation stays local. CPU in real time, GPU asynchronously.

This is not an anti-model position. It's the opposite. It treats frontier inference as the most valuable computation available, which is exactly why you don't waste it in a position where its queue time is your dead air.

A deterministic path doesn't hallucinate, doesn't time out, doesn't emit malformed JSON, and doesn't wait in a frontier lab's queue.

A word on the industry's proposed alternative, which is not to fix the cascade but to dissolve it, fold everything into one realtime speech-to-speech model.

The naturalness gains are real. Prosody, duplex conversation, genuine wins for expressive use cases, and I'd be lying if I said the demos weren't impressive.

But look at what dissolving the seams removes: every place structure could live. No text boundary means no splice point, so nothing is ever reusable. No redaction point, so everything the caller says goes into the model.

The meter runs on every millisecond, on GPU, by design. The cascade's supposed weakness is that it's stitched together, but it’s also what makes it fixable. The seams are the product.

Here's the strangest thing about this industry, and the fastest way to see how young its architecture is. This solution is about a lot more than caching, however caching is bizarrely underused.

Every expensive computation in computing's history got a cache put in front of it. Your CPU has three layers of them. Your DNS resolver answers a name once and serves it from memory until the TTL expires.

Databases build materialized views so the expensive query runs once and gets read forever. Code gets compiled once and executed a billion times — nobody has ever paid a per-run compile fee.

Render once, serve twenty thousand times for the price of bytes; HTTP has a status code, 304, whose whole meaning is "you already have this answer, pay nothing." It's not even a computing pattern, really. A song is recorded once and streamed a billion times. A die is machined once and stamps a million parts.

Every expensive computation ever invented got a cache in front of it. Except this one. Voice AI runs the most expensive computation in history, across a range of modalities, millions of times a day, identically, at full price, with no reuse tier anywhere in the stack.

And before anyone points at prompt caching: that's a cache on the seller's side of the meter, sold back to you as a discount. The cached tokens still get billed, and the generation still runs fresh every time.

Production teams hand-roll greeting caches and filler-phrase caches around their TTS providers; one healthcare deployment reports 90% cache hit rates on common phrases; there's an open feature request on a leading framework asking, more or less politely, why there's no built-in way to reuse already-synthesized speech across calls.

Many vendors whose synthesis is being cached publishes a how-to guide for building the workaround yourself. Everyone has admitted the layer should exist except the people who'd bill less if it did. But that's the next essay.

A customer- and use-case-specific corpus of reusable components, scoped to their governance standards, generated ahead of time and spliced at delivery with the live data points — the appointment time, the name, the number — so that much of what an agent repeats can come from memory, and synthesis is saved for what's genuinely new.

It's live. Check the Docs.

And when a turn genuinely needs a frontier model, the request fires early and the answer arrives while the agent is still talking — the GPU round trip happens behind the speech, never in the silence.

Rauch had a name for the web's version of this: negative latency. The answer exists before the question finishes. Voice's version works the same way, for the same reason.

Don't take our word for it. That healthcare team hand-rolling their own cache hits 90% on common phrases, and the teams pre-caching greetings measured the answer-rate difference in points, not milliseconds.

With the layer built in rather than bolted on, we've seen over 90% reuse across LLM and TTS in exceptionally tuned use cases, and most customers get to 50% across both in a few hours.

A sh*tload, in other words. There is no polite number for buying the same answer 20,000 times a day.

So ask the question the way a buyer should ask it, holding the invoice:

Why do I buy the same answer 20,000 times a day?

Because the architecture refuses to remember.

• Latency stops being a benchmark war and becomes physics you own. Nothing GPU-shaped in the hot path, nothing queued, nothing intercontinental — your UX stops depending on another company's peak traffic.

• Cost stops scaling with conversation length and starts scaling with the genuinely hard fraction of the work. The meter only runs on the exception. And notice the reliability that comes bundled in, because deterministic paths don't have bad days.

• And the boundary problem — data locked inside hospital systems, bank cores, telco stacks, intelligence locked out in a handful of GPU regions — mostly dissolves, because an agent that is mostly structure doesn't need to cross the boundary.

Structure runs anywhere, including inside the walls, next to the data that can't move. CPUs exist everywhere. H100s don't. The frontier gets consulted by exception, redacted questions out, answers back. The data stays.

I might be wrong about how fast this happens. I don't think I'm wrong about the direction, because the direction has a thirty-year undefeated record and voice would have to be the first medium in computing history to permanently exempt itself from it.

The industry's best engineers are streaming the wait into smaller and smaller pieces. Overlapping it, masking it, benchmarking the remains.

All of it is polishing the proverbial. The answer was never faster inference.