TODOforAI open-sources voiceloop, a browser voice agent loop, plus a public benchmark

What the library does

According to a dev.to post from the TODOforAI team, voiceloop is a zero-dependency JavaScript library that keeps the entire voice-agent loop — voice activity detection, speech recognition, an LLM call and text-to-speech — inside the browser. The team says it built the library after failing to find a stack with the responsiveness it wanted: closed APIs could not be self-hosted, and the open frameworks it tried either talked over users or heard their own output through the speakers and cut themselves off mid-reply.

Several design choices stand out. Barge-in is triggered by newly transcribed words rather than raw microphone energy, so the agent's own voice leaking into the mic is not mistaken for an interruption. A word-matching echo filter compares what the microphone picks up against what the agent is currently saying; the developers report zero self-interruptions across 30 echo-coupled turns even with hardware echo cancellation switched off. Responsiveness comes from overlap: the TTS engine starts speaking the first sentence while the LLM is still generating the second, and the LLM request fires speculatively during the user's end-of-turn pause. Silero VAD and Piper TTS run as WebAssembly in the tab, so a fully local configuration needs no cloud round-trip, and turn serialization — rapid turns, tool results, holds and replays — is covered by 178 tests.

Every stage is pluggable: any OpenAI-compatible chat endpoint for the model, four STT providers (Web Speech, ElevenLabs Scribe, Deepgram Flux and Speechmatics), and swappable TTS with Piper local, ElevenLabs cloud or a custom engine. The integration example in the post is a single VoiceAgent constructor pointed at a proxied endpoint plus a start call after an audio-unlock gesture.

The benchmark and the numbers

All figures come from the developers' own runs of voice-agent-bench, a black-box rig they published alongside the library. A scripted caller plays byte-identical pre-generated speech into a virtual mic, the agent's speaker output is recorded, and every metric is derived from the audio alone — so any system that produces sound, closed or open, can be measured without integration work. Systems get the same scripted conversations and, where possible, the same fixed mock LLM with a 300ms time-to-first-token, which isolates the voice loop from the model. Results pool five six-turn conversations (n=30) and report medians and p95, because the developers say single runs jitter by around plus or minus 300ms.

On the smalltalk scenario, voiceloop with Deepgram Flux STT and ElevenLabs Flash TTS posted an 862ms median voice-to-voice time with a 1067ms p95 — the tightest tail in the table. OpenAI Realtime measured a nearly identical 866ms median but a 1644ms p95; the developers flag that Realtime is speech-to-speech and could not use the fixed mock LLM, so that row is not fully comparable. Pipecat 1.8.1 on the same providers as voiceloop came in at 1046ms median with a 3573ms p95, meaning roughly one turn in twenty took over three seconds. ElevenLabs ConvAI measured 1454ms. The free, fully local Piper configuration landed at 974ms, while the zero-key Web Speech plus Piper default — which needs no account anywhere — took 2113ms, about 1.2 seconds slower to close a turn than cloud STT.

Echo and hesitation behavior

The echo scenario is where the stacks diverge most, according to the post. Each system was fed its own voice back through the mic at -15dB with a 30ms delay and no acoustic echo cancellation, approximating a laptop with its speakers on. Pipecat cut itself off on 20 of 30 turns and OpenAI Realtime on 17; voiceloop recorded zero self-cuts and finished echo-coupled turns in 930ms, close to its clean-audio time. On hesitation — a user pausing mid-sentence — every stack except Pipecat backed off, and voiceloop entered on 2 of 30 such turns, yielding within 420ms.

Availability

The library is MIT-licensed with zero dependencies, installable from npm as @todoforai/voiceloop, and a browser demo runs without any API keys. Tuning parameters for VAD thresholds, end-of-turn debounces, barge-in lengths and echo-match thresholds are exposed in a tuning file, with defaults set to whatever won on the bench. The team invites others to add competing systems to the benchmark.

Why it matters

Latency claims in voice AI are usually self-reported and hard to verify; by shipping the measurement rig alongside the library, the developers let others reproduce or contest the numbers rather than take them on faith — though these remain the authors' own runs, and independent replication is pending. The echo findings point at a failure mode that affects real browser deployments on ordinary laptop hardware, and the word-level filter appears to remove it at no latency cost once it classifies correctly. A local-first pipeline closing turns near the one-second mark with no cloud TTS, combined with an MIT license and zero dependencies, lowers the barrier for product teams adding voice without vendor lock-in or per-minute speech costs. If competitors take up the invitation to join the public benchmark, vendor latency marketing could become something checkable.