Table of Contents
Quick Answer
Chain three models: Whisper (speech → text), an LLM (text → response), TTS like OpenVoice or StyleTTS (text → speech). Stream between steps for sub-second latency. Deploy as a web app with WebRTC mic access or a mobile app via Capacitor.
- Time to working demo: 1-2 days
- Cost: $0.01-0.05 per 60-second conversation
- Latency target: <800ms total
What You'll Need
- Whisper API or local whisper.cpp
- Streaming LLM (OpenAI-compatible)
- TTS: StyleTTS 2, OpenVoice, or hosted (Cartesia, Deepgram Aura)
- Next.js + WebRTC for web; Capacitor for mobile
Steps
- Set up mic capture. Use MediaRecorder API. Ask AI: "Generate a React hook that captures 16kHz mono audio from the mic and emits 100ms chunks as WebM."
- Stream STT. Send audio chunks to Whisper API via WebSocket or HTTP stream. For local, use whisper.cpp compiled to WASM. Target: first partial transcript <300ms.
- VAD (voice activity detection). Use Silero VAD (WASM build) to detect end-of-speech. Otherwise you wait forever for user to "finish."
- Trigger LLM on end-of-speech. Stream transcript to LLM. Prompt: "You are a concise voice assistant. Keep answers under 40 words unless asked for detail."
- Stream TTS. As LLM tokens arrive, buffer to sentence boundaries, send each sentence to TTS, play audio chunks as they arrive. This is the key to low latency.
- Barge-in support. If user starts speaking while TTS plays, immediately stop playback and start new STT. Use a state machine: IDLE → LISTENING → THINKING → SPEAKING.
- Deploy. Web: Next.js to Vercel/Coolify. Mobile: wrap in Capacitor, request mic permission on first launch.
- Measure latency. Log: mic-stop → first audio byte. Aim <800ms. Profile and optimize slowest step.
Common Mistakes
- No streaming: Waiting for full transcript + full LLM + full TTS = 5s latency. Stream everything.
- Ignoring barge-in: Users hate being talked over. Detect interruption immediately.
- No VAD: Silence detection via volume threshold is unreliable. Use Silero.
- Long LLM responses: Force max_tokens short. Voice users want brevity.
- No echo cancellation: Mic picks up TTS speaker output. Enable echoCancellation: true.
Top Tools
Tool
Best For
Price
Whisper API
STT
$0.006/min
Cartesia
Low-latency TTS
$0.013/1K chars
StyleTTS 2
Self-hosted TTS
Free
Silero VAD
End-of-speech
Free
LiveKit
WebRTC infra
Free tier
FAQs
Q: Can this run fully offline on phones?
Yes — whisper.cpp + small LLM (Phi-3) + on-device TTS. Quality drops, privacy wins.
Q: How do I reduce latency below 500ms?
Self-host all 3 models on same GPU, use streaming, skip hosted APIs.
Q: What about non-English?
Whisper supports 90+ languages. TTS quality varies — test Cartesia or self-hosted XTTS.
Q: Can I clone my own voice?
Yes — OpenVoice and XTTS support voice cloning from 10 seconds of audio.
Q: Is WebRTC mandatory?
No — plain HTTPS + MediaRecorder works. WebRTC improves low-latency duplex.
Q: How much does ChatGPT Voice cost to replicate?
Infra-wise, ~$0.02/min all-in on good architecture.
Conclusion
Voice is the next interface. Streaming at every step is the secret to feeling magical. Build one narrow voice assistant (doctor's scribe, cooking helper, language tutor) and nail the latency. Everything else follows.