The goal of building a sophisticated voice AI assistant like Iron Man's JARVIS is now more achievable thanks to modern technologies.
Fictional AI assistants set a high bar for natural voice interaction that early real-world technologies like Siri failed to meet until the arrival of LLMs.
To avoid the unsettling 'uncanny valley' in voice AI, systems must handle non-linear conversations, interruptions, and the subtle timing of human speech.
A modern voice AI stack combines speech-to-text, an LLM, and text-to-speech, offering more control and better performance than current speech-to-speech models.
The demo shows how to implement interruption handling by stopping the AI's audio output and feeding the context of the interruption back into the LLM prompt.
Voice interstitials are pre-emptive audio messages that inform the user an action is in progress, preventing the perception of a system failure during long tasks.
An effective voice AI system is not a single monolithic agent but a constellation of smaller, faster models for specific tasks like checking for wake words or playing interstitials.
Twilio's Conversation Relay simplifies development by managing the complex audio pipeline, including speech-to-text, text-to-speech, and interruption handling, via a WebSocket API.
26:30Simon A.T. Jiménez
31:12Daniel Töws
25:17Scott Hanselman
32:26Krzysztof Cieślak
24:22Philipp Schmid
28:27PJ Hagerty
29:03Jörg Neumann
28:37Sami Ekblad
Jobs that call for the skills explored in this talk.
Commerz Direktservice GmbH
Duisburg, Germany
Analog Devices