The Babel fish from "Hitchhiker's Guide to the Galaxy" serves as the inspiration for a real-time audio translation project.
A demonstration of the initial 2019 architecture using GCloud reveals a significant latency of over ten seconds for a simple translation.
Switching to modern, specialized APIs like Deepgram and 11 Labs significantly cuts the total processing time from twelve to five seconds.
Adopting a streaming approach provides a major performance boost, but a naive implementation results in chaotic and low-quality audio output.
Chunking data based on sentence punctuation controls the streaming waterfall, improving the quality of generated audio without sacrificing speed.
Running a smaller, local AI model like Whisper on the edge eliminates cross-continental network latency and provides near-instantaneous results.
Implementing caching, from simple request matching to semantic search with vector databases, avoids redundant generation and speeds up common queries.
Fine-tuning performance involves optimizing prompts to generate fewer tokens and improving perceived speed with clear loading states for the user.
A final recap covers the essential strategies for building fast Gen AI experiences, including streaming, edge computing, caching, and prompt optimization.
30:36Christian Liebel
30:13Maxim Salnikov
21:21Stanislas Girard
31:39Rainer Stropek
32:27Christian Liebel
26:54Duan Lightfoot
35:16Ankit Patel
32:13Cedric Clyburn
Jobs that call for the skills explored in this talk.
Conrad Electronic SE