#181 — How Gradium is beating Big Tech at audio AI

Audio models are 50x+ cheaper to train than frontier text models. Moshi (Kyutai's full-duplex audio model) used 7B parameters and 2.1T tokens, while Llama 3.1 required 405B parameters and 15T tokens—orders of magnitude more expensive. A small team of 4 researchers built Moshi from scratch in 6 months with limited non-profit funding, demonstrating that audio AI doesn't require massive compute clusters or thousand-person teams.

Full-duplex audio AI (like Kyutai's Moshi) models conversations as two simultaneous audio streams—user and machine—enabling natural interruptions, backchanneling ('uh-huh', 'I see'), and 160ms response latency (faster than human conversation at 200-250ms). This contrasts with turn-based models (like OpenAI's early Advanced Voice Mode) that can't handle simultaneous speech and create awkward pauses. For founders, full-duplex enables natural voice interfaces for customer service, therapy apps, language learning, and collaborative tools where conversation flow matters.

Kyutai uses a simple assessment: 'If you put them in a data center with a gun to their head and no internet, can they implement their stuff from scratch?' Look for researchers who can both conceive architectural insights AND implement them—not 'idea guys' who need implementation teams. The AI research hierarchy inverted with deep learning: ideas became cheap while implementation became hard. Neil Zeghidour, Alex Défossez, Olivier Teboul, and Laurent Mazaré (Gradium's founding team) all came from underfunded audio divisions at Google Brain, Meta, and Facebook AI Research where they shipped production code, not just papers.

Significantly less than text models. Moshi was pretrained on 7M hours of audio with transcripts, post-trained on 2,000 hours (Fisher dataset), and instruction-finetuned on 20k+ hours of synthetic dialogue. Compare this to text LLMs trained on trillions of tokens scraped from the entire internet. The key difference: a Stack Overflow answer teaches you something; most phone conversations are filler. Kyutai demonstrated that focused, high-quality conversational data plus synthetic dialogue generation can overcome audio data scarcity without massive scraping infrastructure.

Depends on your latency and customization needs. Gradium offers 160ms theoretical latency with full-duplex architecture (simultaneous speech, natural interruptions), multilingual support, and the ability to fine-tune on your domain. OpenAI's turn-based model can't handle simultaneous speech and historically couldn't be interrupted until recent updates. If you're building conversational AI where natural dialogue flow matters (therapy, language learning, customer service), full-duplex architecture is critical. Gradium also offers production-ready APIs at gradium.ai versus consumer-focused ChatGPT integration.

Three structural reasons: (1) Big labs considered audio 'solved'—when Neil Zeghidour joined Google Brain in 2019, management treated voice as finished while pouring resources into text/vision. Meta's Seamless and Google's speech repos haven't been updated in years. (2) Audio requires deep domain expertise over brute compute—an audio codec needs understanding of acoustic psychophysics and human hearing, not just scaling. A text tokenizer is basically a dictionary; Kyutai's Mimi codec is applied physics. (3) Multimodal tradeoffs—at big labs, voice teams negotiate with text, vision, and code teams for model capacity. Dedicated focus wins.

Two key innovations from Kyutai: (1) Multi-stream modeling—instead of modeling one audio stream (the user), model two simultaneously (user + machine). When the machine isn't speaking, it produces silence or backchanneling. This simple architectural shift solved the turn-taking problem that stumped researchers for years. (2) Mimi codec—combines semantic tokens (content/words) and acoustic tokens (voice style/emotion) into a single real-time generation process. Previous approaches used non-causal models like WavLM that needed future audio to process current embeddings, making real-time impossible. Moshi generates 8 tokens per timestep: 1 semantic, 7 acoustic.

Kyutai completely gave up on quantitative measures and only trusted humans doing blind tests and extensive listening sessions supplemented with freelancers. This evaluation problem is well-documented in music generation—good conversations are subjective and can't be reduced to metrics like BLEU scores or perplexity. For founders building audio products, this means: (1) Budget significant time for human evaluation, (2) Use blind A/B testing with real users, (3) Don't trust automated metrics alone, (4) Accept that 'better' is contextual to your specific use case (customer service needs different qualities than therapy or entertainment).

The research-to-product pipeline model: Kyutai (non-profit) handles fundamental research and ships open-source prototypes (Moshi, Hibiki) that become foundational for others (NVIDIA's PersonaPlex, Qwen3-TTS). Gradium raised $70M to bridge the last mile from research prototypes to production-ready models with API access. This approach works because: (1) Open research attracts top talent who value transparency, (2) Research prototypes prove technical feasibility without production polish, (3) Big labs are distracted by multimodal tradeoffs—voice negotiates with text/vision/code teams for resources. In months, Gradium shipped multilingual models competing with best-in-class by focusing exclusively on audio.

Choose audio-first when: (1) Hands-free or eyes-free interaction is critical (driving, cooking, accessibility), (2) Emotional context and tone matter (therapy, customer service, language learning), (3) Real-time collaboration requires natural turn-taking (virtual assistants, meeting facilitation), (4) Your users have literacy barriers or prefer speaking to typing. Many researchers believe audio will become the dominant AI modality—every sci-fi AI (HAL 9000, Jarvis, Her) speaks naturally. The technology finally caught up in 2024 when Kyutai shipped the first full-duplex model before OpenAI's Advanced Voice Mode and a year before xAI's demo.

While Gradium is early-stage (founded from Kyutai in 2024, raised $70M), their technology enables: (1) Multilingual customer service with real-time translation in the speaker's voice (Hibiki demo), (2) Voice assistants with natural interruptions for hands-free workflows, (3) Real-time speech-to-speech translation running on-device (shipped months before Apple's), (4) Text-to-speech and speech-to-text APIs that became foundations for NVIDIA's PersonaPlex and Qwen3-TTS. Founders can access production-ready models at gradium.ai and review API documentation to integrate full-duplex conversational AI without building codec and turn-taking infrastructure from scratch.

Kyutai built Moshi (the first full-duplex conversational AI) with 4 researchers in 6 months on a non-profit budget orders of magnitude smaller than big labs—demonstrating that focused execution beats massive budgets. Gradium then raised $70M to productionize Kyutai's research into API-ready models, suggesting you need: (1) Small exceptional team for R&D phase (under $5M seed/Series A), (2) Larger round ($50-100M) to bridge research-to-production gap with polished models, API infrastructure, and enterprise sales. This is 50x+ cheaper than frontier text model development because audio models are dramatically smaller (7B vs. 405B parameters) and require less training data.

Kyutai is the first and only open audio research lab, named for the Japanese word for 'sphere,' founded in 2023 with funding from Eric Schmidt and two French billionaires. Unlike OpenAI (closed models, text-focused) or Anthropic (safety-focused, multimodal), Kyutai exclusively researches audio AI and publishes all findings open-source. Their major releases include Moshi (first full-duplex voice model) and Hibiki (real-time voice translation). The lab was created because competitive pressure between big labs meant fewer papers were being published. Kyutai builds research prototypes; their commercial spinout Gradium ($70M raised) productionizes them into APIs.

A neural audio codec is an AI-powered encoder-decoder that compresses audio into tokens a model can process—like embeddings for audio. Traditional codecs (MP3, AAC) use manual, domain-specific rules: speech codecs for speech, music codecs for music, incompatible with each other. SoundStream (Neil Zeghidour's breakthrough at Google Brain) trained a neural network to compress speech, music, and general audio at speech-codec bitrates, working as well as domain-specific codecs. Kyutai's Mimi codec advanced this by combining semantic tokens (content/words) and acoustic tokens (voice style/emotion) into single real-time generation, enabling 160ms latency. Without neural codecs, real-time conversational AI wouldn't be possible.

Semantic tokens represent the content of speech—the words and their meaning. Acoustic tokens represent the style—how the voice sounds, emotion, prosody, speaker characteristics. Previous audio models (like AudioLM) used hierarchical approaches: generate semantic tokens first (content), then acoustic tokens (style) separately. Problem: prevailing semantic token models (WavLM) are non-causal and need future audio to compute current embeddings—impossible for real-time. Kyutai's Mimi codec combines both into single generation via vector quantization: when Moshi speaks, it produces 8 tokens per timestep (1 semantic, 7 acoustic) causally in real-time. This architecture enables voice cloning, emotion, and natural prosody at 160ms latency.

Backchanneling is verbal feedback ('uh-huh', 'yes', 'I see', 'mhm') listeners provide during conversations to signal understanding, agreement, or engagement. Full-duplex models like Moshi produce backchanneling naturally because they model two simultaneous audio streams—when the machine isn't speaking full sentences, it can produce backchannels while you talk. Turn-based models (OpenAI's early Advanced Voice Mode) can't backchannel because only one party can speak at a time. Research shows backchanneling makes conversations feel more lively, natural, and engaging—critical for therapy apps, language learning, customer service, and any use case where rapport matters. Japanese conversation norms rely heavily on backchanneling, making it essential for multilingual products.

The Fisher dataset contains 2,000 hours of English phone conversations with separated audio channels for each speaker—making it ideal for training full-duplex conversational models. Unlike most audio datasets that mix speakers into one channel, Fisher's separated streams let researchers model how two people actually converse simultaneously with interruptions, backchanneling, and overlapping speech. Kyutai used Fisher for Moshi's post-training after pre-training on 7M hours of general audio. For founders: Fisher is publicly available through the Linguistic Data Consortium (LDC), but you'll need similar separated-channel conversational data to train full-duplex models for your specific domain.

Kyutai's Moshi can run entirely on mobile devices, demonstrating that real-time conversational AI doesn't always require cloud infrastructure. At 7B parameters, Moshi is dramatically smaller than multimodal models (Llama 3.1: 405B parameters), making on-device deployment feasible with modern smartphone chips. Kyutai's real-time speech-to-speech translation shipped on-device months before Apple's. On-device audio AI benefits: (1) No latency from network calls, (2) Privacy—no data leaves the device, (3) Works offline, (4) No per-request API costs. Trade-offs: On-device models have less world knowledge and require optimization for mobile inference. For many voice interface use cases (voice assistants, translation, transcription), on-device is preferable.

Turn-taking is how conversation partners decide when to speak, pause, or interrupt—considered one of the hardest problems in audio AI. Challenges: (1) When is the user done talking vs. just thinking? (2) How do you handle interruptions naturally? (3) Should the AI ever interrupt, and when? (4) How do you distinguish meaningful silence from conversation completion? Turn-based models (OpenAI's approach) predict when users finish talking—but if you pause thinking, the model talks anyway. You can't interrupt it easily. Full-duplex models (Kyutai's architecture) solve this by modeling two simultaneous streams, letting both parties interrupt naturally like phone conversations. This architectural shift mostly solved turn-taking, enabling natural dialogue for customer service, therapy, and collaborative applications.

Kyutai instruction-finetuned Moshi on 20k+ hours of synthetic dialogue—AI-generated conversations used to supplement scarce real conversational data. Process typically involves: (1) Generate conversation scripts using text LLMs with prompts for realistic dialogue scenarios, (2) Convert scripts to audio using text-to-speech with varied voices, emotions, and speaking styles, (3) Add realistic acoustic properties (room acoustics, background noise, cross-talk), (4) Fine-tune the audio model on synthetic conversations. Benefits: Overcome audio data scarcity, control for specific conversational patterns (interruptions, backchanneling, turn-taking), create domain-specific dialogues (medical consultations, customer service), avoid privacy issues with real conversations. For founders: Synthetic generation is cost-effective for specialized domains where real conversational data doesn't exist at scale.

Causal models process data sequentially—they only use past and present information to make predictions, enabling real-time processing as audio arrives. Non-causal models use future information to compute current outputs—they must see the entire audio sequence before processing, making real-time impossible. Problem: prevailing open models for semantic tokens (like WavLM) are non-causal—they need future audio to compute current embeddings. Kyutai's breakthrough was making the entire Moshi architecture causal by combining semantic and acoustic token generation into one real-time process via Mimi codec. For founders building real-time voice interfaces: causal architectures are mandatory for conversational AI, voice assistants, live translation, or anything requiring immediate response.

Audio transformers are architecturally similar to text transformers (both use attention mechanisms, similar layer structures) but differ in input representations and optimization targets. Key differences: (1) Input tokens—audio uses compressed acoustic tokens from neural codecs (Mimi generates 8 tokens per timestep), text uses subword tokens from tokenizers. (2) Sequence length—audio requires many more tokens per second (Moshi: 12.5 Hz frame rate) than text, making efficiency critical. (3) Evaluation—text has clear metrics (perplexity, accuracy), audio is subjective requiring human evaluation. Benefits of similarity: audio models inherit LLM advances (RLHF, distillation, hardware optimizations). Kyutai's Moshi uses Temporal+Depth transformers, adding architectural innovations beyond standard transformers.

Mimi is Kyutai's state-of-the-art neural audio codec that enables real-time voice generation by combining semantic tokens (content) and acoustic tokens (style) into a single causal generation process. Built on Neil Zeghidour's earlier SoundStream work at Google Brain, Mimi uses vector quantization summing to produce 8 tokens per timestep: 1 semantic (what's being said), 7 acoustic (how it sounds). Previous approaches modeled these separately and hierarchically, requiring non-causal models that couldn't process real-time. Mimi's innovation: everything is causal—the codec processes audio as it arrives, generating all tokens simultaneously with 160ms latency. This architectural breakthrough enabled Moshi's full-duplex conversations. For founders: Mimi is open-source and can be adapted for custom voice AI applications.

Hibiki translates speech into another language in real-time using the speaker's original voice—not a synthetic voice. Unlike traditional translation (speech→text→translate→new voice), Hibiki uses speech-to-speech translation with full-duplex architecture, preserving prosody, emotion, and speaker characteristics across languages. Technical approach: DSM (Delayed Streams Modeling) extends Kyutai's multi-stream architecture for bidirectional audio translation. Challenge: where do you find data of the same person in the same voice saying the same thing in multiple languages? Kyutai's solution uses architectural innovations and training techniques to transfer voice characteristics across languages. Hibiki ran on-device months before Apple's real-time translation. Use cases: International calls, multilingual customer service, global meetings, language learning.

SoundStream (2021) was the first neural audio codec, created by Neil Zeghidour at Google Brain, that could compress speech, music, and general audio at bitrates normally targeted by speech-only codecs. Before SoundStream, audio codecs were domain-specific with manual rules: speech codecs for speech (Opus, AMR), music codecs for music (MP3, AAC)—incompatible and requiring different engineering. SoundStream trained an encoder-decoder neural network to learn compression end-to-end, working as well as domain-specific codecs with ergonomic benefits. Impact: (1) Unified approach for all audio types, (2) Enabled neural audio generation by providing trainable compression, (3) Led to voice cloning discovery when Neil tested generation, (4) Became foundation for Gemini Live. SoundStream evolved into Kyutai's Mimi codec, enabling real-time conversational AI.

AudioLM (Google research by Olivier Teboul and Neil Zeghidour) introduced the concept of separating audio into semantic tokens (content/words) and acoustic tokens (voice style/emotion), using hierarchical generation: first semantic, then acoustic. This architecture enabled high-quality audio generation with controllable voice characteristics—a major advancement for voice cloning and synthesis. Problem: AudioLM's hierarchical approach required non-causal models for semantic tokens (like WavLM) that need future audio to compute current embeddings—making real-time impossible. Kyutai's contribution: Mimi codec combines both token types into single causal generation, solving the real-time problem. AudioLM's semantic/acoustic separation insight remains foundational—nearly all modern voice AI uses this concept, including Gradium's production models.

Use production-ready APIs from specialized audio AI companies rather than building from scratch. Gradium offers full-duplex conversational AI APIs with 160ms latency, multilingual support, and natural turn-taking—no need to build codecs, implement multi-stream architectures, or solve turn-taking yourself. Alternative approaches: (1) OpenAI's Advanced Voice Mode (turn-based, consumer-focused), (2) Open-source models—Kyutai's Moshi is open-source but requires productionization work, (3) Specialized APIs—Deepgram (transcription), ElevenLabs (TTS), AssemblyAI (STT) for component-based architectures. For founders: Start with APIs to validate product-market fit, consider custom models only when differentiation requires it. The $70M Gradium raised to productionize Kyutai's research shows the research-to-product gap is real.

Many AI researchers believe audio will dominate because: (1) Every sci-fi AI speaks naturally—HAL 9000, Jarvis, Samantha in Her—suggesting cultural intuition that voice is the ultimate interface. (2) Hands-free, eyes-free interaction is superior for most contexts (driving, cooking, walking, accessibility). (3) Natural for humans—we've evolved to communicate verbally for hundreds of thousands of years; reading/writing is only ~5,000 years old. (4) Emotional context—tone, prosody, and emotion are inherent to audio, making it superior for customer service, therapy, and social applications. (5) Lower barrier—speaking requires no literacy, typing skills, or screen access. Technology finally caught up in 2024 when Kyutai shipped full-duplex conversational AI with natural turn-taking, making this vision practically feasible.

Moshi's 7B parameters make it deployable on modern GPUs, edge devices, and even mobile phones—dramatically more accessible than massive multimodal models (Llama 3.1: 405B parameters). Minimum requirements for inference: (1) GPU memory: ~14GB for FP16, ~7GB for INT8 quantization, (2) Compute: Modern GPU (NVIDIA RTX 3090/4090, A100, H100) or Apple Silicon (M1/M2/M3 with sufficient RAM), (3) Mobile: Flagship smartphones with neural accelerators (iPhone 12+, high-end Android). For training from scratch: (1) Compute cluster for pre-training (though 50x+ cheaper than text models), (2) 7M hours audio + transcripts, (3) Post-training on 2k-20k hours conversational data. For founders: Inference is accessible; training from scratch requires expertise but fine-tuning Moshi on domain data is feasible.

Amplify Partners invested in Gradium (disclosed in article) based on the thesis that small specialized teams can outcompete Big Tech in audio AI despite losing in text models. Key investment criteria: (1) Domain expertise over scale—audio requires deep understanding of acoustics, psychophysics, and compression theory that can't be solved by throwing compute at the problem. (2) Economics favor startups—audio models are 50x+ cheaper to train than frontier text models, removing Big Tech's capital advantage. (3) Organizational advantages—focused audio labs avoid multimodal tradeoffs where voice teams negotiate with text/vision teams for resources. (4) Research-to-product pipeline—Kyutai handles fundamental research, Gradium productionizes into APIs. For founders pitching audio AI: emphasize technical depth, team's audio research pedigree, and specific use cases where your approach beats general-purpose models.

Typical human conversation has 200-250ms turn-taking latency—the pause between one person finishing and another starting to speak. Kyutai's Moshi achieves 160ms theoretical latency, faster than human conversation, making it imperceptibly responsive. Neil notes Moshi is 'in some senses even too fast' for natural conversation. For comparison: (1) Traditional voice assistants (Siri, Alexa): 1-3 seconds including network latency, (2) OpenAI Advanced Voice Mode: ~500-800ms (estimates), (3) xAI's voice demo: Higher latency than Moshi (2024-2025). Technical contributors to latency: codec compression/decompression, model inference, network transmission (for cloud models), turn-taking prediction (for turn-based models). For founders: sub-200ms latency is critical for natural conversation; anything above feels robotic and forces users to wait awkwardly.

Before 2020, audio papers submitted to top ML conferences (NeurIPS, ICML, ICLR) had lower acceptance rates than vision or text papers—not because of quality, but because audio was culturally considered less important and 'uncool.' This created a self-reinforcing cycle: (1) Fewer audio researchers meant fewer reviewers comfortable evaluating audio papers, (2) Lower acceptance rates discouraged researchers from entering audio, (3) Less funding and prestige flowed to audio divisions at big labs, (4) Top PhD students chose vision or NLP over audio. Impact: This neglect created opportunity—audio became underexplored whitespace where small teams with domain expertise could make disproportionate breakthroughs. Kyutai's founders all came from underfunded audio divisions that were 'cooking in relative obscurity.' For founders: Look for domains with similar dynamics—culturally dismissed but technically promising.

Research-to-product pipeline model: Non-profit labs (Kyutai) conduct fundamental research and publish open-source, while for-profit spinouts (Gradium) productionize into commercial APIs. Benefits: (1) Non-profit attracts research talent who value open publication over equity, (2) Funding sources—non-profits access philanthropic capital (Eric Schmidt, French billionaires), for-profits raise VC ($70M Series A), (3) Division of labor—researchers focus on breakthroughs, engineers focus on production polish, (4) Faster iteration—Kyutai ships prototypes proving feasibility, Gradium makes them enterprise-ready in months. Examples: OpenAI started non-profit then added for-profit arm (different structure), DeepMind (acquired by Google), Anthropic (PBC structure). For founders: This model works when fundamental research is expensive/risky but productionization has clear commercial path.

Key open-source models: (1) Moshi (Kyutai)—first full-duplex conversational AI, 7B parameters, runs on mobile. (2) Hibiki (Kyutai)—real-time speech-to-speech translation preserving speaker voice. (3) SoundStream (Google)—neural audio codec. (4) AudioLM (Google)—semantic/acoustic token separation. (5) Seamless (Meta)—multilingual translation (unmaintained). Datasets: (1) Fisher (LDC)—2,000 hours phone conversations with separated channels for full-duplex training. (2) Voxpopuli—400k hours multilingual speech. (3) MADLAD-400—machine translation across 400+ languages. (4) LibriSpeech—1,000 hours English audiobooks for ASR. For founders: Start with Moshi for conversational AI, Fisher for training full-duplex models, consider fine-tuning open models on domain data before training from scratch.