VoxAudio: Vocalized Audio Synthesis via
Multi-Reward Autoregressive Flow Matching
Abstract
Vocalized audio synthesis, the task of generating audio in which intelligible speech is embedded within an environmental soundscape, underpins applications such as podcast production and video dubbing. Existing Text-to-Audio (T2A) systems either reduce quoted speech to unintelligible vocal murmur or delegate it to a separate TTS model with post-hoc mixing, which forfeits control over when speech occurs and how it interacts with the scene.
We present VoxAudio, a causal autoregressive flow matching model that addresses this problem from three complementary aspects. At the architecture level, chunk-wise causal factorization with independent per-chunk noise levels lets audio be emitted through sliding-window streaming inference with KV caching at variable target durations; to enable inference at arbitrary chunk granularities, the model is pretrained with randomized chunk boundaries. At the preference level, multi-reward Negative-aware FineTuning (NFT) jointly optimizes semantic fidelity, linguistic accuracy, aesthetic quality, and temporal grounding; to prevent the policy from inflating the linguistic reward by suppressing the environmental background, this reward is computed on vocals isolated by source separation. At the data level, we build VoxCorpus, a large-scale corpus whose captions quote the verbatim transcript of embedded speech with time intervals, and VoxBench, an interval-annotated benchmark with a temporal-grounding metric. Experiments on four benchmarks spanning general audio, speech, and unified vocalized audio validate the effectiveness and efficiency of VoxAudio.
Vocalized Scene Audio
Prompts ask for intelligible speech embedded in an environmental soundscape.
VoxBench8.2sA streamer broadcasting a late-night coding session says, 'Compiling now - wish me luck!' amid keyboard clatter and CPU fan whirring.
VoxBenchA patient in a quiet examination room listens to their own heartbeat while waiting for a doctor, saying: "I think it's just stress, but let's check.
VoxBenchA woman comments on the peacefulness, saying: "It's so serene here, isn't it?" while birds chirp and a stream flows in the background.
MECAT-enA male voice in a low-quality recording recites abstract phrases, saying, "for everyone we know. Dissociative. Hyper-associative." amid the ambient sounds of a flowing stream.
MECAT-enA male speaker discussing code structure explains, "timestamp itself. So the timestamp is the actual string that we get way up here. So if you look at this timestamp right here," amid intermittent keyboard typing.
MECAT-enA man explaining technical concepts says, "vector 3 which adds to whatever position we're at. So we're going to take, in this case, the position." amid intermittent clicking sounds.
MECAT-enA sound engineer working on a track says, "That was a little angle efficient. No, I'm sorry." amid experimental electronic music and abrupt thuds.
Speech Generation
Speech synthesized from Seed-TTS-eval text. VoxAudio synthesizes from text alone, whereas the dedicated TTS baselines (F5-TTS, CosyVoice3) additionally receive a reference utterance for voice cloning; Dasheng-AudioGen is the unified text-to-audio baseline.
Its European headquarters is in Waterloo, Belgium.
She was involved in several record productions.
A Roman Road ran north from Hamworthy to Badbury Rings, a Roman transport hub.
General Audio
Text-to-audio generation on AudioCaps captions without any speech requirement, compared with the ground-truth recording, a dedicated T2A system (TangoFlux), and the unified baseline (Dasheng-AudioGen).
A toilet flushing water as music plays in the background
The clinking of glasses with some rustling
A telephone dialing tone followed by a plastic switch flipping on and off
Variable-Duration Streaming Generation
VoxAudio generates at the target duration requested in the prompt via duration conditioning.
8sThe slow, deliberate footsteps on a stone floor, accompanied by the creaking of an old, heavy wooden door being pushed open, the rustling of a dark, flowing cloak, and the faint, magical hum of energy building up, culminating in a sudden, intense whoosh as a mystical portal is activated
13sGentle river flowing with the sound of water lapping against the shore, accompanied by the soft rustling of leaves in a light breeze and the occasional chirping of birds
20sA mother calls her child for dinner, saying “Dinner's ready, sweetie!” as the hallway echoes with the sound of a vacuum cleaner.
30sThe rhythmic thud of a basketball being dribbled on a hardwood court, echoing through a large gymnasium, interspersed with the squeaking of sneakers and the occasional metallic clang as the ball hits the rim, all underlaid by the ambient hum of an excited crowd
© 2026 VoxAudio Project.