Bass Accompaniment Generation via Latent Diffusion

Abstract
The ability to automatically generate music that appropriately matches an arbitrary input track is a challenging task. We present a novel controllable system for generating single stems to accompany input musical mixes of any length. At the core of our method are audio autoencoders that efficiently compress audio waveform samples into invertible latent representations, and a conditional latent diffusion model that takes as input the latent encoding of a mix and generates the latent encoding of a corresponding stem. To provide control over the timbre of generated samples, we introduce a technique to ground the latent space to a user-provided reference style during diffusion sampling. For further improving audio quality, we adapt classifier-free guidance to avoid distortions at high guidance strengths when generating an unbounded latent space. We train our model on a dataset of pairs of mixes and matching bass stems. Quantitative experiments demonstrate that, given an input mix, the proposed system can generate basslines with user-specified timbres. Our controllable conditional audio generation framework represents an important advance for creating generative AI tools to assist musicians in music production.

Examples

We showcase basslines generated by our system using mixes from MUSDB18. We randomly mix 1 to 3 stems (Drums, Vocals, Misc) for each track and use the resulting mixes as input conditioning for our system. All samples are converted to MP3.

Style Grounding

We generate samples using a randomly cropped ~23 second bass sample from MUSDB18 as the style reference using the proposed grounding technique. The input mix and the style bass sample do not belong to the same original MUSDB18 track.

No Grounding

In the following examples, the system is not grounded to any specific style.

Longer Generated Basslines

We showcase basslines generated by our system using input mixes with a different (longer) length to the samples used during training. Thanks to a modified attention mechanism, the system generates coherent samples even when mixes with different length are used as input.