Open-Robust Lab

Research / AURA

AURA

A unified multimodal framework for conversational music editing.

Research release Updated September 13, 2026
Code — AURAModel — AURADataset — AURA-Chat-EditLive demoPaper
AURA architecture showing multimodal inputs, the language model, concept and audio projectors, and the MusicGen decoder.

What this project does

AURA turns music editing into a conversation. A user provides a song and a natural-language request—such as adding an instrument, removing a stem, or changing a mood—and the system replies conversationally before rendering the edited audio.

Why it matters

Most instruction-guided music editors handle each request in isolation. Creative work is usually iterative: people listen, respond, revise, and refine. AURA is designed around that dialogue, using conversation history alongside the audio so an editing session can develop over multiple turns.

Method overview

The framework combines a multimodal language-model “thinker” with a music generation bridge. The thinker interprets the request and emits typed edit tokens. Those tokens, together with frame-aligned source features, condition a frozen MusicGen backbone through the BiFAM fusion mechanism. Localized edits are anchored to preserve audio outside the requested segment and crossfaded at the boundaries.

Evaluation highlights

The paper evaluates AURA on Slakh2100 and MoisesDB, reporting improvements in edit correctness and preservation over instruction-guided baselines. The public repository includes training, inference, serving, and evaluation code for reproducing the documented workflow.

Release artifacts

The code, AURA adapter checkpoints, AURA-Chat-Edit dataset, and arXiv paper are public. The model card reports approximately 2.3 GB of AURA adapters; the Qwen and MusicGen base models are downloaded separately and remain subject to their own terms. A public research demo is also available for exploring the project in the browser.

Demo

The AURA interactive demo presents the project as a research demonstration. Review its on-page guidance before uploading or editing audio, and only use material you have permission to use.

Listen to two examples

Each comparison contains the original input, the held-out reference target, and AURA’s generated edit. The metrics shown are the per-sample values reported by the official demo; FAD is not included because it is a corpus-level metric.

Hear every published example on the full AURA demo →

Scope and limitations Read before using

Intended use

  • Research on conversational and instruction-guided music editing.
  • Editing supported audio with natural-language requests in research settings.
  • Reproducing and extending the published AURA evaluations.

Out of scope

  • Editing material without the rights or permission required to use it.
  • Treating generated audio as guaranteed to preserve every unaffected detail.

Known limitations

  • The public demo is a research demonstration and should not be treated as a production audio editor.
  • Running the full system requires separate large base models and GPU resources.
  • Published evaluation covers specific music datasets and edit categories and may not represent every genre, recording condition, or request.
  • The base Qwen and MusicGen models retain their own licenses and usage terms.
CitationarXiv
BibTeX
@misc{trinh2026auraunifiedmultimodalframework,
  title={AURA: Unified Multimodal Framework for Conversational Music Editing},
  author={Quoc-Huy Trinh and Minh-Van Nguyen and Debesh Jha},
  year={2026},
  eprint={2609.14344},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2609.14344}
}
Related projects1