Can a Recurrent LLM Write a Song? Inside LiuBai-RWKV + YuE2

Xiaol.x

AI-assisted summary of Can a Recurrent LLM Write a Song? Inside LiuBai-RWKV + YuE2 — original source. Check the original for full context.

Source published: 2026-09-25

Key ideas

LiuBai-RWKV replaces the transformer autoregressive stage in YuE2 with a 3-billion-parameter RWKV-7 model, generating discrete semantic tokens with a constant-memory footprint instead of an expanding KV-cache. (0:00)
To bypass the unreleased YuE2 audio encoder, the author created R1A, an inverse tokenizer that converts MERT-v2 features to YuE2 semantic tokens with a 0.981 MERT cosine similarity and ~0.33 WER. (1:52)
Interleaving lyrics and audio tokens at the section level (R3.1 format) reduced generation Word Error Rate from 1.37 to 1.05 compared to monolithic serialization (G3), but distinct lyric word overlap remains stuck at roughly 10%. (5:09)
The model reliably renders genre timbre, song structure, and vocal textures across six diverse styles, but exact lyric adherence remains the core unsolved failure mode of the recurrent architecture. (5:57)

Detailed notes

## Background Published on YouTube by Xiaol.x on September 25, 2026, this video introduces LiuBai-RWKV, an experimental 3-billion-parameter RWKV-7 music generator designed to turn style prompts and lyrics into YuE2 semantic tokens for stereo audio rendering. According to the page description, the project is presented as an honest research build rather than a finished commercial Suno replacement, detailing recurrent memory mechanics, token serialization, and why lyric adherence remains an unsolved challenge.

## TL;DR LiuBai-RWKV replaces the standard transformer in YuE2's music generation pipeline with an RWKV-7 architecture, maintaining a constant-memory recurrent state while autoregressively emitting discrete semantic tokens at 25 codes per second. Because the upstream audio encoder was unreleased, the author developed R1A—a custom bridge mapping MERT-v2 features to YuE2 semantic tokens—to prepare the Suno-94k dataset for training across four A100 GPUs. Restructuring the sequence layout from monolithic lyric blocks (G3) to section-interleaved blocks (R3.1) proved crucial, significantly lowering Word Error Rate and preventing total vocal collapse. Despite successfully generating realistic instrumentals, vocal textures, and verse-chorus arrangements across six diverse genres, the model still frequently produces phonetic babble rather than requested lyrics, highlighting the ongoing difficulty of cross-modal alignment without cross-attention.

## Atmosphere & style The presentation carries the grounded, unpretentious tone of an open-source engineering lab notebook, pointedly avoiding commercial AI hype in favor of raw audio and transparent ablation metrics. Xiaol.x communicates with methodical, analytical clarity, guiding viewers through architectural trade-offs and pragmatic workarounds before playing uncurated, warts-and-all audio demonstrations. The resulting experience feels intellectually candid and deeply authentic, balancing genuine enthusiasm for recurrent sequence modeling with an unflinching examination of the model's vocal alignment failures.

## Bulletin - [00:00](https://www.youtube.com/watch?v=NyJZCDt6wws&t=0s) LiuBai-RWKV replaces the transformer autoregressive stage in YuE2 with a 3-billion-parameter RWKV-7 model, generating discrete semantic tokens with a constant-memory footprint instead of an expanding KV-cache. - [01:52](https://www.youtube.com/watch?v=NyJZCDt6wws&t=112s) To bypass the unreleased YuE2 audio encoder, the author created R1A, an inverse tokenizer that converts MERT-v2 features to YuE2 semantic tokens with a 0.981 MERT cosine similarity and ~0.33 WER. - [05:09](https://www.youtube.com/watch?v=NyJZCDt6wws&t=309s) Interleaving lyrics and audio tokens at the section level (R3.1 format) reduced generation Word Error Rate from 1.37 to 1.05 compared to monolithic serialization (G3), but distinct lyric word overlap remains stuck at roughly 10%. - [05:57](https://www.youtube.com/watch?v=NyJZCDt6wws&t=357s) The model reliably renders genre timbre, song structure, and vocal textures across six diverse styles, but exact lyric adherence remains the core unsolved failure mode of the recurrent architecture.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=NyJZCDt6wws&t=0s) LiuBai-RWKV acts as a high-level musical planner that outputs discrete YuE2 semantic tokens instead of generating raw audio directly. - [00:00](https://www.youtube.com/watch?v=NyJZCDt6wws&t=0s) The semantic token vocabulary contains 32,768 codes emitted at 25 tokens per second, where each token governs roughly 40 milliseconds of audio. - [01:52](https://www.youtube.com/watch?v=NyJZCDt6wws&t=112s) YuE2's downstream non-autoregressive (NAR) acoustic model and audio VAE are kept intact to handle waveform synthesis. - [01:52](https://www.youtube.com/watch?v=NyJZCDt6wws&t=112s) Because YuE2 lacked an upstream audio encoder, the custom R1A tokenizer was built to map MERT-v2 audio representations directly into YuE2 tokens. - [03:36](https://www.youtube.com/watch?v=NyJZCDt6wws&t=216s) Training was conducted on a cleaned subset of 93,058 songs from the Suno-94k corpus across 4× NVIDIA A100 GPUs using BF16 precision. - [03:36](https://www.youtube.com/watch?v=NyJZCDt6wws&t=216s) RWKV-7 uses fixed-size recurrent state updates to avoid the linear memory scaling of transformer KV-caches during streaming audio generation. - [05:09](https://www.youtube.com/watch?v=NyJZCDt6wws&t=309s) Monolithic sequence formatting (G3) failed because placing lyrics thousands of tokens before audio caused the network to behave as an unconditional generator. - [05:09](https://www.youtube.com/watch?v=NyJZCDt6wws&t=309s) The R3.1 serialization format solves long-distance credit assignment by interleaving local section metadata, lyrics, and audio code blocks. - [05:57](https://www.youtube.com/watch?v=NyJZCDt6wws&t=357s) Testing across punk, Motown, metal, reggaeton, jazz, and ambient prompts showed strong genre adaptation and vocal tone alongside persistent phonetic drift. - [06:45](https://www.youtube.com/watch?v=NyJZCDt6wws&t=405s) R3.1 beat the G3 baseline in 15 out of 20 evaluation prompts and imposed a +0.106 nats penalty under permuted text, confirming real conditioning on local lyric blocks. - [07:50](https://www.youtube.com/watch?v=NyJZCDt6wws&t=470s) Future research focuses on finer line-level interleaving, timestamp forced-alignment, and audio-grounded CTC or ASR loss objectives to improve lyric fidelity.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=NyJZCDt6wws&t=0s) Rethinking Song Generation: LiuBai-RWKV and Semantic Tokens

The project explores whether a language model can compose complete songs without relying on a continually ballooning transformer attention cache. LiuBai-RWKV tackles this as a research experiment using a 3-billion-parameter RWKV-7 architecture. Instead of outputting raw audio waveforms directly, the model takes a text style caption alongside written lyrics and autoregressively predicts a stream of discrete semantic music tokens. These tokens are subsequently handed off to YuE2's rendering pipeline to produce stereo audio. The creator emphasizes that this is an honest research build rather than a finished commercial Suno replacement, characterized by real checkpoints, uncurated audio samples, and distinct failure modes.

```mermaid flowchart LR accTitle: LiuBai-RWKV System Pipeline accDescr: Pipeline showing inputs feeding into RWKV-7 to produce semantic tokens, which are decoded by YuE2 into audio. subgraph Inputs ["Input Prompts"] A["Style Caption"] B["Structured Lyrics"] end subgraph Core ["LiuBai (Planner)"] C["RWKV-7 3B LLM"] end subgraph Renderer ["YuE2 Stack (Renderer)"] D["Semantic Tokens\n(32,768 Codebook)"] E["NAR Acoustic Model"] F["Audio VAE"] G["Stereo Audio"] end A --> C B --> C C --> D D --> E E --> F F --> G ```

Within this design, LiuBai acts as the high-level musical planner, predicting compact structural instructions while leaving acoustic realization to a fixed renderer. Songs are broken into standard structural units—intros, verses, choruses, and bridges. At runtime, each generated section is framed by dedicated boundary markers, beginning with `<sos>` and `<soa>`, followed by a sequence of YuE2 semantic codes, and terminating with `<eoa>`. Rather than representing discrete notes or raw sample slices, these tokens function as dense musical blueprints:

- **Semantic Codebook:** 32,768 discrete token IDs. - **Vocabulary Placement:** In the R3.1 setup, audio codes occupy token IDs 65,544 and above inside the RWKV vocabulary. - **Temporal Resolution:** Emitted at 25 codes per second, meaning each semantic token spans approximately 40 milliseconds of audio. - **Rendering Pipeline:** YuE2 expands the compact semantic instructions into dense acoustic details via a non-autoregressive (NAR) stage before the VAE decodes the final stereo waveform.

---

[01:52](https://www.youtube.com/watch?v=NyJZCDt6wws&t=112s) Decoupling the Architecture and the R1A Audio Inverse Tokenizer

YuE2 serves as both the conceptual benchmark and the rendering partner for this build. Under standard YuE2 operation, an autoregressive language model generates semantic tokens, a non-autoregressive model builds acoustic representations, and a variational autoencoder (VAE) synthesizes the waveform. LiuBai preserves the latter two stages unchanged, replacing only the text-to-semantic-token language modeling stage with RWKV-7. This clean architectural separation holds the token representation and rendering engine fixed, providing a controlled testbed to see if a recurrent model can handle complex musical composition. The computational upside is substantial—retaining a constant-size memory state during inference—but the model faces the steep challenge of learning the alignment between lyrics, song sections, and musical tokens without cross-attention.

A significant hurdle arose during training: YuE2 had not released the upstream audio encoder required to tokenize raw song data into YuE2 semantic tokens. To circumvent this, the author developed a custom inverse tokenizer dubbed R1A:

- **Feature Extraction:** R1A maps from MERT-v2 self-supervised audio representations directly to predicted YuE2 semantic tokens. - **Teacher-Student Alignment:** It is trained alongside a joint teacher that runs predicted tokens through the frozen YuE2 rendering stack across 20 unseen evaluation songs. - **Validation Results:** The resulting R1A tokenizer head achieved a Word Error Rate (WER) of ~0.33 when transcribed against reference text, with an audio feature alignment score of 0.981 MERT cosine similarity. - **Scope of Metrics:** These figures confirm that the synthetic token bridge and round-trip audio reconstruction are reliable enough to generate training data for the subsequent RWKV experiments.

---

[03:36](https://www.youtube.com/watch?v=NyJZCDt6wws&t=216s) Training Scale, Recurrent Memory, and Serialization Formats

The training pipeline relies primarily on the Suno-94k dataset, which consists of 94,174 songs totaling roughly 5,423 hours of music annotated with style tags and structured lyrics. Following data cleaning and filtering against the YuE2 pipeline requirements, the corpus settled at 93,058 usable songs. The baseline RWKV model was trained from scratch across 4× NVIDIA A100 GPUs using BF16 mixed-precision arithmetic, an 8,000-token context window, and an expanded vocabulary accommodating the 32,768 semantic audio codes.

Traditional transformer decoders store past keys and values in an expanding KV-cache, causing memory requirements to scale linearly with song length. By contrast, RWKV-7 updates a fixed-size recurrent hidden state at every step. This makes streaming generation attractive because the computational memory footprint never balloons. However, the author cautions that recurrent state updates are not "a magic musical memory"; the network must still deduce timing, repetition patterns, and genre shifts purely from sequence structure.

```mermaid flowchart TD accTitle: G3 vs R3.1 Sequence Serialization Comparison accDescr: Diagram illustrating the structural differences between the monolithic G3 sequence and the section-interleaved R3.1 format. subgraph G3 ["G3 Monolithic Format (Long Distance)"] direction LR G1["[Style Caption]"] --> G2["[Full Lyrics Block]"] --> G3A["[Audio Tokens: Section 1]"] --> G4["[Audio Tokens: Section 2]"] end subgraph R31 ["R3.1 Interleaved Format (Local Conditioning)"] direction LR R1["[Header]"] --> R2["[Sec 1 Text]"] --> R3A["[Sec 1 Audio]"] --> R4["[Sec 2 Text]"] --> R5["[Sec 2 Audio]"] end ```

The physical serialization of sequence tokens proved to be the decisive factor in whether the model learned to associate lyrics with music:

- **G3 Format (Monolithic Whole-Song Layout):** The earlier G3 training run concatenated metadata and lyrics entirely at the front of the sequence, followed by an unbroken block of audio tokens. Because lyric text sat thousands of tokens away from its corresponding audio, the network functioned as an unconditional music generator and failed to link specific words to vocal passages. - **R3.1 Format (Interleaved Section Layout):** R3.1 restructured sequences into repeating local blocks: a section marker (e.g., verse, chorus), the specific lyric text for that section, the audio start token (`<soa>`), the local YuE2 semantic codes, and an audio end token (`<eoa>`). Interleaving text and audio blocks drastically reduced the credit assignment distance, enabling section-by-section decoding.

---

[05:57](https://www.youtube.com/watch?v=NyJZCDt6wws&t=357s) Evaluation Across Six Genres and the Lyric Adherence Dilemma

To verify real-world generalization rather than relying on cherry-picked highlights, generation was evaluated across six diverse prompts from held-out evaluation sets:

1. **Punk:** *"Don't try to understand me"* 2. **Motown:** *"Curly hair"* 3. **Industrial Metal:** *"DB82Xure"* 4. **Reggaeton:** *"Bailando"* 5. **Jazz:** *"Night and Day"* 6. **Ambient:** *"Não gosto de frutas"*

The listening tests showcase LiuBai's core strengths alongside its most glaring failure mode. The model easily captures surface musical qualities: instrumental timbre matches the requested genres, vocal deliveries possess realistic human textures, and arrangements adhere to conventional verse-chorus song structures. Yet when listening closely to vocal performance, the generated singing often diverges into phonetic babble or hallucinated phrases that bear no resemblance to the input prompts: "The model can learn a musical surface before it learns reliable text to voice alignment."

Comparative evaluation against the monolithic G3 baseline shows measurable architectural gains:

---

[07:50](https://www.youtube.com/watch?v=NyJZCDt6wws&t=470s) Next Research Steps and Open-Source Roadmap

The quantitative and qualitative evidence points to a single core conclusion: section-level interleaving significantly stabilizes song structure and local conditioning, but true lyric adherence remains an unsolved challenge for the recurrent architecture. To close the remaining gap between musical form and phonetic precision, several concrete research directions are planned:

- **Fine-Grained Alignment:** Expanding forced-alignment preprocessing to provide precise timestamp boundaries linking lyrics directly to audio blocks in the training dataset. - **Line-Level Block Serialization:** Moving beyond stanza-level blocks toward finer-grained, line-by-line interleaving to bring text prompts into even closer proximity with their corresponding audio codes. - **Acoustic and Perceptual Loss Objectives:** Introducing loss functions anchored to the rendered audio signal, such as Connectionist Temporal Classification (CTC) losses, automatic speech recognition (ASR) reward signals, or flow-matching losses derived from the joint teacher model. - **Open-Source Licensing:** Working around license constraints tied to the upstream YuE2 ecosystem so LiuBai-RWKV can be released cleanly as an open-source toolchain.

The video concludes by inviting developers to inspect the project repository, which documents these limitations, training run metrics, and negative results openly. Rather than being swayed by single polished audio clips, observers are encouraged to track the project's ongoing pull requests and check whether future runs improve the core lyric binding metric.

## Notable quotes > [00:00](https://www.youtube.com/watch?v=NyJZCDt6wws&t=0s) This is an honest research build—not a finished Suno replacement. — Xiaol.x

> [00:00](https://www.youtube.com/watch?v=NyJZCDt6wws&t=0s) The model captures genre, vocal texture, and song-like structure, but often fails to sing the requested lyrics accurately. — Xiaol.x

> [01:52](https://www.youtube.com/watch?v=NyJZCDt6wws&t=112s) The computational upside is substantial—retaining a constant-size memory state during inference—but the model faces the steep challenge of learning the alignment between lyrics, song sections, and musical tokens without cross-attention. — Xiaol.x

> [03:36](https://www.youtube.com/watch?v=NyJZCDt6wws&t=216s) recurrent state updates are not 'a magic musical memory'; the network must still deduce timing, repetition patterns, and genre shifts purely from sequence structure. — Xiaol.x

> [03:36](https://www.youtube.com/watch?v=NyJZCDt6wws&t=216s) Because lyric text sat thousands of tokens away from its corresponding audio, the network functioned as an unconditional music generator and failed to link specific words to vocal passages. — Xiaol.x

> [05:57](https://www.youtube.com/watch?v=NyJZCDt6wws&t=357s) The model can learn a musical surface before it learns reliable text to voice alignment. — Xiaol.x

> [05:57](https://www.youtube.com/watch?v=NyJZCDt6wws&t=357s) R3.1 eliminated catastrophic collapse where singing degraded completely — Xiaol.x

> [07:50](https://www.youtube.com/watch?v=NyJZCDt6wws&t=470s) section-level interleaving significantly stabilizes song structure and local conditioning, but true lyric adherence remains an unsolved challenge for the recurrent architecture. — Xiaol.x

## Mentioned LiuBai-RWKV, RWKV-7, YuE, YuE2, Suno, Suno-94k, MERT-v2, MMS forced alignment, HeartTranscriptor, BlinkDL, M-A-P, Xiaol.x, NVIDIA A100, R1A

## Video description I’m building LiuBai-RWKV, a 3-billion-parameter RWKV-7 music generator that turns style and lyrics into YuE2 semantic tokens, then renders them into stereo audio.

This episode explains:

- How semantic music tokens represent a song - How RWKV maintains recurrent memory during generation - Where YuE2’s AR, NAR, and VAE stages fit - How R3.1 improves over the earlier G3 format - Six generated songs across punk, Motown, metal, reggaeton, jazz, and ambient - Why lyric adherence is still the central unsolved problem

This is an honest research build—not a finished Suno replacement. The model captures genre, vocal texture, and song-like structure, but often fails to sing the requested lyrics accurately.

Chapters:

00:00 The hook 00:50 What semantic tokens look like 01:52 Where YuE2 fits 02:40 The missing audio encoder 03:36 Data and training 04:27 What RWKV remembers 05:09 How a song is serialized 05:57 Six generated songs 06:45 The evidence 07:50 What comes next 08:42 Closing thoughts

Code, samples, and experiment logs: https://github.com/xiaol/LiuBai-RWKV

Credits: RWKV-7 by BlinkDL; YuE/YuE2 by M-A-P; MERT-v2; MMS forced alignment; HeartTranscriptor; suno-94k.

[Source](https://www.youtube.com/watch?v=NyJZCDt6wws)

中文简报

循环大模型能写歌吗？深入解析 LiuBai-RWKV + YuE2 LiuBai-RWKV 在 YuE2 的音乐生成流程中以 RWKV-7 架构取代了标准 Transformer，在维持恒定内存循环状态的同时，以每秒 25 个代码的速率自回归输出离散语义词元。由于上游音频编码器未发布，作者开发了 R1A——一种将 MERT-v2 特征映射到 YuE2 语义词元的自定义桥接器，以此处理 Suno-94k 数据集并在四张 A100 GPU 上完成训练。将序列结构从整块歌词格式（G3）重构为分段交错块格式（R3.1）被证明至关重要，此举显著降低了字错率并避免了人声彻底崩溃。尽管该模型在六种不同流派中成功生成了逼真的器乐编配、人声质感和主副歌结构，但它仍频繁产生类似发音杂音而非指定歌词，凸显出在不使用交叉注意力机制的情况下进行跨模态对齐的持续难题。 LiuBai-RWKV 用一个 30 亿参数的 RWKV-7 模型替换了 YuE2 中的 Transformer 自回归阶段，以恒定内存占用生成离散语义词元，无需不断膨胀的 KV-cache。 (0:00) 为绕过未发布的 YuE2 音频编码器，作者开发了逆分词器 R1A，以 0.981 的 MERT 余弦相似度和约 0.33 的 WER 将 MERT-v2 特征转换为 YuE2 语义词元。 (1:52) 与整体式序列化（G3）相比，在分段层级交错排列歌词与音频词元（R3.1 格式）将生成的字错率从 1.37 降低到 1.05，但独立歌词词汇的重合度仍停留在约 10%。 (5:09) 该模型能够在六种不同风格中可靠呈现流派音色、歌曲结构和人声质感，但精确依循歌词仍是该循环架构核心的未解失效模式。 (5:57)

Reading summary

LiuBai-RWKV explores whether a 3-billion-parameter recurrent model can replace transformer decoders in full-song generation by pairing constant-memory sequence planning with YuE2's rendering engine. Research by creator Xiaol.x demonstrates that while recurrent architectures can master structural arrangements and genre styles, binding generated vocals to written lyrics requires strict section-level data serialization. Decoupling Song Planning from Acoustic Rendering (0:00) LiuBai-RWKV tests musical composition without the expanding key-value cache of transformer decoders by utilizing an RWKV-7 architecture. Instead of synthesizing raw waveforms directly, the 3-billion-parameter model functions as a structural planner that autoregressively predicts discrete semantic tokens from a style caption and lyric prompt. These semantic tokens are subsequently handed to YuE2's frozen non-autoregressive acoustic stage and variational autoencoder to synthesize final stereo audio. In this decoupled pipeline, the semantic tokens operate as musical blueprints rather than raw audio slices. The system employs a 32,768-token codebook mapped to token IDs 65,544 and above within the RWKV vocabulary. Emitted at 25 tokens per second—each spanning roughly 40 milliseconds—the tokens allow the recurrent network to maintain a constant-size hidden memory state during inference while delegating dense acoustic synthesis to the downstream renderer. Bridging Semantic Tokenization with R1A (1:52) Because the YuE2 project had not released an upstream audio encoder to convert raw music into its semantic token format, Xiaol.x developed a custom inverse tokenizer named R1A. This module translates MERT-v2 self-supervised audio representations directly into predicted YuE2 semantic codes, utilizing a joint teacher setup validated across 20 unseen evaluation songs. Evaluation of the R1A bridge demonstrated a Word Error Rate of approximately 0.33 against reference transcriptions and an audio alignment score of 0.981 MERT cosine similarity. This validated token bridge allowed the author to prepare training data from the Suno-94k dataset, which was filtered down to 93,058 usable songs (roughly 5,423 hours) and trained from scratch on 4× NVIDIA A100 GPUs using BF16 precision and an 8,000-token context window. Solving Conditioning Failure Through Interleaved Serialization (3:36) Without transformer cross-attention mechanisms, the physical serialization of training tokens proved critical for teaching the recurrent network to associate lyrics with music. The project's initial G3 training run concatenated metadata and lyrics at the beginning of the sequence, followed by an unbroken block of audio tokens. Because lyrics sat thousands of tokens away from corresponding vocal passages, the network effectively operated as an unconditional music generator that ignored text prompts. To bridge this distance, the revised R3.1 format interleaved lyrics and audio into localized section blocks containing section headers, specific lyric stanzas, and audio tokens framed by dedicated start (<soa>) and end (<eoa>) markers. This reduced the credit assignment distance, resulting in a +0.106 nats teacher-forcing loss penalty when text was permuted and improving head-to-head win rates over the monolithic baseline from 5 to 15 out of 20 evaluation prompts. Stylistic Fluency Versus Phonetic Lyric Hallucination (5:57) Evaluation across six distinct musical genres—punk, Motown, industrial metal, reggaeton, jazz, and ambient—revealed that LiuBai reliably captures surface musical elements. Across runs, instrumental timbres matched requested genres, arrangements respected conventional verse-chorus structures, and stylistic MERT cosine similarity held stable at roughly 0.79. However, fine-grained lyric adherence remains a major failure mode. Listening tests revealed that vocal performances frequently devolve into phonetic babble or hallucinated phrases unrelated to input prompts. While R3.1 improved mean Word Error Rate from G3's 1.37 down to 1.05 and prevented complete vocal collapse, generated singing retained only about 10% distinct lyric word overlap, confirming the author's observation that the model learns musical surface features well before achieving reliable text-to-voice alignment. Xiaol.x concludes that while section interleaving stabilizes song structure, precise lyric adherence in recurrent architectures remains unresolved. Future development plans include timestamped forced alignment, line-by-line block serialization, and audio-level loss objectives such as CTC or speech-recognition rewards, alongside resolving YuE2 licensing dependencies prior to an open-source toolchain release.

Source quotes

[0:00] Xiaol.x: This is an honest research build—not a finished Suno replacement. [0:00] Xiaol.x: The model captures genre, vocal texture, and song-like structure, but often fails to sing the requested lyrics accurately. [1:52] Xiaol.x: The computational upside is substantial—retaining a constant-size memory state during inference—but the model faces the steep challenge of learning the alignment between lyrics, song sections, and musical tokens without cross-attention. [3:36] Xiaol.x: recurrent state updates are not 'a magic musical memory'; the network must still deduce timing, repetition patterns, and genre shifts purely from sequence structure. [3:36] Xiaol.x: Because lyric text sat thousands of tokens away from its corresponding audio, the network functioned as an unconditional music generator and failed to link specific words to vocal passages. [5:57] Xiaol.x: The model can learn a musical surface before it learns reliable text to voice alignment. [5:57] Xiaol.x: R3.1 eliminated catastrophic collapse where singing degraded completely [7:50] Xiaol.x: section-level interleaving significantly stabilizes song structure and local conditioning, but true lyric adherence remains an unsolved challenge for the recurrent architecture.

Original transcript

[0:00] What if [music] a language model could [0:02] write a song without carrying a growing [0:04] attention cache? I'm building that [0:06] experiment in Leu by RWKV, [0:12] a 3 billion parameter [music] RWV [0:16] 7 model that reads a style caption and [0:18] lyrics predicts a stream of music tokens [0:22] and hands those tokens to UE2 [0:26] to render into audio. This is not a [0:29] finished pseudo replacement. It is a [0:32] research build with real checkpoints, [0:35] real samples, and very visible failure [0:37] modes. In this episode, I'll show the [0:40] whole path, explain what RWKV is doing, [0:44] compare the design with UE2, and then [0:47] listen to the result honestly. Here is [0:50] what RWKV predicts. Not a waveform, but [0:54] UE2 semantic tokens. A section starts [0:58] with SOS and S SOA then UE2 codes and [1:02] ends with EOA. The code book has 32,768 [1:09] IDs. R3 one places them from 65,544 [1:14] upward in the RWKV vocabulary. At 25 [1:19] codes per second, each code covers about [1:22] 40 milliseconds. [1:25] A held out song is split into sections. [1:28] Intro, verse, chorus, bridge. A token is [1:32] not a note or audio sample. It is a [1:34] compact musical instruction. [1:37] U2 [1:39] expands it into acoustic detail and the [1:42] VAE renders stereo audio. Lubi can [1:47] replace the planner while the renderer [1:49] stays fixed. [1:51] UE2 is the comparison point and the [1:54] rendering partner. It provides the [1:57] semantic token vocabulary, the non-auto [2:00] reggressive acoustic stage and the VAE [2:03] that turns the rendered representation [2:06] into a waveform. Liubi does not replace [2:09] every part of UE2. It replaces the text [2:14] to semantic token language model stage [2:17] with RWKV7. [2:19] That gives us a clean experiment. Hold [2:22] the token space and renderer steady. [2:24] Then ask whether a recurrent language [2:26] model can learn the structure of songs. [2:29] The practical benefit is a constant size [2:31] recurrent state during generation. [2:33] The practical cost is that the model has [2:36] to learn a difficult binding between [2:37] words, sections, and audio codes. Um [2:41] there was an awkward obstacle. UE2 did [2:44] not release the audio encoder needed to [2:46] turn the training songs into UE2 [2:48] semantic tokens. Lubai therefore trains [2:51] an inverse tokenizer called R1A. [2:55] It starts from MERT v2 audio features, [2:59] predicts UE2 semantic tokens and is [3:03] trained with a joint teacher that runs [3:05] those tokens through the frozen UE2 [3:08] stack on 20 unseen real songs. The [3:12] accepted R1A head reached a word error [3:17] rate of about 0.33 [3:20] against the original transcript and a [3:22] mer cosign of 0.981. [3:26] Those numbers describe the token bridge [3:28] and round trip, not Lubai's lyric [3:31] generation. They are the foundation that [3:33] makes the next experiment possible. The [3:36] main corpus is Sunno 94K [3:40] 94,174 [3:42] songs about 5,423 [3:45] hours with style captions and structured [3:48] lyrics. After filtering, the UEE2 corpus [3:52] contains 93,58 [3:54] songs. The first RWKV run used 4 A100 [3:59] GPUs, Bflat 16 training, an 8,000 token [4:04] context, and a vocabulary enlarged for [4:08] the 32,7 to68 semantic codes. The model [4:13] was trained from scratch for this music [4:15] stage, then continued into the section [4:17] format. That last detail matters because [4:20] a full song can put a lyric line [4:22] thousands of tokens away from the audio. [4:25] that should sing it. RWKV is auto [4:29] reggressive like a language model, but [4:30] its inference state is carried forward [4:33] recurrently. Instead of appending every [4:35] pass key and value to a growing [4:38] transformer cache, each new token [4:40] updates a fixed size state. The state is [4:44] not a magic musical memory. It is simply [4:47] the object the next token reads. For [4:50] music, that trade-off is attractive. [4:54] generation can stream through a long [4:56] sequence with memory that does not grow [4:58] with the song length. The model still [5:01] has to learn timing, repetition, section [5:03] changes, and style from the data. [5:05] Efficient memory does not remove the [5:07] learning problem. The earlier G3 run [5:11] used a whole song layout, caption and [5:14] lyrics first, then one long audio token [5:17] block. It learned an unconditional music [5:20] language model, but it barely connected [5:23] local lyrics to local audio. R31 [5:28] changes the serialization. A header is [5:31] followed by repeated blocks, a section [5:34] marker. That section's lyric text, its [5:36] audio start marker, its semantic codes, [5:39] and an end marker. Now, the text and the [5:42] audio it should control are close [5:45] together. During generation, the model [5:47] decodes section by section. This is a [5:50] small change in the sequence format, but [5:53] it directly attacks the credit [5:55] assignment distance. [5:57] Here are six generated [music] songs [5:59] from held out prompts. So, this is not [6:01] one cherrypicked clip. We start with [6:03] punk on don't try to understand me then [6:06] move through Mottown on curly hair [6:08] industrial metal on DB82Xure [6:11] reggaon on baigo jazz on ng I a ng I [6:17] aotry [6:19] and ambient on now ghostto [music] de [6:22] frutas listen for what the model already [6:26] gets across these genres coherent vocal [6:29] texture recognizable style and songlike [6:32] sections, [music] [6:33] then listen for what it misses. The [6:36] words are often fluent but unrelated to [6:38] the requested lyrics. The model can [6:40] learn a musical surface before it learns [6:42] reliable text to voice alignment. On 20 [6:46] held out prompts, R3 [6:49] 1 reaches a mean word error rate of 1.05 [6:53] against the requested lyrics versus 1.37 [6:57] for G3. R3 [7:00] one wins 15 of the 20 prompts and [7:04] removes the most catastrophic collapse [7:08] songs. A teacher force test also shows [7:11] real local binding. The loss for [7:14] permuted section text is about 0.106NATs [7:17] 106 knats higher than for the match [7:20] text, but the rendered transcript [7:25] contains only about 10% distinct lyric [7:28] word overlap. Style similarity is [7:31] roughly unchanged [7:33] with a mer cosign near 0.79. [7:38] So the evidence supports a narrow claim. [7:42] um section interle improves local [7:45] conditioning and song stability while [7:47] full lyric adherence remains unsolved. [7:50] The failure points to the next [music] [7:53] experiments. [7:55] First, align more songs so the training [7:58] target has reliable lyric to audio [8:01] boundaries. Second, move from stanza [8:05] level blocks to line level blocks and [8:08] shorten the text to audio distance [8:10] again. Third, if the binding remains [8:13] weak, add an objective tied to the [8:16] rendered sound, an ASR or CTC reward, or [8:20] the joint teachers flow loss. There is [8:23] also a release question. UE2 and its [8:26] renderer carry their own model and [8:28] license constraints. So, an open [8:30] distribution of LUI needs those [8:31] dependencies documented and eventually [8:33] replaced or cleared. The repository [8:37] records the limitations [8:39] instead of hiding them. That is Liub by [8:43] RWKV today. A recurrent language model [8:46] that can plan and render convincing [8:48] musical surfaces. A UE2 bridge that [8:51] makes the experiment audible and a [8:55] clearer unsolved problem at the center [8:58] making the voice sing the words the user [9:00] actually wrote. The code, samples, [9:03] training logs, and failed baselines are [9:04] open in the repository. If you want to [9:07] follow the next run, start with the [9:09] status log, listen to the R3, 1, and G3 [9:13] pairs, and watch the lyric binding [9:15] metric rather than a single impressive [9:17] clip.