Ubuyobozi bwa Audio AI

UmuzikiLM: Hierarchical Semantic na Acoustic Tokens

UmuzikiLM ni __AIU_PROTECTED_11 __ 'inyandiko-yumuziki yerekana amajwi-maremare yerekana amajwi maremare binyuze murwego rwibimenyetso byerekana imiterere yumuziki hamwe nibimenyetso bya acoustic kugirango bisobanuke neza.

Umva iki gitaboAI-byavuzwe · ~7 min

Kuriyi page8 min soma
  1. Ibyingenzi byingenzi
  2. Kwibira cyane
  3. Trace one prompt through MusicLM's stages
  4. Gushyira mu bikorwa Isi
  5. Ejo hazaza h'umuzikiLM: Hierarchical Semantic na Acoustic Tokens
  6. Kubishyira mubikorwa
  7. Inkomoko no gusoma
  8. Komeza Ubushakashatsi
  9. Ibibazo bikunze kubazwa

Ibyingenzi byingenzi

  1. MusicLM, posted by Google Research in January 2023, generates 24 kHz music from text that stays consistent over several minutes.
  2. It plans music with 25-per-second semantic tokens from w2v-BERT, then renders sound with 600-per-second acoustic tokens from the SoundStream codec.
  3. MuLan's shared music and text embedding lets MusicLM train on audio alone and accept text descriptions only at generation time.
  4. In 1,200 listening ratings MusicLM won 312 comparisons versus 158 for Riffusion and 97 for Mubert, while real recordings won 472.
  5. The paper found exact memorization below 0.2% but approximate matches for about 1% of examples, and the models were not released.

Kwibira cyane

MusicLM is a generative model that writes music as audio, not as sheet music or MIDI notes. A user types a description of genre, mood, instruments and tempo, and the system produces a recording that tries to match it. The research paper, MusicLM: Generating Music From Text by Andrea Agostinelli, Timo Denk and eleven colleagues at Google Research, was posted on January 26, 2023. It reports music at a 24 kHz sample rate that remains coherent over several minutes, and it compares favorably with two earlier text-to-music systems, Mubert and Riffusion, on both audio quality and faithfulness to the text.

The central idea is to treat music like language. Raw audio has tens of thousands of samples per second, far too many for a model to predict one by one over minutes. MusicLM therefore converts audio into tokens, short discrete codes, at two levels of detail. Semantic tokens come from w2v-BERT, a 600-million-parameter self-supervised audio model; MusicLM takes embeddings from its seventh layer and maps each to the nearest of 1,024 clusters, giving 25 tokens per second that capture melody, rhythm and long-range structure. Acoustic tokens come from SoundStream, a neural audio codec that compresses 24 kHz audio into 50 frames per second, each described by 12 codes from a residual vector quantizer, which works out to 600 tokens per second and a bitrate of 6 kbps. Acoustic tokens preserve timbre and recording detail well enough to be decoded back into sound.

The third ingredient connects text to music. MuLan, a Google model described in August 2022, was trained on 44 million music recordings totaling 370,000 hours paired with loosely associated text, so that a clip and a description of it land near each other in a shared embedding space. This lets MusicLM train on audio alone: during training it conditions on the MuLan embedding of the music itself, and at generation time it swaps in the MuLan embedding of the user's text. That trick removes the need for millions of hand-written captions.

Generation then runs as a hierarchy of separate transformer models, each predicting the next token from the ones before it. The semantic stage reads the MuLan tokens and generates semantic tokens, which sketch the musical plan. The acoustic stage, split into a coarse model for the first four SoundStream quantizer levels and a fine model for the remaining eight, generates acoustic tokens conditioned on both the MuLan tokens and the semantic tokens. The SoundStream decoder finally turns acoustic tokens into a waveform. Each stage is a decoder-only transformer of 24 layers with about 430 million parameters, and sampling temperatures of 1.0, 0.95 and 0.4 for the semantic, coarse and fine stages trade variety against stability.

MusicLM builds directly on earlier Google work. SoundStream, the codec, appeared in July 2021. AudioLM, posted in September 2022, introduced the hybrid of semantic and acoustic tokens and showed it could continue speech and produce coherent piano music from raw audio alone. MuLan supplied the text link in August 2022. MusicLM combined the three in January 2023, added melody conditioning through a small vision-transformer-style model trained on mel spectrograms, and released the MusicCaps evaluation set. In May 2023 Google opened sign-ups for a public version in AI Test Kitchen.

The paper measures quality in three ways on MusicCaps captions. Frechet Audio Distance (FAD) compares statistics of generated and real audio, where lower is better; MusicLM scored 4.0 on the VGGish variant against 9.6 for Mubert and 13.4 for Riffusion. MuLan Cycle Consistency, which checks how well the generated audio matches the text in MuLan's embedding space, was 0.51 for MusicLM versus 0.32 and 0.34 for the baselines. In 1,200 pairwise listening ratings, MusicLM won 312 comparisons, Riffusion 158 and Mubert 97, while the real reference recordings won 472, showing a clear remaining gap to human-made music. Compared with Riffusion, which fine-tunes an image diffusion model on spectrograms, and Mubert, which assembles pre-recorded sounds, MusicLM generates the audio itself. Compared with later single-stage models such as MusicGen, its cascade of several models is more complex to train and run.

The authors document several limits. Training data shapes the output, so music from cultures underrepresented in the data may be poorly served, and the paper raises concerns about cultural appropriation. Lyrics, vocal quality and song structure such as verse and chorus are all listed as future work. To test copying, the team measured memorization: exact matches stayed below 0.2% even with a ten-second prompt, but for about 1% of examples they found an approximate match to training audio. Because of the risk of misappropriating creative content, the paper states that the team had no plans to release the models at that point, so MusicLM itself is not available as open weights.

In practice, MusicLM is expensive to reproduce. The semantic and acoustic stages were trained on about five million audio clips, roughly 280,000 hours of music, and three 430-million-parameter transformers plus w2v-BERT, SoundStream and MuLan must run in sequence to produce a clip. For most people, the realistic ways to engage with the work are listening to the official examples page, using MusicCaps to evaluate other models, or experimenting with openly released successors that adopt similar token-based ideas.

03Urugero rwakazi

Trace one prompt through MusicLM's stages

  1. Start with a caption such as 'Hip hop song with violin solo' and pass it through MuLan's text encoder to get an embedding, which is quantized into MuLan tokens.

  2. Run the semantic transformer to generate 25 semantic tokens per second; for a 10-second clip that is 250 tokens describing melody, rhythm and structure.

  3. Run the coarse acoustic transformer to predict the first four SoundStream codebook levels for 50 frames per second, conditioned on the MuLan and semantic tokens.

  4. Run the fine acoustic transformer to predict the remaining eight codebook levels, bringing the total to 600 acoustic tokens per second, or 6,000 for 10 seconds.

  5. Feed the acoustic tokens to the SoundStream decoder to produce a 24 kHz waveform.

  6. Compare the result with the official examples page and score your own judgment against the caption, noting where instruments or mood differ.

Ibyo byerekana

The learner ends up with a step-by-step token budget for a 10-second clip, 250 semantic and 6,000 acoustic tokens, and a clear picture of why the hierarchy exists: planning happens on a short sequence, and detail is added only after the structure is set. The arithmetic can be checked against the token rates stated in the paper.

Gushyira mu bikorwa Isi

Public demo: from May 10, 2023, Google let people sign up to try MusicLM in AI Test Kitchen on the web, Android and iOS, where each text prompt returned two versions of a track to compare.

Humming to arrangement: the MusicLM paper shows a melody-conditioning mode that takes a whistled or hummed tune and renders it in the style named by a text caption, such as an orchestral or jazz arrangement.

Research benchmark: MusicCaps, released with the paper under a CC BY-SA 4.0 license, provides 5,521 ten-second clips from AudioSet, each with a free-text caption and a list of musical aspects written by professional musicians.

Model comparisons: later systems, such as Meta's MusicGen published at NeurIPS 2023, evaluate on MusicCaps and contrast their single-stage design with cascaded, multi-model pipelines like MusicLM's.

Ejo hazaza h'umuzikiLM: Hierarchical Semantic na Acoustic Tokens

The MusicLM paper names its own next steps: lyrics generation, better text conditioning and vocal quality, modeling high-level song structure such as introduction, verse and chorus, and generating at a higher sample rate than 24 kHz. Research since 2023 has pursued simpler architectures as well. Meta's MusicGen, published at NeurIPS 2023, uses a single-stage transformer with interleaved codec tokens instead of a cascade and released code and models through its AudioCraft repository, which made comparisons on the MusicCaps benchmark easier for other groups. Evaluation remains an open problem, because metrics such as Frechet Audio Distance capture audio statistics rather than musicality, and MusicLM's own listening test showed real recordings still preferred. The legal and ethical questions raised in the paper are also unresolved: training data provenance, the roughly 1% of examples with approximate matches to training audio, and the effect of generated music on working musicians and on cultures underrepresented in training sets. How music generators are licensed, labeled and compensated will shape which research systems become products.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Kugera no kugera. Itezimbere kugerwaho binyuze mu kwandukura, kuvuga, no guhuza amajwi.
  • Igiciro na bije. Amatsinda yibitangazamakuru arashobora kohereza amajwi yihuse hamwe na bije nto.
  • Umuvuduko n'igipimo. Sisitemu ireba abakiriya irashobora gutunganya imikoranire ivugwa murwego runini.

Witondere

  • Gukoresha nabi amajwi no kwigira ibyago byiyongera mugihe uruhushya rubuze.
  • Ukuri kurashobora kugabanuka hejuru yimvugo, imvugo, cyangwa urusaku rwibidukikije.
  • Amajwi yubukorikori arashobora kwibeshya kumvugo yukuri nta kirango gisobanutse.

Intambwe ikurikira

  1. Shaka uruhushya rusobanutse rwo gufata amajwi, gukoroniza, no gukoresha.
  2. Ikizamini cyiza mubiganiro bitandukanye hamwe nuburyo bwimbere.
  3. Sobanura igihe umuntu agomba gusuzuma cyangwa kwemeza ibisubizo.
  4. Andika amajwi yubukorikori kandi ugumane inyandiko zerekana kubazwa.

Inkomoko no gusoma

  1. arXiv (Agostinelli et al., Google Research, 2023)MusicLM: Generating Music From Text
  2. Google ResearchMusicLM examples
  3. arXiv (Borsos et al., 2022)AudioLM: a Language Modeling Approach to Audio Generation
  4. arXiv (Zeghidour et al., 2021)SoundStream: An End-to-End Neural Audio Codec
  5. arXiv (Huang et al., 2022)MuLan: A Joint Embedding of Music Audio and Natural Language
  6. Google on Hugging Facegoogle/MusicCaps
  7. GoogleHow to try MusicLM from Google's AI Test Kitchen
  8. arXiv / NeurIPS 2023 (Copet et al.)Igisekuru cyoroshye kandi kigenzurwa

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the MusicLM: Hierarchical Semantic and Acoustic Tokens quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

UmuzikiLM ni iki: Hierarchical Semantic na Acoustic Tokens?

UmuzikiLM ni __AIU_PROTECTED_11 __ 'inyandiko-yumuziki yerekana amajwi-maremare yerekana amajwi maremare binyuze murwego rwibimenyetso byerekana imiterere yumuziki hamwe nibimenyetso bya acoustic kugirango bisobanuke neza.

How does MusicLM formulate conditional music generation?

The paper casts conditional music generation as hierarchical sequence-to-sequence modeling, generating semantic and acoustic token sequences.

Ni uruhe ruhare rw'ibimenyetso bisobanura muri MuzikaLM?

Ibimenyetso bya semantique (kuva kuri w2v-BERT) bifata ibintu bito, bigenda bihinduka buhoro buhoro nka melody na rhythm mugihe kirekire.

Nibihe bice bitanga ibisobanuro byiza bya acoustic nka timbre nuburyo?

Ibimenyetso bya Acoustic biva muri SoundStream neural codec kandi bigashyiraho amakuru meza nka timbre nuburyo.

Nigute MusicLM ihuza inyandiko yihuta kumajwi yumuziki?

MuLan yatojwe kugirango ibisobanuro byanditse hamwe nu ikarita yerekana amajwi ahantu hafi aho hasangiwe gushyiramo, bigahindura inyandiko.

Ni ukubera iki igishushanyo mbonera, cyateguwe gifasha imiterere ndende?

Ibimenyetso bidasanzwe bihinduka gahoro gahoro, bityo Transformer irashobora kwerekana imiterere yigihe kirekire neza mbere yuko acoustic yuzuye yuzuye.