Xeetu music

Music genre classification takes an audio file and assigns it one or more genre labels such as classical, country, disco, hip-hop, jazz, metal, pop, reggae, rock or blues. It is one of the oldest tasks in music information retrieval (MIR), the research field that builds software to search, organize and analyze music. The task sounds simple but has a philosophical problem at its core: genres are, as Tzanetakis and Cook wrote in 2002, labels created by humans with no strict definitions or boundaries, shaped by marketing, history and culture. A classifier can only learn whatever regularities its training labels happen to encode.

Every system follows the same basic pipeline. First, audio is converted to a standard format, often mono at 22,050 or 16,000 samples per second, and cut into excerpts, commonly 30 seconds. Second, the audio is turned into a representation that exposes musical structure. The most common is a mel-spectrogram, an image-like grid showing how much energy falls in each frequency band over time, with bands spaced to match human pitch perception. Third, a model maps that representation to scores for each genre. Fourth, predictions from many short windows are averaged to label the whole track. Systems differ mainly in step three: whether the features are designed by hand or learned.

The classic hand-crafted approach was set out by George Tzanetakis and Perry Cook in IEEE Transactions on Speech and Audio Processing in July 2002. They computed three families of features. Timbral texture features describe the sound's color: the spectral centroid (where the center of mass of the spectrum lies, which correlates with brightness), spectral rolloff, spectral flux (how quickly the spectrum changes), zero-crossing rate (a rough measure of noisiness) and the first five mel-frequency cepstral coefficients (MFCCs, a compact summary of the spectral envelope borrowed from speech recognition). Rhythmic features came from a beat histogram covering 40 to 200 beats per minute. Pitch features came from a histogram of detected pitches. Simple statistical classifiers, including Gaussian mixture models and k-nearest neighbors, then labeled the clips, reaching 61% accuracy on ten genres.

Their dataset of 1,000 thirty-second clips, 100 for each of ten genres, recorded at 22,050 Hz from radio, CDs and MP3 files, became known as GTZAN and the standard benchmark. In 2013, Bob Sturm counted at least 100 published works that used it and documented serious faults: 50 exact repeated excerpts, 21 further repetitions of the same recording, mislabeled clips and distorted audio. Artist repetition is heavy; the 100 blues excerpts come from only nine artists, and more than a third of the reggae excerpts are by Bob Marley. When the same artist appears in both training and test sets, a model can score well by recognizing the artist or recording rather than the genre, which inflates reported accuracy.

Deep learning shifted the field around 2016. Keunwoo Choi, George Fazekas and Mark Sandler showed at ISMIR 2016 that fully convolutional networks reading mel-spectrograms reached state-of-the-art music tagging on the MagnaTagATune dataset, and that deeper networks did better on the larger Million Song Dataset, concluding that mel-spectrograms are an effective input. Convolutional networks learn their own filters for patterns such as drum hits, harmonic textures and vocal styles, removing the need to design features by hand. Larger datasets followed, notably FMA in 2017 with 106,574 tracks and 161 genres arranged in a hierarchy, with Creative Commons audio that researchers can share.

The current approach is transfer learning from large pretrained models. PANNs (Kong and colleagues, 2020) trained convolutional networks on AudioSet, a collection of about 1.9 million YouTube clips labeled with 527 sound classes, then fine-tuned them on smaller tasks. On GTZAN, the fine-tuned network scored 91.5% accuracy against 75.8% when trained from scratch, while a specialized prior system had reported 93.9%. MERT (ICLR 2024) went further with self-supervised pretraining on music audio, learning to predict hidden targets from an acoustic codec and a constant-Q transform, a pitch-aligned spectrogram, and was evaluated across 14 tasks including genre classification.

Compared with the alternatives, audio-based classification has distinct strengths and weaknesses. Metadata-based approaches use artist, label or user tags; they are cheap and often accurate for well-known music but fail for new uploads with no history. Collaborative filtering infers similarity from listening behavior rather than sound, which captures cultural context that audio cannot. Lyrics-based models help for genres defined by content or language. Audio classifiers work on any recording, including brand-new or unreleased tracks, which is why they are useful for cold-start tagging, but they see only the sound and miss context.

The limitations are significant. Genre boundaries are subjective, overlapping and culture-specific, so a single label per track is often wrong by design; FMA's hierarchy and multi-label tagging are partial answers. Benchmark numbers can mislead: Sturm showed that GTZAN's repetitions and mislabelings affect different systems differently, so accuracy rankings on it are not reliable without artist-filtered splits. GTZAN's ten categories are all Western popular and classical styles, so models trained on it say nothing about most of the world's music. Models can latch onto production artifacts, such as recording era or audio compression, instead of musical traits. In practice, fine-tuning a pretrained model on a dataset the size of GTZAN is a small job, while pretraining models such as PANNs on about 1.9 million clips, or MERT at up to 330 million parameters, requires far larger compute.