Can you compose a new Bach chorale using recurrent and/or convolutional neural networks? 🎼 🎶 🎹
Source: Photo by Joey Huang on Unsplash
This exercise adapts Géron et al.'s Jupyter notebook exercises for chapter 13 (License) of his book “Hands-On Machine Learning with Scikit-Learn and PyTorch”.
Part I: Setup¶
First, let’s import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We’ll also check that Python 3.9 or later is installed, as well as PyTorch ≥2.0.
import sys
# Is this notebook running on Colab or Kaggle?
IS_COLAB = "google.colab" in sys.modules
IS_KAGGLE = "kaggle_secrets" in sys.modules
import torch
import torch.nn as nn
import torch.utils.tensorboard
if not torch.cuda.is_available() and not torch.backends.mps.is_available():
print("No GPU was detected. LSTMs and CNNs can be very slow without a GPU.")
if IS_COLAB:
print("Go to Runtime > Change runtime and select a GPU hardware accelerator.")
if IS_KAGGLE:
print("Go to Settings > Accelerator and select GPU.")
device = "cpu"
else:
device = "cuda" if torch.cuda.is_available() else "mps"
# Common imports
import numpy as np
import os
from pathlib import Path
# to make this notebook's output stable across runs
np.random.seed(42)
torch.manual_seed(42)
# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)
# Where to save the figures
IMAGES_PATH = "_files"
os.makedirs(IMAGES_PATH, exist_ok=True)
def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):
path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
print("Saving figure", fig_id)
if tight_layout:
plt.tight_layout()
plt.savefig(path, format=fig_extension, dpi=resolution)
# Loading Tensorboard
%load_ext tensorboardLet’s import two more libraries:
import pooch # Import the pooch library to load data from URL
import pandas as pd # Import the pandas library to handle arraysSecond, let’s load the data:
url = "https://raw.githubusercontent.com/gse-unil/2026_MLEES_book/main/data/part-III/jsb_chorales.tgz"files = pooch.retrieve(url, processor=pooch.Untar(),
known_hash='sha256:e722fbc866fc1ba0bac56a6fcad95d7e3e584da80173e13932e3da441440efd7')# Finds the directory containing the files and ...
jsb_chorales_dir = Path(files[0]).parent.parent
# ... Sort these files into training/validation/test
train_files = sorted(jsb_chorales_dir.glob("train/chorale_*.csv"))
valid_files = sorted(jsb_chorales_dir.glob("valid/chorale_*.csv"))
test_files = sorted(jsb_chorales_dir.glob("test/chorale_*.csv"))# Load the chorales from the training, validation, and test set
def load_chorales(filepaths):
return [pd.read_csv(filepath).values.tolist() for filepath in filepaths]
train_chorales = load_chorales(train_files)
valid_chorales = load_chorales(valid_files)
test_chorales = load_chorales(test_files)Third, let’s define the functions Géron implemented to listen to these chorales. According to Géron:
“You don’t need to know how this works, but if you’re interested, look at the pretty_midi library for more sophisticated MIDI generation. You could also use the music21 library.” This function just synthesizes a sine wave per note, so no deep-learning library is needed here — it works exactly the same whether the chorale came from a PyTorch or a Keras model.
from IPython.display import Audio
def notes_to_frequencies(notes):
# Frequency doubles when you go up one octave; there are 12 semi-tones
# per octave; Note A on octave 4 is 440 Hz, and it is note number 69.
return 2 ** ((np.array(notes) - 69) / 12) * 440
def frequencies_to_samples(frequencies, tempo, sample_rate):
note_duration = 60 / tempo # the tempo is measured in beats per minutes
# To reduce click sound at every beat, we round the frequencies to try to
# get the samples close to zero at the end of each note.
frequencies = np.round(note_duration * frequencies) / note_duration
n_samples = int(note_duration * sample_rate)
time = np.linspace(0, note_duration, n_samples)
sine_waves = np.sin(2 * np.pi * frequencies.reshape(-1, 1) * time)
# Removing all notes with frequencies ≤ 9 Hz (includes note 0 = silence)
sine_waves *= (frequencies > 9.).reshape(-1, 1)
return sine_waves.reshape(-1)
def chords_to_samples(chords, tempo, sample_rate):
freqs = notes_to_frequencies(chords)
freqs = np.r_[freqs, freqs[-1:]] # make last note a bit longer
merged = np.mean([frequencies_to_samples(melody, tempo, sample_rate)
for melody in freqs.T], axis=0)
n_fade_out_samples = sample_rate * 60 // tempo # fade out last note
fade_out = np.linspace(1., 0., n_fade_out_samples)**2
merged[-n_fade_out_samples:] *= fade_out
return merged
def play_chords(chords, tempo=160, amplitude=0.1, sample_rate=44100, filepath=None):
'''
Reads chords (sets of 4 notes) in a chorale (list of chords)
and outputs an audio file that can be played.
Arguments:
chords: A list of chords, for instance a full chorale
Optional arguments:
tempo: The tempo of the music
amplitude: The amplitude of the sine waves to be played
sample_rate: How many frequencies are sampled ~ music quality
'''
samples = amplitude * chords_to_samples(chords, tempo, sample_rate)
if filepath:
from scipy.io import wavfile
samples = (2**15 * samples).astype(np.int16)
wavfile.write(filepath, sample_rate, samples)
return display(Audio(filepath))
else:
return display(Audio(samples, rate=sample_rate))Part II: Preliminary Data Analysis and Preprocessing¶
The dataset is composed of 382 chorales composed by Johann Sebastian Bach. Each chorale is 100 to 640 time steps long, and each time step contains 4 integers, where each integer corresponds to a note’s index on a piano (except for the value 0, which means that no note is played).
Our goal is to train a model—recurrent, convolutional, or both—that can predict the next time step (four notes), given a sequence of time steps from a chorale. Once trained, we can use this model to generate Bach-like music, one note at a time. We can do this by giving the model the start of a chorale and asking it to predict the next time step, then appending these time steps to the input sequence and asking the model for the next note, and so on.
Q1) Check that notes range from min_note = 36 (which is C1, i.e. C/Do on octave 1) to max_note = 81 (which is A5, i.e. A/La on octave 5). Also, verify that there are n_notes = 47 different notes.¶
First, explore the dataset:
# Explore the dataset: How is it structured? What does a sample look like? etc.Second, let’s group all of the chorales’ notes in a set called notes:
notes = set() # Initialize the notes with an empty set
for chorales in (_____, _____, _____): # Loop through chorales in training/validation/test sets
for chorale in _____: # Loop through all chorales
for chord in _____: # Loop through chords within a chorale
notes |= set(chord) # Add notes that are in chord but not yet in notesThird, calculate min_note and max_note.
Hint: Be careful to exclude 0 when calculating min_note.
min_note = __________
max_note = __________# This will return an error message if your code is erroneous
assert min_note == 36
assert max_note == 81Finally, calculate the total number of notes n_notes.
# Write your code below# This will return an error message if your code is erroneous
assert n_notes == 47Q2) What is the training/validation/test split?¶
Hint: You may use the len build-in function to get the length of a list
# Calculate the number of chorales in each of the training/validation/test sets# What is the training/validation/test split (in %)?Q3) Listen to a few chorales 🎵¶
Hint 1: Use the play_chords function defined above. Type play_chords? to display its documentation.
Hint 2: You can start by selecting chorales from the training set train_chorales.
# Explore `train_chorales` and `play_chords` to get some intuition# Use `play_chords` to produce chorales you can listen toDivine! 🎶
In order to be able to generate new chorales, we want to train a model that can predict the next chord given all previous chords. If we naively try to predict the next chord in one shot, predicting all 4 notes at once, the different notes in the chord would not be well correlated (imagine training a model to write a story where each word is chosen independently of the words that were previously written: the story would not make much sense). So instead we will predict one note at a time. To do this, we first turn each chord into an arpegio (i.e. a sequence of notes rather than a simultaneous chord), by defining create_target, which flattens the last dimension:
def create_target(batch):
X = batch[:, :-1]
Y = batch[:, 1:] # predict next note in each arpegio, at each step
return X, YWe will also shift the values so that they range from 0 to 46, where 0 represents silence, and values 1 to 46 represent notes 36 (C1) to 81 (A5).
def preprocess(window):
window = torch.where(window == 0, window, window - min_note + 1) # shift values
return window.reshape(-1) # convert to arpegioAnd we will train the model on windows of 128 notes (i.e., 32 chords).
Since the dataset fits in memory, we can preprocess the chorales with plain Python and build a PyTorch Dataset — this is the direct equivalent of the tf.data windowing pipeline Géron builds in his own chapter 13, just written for PyTorch’s Dataset/DataLoader pair instead.
class ChoraleWindows(torch.utils.data.Dataset):
def __init__(self, chorales, window_size=32, window_shift=16):
self.windows = []
for chorale in chorales:
chorale = torch.tensor(chorale, dtype=torch.long)
for start in range(0, len(chorale) - window_size, window_shift):
self.windows.append(preprocess(chorale[start:start + window_size + 1]))
def __len__(self):
return len(self.windows)
def __getitem__(self, idx):
return self.windows[idx]
def bach_dataset(chorales, batch_size=32, shuffle=False, window_size=32, window_shift=16):
dataset = ChoraleWindows(chorales, window_size=window_size, window_shift=window_shift)
return torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=shuffle)Note that this bach_dataset function is designed to output the sequence using the shape (batch_size, number_of_notes) because the sequence is then fed into an Embedding layer. create_target then splits each batch into an input arpegio and the target arpegio shifted by one note, which we’ll apply after loading each batch from the DataLoader.
Q4) Use the function bach_dataset above to create the training, validation, and test sets. Shuffle the training set.¶
train_set = bach_dataset(_________, __________=_____)
valid_set = bach_dataset(_________)
test_set = bach_dataset(_________)We could feed the note values directly to the model, as floats, but this would probably not give good results. Indeed, the relationships between notes are not that simple: for example, if you replace a C3 with a C4, the melody will still sound fine, even though these notes are 12 semi-tones apart (i.e., one octave). Conversely, if you replace a C3 with a C#3, it’s very likely that the chord will sound horrible, despite these notes being just next to each other. So we will use an Embedding layer to convert each note to a small vector representation (see Géron Chapter 13 for more details on embeddings). We will use 5-dimensional embeddings, so the output of this first layer will have a shape of [batch_size, window_size, n_embedding_dims=5].
# Choose the number of embedding dimensions here. Géron recommends 5.
n_embedding_dims = ______Now implement a small WaveNet (we recommend starting with no more than 5 layers).
Hint 1: You need to start with an embedding layer to convert integer notes into a vector of length n_embedding_dims. For that purpose, the syntax is: nn.Embedding(num_embeddings=n_notes, embedding_dim=n_embedding_dims). num_embeddings is the number of possible integer categories of the notes we would like to convert, while embedding_dim is the number of embedding dimensions.
Hint 2: A WaveNet is a sequence of Conv1d layers with increased dilation rate. Unlike Keras, PyTorch’s nn.Conv1d has no padding="causal" mode, so we’ve provided a small CausalConv1d wrapper below that left-pads its input to keep the convolution from looking at future notes. Below is a WaveNet with 3 layers and a constant number of filters equal to 128 (increase the filter size for more representation power). Note the increase in the dilation rate.
CausalConv1d(128, 128, kernel_size=2, dilation=2), nn.ReLU(),
CausalConv1d(128, 128, kernel_size=2, dilation=4), nn.ReLU(),
CausalConv1d(128, 128, kernel_size=2, dilation=8), nn.ReLU(),Hint 3: Unlike Keras, nn.Conv1d expects its input with the channel dimension before the sequence dimension — (batch_size, channels, window_size) — while nn.Embedding’s output (and nn.Linear’s input) has the channel dimension last. The WaveNet module below already permutes between the two conventions for you; you only need to fill in the convolutional stack and the final output layer.
Hint 4: For the final layer, you need to output a score for each of the n_notes possible notes — leave it as a plain nn.Linear, with no activation. Unlike Keras, nn.CrossEntropyLoss (which we’ll use to compile the model) expects raw logits, not softmax probabilities.
class CausalConv1d(nn.Module):
"""A Conv1d that only looks at past time steps, by left-padding its input."""
def __init__(self, in_channels, out_channels, kernel_size, dilation):
super().__init__()
self.left_padding = (kernel_size - 1) * dilation
self.conv = nn.Conv1d(in_channels, out_channels, kernel_size, dilation=dilation)
def forward(self, x):
x = nn.functional.pad(x, (self.left_padding, 0))
return self.conv(x)
class WaveNet(nn.Module):
def __init__(self, n_notes, n_embedding_dims):
super().__init__()
self.embedding = nn.Embedding(num_embeddings=______, embedding_dim=______)
self.conv_stack = nn.Sequential(
__________________________________________,
__________________________________________,
)
self.output_layer = nn.Linear(________, ________)
def forward(self, x):
x = self.embedding(x) # (batch, window_size, n_embedding_dims)
x = x.permute(0, 2, 1) # (batch, n_embedding_dims, window_size), for Conv1d
x = self.conv_stack(x)
x = x.permute(0, 2, 1) # (batch, window_size, filters), for Linear
return self.output_layer(x) # (batch, window_size, n_notes) logits
model = WaveNet(n_notes, n_embedding_dims)# Check that your model looks right
print(model)Q6) Set up your model’s loss and optimizer using nn.CrossEntropyLoss as the loss since your outputs are logits over 47 classes, and torchmetrics.Accuracy as an additional metric to monitor during training.¶
Hint: Potential PyTorch optimizers are listed at this link
import torchmetrics
# Choose your optimizer
optimizer = torch.optim._______________________(model.parameters())# Set up your loss and metric
loss_fn = ____________
accuracy = torchmetrics.Accuracy(task="multiclass", num_classes=n_notes)Q7) Train your model on the training set and plot the learning curves. Is your model overfitting?¶
Hint 1: You may use 20 epochs for training and a patience of 20 epochs for your early stopping.
Hint 2: To plot your learning curves with Tensorboard, fill out the information in the cell below
#Change this number and rerun this cell whenever you want to change runs
run_index = ___ # it should be an integer, e.g. 1
run_logdir = os.path.join(os.curdir, "my_bach_logs", "run_{:03d}".format(run_index))
print(run_logdir)As in the artificial-neural-networks and deep-computer-vision exercises, since PyTorch has no EarlyStopping/ModelCheckpoint callbacks, we track them manually: a patience value, a best_val_loss tracker, and a checkpoint path to pass to torch.save().
For the checkpoint, we recommend monitoring the validation loss to avoid overfitting.
patience = _____
best_val_loss = ______
epochs_without_improvement = 0
checkpoint_path = "my_bach_model.pt"
writer = torch.utils.tensorboard.SummaryWriter(run_logdir)n_epochs = ___
for epoch in range(n_epochs):
# Training phase
model.train()
train_losses = []
accuracy.reset()
for batch in ____: # training data
X, Y = create_target(batch)
optimizer.zero_grad()
logits = model(X)
loss = loss_fn(logits.permute(0, 2, 1), Y) # CrossEntropyLoss wants classes on dim 1
loss.backward()
optimizer.step()
train_losses.append(loss.item())
accuracy.update(logits.reshape(-1, n_notes), Y.reshape(-1))
train_accuracy = accuracy.compute().item()
# Validation phase
model.eval()
val_losses = []
accuracy.reset()
with torch.no_grad():
for batch in ____: # validation data
X, Y = create_target(batch)
logits = model(X)
val_losses.append(loss_fn(logits.permute(0, 2, 1), Y).item())
accuracy.update(logits.reshape(-1, n_notes), Y.reshape(-1))
val_loss = np.mean(val_losses)
val_accuracy = accuracy.compute().item()
print(f"Epoch {epoch + 1}/{n_epochs} - loss: {np.mean(train_losses):.4f} - accuracy: {train_accuracy:.4f}"
f" - val_loss: {val_loss:.4f} - val_accuracy: {val_accuracy:.4f}")
writer.add_scalar("val_loss", val_loss, epoch)
writer.add_scalar("val_accuracy", val_accuracy, epoch)
if val_loss < best_val_loss:
best_val_loss = val_loss
epochs_without_improvement = 0
torch.save(model.state_dict(), checkpoint_path)
else:
epochs_without_improvement += 1
if epochs_without_improvement >= patience:
breakVisualize your learning curves using Tensorboard:
%tensorboard --logdir=./my_bach_logs --port=8888 # Pick any 4 digits for the portQ8) To double check whether your model is overfitting, evaluate it on the test set, reloading the best checkpoint first.¶
# Rollback to the best checkpoint, then evaluate the model on the test set
model.load_state_dict(torch.load(____, weights_only=True))
model.____()
accuracy.reset()
with torch.no_grad():
for batch in ____:
X, Y = create_target(batch)
logits = model(X)
accuracy.____(logits.reshape(-1, n_notes), Y.reshape(-1))
print(f"Test accuracy: {accuracy.compute().item():.4f}")Ideally, you should reach an accuracy of at least 40%. If you don’t, you may:
Increase the number of trainable parameters in your model, e.g. by increasing the filter size,
Train your model for more epochs, or
Adjust your checkpoint, e.g. to monitor your validation loss (‘val_loss’) to avoid overfitting.
Now let’s write a function that will generate a new chorale. We will give it a few seed chords, it will convert them to arpegios (the format expected by the model), and use the model to predict the next note, then the next, and so on. In the end, it will group the notes 4 by 4 to create chords again, and return the resulting chorale.
def generate_chorale(model, seed_chords, length):
model.eval()
arpegio = preprocess(torch.tensor(seed_chords, dtype=torch.long))
arpegio = arpegio.reshape(1, -1)
with torch.no_grad():
for chord in range(length):
for note in range(4):
next_note_logits = model(arpegio)[:, -1, :]
next_note = next_note_logits.argmax(dim=-1, keepdim=True)
arpegio = torch.cat([arpegio, next_note], dim=1)
arpegio = torch.where(arpegio == 0, arpegio, arpegio + min_note - 1)
return arpegio.reshape(-1, 4)Q9) Using seed chords from the test set, generate your first chorale! 🎼¶
Extract some seed_chords from the test set test_chorales.
Hint: You can simply use the first 5-10 chords of one of the test chorales.
seed_chords = ________[_____][____:______]and play them 😃
Hint: You may use the function play_chords defined above.
# Play the seed chordsNow we are ready to generate our first chorale! Let’s ask the function to generate n_generated more chords:
n_generated = ____________ # Choose a number of chords to generate (e.g., 20-100)new_chorale = generate_chorale(model, seed_chords, n_generated)
play_chords(new_chorale)From Géron:
"This approach has one major flaw: it is often too conservative. Indeed, the model will not take any risk, it will always choose the note with the highest score, and since repeating the previous note generally sounds good enough, it’s the least risky option, so the algorithm will tend to make notes last longer and longer. Pretty boring. Plus, if you run the model multiple times, it will always generate the same melody.
So let’s spice things up a bit! Instead of always picking the note with the highest score, we will pick the next note randomly, according to the predicted probabilities. For example, if the model predicts a C3 with 75% probability, and a G3 with a 25% probability, then we will pick one of these two notes randomly, with these probabilities. We will also add a temperature parameter that will control how “hot” (i.e., daring) we want the system to feel. A high temperature will bring the predicted probabilities closer together, reducing the probability of the likely notes and increasing the probability of the unlikely ones."
def generate_chorale_v2(model, seed_chords, length, temperature=1):
model.eval()
arpegio = preprocess(torch.tensor(seed_chords, dtype=torch.long))
arpegio = arpegio.reshape(1, -1)
with torch.no_grad():
for chord in range(length):
for note in range(4):
next_note_logits = model(arpegio)[:, -1, :]
rescaled_logits = torch.log_softmax(next_note_logits, dim=-1) / temperature
next_note = torch.distributions.Categorical(logits=rescaled_logits).sample().reshape(1, 1)
arpegio = torch.cat([arpegio, next_note], dim=1)
arpegio = torch.where(arpegio == 0, arpegio, arpegio + min_note - 1)
return arpegio.reshape(-1, 4)Q10) Using the function generate_chorale_v2, generate 3 chorales using this new function: one cold (temperature<1), one medium (temperature=1), and one hot (temperature>1).¶
new_chorale_v2_cold = generate_chorale_v2(____,____,____,____)
play_chords(new_chorale_v2_cold, filepath="bach_cold.wav")new_chorale_v2_medium = generate_chorale_v2(____,____,____,____)
play_chords(new_chorale_v2_medium, filepath="bach_medium.wav")new_chorale_v2_hot = generate_chorale_v2(____,____,____,____)
play_chords(new_chorale_v2_hot, filepath="bach_hot.wav")Part IV: Generating a Masterpiece Using Recurrent Neural Networks¶
Q11) Improve your model’s accuracy by adding batch normalization and at least one recurrent neural network layer at the end of your model.¶
Hint 1: Consider adding a Long Short-Term Memory, a Gated Recurrent Unit, or a MultiheadAttention layer.
Hint 2: Batch normalization layers are documented at this link, and you may insert them between any layers to accelerate convergence during training.
Hint 3: You may reuse some of the code you wrote for Q5-Q8.
Hint 4: If you would like to be more systematic about your model architecture choices, you can optimize hyperparameters of your model, such as the number of filters, the layer parameters, the learning rate, the optimizer, etc. using hyperparameter optimization libraries such as Optuna, as shown in the artificial-neural-networks exercises.
# Redesign your model's architecture# Set up its loss, optimizer, and accuracy metric# Define your early-stopping / checkpointing / TensorBoard tracking (they can really improve the accuracy if well-chosen!)# Train your model# Evaluate its accuracy on the test setYou should be able to reach accuracy values larger than 60% at this stage 😲
Q12) Compose a masterpiece.¶
Hint 1: You may reuse some of the code you wrote for Q9 and Q10.
Hint 2: Experiment with other seeds, lengths and temperatures to compose your masterpiece.
# Choose and play the seed chords# Generate a new chorale that's even more beautiful than the previous oneYou can try a fun social experiment: send your friends a few of your favorite generated chorales, plus the real chorale, and ask them to guess which one is the real one!
Check out Google’s Coconet model, which was used for a nice Google doodle about Bach.