Lately snippets of the Prelude from Bach's English Suite No. 3 (BWV 808) have been playing unbidden in my mind. To scratch the itch, I trained a tiny transformer model to memorize the piece so that I could listen to it in the novel audio format of a neural network. Press the play button below to start running the forward passes of the model which drive the playback. Increasing the temperature enough will produce a cacophony.
I started with Gary Bricault's MIDI arrangement of the Prelude. Tokens represent the beginning and end of the sequence, the note on/note off MIDI messages, and advancements of the clock denominated in steps that fit 90 to a measure. This turns the piece into a sequence of 7,905 tokens, and my goal was to train a small transformer to memorize this sequence so that feeding it just the beginning-of-sequence token would result in a perfect playback.
The longest repeated subsequence is 251 tokens long, which means that the context length must be at least 252 tokens for the model to be able to produce by greedy decoding the two different 252nd tokens which follow those repetitions. I used a 256-token context window. The smallest transformer I successfully trained — the one which plays on this page — has 9,184 parameters and 36 KB of weights. It was necessary to ensure that the logit difference between the model's top prediction and the runner-up was large enough in each case that moving from PyTorch to the browser would not flip the ordering due to rounding differences.
A small JavaScript inference engine runs on this page, tested to ensure it reproduces exactly the same output as PyTorch. Inference runs much faster than necessary, so the model runs ahead of playback in batches. The output sequence is turned into sound by translating the tokens into Tone.js function calls rather than MIDI messages. Those function calls play the piano samples which you hear. A humanization step in the translation process adds small imperfections to the performance.