Tiny-LLM: Learning the very basics

This project shows builds the very simplest "LLMs", with toy programs that you can inspect every detail. In this page, we inspect how tokenization works, specifically how we get some text into something that we send to the neurla network.

How the text becomes tokens

A neural network does not work directly with letters or words. It first converts text into tokens: integer IDs that the model can use as array indexes.

Tiny-Net deliberately uses an extremely simple tokenizer. Each byte is a token, so lowercase a, b, a space, a comma, and a newline all have their own numeric token IDs. The model has a vocabulary of 256 possible byte values.

Embedding each token

A token ID by itself is just a number. Tiny-Net therefore looks it up in an embedding table. Each token has a small learned vector associated with it. The concept of embedding is converting a token (in our case, a character) into a multi dimensional vector (in our case, 2 dimensions). The vector for each token starts randomized. Then during training, the vector is adjusted. What we see here is how similar meaning things, like puncuation and vowels, move to cluster together, while destinctive things move apart -- indeed, those clusters move away from each other.

In this demonstration the embedding dimension is exactly 2. That means every token is represented by two learned numbers:

Those two values can be used directly as the token's X and Y coordinates on a 2D plot. There is no PCA, t-SNE, or other projection involved. The points you see are the actual embedding weights inside the network.

How Tiny-Net predicts the next character

During training, Tiny-Net takes one character from the corpus and tries to predict the character that follows it. For example, when it sees t in the word the, the correct next token is h.

The model compares its prediction with the real next character and uses backpropagation to slightly adjust the weights. This includes changing the two numbers in the current token's embedding. Repeating this process many times causes the points on the plot to move.

Characters that benefit from similar internal representations may move closer together, while others may separate. The visualization lets you watch that organization emerge during training.