Tiny-Net: learned token embeddings in 2D

Each point is the actual 2-element embedding vector learned for that byte token. The network predicts the next byte from the current byte.

Redraw every N training steps Learning rate

Loading embedded corpus…

Step: 0 EMA loss: — Corpus bytes: — Plot range: ±0.10

Network dimensions

token byte → embedding[2] → hidden[64] → tanh → logits[256] → softmax. The plotted tokens are a-z plus , . : ; -. Training uses adjacent byte pairs from the corpus embedded directly in this HTML file, one SGD update per step.