
Pico GPT
A full-sized language model hides too much when I want to trace how tokens, attention, Transformer blocks, training and generation fit together.
A 232,608-parameter decoder-only model with character tokenization, learned embeddings, causal attention, training, validation and text generation.
- Python
- PyTorch
- Transformers


