An interactive Transformer language model trained on the Tiny Shakespeare dataset. 📜
Inspired by Andrej Karpathy, this project helped me understand the core concepts behind Transformers, like self-attention and sequence generation. I’m now working on scaling this up to create a GPT-2 clone! 🚀
- Self-Attention Mechanism: Helps the model focus on relevant tokens in the sequence.
- Multi-Head Attention: Captures diverse relationships between tokens for richer understanding.
- Positional Embeddings: Gives the model a sense of token order, crucial for sequences.
- Layer Normalization: Stabilizes and accelerates training for better convergence.
- Feedforward Neural Network: Processes attention outputs to extract complex features.
- Token and Positional Embeddings: Converts characters into meaningful vector representations.
- Greedy & Multinomial Sampling: Enables flexible and creative text generation.
- 🧠 Parameters: ~3 Million
- 📚 Dataset: Tiny Shakespeare (~1MB of Shakespeare’s works)
- ⚡ Training Split: 90% training, 10% validation
- 📊 Loss Estimation: Robust evaluation on both train and validation sets
After training, the model can generate Shakespeare-like text — it’s incredible to see how it captures the style and rhythm of classic literature! ✨
I’m working on scaling this up into a GPT-2 clone, and I’m exploring optimizations like rotary embeddings and flash attention. If you have ideas or suggestions, I’d love to hear them! Let’s keep learning and building together. 🚀
A huge shoutout to Andrej Karpathy for making complex concepts like self-attention, multi-head attention, and layer normalization so understandable — your teachings inspired this project! 🙌
#Demo_pic
