A from-scratch PyTorch LLM implementing Sparse Mixture-of-Experts (MoE) with Top-2 gating. Integrates modern Llama-3 components (RMSNorm, SwiGLU, RoPE, GQA) and a custom-coded Byte-Level BPE tokenizer. Pre-trained on a curated corpus of existential & dark philosophical literature.
python pytorch transformer moe from-scratch mixture-of-experts bpe gqa llm rmsnorm swiglu existential-ai llama-3-architecture rope-embeddings custom-tokenizer
-
Updated
Dec 1, 2025 - Python