OpenArch is "PyTorch implementations of modern open-source LLM architectures (Llama, Qwen, DeepSeek, Gemma, GPT-OSS, Kimi, and more) -- written from scratch for readability and learning, based on Sebastian Raschka's LLM Architecture Gallery."

Looks like I super valuable resource for all of you wanting to learn LLMs. (I do too but I don't know where to find the time.)

"This repository contains hand-written PyTorch implementations of the model architectures cataloged in Sebastian Raschka's LLM Architecture Gallery. Each model is implemented to the best of my knowledge from the original papers, technical reports, reference config.json files, and the excellent writeups by Sebastian Raschka and Machine Learning Mastery."

"The goal is not to compete with transformers or other production libraries. The goal is clarity and learning: a single readable file per architecture, with the structural choices (attention type, normalization, layer mix, MoE routing, positional encoding) made explicit and easy to compare side-by-side."

"Modern LLM architectures share a common skeleton but differ in dozens of small, important choices:"

• "Attention: MHA, GQA, MQA, MLA, sliding-window, linear/DeltaNet hybrids"
• "Normalization: pre-norm, post-norm, QK-Norm, sandwich norm, RMSNorm"
• "Positional encodings: RoPE, NoPE, partial RoPE, YaRN"
• "Decoder type: dense vs sparse MoE (with or without shared experts), hybrid Mamba/attention"
• "Training-time tricks: Multi-token-prediction, latent experts, gated attention"

"Reading the official model code can be hard because production repos optimize for speed, sharding, and backward compatibility. This repo optimizes for reading."

anuj0456 / OpenArch

#solidstatelife #ai #genai #llms #pytorch