How LLMs Work
Attention, MoE, reasoning models, and how big models get small.
Everything an LLM does traces back to a handful of architectural ideas. This guide walks through them in order: the attention mechanism underneath every transformer, the Mixture-of-Experts trick behind the biggest frontier models, how reasoning models think before answering, and the distillation and fine-tuning techniques that shrink frontier capability onto a single GPU — or a phone.
The core architecture
The math and machinery inside every modern model.
01Attention Mechanism and Transformers: The Math Behind Every LLM
Every LLM runs on the Transformer attention mechanism. Learn scaled dot-product attention, multi-head attention, and positional encoding from scratch.
02Mixture of Experts (MoE): The Architecture Behind GPT-4, Mixtral, and Grok
Mixture of Experts scales model capacity without scaling compute. Instead of activating all parameters for every token, MoE routes each token to a small subset of specialist net...
03Reasoning Models Explained: How o1, o3, and DeepSeek R1 Think Before They Answer
Reasoning models like o1, o3, and DeepSeek R1 don't just predict the next token. They think through problems step by step before answering. Here is what that actually means, why...
Beyond text
Models that see and models that draw.
04Vision Language Models (VLMs): How GPT-4o, Claude, and LLaVA Understand Images
Vision Language Models bridge the gap between pixels and language. This post covers how CLIP encodes images, how visual tokens are projected into the language space, how cross-a...
05Diffusion Models Explained: The Math-Free Guide to How Stable Diffusion and DALL-E Work
Diffusion models generate images by learning to reverse a noise process. No math required. Here is the intuition behind Stable Diffusion, DALL-E, and Midjourney, and why this ap...
Making models smaller
Distillation, LoRA, PEFT, and running LLMs at the edge.
06Knowledge Distillation: How Small Models Learn from Big Ones
Knowledge distillation trains a small student model to learn from a large teacher, retaining most capability at a fraction of the cost. Learn how it works.
07LoRA and QLoRA: Fine-Tuning LLMs on a Single GPU
Full fine-tuning a 7B parameter model requires 112 GB of VRAM and costs thousands of dollars. LoRA reduces trainable parameters by 99% by decomposing weight updates into low-ran...
08PEFT Methods Explained: LoRA, QLoRA, and Adapter-Based Fine-Tuning
Full fine-tuning a 7B model costs thousands in GPU hours. LoRA and QLoRA achieve comparable quality by training a fraction of the parameters. The math, the memory savings, and t...
09Edge AI: Running LLMs on Your Phone Without the Cloud
LLMs no longer require a data center. Phi-3, Gemma, and Apple Intelligence run directly on device, with no data leaving your phone. Here is how on-device AI works, why it matter...
Training data & open models
Where capability comes from, and who gives it away.
10Synthetic Data: How AI Trains Itself on AI-Generated Data
Real-world data is expensive, biased, and often private. Synthetic data lets AI systems generate their own training sets, and it is becoming one of the most powerful techniques ...
11Best Open-Source LLMs in 2026
Llama, Mistral, Qwen, and Gemma have each carved out distinct strengths in 2026. A practical comparison across benchmark performance, context length, licensing, and what each mo...
Deep Learning
Prefer a structured path? The self-paced Deep Learning course builds these ideas up from single neurons, with quizzes and labs.
Start the courseTools & resources
Free utilities that pair with this guide.
Keep going
Guides that pick up where this one ends.