ai-engineering-from-scratch

Journey • Get Started • Toolkit • Roadmap • Contribute • Glossary

---

260+ hands-on lessons across 20 phases. From linear algebra to autonomous agent swarms. Python, TypeScript, Rust, Julia. Every lesson produces something reusable: prompts, skills, agents, MCP servers.

You learn AI. You build real things. You ship tools others can use.

	Other Courses	This Course
Scope	One slice (NLP or Vision or Agents)	Everything: math, ML, DL, NLP, vision, speech, transformers, LLMs, agents, swarms
Languages	Python only	Python, TypeScript, Rust, Julia
Output	"I learned something"	A portfolio of tools, prompts, skills, and agents
Depth	Surface-level or theory-heavy	Build from scratch first, then use frameworks
Format	Videos or docs	Runnable code + notebooks + docs + web app

---

The Journey

Phase 0: Setup & Tooling 12 lessons Get your environment ready for everything that follows. # Lesson Type Lang 01 Dev Environment Build Python, Node, Rust 02 Git & Collaboration Learn -- 03 GPU Setup & Cloud Build Python 04 APIs & Keys Build Python, TS 05 Jupyter Notebooks Build Python 06 Python Environments Build Python 07 Docker for AI Build Python 08 Editor Setup Build -- 09 Data Management Build Python 10 Terminal & Shell Learn -- 11 Linux for AI Learn -- 12 Debugging & Profiling Build Python

Phase 1: Math Foundations 22 lessons   The intuition behind every AI algorithm, through code.

#	Lesson	Type	Lang
01	Linear Algebra Intuition	Learn	Python, Julia
02	Vectors, Matrices & Operations	Build	Python, Julia
03	Matrix Transformations & Eigenvalues	Build	Python, Julia
04	Calculus for ML: Derivatives & Gradients	Learn	Python
05	Chain Rule & Automatic Differentiation	Build	Python
06	Probability & Distributions	Learn	Python
07	Bayes' Theorem & Statistical Thinking	Build	Python
08	Optimization: Gradient Descent Family	Build	Python
09	Information Theory: Entropy, KL Divergence	Learn	Python
10	Dimensionality Reduction: PCA, t-SNE, UMAP	Build	Python
11	Singular Value Decomposition	Build	Python, Julia
12	Tensor Operations	Build	Python
13	Numerical Stability	Build	Python
14	Norms & Distances	Build	Python
15	Statistics for ML	Build	Python
16	Sampling Methods	Build	Python
17	Linear Systems	Build	Python
18	Convex Optimization	Build	Python
19	Complex Numbers for AI	Learn	Python
20	The Fourier Transform	Build	Python
21	Graph Theory for ML	Build	Python
22	Stochastic Processes	Learn	Python

Phase 2: ML Fundamentals 18 lessons   Classical ML - still the backbone of most production AI.

#	Lesson	Type	Lang
01	What Is Machine Learning	Learn	Python
02	Linear Regression from Scratch	Build	Python
03	Logistic Regression & Classification	Build	Python
04	Decision Trees & Random Forests	Build	Python
05	Support Vector Machines	Build	Python
06	KNN & Distance Metrics	Build	Python
07	Unsupervised Learning: K-Means, DBSCAN	Build	Python
08	Feature Engineering & Selection	Build	Python
09	Model Evaluation: Metrics, Cross-Validation	Build	Python
10	Bias, Variance & the Learning Curve	Learn	Python
11	Ensemble Methods: Boosting, Bagging, Stacking	Build	Python
12	Hyperparameter Tuning	Build	Python
13	ML Pipelines & Experiment Tracking	Build	Python
14	Naive Bayes	Build	Python
15	Time Series Fundamentals	Build	Python
16	Anomaly Detection	Build	Python
17	Handling Imbalanced Data	Build	Python
18	Feature Selection	Build	Python

Phase 3: Deep Learning Core 13 lessons   Neural networks from first principles. No frameworks until you build one.

#	Lesson	Type	Lang
01	The Perceptron: Where It All Started	Build	Python
02	Multi-Layer Networks & Forward Pass	Build	Python
03	Backpropagation from Scratch	Build	Python
04	Activation Functions: ReLU, Sigmoid, GELU & Why	Build	Python
05	Loss Functions: MSE, Cross-Entropy, Contrastive	Build	Python
06	Optimizers -SGD, Momentum, Adam, AdamW	Build	Python
07	Regularization -Dropout, Weight Decay, BatchNorm	Build	Python
08	Weight Initialization & Training Stability	Build	Python
09	Learning Rate Schedules & Warmup	Build	Python
10	Build Your Own Mini Framework	Build	Python
11	Introduction to PyTorch	Build	Python
12	Introduction to JAX	Build	Python
13	Debugging Neural Networks	Learn	Python

Phase 4: Computer Vision 16 lessons   From pixels to understanding - image, video, and 3D.

#	Lesson	Type	Lang
01	Image Fundamentals: Pixels, Channels, Color Spaces	Learn	Python
02	Convolutions from Scratch	Build	Python
03	CNNs: LeNet to ResNet	Build	Python
04	Image Classification	Build	Python
05	Transfer Learning & Fine-Tuning	Build	Python
06	Object Detection -YOLO from Scratch	Build	Python
07	Semantic Segmentation -U-Net	Build	Python
08	Instance Segmentation -Mask R-CNN	Build	Python
09	Image Generation -GANs	Build	Python
10	Image Generation -Diffusion Models	Build	Python
11	Stable Diffusion -Architecture & Fine-Tuning	Build	Python
12	Video Understanding -Temporal Modeling	Build	Python
13	3D Vision: Point Clouds, NeRFs	Build	Python
14	Vision Transformers (ViT)	Build	Python
15	Real-Time Vision: Edge Deployment	Build	Python, Rust
16	Build a Complete Vision Pipeline	Build	Python

Phase 5: NLP: Foundations to Advanced 18 lessons   Language is the interface to intelligence.

#	Lesson	Type	Lang
01	Text Processing: Tokenization, Stemming, Lemmatization	Build	Python
02	Bag of Words, TF-IDF & Text Representation	Build	Python
03	Word Embeddings: Word2Vec from Scratch	Build	Python
04	GloVe, FastText & Subword Embeddings	Build	Python
05	Sentiment Analysis	Build	Python
06	Named Entity Recognition (NER)	Build	Python
07	POS Tagging & Syntactic Parsing	Build	Python
08	Text Classification -CNNs & RNNs for Text	Build	Python
09	Sequence-to-Sequence Models	Build	Python
10	Attention Mechanism -The Breakthrough	Build	Python
11	Machine Translation	Build	Python
12	Text Summarization	Build	Python
13	Question Answering Systems	Build	Python
14	Information Retrieval & Search	Build	Python
15	Topic Modeling: LDA, BERTopic	Build	Python
16	Text Generation	Build	Python
17	Chatbots: Rule-Based to Neural	Build	Python
18	Multilingual NLP	Build	Python

Phase 6: Speech & Audio 12 lessons   Hear, understand, speak.

#	Lesson	Type	Lang
01	Audio Fundamentals: Waveforms, Sampling, FFT	Learn	Python
02	Spectrograms, Mel Scale & Audio Features	Build	Python
03	Audio Classification	Build	Python
04	Speech Recognition (ASR)	Build	Python
05	Whisper: Architecture & Fine-Tuning	Build	Python
06	Speaker Recognition & Verification	Build	Python
07	Text-to-Speech (TTS)	Build	Python
08	Voice Cloning & Voice Conversion	Build	Python
09	Music Generation	Build	Python
10	Audio-Language Models	Build	Python
11	Real-Time Audio Processing	Build	Python, Rust
12	Build a Voice Assistant Pipeline	Build	Python

Phase 7: Transformers Deep Dive 14 lessons   The architecture that changed everything.

#	Lesson	Type	Lang
01	Why Transformers: The Problems with RNNs	Learn	--
02	Self-Attention from Scratch	Build	Python
03	Multi-Head Attention	Build	Python
04	Positional Encoding: Sinusoidal, RoPE, ALiBi	Build	Python
05	The Full Transformer: Encoder + Decoder	Build	Python
06	BERT -Masked Language Modeling	Build	Python
07	GPT -Causal Language Modeling	Build	Python
08	T5, BART -Encoder-Decoder Models	Build	Python
09	Vision Transformers (ViT)	Build	Python
10	Audio Transformers -Whisper Architecture	Build	Python
11	Mixture of Experts (MoE)	Build	Python
12	KV Cache, Flash Attention & Inference Optimization	Build	Python, Rust
13	Scaling Laws	Learn	Python
14	Build a Transformer from Scratch	Build	Python

Phase 8: Generative AI 14 lessons   Create images, video, audio, 3D, and more.

#	Lesson	Type	Lang
01	Generative Models: Taxonomy & History	Learn	--
02	Autoencoders & VAE	Build	Python
03	GANs: Generator vs Discriminator	Build	Python
04	Conditional GANs & Pix2Pix	Build	Python
05	StyleGAN	Build	Python
06	Diffusion Models -DDPM from Scratch	Build	Python
07	Latent Diffusion & Stable Diffusion	Build	Python
08	ControlNet, LoRA & Conditioning	Build	Python
09	Inpainting, Outpainting & Editing	Build	Python
10	Video Generation	Build	Python
11	Audio Generation	Build	Python
12	3D Generation	Build	Python
13	Flow Matching & Rectified Flows	Build	Python
14	Evaluation: FID, CLIP Score	Build	Python

Phase 9: Reinforcement Learning 12 lessons   The foundation of RLHF and game-playing AI.

#	Lesson	Type	Lang
01	MDPs, States, Actions & Rewards	Learn	Python
02	Dynamic Programming	Build	Python
03	Monte Carlo Methods	Build	Python
04	Q-Learning, SARSA	Build	Python
05	Deep Q-Networks (DQN)	Build	Python
06	Policy Gradients -REINFORCE	Build	Python
07	Actor-Critic -A2C, A3C	Build	Python
08	PPO	Build	Python
09	Reward Modeling & RLHF	Build	Python
10	Multi-Agent RL	Build	Python
11	Sim-to-Real Transfer	Build	Python
12	RL for Games	Build	Python

Phase 10: LLMs from Scratch 14 lessons   Build, train, and understand large language models.

#	Lesson	Type	Lang
01	Tokenizers: BPE, WordPiece, SentencePiece	Build	Python
02	Building a Tokenizer from Scratch	Build	Python
03	Data Pipelines for Pre-Training	Build	Python
04	Pre-Training a Mini GPT (124M)	Build	Python
05	Distributed Training, FSDP, DeepSpeed	Build	Python
06	Instruction Tuning -SFT	Build	Python
07	RLHF -Reward Model + PPO	Build	Python
08	DPO -Direct Preference Optimization	Build	Python
09	Constitutional AI	Build	Python
10	Evaluation -Benchmarks, Evals	Build	Python
11	Quantization: INT8, GPTQ, AWQ, GGUF	Build	Python, Rust
12	Inference Optimization	Build	Python
13	Building a Complete LLM Pipeline	Build	Python
14	Open Models: Architecture Walkthroughs	Learn	Python

Phase 11: LLM Engineering 13 lessons   Put LLMs to work in production.

#	Lesson	Type	Lang
01	Prompt Engineering: Techniques & Patterns	Build	Python
02	Few-Shot, CoT, Tree-of-Thought	Build	Python
03	Structured Outputs	Build	Python, TS
04	Embeddings & Vector Representations	Build	Python
05	Vector Databases	Build	Python, TS
06	RAG -Retrieval-Augmented Generation	Build	Python, TS
07	Advanced RAG -Chunking, Reranking	Build	Python
08	Fine-Tuning with LoRA & QLoRA	Build	Python
09	Function Calling & Tool Use	Build	Python, TS
10	Evaluation & Testing	Build	Python
11	Caching, Rate Limiting & Cost	Build	Python, TS
12	Guardrails & Safety	Build	Python
13	Building a Production LLM App	Build	Python, TS

Phase 12: Multimodal AI 11 lessons   See, hear, read, and reason across modalities.

#	Lesson	Type	Lang
01	Multimodal Representations	Learn	--
02	CLIP: Vision + Language	Build	Python
03	Vision-Language Models	Build	Python
04	Audio-Language Models	Build	Python
05	Document Understanding	Build	Python
06	Video-Language Models	Build	Python
07	Multimodal RAG	Build	Python, TS
08	Multimodal Agents	Build	Python, TS
09	Text-to-Image Pipelines	Build	Python
10	Text-to-Video Pipelines	Build	Python
11	Any-to-Any Models	Learn	Python

Phase 13: Tools & Protocols 10 lessons   The interfaces between AI and the real world.

#	Lesson	Type	Lang
01	Function Calling Deep Dive	Build	Python, TS
02	Tool Use Patterns	Build	TS
03	MCP: Model Context Protocol	Learn	--
04	Building MCP Servers	Build	TS, Python
05	Building MCP Clients	Build	TS, Python
06	MCP Resources, Prompts & Sampling	Build	TS
07	Structured Output Schemas	Build	TS, Python
08	API Design for AI	Build	TS
09	Browser Automation & Web Agents	Build	TS
10	Build a Complete Tool Ecosystem	Build	TS, Python

Phase 14: Agent Engineering 15 lessons   Build agents from first principles.

#	Lesson	Type	Lang
01	The Agent Loop	Build	Python, TS
02	Tool Dispatch & Registration	Build	TS
03	Planning: TodoWrite, DAGs	Build	TS
04	Memory: Short-Term, Long-Term, Episodic	Build	TS, Python
05	Context Window Management	Build	TS
06	Context Compression & Summarization	Build	TS
07	Subagents: Delegation	Build	TS
08	Skills & Knowledge Loading	Build	TS
09	Permissions, Sandboxing & Safety	Build	TS, Rust
10	File-Based Task Systems	Build	TS
11	Background Task Execution	Build	TS
12	Error Recovery & Self-Healing	Build	TS
13	Hooks: PreToolUse, PostToolUse	Build	TS
14	Eval-Driven Agent Development	Build	Python, TS
15	Build a Complete AI Agent	Build	TS

Phase 15: Autonomous Systems 11 lessons   Agents that run without human intervention safely.

#	Lesson	Type	Lang
01	What Makes a System Autonomous	Learn	--
02	Autonomous Loops	Build	TS, Python
03	Self-Healing Agents	Build	TS
04	AutoResearch: Autonomous Research	Build	TS, Python
05	Eval-Driven Loops	Build	TS
06	Human-in-the-Loop	Build	TS
07	Continuous Agents	Build	TS
08	Cost-Aware Autonomous Systems	Build	TS
09	Monitoring & Observability	Build	TS, Rust
10	Safety Boundaries	Build	TS
11	Build an Autonomous Coding Agent	Build	TS

Phase 16: Multi-Agent & Swarms 14 lessons   Coordination, emergence, and collective intelligence.

#	Lesson	Type	Lang
01	Why Multi-Agent	Learn	--
02	Agent Teams: Roles & Delegation	Build	TS
03	Communication Protocols	Build	TS
04	Shared State & Coordination	Build	TS, Rust
05	Message Passing & Mailboxes	Build	TS
06	Task Markets	Build	TS
07	Consensus Algorithms	Build	TS, Rust
08	Swarm Intelligence	Build	Python, TS
09	Agent Economies	Build	TS
10	Worktree Isolation	Build	TS
11	Hierarchical Swarms	Build	TS
12	Self-Organizing Systems	Build	TS, Rust
13	DAG-Based Orchestration	Build	TS, Rust
14	Build an Autonomous Swarm	Build	TS, Rust

Phase 17: Infrastructure & Production 11 lessons   Ship AI to the real world.

#	Lesson	Type	Lang
01	Model Serving	Build	Python
02	Docker for AI Workloads	Build	Python, Rust
03	Kubernetes for AI	Build	Python
04	Edge Deployment: ONNX, WASM	Build	Python, Rust
05	Observability	Build	TS, Rust
06	Cost Optimization	Build	TS
07	CI/CD for ML	Build	Python
08	A/B Testing & Feature Flags	Build	Python, TS
09	Data Pipelines	Build	Python, Rust
10	Security: Red Teaming, Defense	Build	Python, TS
11	Build a Production AI Platform	Build	Python, TS, Rust

Phase 18: Ethics, Safety & Alignment 6 lessons   Build AI that helps humanity. Not optional.

#	Lesson	Type	Lang
01	AI Ethics: Bias, Fairness	Learn	--
02	Alignment: What & Why	Learn	--
03	Red Teaming & Adversarial Testing	Build	Python
04	Responsible AI Frameworks	Learn	--
05	Privacy: Differential Privacy, FL	Build	Python
06	Interpretability: SHAP, Attention	Build	Python

Phase 19: Capstone Projects 5 projects   Prove everything you learned.

#	Project	Combines	Lang
01	Build a Mini GPT & Chat Interface	Phases 1, 3, 7, 10	Python, TS
02	Build a Multimodal RAG System	Phases 5, 11, 12, 13	Python, TS
03	Build an Autonomous Research Agent	Phases 14, 15, 6	TS, Python
04	Build a Multi-Agent Dev Team	Phases 14, 15, 16, 17	TS, Rust
05	Build a Production AI Platform	All phases	Python, TS, Rust

---

Course Output: The Toolkit

Every lesson produces something reusable. By the end you have:

outputs/
├── prompts/          Prompt templates for every AI task
├── skills/           SKILL.md files for AI coding agents
├── agents/           Agent definitions ready to deploy
└── mcp-servers/      MCP servers you built during the course

Real tools. Install them with SkillKit, plug them into Claude Code, Cursor, or any AI agent.

---

How Each Lesson Works

phases/XX-phase-name/NN-lesson-name/
├── code/           Runnable implementations (Python, TS, Rust, Julia)
├── notebook/       Jupyter notebooks for experimentation
├── docs/
│   └── en.md       Lesson documentation
└── outputs/        Prompts, skills, agents produced by this lesson

Every lesson follows 6 steps:

1. Motto - one-line core idea
2. Problem - why this matters
3. Concept - visual diagrams and intuition
4. Build It - implement from scratch
5. Use It - same thing with real frameworks
6. Ship It - the prompt, skill, or agent this lesson produces

---

Getting Started

git clone https://github.com/rohitg00/ai-engineering-from-scratch.git
cd ai-engineering-from-scratch

python phases/00-setup-and-tooling/01-dev-environment/code/verify.py

python phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py

Prerequisites

• You can write code (Python or any language)
• You want to understand how AI actually works

---

Contributing

See CONTRIBUTING.md for how to add lessons, translations, and outputs.

Want to fork this for your team or school? See FORKING.md.

See ROADMAP.md for progress tracking.

---

MIT License. Use it however you want.