Deep Generative Models Roadmap
In recent years, large language models (LLMs) have become a hot topic, which is also highly relevant to the author’s PhD research. This roadmap aims to share various course materials and references that the author found useful while getting familiar with and delving into the field of deep generative models. It’s intended to help practitioners in related areas or anyone interested in the underlying principles of generative models. Due to limited time outside of research, the author has not completed all the course assignments; updates will be added gradually when time permits.
In fact, LLMs are just one branch of deep generative models. Other types such as VAEs, GANs, Diffusion Models, and Flows still play an important role in the broader domain of "generation." The term AIGC (AI-generated content) broadly refers to these technologies.
Recommended courses for learning:
-
MIT 6.S184: Generative AI with Stochastic Differential Equations: An introductory GenAI course offered during MIT's IAP term. It explains the mathematical foundations behind Flow Matching and Diffusion Models from the perspective of differential equations, accompanied by simple hands-on labs to help students grasp the concepts through practice. Ideal for those interested in the underlying mathematical principles.
-
MIT 6.S978: Deep Generative Models: Taught by MIT’s rising star Prof. Kaiming He, this course covers fundamental theories and cutting-edge papers related to various generative models. The assignments include well-prepared scaffold code. While not overly difficult, they help deepen understanding and provide a quick, comprehensive view of the field.
-
UCB CS294-158-SP24: Deep Unsupervised Learning: Taught by reinforcement learning giant Pieter Abbeel. Compared to the MIT course, this one is more comprehensive and includes lecture videos and slides. The homework only provides test code, so students must implement model architecture and training code themselves. Though demanding, it’s ideal for those who want hands-on experience in training models. As is well known, there are many practical tricks in deep learning, and the devil is often in the details. Nothing teaches those details better than training a model yourself.
-
CMU 10423: Generative AI: CMU’s GenAI course focuses more on large language models compared to the previous two, but shares much of the same content otherwise. The assignments are quite engaging and worth trying out in your spare time.
The GPT series by OpenAI has demonstrated remarkable performance under the guidance of scaling laws, especially in mathematics and coding. If you are primarily interested in LLMs, the following courses are recommended:
-
Stanford CS336: Language Modeling from Scratch: As the title suggests, this course teaches you to build all the core components of an LLM from scratch, such as the tokenizer, model architecture, training optimizer, low-level operators, data cleaning, and post-training algorithms. Each assignment has a 40-50 page PDF handout—very rigorous. Highly recommended if you want to fully understand every low-level detail of LLMs.
-
CMU 15-779: Advanced Topics in Machine Learning Systems (LLM Edition) / EN: A systems and performance-oriented course that explains how high-level models are decomposed into low-level kernels and executed efficiently on heterogeneous accelerators and in distributed environments. Topics include CUDA, ML compilation, graph-level optimizations, auto-parallelization, and LLM serving/inference acceleration, along with weekly paper reviews and a final systems project.
-
CMU 11868: Large Language Model Systems: This CMU course focuses on system-level optimization of LLMs, including GPU acceleration, distributed training/inference, and cutting-edge techniques. Great for students in systems research to gain a holistic understanding of the field. (Disclosure: One of my papers on PD decoupling is included in the syllabus, hence the personal recommendation.) Assignments involve implementing a mini-PyTorch framework and then building system-level LLM optimizations on top of it.
-
CMU 11667: Large Language Models: Methods and Applications and CMU 11711: Advanced NLP: Compared to the previous two, these courses focus more on higher-level algorithms and applications. Each lecture includes many recommended readings, making them suitable for gaining a broad understanding of LLM research frontiers. You can then dive deeper into any subfield that interests you based on the reading materials.
In addition to courses, the following resources are also highly recommended:
- Awesome-LLM: A curated list of LLM-related resources.
- LLMSys-PaperList: A collection of system-related papers on LLMs.
- MLsys-Guide: A beginner’s guide to deep learning systems.
Deep Generative Models Roadmap
In recent years, large language models (LLMs) have become a hot topic, which is also highly relevant to the author’s PhD research. This roadmap aims to share various course materials and references that the author found useful while getting familiar with and delving into the field of deep generative models. It’s intended to help practitioners in related areas or anyone interested in the underlying principles of generative models. Due to limited time outside of research, the author has not completed all the course assignments; updates will be added gradually when time permits.
In fact, LLMs are just one branch of deep generative models. Other types such as VAEs, GANs, Diffusion Models, and Flows still play an important role in the broader domain of "generation." The term AIGC (AI-generated content) broadly refers to these technologies.
Recommended courses for learning:
-
MIT 6.S184: Generative AI with Stochastic Differential Equations: An introductory GenAI course offered during MIT's IAP term. It explains the mathematical foundations behind Flow Matching and Diffusion Models from the perspective of differential equations, accompanied by simple hands-on labs to help students grasp the concepts through practice. Ideal for those interested in the underlying mathematical principles.
-
MIT 6.S978: Deep Generative Models: Taught by MIT’s rising star Prof. Kaiming He, this course covers fundamental theories and cutting-edge papers related to various generative models. The assignments include well-prepared scaffold code. While not overly difficult, they help deepen understanding and provide a quick, comprehensive view of the field.
-
Stanford CS236: Deep Generative Models: A classic Stanford course taught by Stefano Ermon. Starting from Autoregressive Models and Maximum Likelihood Learning, it systematically covers VAEs, Normalizing Flows, GANs, EBMs, Score-based Models, and Diffusion Models. Compared with MIT 6.S184, CS236 takes a more probabilistic modeling perspective and derives recurring concepts such as the ELBO, Change of Variables, and Score Matching from first principles. There is a fair amount of math, but the course progresses naturally. If you want to understand why different generative models are designed the way they are rather than merely knowing how to train a Diffusion Model, this is an excellent course to follow end-to-end.
-
UCB CS294-158-SP24: Deep Unsupervised Learning: Taught by reinforcement learning giant Pieter Abbeel. Compared to the MIT course, this one is more comprehensive and includes lecture videos and slides. The homework only provides test code, so students must implement model architecture and training code themselves. Though demanding, it’s ideal for those who want hands-on experience in training models. As is well known, there are many practical tricks in deep learning, and the devil is often in the details. Nothing teaches those details better than training a model yourself.
-
CMU 10423: Generative AI: CMU’s GenAI course focuses more on large language models compared to the previous two, but shares much of the same content otherwise. The assignments are quite engaging and worth trying out in your spare time.
The GPT series by OpenAI has demonstrated remarkable performance under the guidance of scaling laws, especially in mathematics and coding. If you are primarily interested in LLMs, the following courses are recommended:
-
Stanford CS336: Language Modeling from Scratch: As the title suggests, this course teaches you to build all the core components of an LLM from scratch, such as the tokenizer, model architecture, training optimizer, low-level operators, data cleaning, and post-training algorithms. Each assignment has a 40-50 page PDF handout—very rigorous. Highly recommended if you want to fully understand every low-level detail of LLMs.
-
CMU 15-779: Advanced Topics in Machine Learning Systems (LLM Edition) / EN: A systems and performance-oriented course that explains how high-level models are decomposed into low-level kernels and executed efficiently on heterogeneous accelerators and in distributed environments. Topics include CUDA, ML compilation, graph-level optimizations, auto-parallelization, and LLM serving/inference acceleration, along with weekly paper reviews and a final systems project.
-
CMU 11868: Large Language Model Systems: This CMU course focuses on system-level optimization of LLMs, including GPU acceleration, distributed training/inference, and cutting-edge techniques. Great for students in systems research to gain a holistic understanding of the field. (Disclosure: One of my papers on PD decoupling is included in the syllabus, hence the personal recommendation.) Assignments involve implementing a mini-PyTorch framework and then building system-level LLM optimizations on top of it.
-
CMU 11667: Large Language Models: Methods and Applications and CMU 11711: Advanced NLP: Compared to the previous two, these courses focus more on higher-level algorithms and applications. Each lecture includes many recommended readings, making them suitable for gaining a broad understanding of LLM research frontiers. You can then dive deeper into any subfield that interests you based on the reading materials.
In addition to courses, the following resources are also highly recommended:
- Awesome-LLM: A curated list of LLM-related resources.
- LLMSys-PaperList: A collection of system-related papers on LLMs.
- MLsys-Guide: A beginner’s guide to deep learning systems.
深度生成模型学习路线
近几年大语言模型成为大热的方向，也和笔者博士期间的课题非常相关。这篇路线图旨在分享笔者在熟悉和深入深度生成模型这一领域过程中学习和参考的各类课程资料，方便相关领域的从业者或者对生成模型的底层原理感兴趣的朋友共同学习。由于笔者科研之余时间有限，很多课程的实验并未完成，等后续有时间完成之后会在该目录下一一添加。
其实，大语言模型只是深度生成模型的一个分支，而其他生成模型例如 VAE，GAN，Diffusion Model，Flow 等等，都还在“生成”这一领域占有重要地位，所谓的 AIGC，就是泛指这一类技术。 推荐学习下列课程：
-
MIT 6.S184: Generative AI with Stochastic Differential Equations: MIT IAP 小学期的 GenAI 入门课程，主要通过微分方程的视角讲解了 Flow Matching 和 Diffusion Model 背后的数学原理，并且配有简单的小实验让学生在实践中理解，适合对底层数学原理感兴趣的同学入门。
-
MIT 6.S978: Deep Generative Models: MIT 新晋明星教授何恺明亲授，涵盖了各种生成模型的基础理论和相关前沿论文，几次作业都有丰富的脚手架代码，难度不高但能加深理解，能对这个领域有个快速全貌了解。
-
UCB CS294-158-SP24: Deep Unsupervised Learning: 强化学习领域的顶级巨佬 Pieter Abbeel 主讲，相比 MIT 的课程内容更加丰富全面，并且有配套课程视频和 Slides。此外课后作业只有测试代码，需要学生自主编写模型架构定义和训练代码，虽然硬核但很适合有志于炼丹的同学练手。众所周知，深度学习理论实践中存在着很多经验技巧，魔鬼往往存在于细节里。没有什么比自己上手训一个模型更能掌握这些细节了。
-
CMU 10423: Generative AI: CMU 的 GenAI 课程，相比前两门课更侧重于大语言模型一些，其他内容和前两门课重合较多。不过课程作业都挺有意思，推荐闲暇时间练练手。
OpenAI 的 GPT 系列让大语言模型在 Scaling Law 的指引下展现出惊人的效果，在数学和代码领域取得了很大进展。如果你主要关注大语言模型这个方向，那么推荐如下课程：
-
Stanford CS336: Language Modeling from Scratch: 正如课程标题写的，在这门课程中你将从头编写大语言模型的所有核心组件，例如 Tokenizer，模型架构，训练优化器，底层算子，训练数据清洗，后训练算法等等。每次作业的 handout 都有四五十页 pdf，相当硬核。如果你想充分吃透大语言模型的所有底层细节，那么非常推荐学习这门课程。
-
CMU 15-779: Advanced Topics in Machine Learning Systems (LLM Edition) / EN: 偏系统与性能优化视角，重点讲清“高层模型如何分解成 kernel 并在异构加速器与分布式环境中高效执行”，覆盖 CUDA、ML 编译、图级优化、自动并行化、LLM Serving 与推理加速等内容，并配套按周论文阅读与期末系统项目。
-
CMU 11868: Large Language Model Systems: CMU 的大语言模型系统课程，侧重底层系统优化，例如 GPU 加速，分布式训练和推理，以及各种前沿技术。非常适合从事系统领域的同学对这个方向有个全貌性的了解。课表里还包含了一篇我发表的 PD 分离相关的文章，因此私心推荐一下。课程作业的话会让你先实现一个迷你 Pytorch，然后在上面实现各种大语言模型的系统级优化。
-
CMU 11667: Large Language Models: Methods and Applications 和 CMU 11711: Advanced NLP: 和前两门课相比，这两门课更偏重上层算法和应用，而且每节课都列举了很多相关阅读材料，适合对大语言模型发展前沿的各个方向都有个粗糙的认识，如果对某个子领域感兴趣的话再寻着参考资料深入学习。
除了课程以外，还有很多不错的资料作为参考：
- Awesome-LLM: 大语言模型相关资料汇总
- LLMSys-PaperList: 大语言模型系统相关论文汇总
- MLsys-Guide: 深度学习系统入门指南
深度生成模型学习路线
近几年大语言模型成为大热的方向，也和笔者博士期间的课题非常相关。这篇路线图旨在分享笔者在熟悉和深入深度生成模型这一领域过程中学习和参考的各类课程资料，方便相关领域的从业者或者对生成模型的底层原理感兴趣的朋友共同学习。由于笔者科研之余时间有限，很多课程的实验并未完成，等后续有时间完成之后会在该目录下一一添加。
其实，大语言模型只是深度生成模型的一个分支，而其他生成模型例如 VAE，GAN，Diffusion Model，Flow 等等，都还在“生成”这一领域占有重要地位，所谓的 AIGC，就是泛指这一类技术。 推荐学习下列课程：
-
MIT 6.S184: Generative AI with Stochastic Differential Equations: MIT IAP 小学期的 GenAI 入门课程，主要通过微分方程的视角讲解了 Flow Matching 和 Diffusion Model 背后的数学原理，并且配有简单的小实验让学生在实践中理解，适合对底层数学原理感兴趣的同学入门。
-
MIT 6.S978: Deep Generative Models: MIT 新晋明星教授何恺明亲授，涵盖了各种生成模型的基础理论和相关前沿论文，几次作业都有丰富的脚手架代码，难度不高但能加深理解，能对这个领域有个快速全貌了解。
-
Stanford CS236: Deep Generative Models: Stanford 的经典生成模型课程，由 Stefano Ermon 主讲，从 Autoregressive Model 和 Maximum Likelihood 出发，系统覆盖 VAE、Normalizing Flow、GAN、EBM、Score-based Model 和 Diffusion Model 等主流生成模型。相比 MIT 6.S184，这门课更偏概率建模视角，ELBO、Change of Variables、Score Matching 等常见概念都会从头推导，数学不少但循序渐进。如果你不满足于“会调 Diffusion”，而是想真正理解各种生成模型为什么这样设计，非常推荐把这门课作为主线系统学习。
-
UCB CS294-158-SP24: Deep Unsupervised Learning: 强化学习领域的顶级巨佬 Pieter Abbeel 主讲，相比 MIT 的课程内容更加丰富全面，并且有配套课程视频和 Slides。此外课后作业只有测试代码，需要学生自主编写模型架构定义和训练代码，虽然硬核但很适合有志于炼丹的同学练手。众所周知，深度学习理论实践中存在着很多经验技巧，魔鬼往往存在于细节里。没有什么比自己上手训一个模型更能掌握这些细节了。
-
CMU 10423: Generative AI: CMU 的 GenAI 课程，相比前两门课更侧重于大语言模型一些，其他内容和前两门课重合较多。不过课程作业都挺有意思，推荐闲暇时间练练手。
OpenAI 的 GPT 系列让大语言模型在 Scaling Law 的指引下展现出惊人的效果，在数学和代码领域取得了很大进展。如果你主要关注大语言模型这个方向，那么推荐如下课程：
-
Stanford CS336: Language Modeling from Scratch: 正如课程标题写的，在这门课程中你将从头编写大语言模型的所有核心组件，例如 Tokenizer，模型架构，训练优化器，底层算子，训练数据清洗，后训练算法等等。每次作业的 handout 都有四五十页 pdf，相当硬核。如果你想充分吃透大语言模型的所有底层细节，那么非常推荐学习这门课程。
-
CMU 15-779: Advanced Topics in Machine Learning Systems (LLM Edition) / EN: 偏系统与性能优化视角，重点讲清“高层模型如何分解成 kernel 并在异构加速器与分布式环境中高效执行”，覆盖 CUDA、ML 编译、图级优化、自动并行化、LLM Serving 与推理加速等内容，并配套按周论文阅读与期末系统项目。
-
CMU 11868: Large Language Model Systems: CMU 的大语言模型系统课程，侧重底层系统优化，例如 GPU 加速，分布式训练和推理，以及各种前沿技术。非常适合从事系统领域的同学对这个方向有个全貌性的了解。课表里还包含了一篇我发表的 PD 分离相关的文章，因此私心推荐一下。课程作业的话会让你先实现一个迷你 Pytorch，然后在上面实现各种大语言模型的系统级优化。
-
CMU 11667: Large Language Models: Methods and Applications 和 CMU 11711: Advanced NLP: 和前两门课相比，这两门课更偏重上层算法和应用，而且每节课都列举了很多相关阅读材料，适合对大语言模型发展前沿的各个方向都有个粗糙的认识，如果对某个子领域感兴趣的话再寻着参考资料深入学习。
除了课程以外，还有很多不错的资料作为参考：
- Awesome-LLM: 大语言模型相关资料汇总
- LLMSys-PaperList: 大语言模型系统相关论文汇总
- MLsys-Guide: 深度学习系统入门指南