Merge pull request #71 from huggingface/unit8

Adding Unit 8

README.md

@@ -31,16 +31,20 @@ This course is **self-paced** you can start when you want 🥳.
 | [Published 🥳](https://github.com/huggingface/deep-rl-class/tree/main/unit5#unit-5-policy-gradient-with-pytorch) | [Policy Gradient with PyTorch](https://huggingface.co/blog/deep-rl-pg)                                     | [Code a Reinforce agent from scratch using PyTorch and train it to play Pong 🎾, CartPole and Pixelcopter 🚁](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit5/unit5.ipynb)                                       |
 | [Published 🥳](https://github.com/huggingface/deep-rl-class/tree/main/unit6#towards-better-explorations-methods-with-curiosity) | [Towards better explorations methods with Curiosity](https://github.com/huggingface/deep-rl-class/tree/main/unit6#towards-better-explorations-methods-with-curiosity)|                                                                                                         |
 | [Published 🥳](https://github.com/huggingface/deep-rl-class/tree/main/unit7#unit-7-advantage-actor-critic-a2c-using-robotics-simulations-with-pybullet-) | [Advantage Actor Critic (A2C)](https://github.com/huggingface/deep-rl-class/tree/main/unit7#unit-7-advantage-actor-critic-a2c-using-robotics-simulations-with-pybullet-) | [Train a bipedal walker and a spider to learn to walk using A2C](https://github.com/huggingface/deep-rl-class/tree/main/unit7#unit-7-advantage-actor-critic-a2c-using-robotics-simulations-with-pybullet-)                                                                                                        |
-| August the 5th | Proximal Policy Optimization (PPO)                       | 🏗️                                                                                                        |
-| August | Decision Transformers and offline Reinforcement Learning | 🏗️                                                                                                        |
+| [Published 🥳](https://github.com/huggingface/deep-rl-class/tree/main/unit8#unit-8-proximal-policy-optimization-ppo-with-pytorch) | [Proximal Policy Optimization (PPO)](https://github.com/huggingface/deep-rl-class/tree/main/unit8#unit-8-proximal-policy-optimization-ppo-with-pytorch)                       | [Code a PPO agent from scratch using PyTorch and bulletproof it with Classical Control Environments](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit8/unit8.ipynb)                                                                                                           |
+| TBA | Decision Transformers and offline Reinforcement Learning | 🏗️                                                                                                        |
 
 
 
 ## The library you'll learn during this course
+Version 1.0 (current):
 - [Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3)
 - [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo)
-- [RLlib](https://docs.ray.io/en/latest/rllib/index.html)
 - [CleanRL](https://github.com/vwxyzjn/cleanrl)
+Version 2.0:
+- + [RLlib](https://docs.ray.io/en/latest/rllib/index.html)
+- + [Sample Factory](https://github.com/alex-petrenko/sample-factory)
+- + [Hugging Face Decision Transformers](https://huggingface.co/blog/decision-transformers)
 - More to come 🏗️
 
 ## The Environments you'll use

unit8/README.md

@@ -0,0 +1,63 @@
+# Unit 8: Proximal Policy Optimization (PPO) with PyTorch
+
+Today we'll learn about Proximal Policy Optimization (PPO), an architecture that improves our agent's training stability by avoiding too large policy updates. To do that, we use a ratio that will indicates the difference between our current and old policy and clip this ratio from a specific range  $[1 - \epsilon, 1 + \epsilon]$. Doing this will ensure that our policy update will not be too large and that the training is more stable.
+
+And then, after the theory, we'll code a PPO architecture from scratch using PyTorch and bulletproof our implementation with CartPole-v1 and LunarLander-v2.
+
+🏆 You'll then be able to **compare your agent’s results with other classmates thanks to a leaderboard** 🔥 👉 https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard
+
+<img src="assets/img/LunarLander.gif" alt="LunarLander"/>
+
+Let's get started 🥳
+
+## Required time ⏱️
+The required time for this unit is, approximately:
+- 1 hour for the theory.
+- 2 hours for the hands-on.
+
+## Start this Unit 🚀
+Here are the steps for this Unit:
+
+1️⃣ 📖 [Read Proximal Policy Optimization Chapter](https://huggingface.co/blog/deep-rl-ppo).
+
+2️⃣ 👩‍💻 Then dive on the hands-on:
+
+The hands-on 👉 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit8/unit8.ipynb)
+
+Thanks to a leaderboard, you'll be able to compare your results with other classmates and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 8 🏆?
+
+The leaderboard 👉 https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard
+
+## Additional readings 📚
+- [Towards Delivering a Coherent Self-Contained Explanation of Proximal Policy Optimization by Daniel Bick](https://fse.studenttheses.ub.rug.nl/25709/1/mAI_2021_BickD.pdf) 
+- [What is the way to understand Proximal Policy Optimization Algorithm in RL?](https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl)
+- [Foundations of Deep RL Series, L4 TRPO and PPO by Pieter Abbeel](https://youtu.be/KjWF8VIMGiY)
+- [OpenAI PPO Blogpost](https://openai.com/blog/openai-baselines-ppo/)
+- [Spinning Up RL PPO](https://spinningup.openai.com/en/latest/algorithms/ppo.html)
+- [Paper Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347)
+- [The 37 Implementation Details of Proximal Policy Optimization](https://ppo-details.cleanrl.dev//2021/11/05/ppo-implementation-details/)
+
+## How to make the most of this course
+
+To make the most of the course, my advice is to:
+
+- **Participate in Discord** and join a study group.
+- **Read multiple times** the theory part and takes some notes
+- Don’t just do the colab. When you learn something, try to change the environment, change the parameters and read the libraries' documentation. Have fun 🥳
+- Struggling is **a good thing in learning**. It means that you start to build new skills. Deep RL is a complex topic and it takes time to understand. Try different approaches, use our additional readings, and exchange with classmates on discord.
+
+## This is a course built with you 👷🏿‍♀️
+
+We want to improve and update the course iteratively with your feedback. **If you have some, please fill this form** 👉 https://forms.gle/3HgA7bEHwAmmLfwh9
+
+## Don’t forget to join the Community 📢
+
+We have a discord server where you **can exchange with the community and with us, create study groups to grow each other and more** 
+
+👉🏻 [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Don’t forget to **introduce yourself when you sign up 🤗**
+
+❓If you have other questions, [please check our FAQ](https://github.com/huggingface/deep-rl-class#faq)
+
+### Keep learning, stay awesome 🤗