From 60d2c0ac6b6cd8882b757cb65e0e196d01d88bf1 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Tue, 2 Aug 2022 09:07:18 +0200 Subject: [PATCH 1/4] Add additional readings PPO unit --- unit8/README.md | 66 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 unit8/README.md diff --git a/unit8/README.md b/unit8/README.md new file mode 100644 index 0000000..3c47b34 --- /dev/null +++ b/unit8/README.md @@ -0,0 +1,66 @@ +# Unit 8: Proximal Policy Optimization (PPO) using Robotics Simulations with PyBullet πŸ€– + +One of the major industries that use Reinforcement Learning is robotics. Unfortunately, **having access to robot equipment is very expensive**. Fortunately, some simulations exist to train Robots: +1. PyBullet +2. MuJoco +3. Unity Simulations + +We're going to learn about Advantage Actor Critic (A2C) and how to use PyBullet. And train a spider agent to walk. + +πŸ† You'll then be able to **compare your agent’s results with other classmates thanks to a leaderboard** πŸ”₯ πŸ‘‰ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard + +![cover](https://github.com/huggingface/deep-rl-class/blob/main/unit7/assets/img/pybullet-envs.gif?raw=true) + +Let's get started πŸ₯³ + +## Required time ⏱️ +The required time for this unit is, approximately: +- 1 hour for the theory. +- 2 hours for the hands-on. + +## Start this Unit πŸš€ +Here are the steps for this Unit: + +1️⃣ πŸ“– [Read Proximal Policy Optimization Chapter](https://huggingface.co/blog/deep-rl-ppo). + +2️⃣ πŸ‘©β€πŸ’» Then dive on the hands-on where you'll train two robots to walk. + +The hands-on πŸ‘‰ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit7/unit7.ipynb) + +Thanks to a leaderboard, you'll be able to compare your results with other classmates and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 7 πŸ†? + +The leaderboard πŸ‘‰ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard + +## Additional readings πŸ“š +- [Towards Delivering a Coherent Self-Contained Explanation of Proximal Policy Optimization by Daniel Bick](https://fse.studenttheses.ub.rug.nl/25709/1/mAI_2021_BickD.pdf) +- [What is the way to understand Proximal Policy Optimization Algorithm in RL?](https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl) +- [Foundations of Deep RL Series, L4 TRPO and PPO by Pieter Abbeel](https://youtu.be/KjWF8VIMGiY) +- [OpenAI PPO Blogpost](https://openai.com/blog/openai-baselines-ppo/) +- [Spinning Up RL PPO](https://spinningup.openai.com/en/latest/algorithms/ppo.html) +- [Paper Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347) +- [The 37 Implementation Details of Proximal Policy Optimization](https://ppo-details.cleanrl.dev//2021/11/05/ppo-implementation-details/) + +## How to make the most of this course + +To make the most of the course, my advice is to: + +- **Participate in Discord** and join a study group. +- **Read multiple times** the theory part and takes some notes +- Don’t just do the colab. When you learn something, try to change the environment, change the parameters and read the libraries' documentation. Have fun πŸ₯³ +- Struggling is **a good thing in learning**. It means that you start to build new skills. Deep RL is a complex topic and it takes time to understand. Try different approaches, use our additional readings, and exchange with classmates on discord. + +## This is a course built with you πŸ‘·πŸΏβ€β™€οΈ + +We want to improve and update the course iteratively with your feedback. **If you have some, please fill this form** πŸ‘‰ https://forms.gle/3HgA7bEHwAmmLfwh9 + +## Don’t forget to join the Community πŸ“’ + +We have a discord server where youΒ **can exchange with the community and with us, create study groups to grow each other and more**Β  + +πŸ‘‰πŸ»Β [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9). + +Don’t forget toΒ **introduce yourself when you sign upΒ πŸ€—** + +❓If you have other questions, [please check our FAQ](https://github.com/huggingface/deep-rl-class#faq) + +### Keep learning, stay awesome πŸ€— From 81558a6a0e63b4c6c6c4c1fa52991160de680cc5 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Thu, 4 Aug 2022 21:34:24 +0200 Subject: [PATCH 2/4] Update README.md --- unit8/README.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/unit8/README.md b/unit8/README.md index 3c47b34..60ab065 100644 --- a/unit8/README.md +++ b/unit8/README.md @@ -1,15 +1,12 @@ -# Unit 8: Proximal Policy Optimization (PPO) using Robotics Simulations with PyBullet πŸ€– +# Unit 8: Proximal Policy Optimization (PPO) with PyTorch -One of the major industries that use Reinforcement Learning is robotics. Unfortunately, **having access to robot equipment is very expensive**. Fortunately, some simulations exist to train Robots: -1. PyBullet -2. MuJoco -3. Unity Simulations +Today we'll learn about Proximal Policy Optimization (PPO), an architecture that improves our agent's training stability by avoiding too large policy updates. To do that, we use a ratio that will indicates the difference between our current and old policy and clip this ratio from a specific range $[1 - \epsilon, 1 + \epsilon]$. Doing this will ensure that our policy update will not be too large and that the training is more stable. -We're going to learn about Advantage Actor Critic (A2C) and how to use PyBullet. And train a spider agent to walk. +And then, after the theory, we'll code a PPO architecture from scratch using PyTorch and bulletproof our implementation with CartPole-v1 and LunarLander-v2. πŸ† You'll then be able to **compare your agent’s results with other classmates thanks to a leaderboard** πŸ”₯ πŸ‘‰ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard -![cover](https://github.com/huggingface/deep-rl-class/blob/main/unit7/assets/img/pybullet-envs.gif?raw=true) +LunarLander Let's get started πŸ₯³ @@ -23,9 +20,9 @@ Here are the steps for this Unit: 1️⃣ πŸ“– [Read Proximal Policy Optimization Chapter](https://huggingface.co/blog/deep-rl-ppo). -2️⃣ πŸ‘©β€πŸ’» Then dive on the hands-on where you'll train two robots to walk. +2️⃣ πŸ‘©β€πŸ’» Then dive on the hands-on: -The hands-on πŸ‘‰ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit7/unit7.ipynb) +The hands-on πŸ‘‰ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit8/unit8.ipynb) Thanks to a leaderboard, you'll be able to compare your results with other classmates and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 7 πŸ†? From b3e11ff4d20641b2957cf126cb363950612f36f9 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 5 Aug 2022 08:26:47 +0200 Subject: [PATCH 3/4] Update README.md --- README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index cd4a82a..0bf86aa 100644 --- a/README.md +++ b/README.md @@ -31,16 +31,20 @@ This course is **self-paced** you can start when you want πŸ₯³. | [Published πŸ₯³](https://github.com/huggingface/deep-rl-class/tree/main/unit5#unit-5-policy-gradient-with-pytorch) | [Policy Gradient with PyTorch](https://huggingface.co/blog/deep-rl-pg) | [Code a Reinforce agent from scratch using PyTorch and train it to play Pong 🎾, CartPole and Pixelcopter 🚁](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit5/unit5.ipynb) | | [Published πŸ₯³](https://github.com/huggingface/deep-rl-class/tree/main/unit6#towards-better-explorations-methods-with-curiosity) | [Towards better explorations methods with Curiosity](https://github.com/huggingface/deep-rl-class/tree/main/unit6#towards-better-explorations-methods-with-curiosity)| | | [Published πŸ₯³](https://github.com/huggingface/deep-rl-class/tree/main/unit7#unit-7-advantage-actor-critic-a2c-using-robotics-simulations-with-pybullet-) | [Advantage Actor Critic (A2C)](https://github.com/huggingface/deep-rl-class/tree/main/unit7#unit-7-advantage-actor-critic-a2c-using-robotics-simulations-with-pybullet-) | [Train a bipedal walker and a spider to learn to walk using A2C](https://github.com/huggingface/deep-rl-class/tree/main/unit7#unit-7-advantage-actor-critic-a2c-using-robotics-simulations-with-pybullet-) | -| August the 5th | Proximal Policy Optimization (PPO) | πŸ—οΈ | -| August | Decision Transformers and offline Reinforcement Learning | πŸ—οΈ | +| [Published πŸ₯³](https://github.com/huggingface/deep-rl-class/tree/main/unit8#unit-8-proximal-policy-optimization-ppo-with-pytorch) | [Proximal Policy Optimization (PPO)](https://github.com/huggingface/deep-rl-class/tree/main/unit8#unit-8-proximal-policy-optimization-ppo-with-pytorch) | [Code a PPO agent from scratch using PyTorch and bulletproof it with Classical Control Environments](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit8/unit8.ipynb) | +| TBA | Decision Transformers and offline Reinforcement Learning | πŸ—οΈ | ## The library you'll learn during this course +Version 1.0 (current): - [Stable-Baselines3](https://github.com/DLR-RM/stable-baselines3) - [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo) -- [RLlib](https://docs.ray.io/en/latest/rllib/index.html) - [CleanRL](https://github.com/vwxyzjn/cleanrl) +Version 2.0: +- + [RLlib](https://docs.ray.io/en/latest/rllib/index.html) +- + [Sample Factory](https://github.com/alex-petrenko/sample-factory) +- + [Hugging Face Decision Transformers](https://huggingface.co/blog/decision-transformers) - More to come πŸ—οΈ ## The Environments you'll use From c4064777629090ace63cf4d2beae100e8450a1df Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 5 Aug 2022 08:27:15 +0200 Subject: [PATCH 4/4] Update README.md --- unit8/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unit8/README.md b/unit8/README.md index 60ab065..1fcb930 100644 --- a/unit8/README.md +++ b/unit8/README.md @@ -24,7 +24,7 @@ Here are the steps for this Unit: The hands-on πŸ‘‰ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit8/unit8.ipynb) -Thanks to a leaderboard, you'll be able to compare your results with other classmates and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 7 πŸ†? +Thanks to a leaderboard, you'll be able to compare your results with other classmates and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 8 πŸ†? The leaderboard πŸ‘‰ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard