Merge pull request #1 from huggingface/main

update from source

DISCORD.Md

@@ -0,0 +1,33 @@
+## Discord 101
+
+Hey there! My name is Huggy, the dog 🐶, and I'm looking forward to train with you during this RL Class! Although I don't know much about bringing sticks (yet), I know one or two things about Discord. So I wrote this guide to help you learn about it!
+
+![Huggy](https://pbs.twimg.com/media/FRXE2eXWUAQTkEM?format=png&name=900x900)
+
+Discord is a free chat platform. If you've used Slack, it's quite similar. There is a Hugging Face Community Discord server with 8000 members you can join with a single click [here](http://hf.co/join/discord). 🐕 So many humans to play with!
+
+Starting in Discord can be a bit intimidating, so let me take you through it.
+
+* Unlike Slack, you have access to most channels. That means that there might be a bit more noise that what you would like. And I hate loud noise **woof**! 🔔 Since channels are gathered by topic, you can mute individual channels or entire categories. For example, the **Audio Lounge** category is about ML for Audio which I don't care about, so I right click the category and then click **Mute**. Ah much better! 
+* I still see so many channels **woof**. You can also collapse a category so you don't see channels that are not of interest.
+* So which channels are interesting to me?
+  * They are in the reinforcement learning lounge.
+  * #rl-announcements have announcements about the course.
+  * #discussions is a place where you can talk with fellow humans.
+  * #unity-ml-agents is a place to talk about ML Agents, a library we'll learn during th ecourse.
+  * #study-groups do you want to form a group with fellow humans? This is the place
+ 
+The HF Community Server has a thriving community of humans interested in many areas, so you can also learn from those. There are paper discussions, events, and many other things. Here are other channels in the community
+  * in #reads people share interesting resources or papers they've found
+  * in #today-im-learning people share what they are learning...today!
+  * in #i-made-this you can show off your amazing projects **woof**. 
+
+
+🦴🐕Was this useful? There are a couple of tips I can share with you:
+* There are voice channels you can use as well, although most people prefer text chat.
+* You can use markdown style. So if you're writing code, you can use that style. Sadly this does not work as well for links.
+* You can open threads as well!
+
+I hope this is useful! And if you have questions, just ask! See you later! 🌳
+
+**Huggy**

README.md

@@ -9,18 +9,22 @@ In this free course, you will:
 - 🏆 **Participate in challenges** where you will evaluate your agents against other teams.
 - 🖌️🎨 **Learn to share your own environments made with Unity and Godot**.
 
-➡️➡️➡️ Don't forget to sign up here: https://forms.gle/4bbgzs3oVZMjgDed7
+➡️➡️➡️ Don't forget to sign up here: http://eepurl.com/h1pElX
 
 The best way to keep in touch is to **join our discord server to exchange with the community and with us** 👉🏻 https://discord.gg/aYka4Yhff9
 
+Are you new to Discord? Check our **discord 101 to get the best practices** 👉 https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md
+
 And don't forget to share with your friends who want to learn 🤗!
 
 ## The Syllabus 🏗️
+This course is **self-paced** you can start when you want 🥳.
+
 | 📆 Publishing date  | 📘 Unit           | 👩‍💻 Hands-on |
 |---------------|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
-| May, the 4th  | An Introduction to Deep Reinforcement Learning           | Train a Deep Reinforcement Learning lander agent to land correctly on the Moon 🌕 using Stable-Baselines3 |
-| May, the 11th | Bonus                                                    | 🎁 it's a suprise 🎁                                                               |
-| May, the 18th | Q-Learning                                               | Train an agent to cross a Frozen lake in this new version of the environment.                                                                                                        |
+| [Published 🥳](https://github.com/huggingface/deep-rl-class/tree/main/unit1#unit-1-introduction-to-deep-reinforcement-learning)  | [An Introduction to Deep Reinforcement Learning](https://github.com/huggingface/deep-rl-class/tree/main/unit1)           | [Train a Deep Reinforcement Learning lander agent to land correctly on the Moon 🌕 using Stable-Baselines3](https://github.com/huggingface/deep-rl-class/blob/main/unit1/unit1.ipynb) |
+| May, the 11th | [Bonus](https://discord.com/channels/879548962464493619/968114737655214080/973937495546925056)                                                    |                                                               |
+| May, the 18th/May, the 20th | [Q-Learning](https://github.com/huggingface/deep-rl-class/blob/main/unit2/README.md)                                               | Train an agent to cross a Frozen lake in this new version of the environment.                                                                                                        |
 | June, the 1st | Deep Q-Learning and improvements                         | Train a Deep Q-Learning agent to play Space Invaders                                                     |
 |               | Policy-based methods                                     | 🏗️                                                                                                        |
 |               | Actor-Critic Methods                                     | 🏗️                                                                                                        |
@@ -69,4 +73,88 @@ And don't forget to share with your friends who want to learn 🤗!
 - More to come 🚧
 
 
+## Prerequisites
+- Good skills in Python 🐍
+- Basics in Deep Learning and Pytorch
 
+If it's not the case yet, you can check these free resources:
+- Python: https://www.udacity.com/course/introduction-to-python--ud1110
+- Intro to Deep Learning with PyTorch: https://www.udacity.com/course/deep-learning-pytorch--ud188
+- PyTorch in 60min: https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html
+
+## FAQ
+**Is this class free?**
+
+Yes, totally free 🥳.
+
+
+**Do I need to have a Hugging Face account to follow the course?**
+
+Yes, to push your trained agents during the hands-on, you need an account (it's free) 🤗.
+
+You can create one here 👉 [https://huggingface.co/join](https://huggingface.co/join)
+
+
+**What’s the format of the class?**
+
+The course consists of **8 Units.** In each of the Units, we'll have:
+
+- **A theory explained part**: an article and a video (based on Deep Reinforcement Learning Course)
+- **A hands-on Google Colab** where you'll learn to use famous Deep RL libraries such as Stable Baselines3, RL Baselines3 Zoo, and RLlib to train your agents in unique environments such as SnowballFight, Huggy the Doggo 🐶, and classical ones such as Space Invaders and PyBullet.
+- **Some optional challenges**: train an agent in another environment, and try to beat the results.
+
+It's not a live course video, so you can watch and read each unit when you want 🤗
+You can check the syllabus here 👉 https://github.com/huggingface/deep-rl-class
+
+
+**What I will do during this course?**
+
+In this free course, you will:
+
+- 📖 Study Deep Reinforcement Learning in theory and practice.
+- 🧑‍💻 Learn to use famous Deep RL libraries such as Stable Baselines3, RL Baselines3 Zoo, and RLlib.
+- 🤖 Train agents in unique environments such as SnowballFight, Huggy the Doggo 🐶, and classical ones such as Space Invaders and PyBullet.
+- 💾 Publish your trained agents in one line of code to the Hub. But also download powerful agents from the community.
+- 🏆 Participate in challenges where you will evaluate your agents against other teams.
+- 🖌️🎨 Learn to share your own environments made with Unity and Godot.
+
+
+**Where do I sign up?**
+
+Here 👉 http://eepurl.com/h1pElX
+
+
+**Where can I find the course?**
+
+**On this repository**, we'll publish every week the links (chapters, hands-ons, videos).
+
+
+**Where can I exchange with my classmates and with you?**
+
+We have a discord server where you **can exchange with the community and with us** 👉🏻 [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9)
+
+Don’t forget to **introduce yourself when you sign up 🤗**
+
+
+**I have some feedback**
+
+We want to improve and update the course iteratively with your feedback. If you have some, please send a mail to thomas.simonini@huggingface.co
+
+
+**How much background knowledge is needed?**
+
+Some prerequisites:
+
+Good skills in **Python** 🐍
+Basics in **Deep Learning and Pytorch**
+
+If it's not the case yet, you can check these free resources:
+
+- Python: [https://www.udacity.com/course/introduction-to-python--ud1110](https://www.udacity.com/course/introduction-to-python--ud1110)
+- Intro to Deep Learning with PyTorch: [https://www.udacity.com/course/deep-learning-pytorch--ud188](https://www.udacity.com/course/deep-learning-pytorch--ud188)
+- PyTorch in 60min: [https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)
+
+
+**Is there a certificate?**
+
+Yes 🎉. You'll **need to upload the eight models with the eight hands-on.**

unit1/README.md

@@ -0,0 +1,76 @@
+# Unit 1: Introduction to Deep Reinforcement Learning
+
+In this Unit, you'll learn the foundations of Deep RL. And **you’ll train your first lander agent 🚀 to land correctly on the Moon 🌕** using Stable-Baselines3 and share it with the community.
+
+<img src="assets/img/LunarLander.gif" alt="LunarLander"/>
+
+You'll then be able to **compare your agent’s results with other classmates thanks to a leaderboard** 🔥.
+
+This course is **self-paced**, you can start whenever you want.
+
+## Required time ⏱️
+The required time for this unit is, approximately:
+- 2 hours for the theory
+- 1 hour for the hands-on.
+
+## Start this Unit 🚀
+Here are the steps for this Unit:
+
+1️⃣ Sign up to our Discord Server. This is the place where you **can exchange with the community and with us, create study groups to grow each other and more** 
+
+👉🏻 [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Are you new to Discord? Check our **discord 101 to get the best practices** 👉 https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md
+
+2️⃣ **Introduce yourself on Discord in #introduce-yourself Discord channel 🤗 and check on the left the Reinforcement Learning section.**
+
+- In #rl-announcements we give the last information about the course.
+- #discussions is a place to exchange.
+- #unity-ml-agents is to exchange about everything related to this library.
+- #study-groups, to create study groups with your classmates.
+
+<img src="assets/img/discord_channels.jpg" alt="Discord Channels"/>
+
+3️⃣ 📖 **Read An [Introduction to Deep Reinforcement Learning](https://huggingface.co/blog/deep-rl-intro)**, where you’ll learn the foundations of Deep RL. You can also watch the video version attached to the article. 👉 https://huggingface.co/blog/deep-rl-intro
+
+4️⃣ 👩‍💻 Then dive on the hands-on, where **you’ll train your first lander agent 🚀 to land correctly on the Moon 🌕 using Stable-Baselines3 and share it with the community.** Thanks to a leaderboard, **you'll be able to compare your results with other classmates** and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 1 🏆?
+
+The hands-on 👉 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit1/unit1.ipynb)
+
+The leaderboard 👉 https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard
+
+You can work directly **with the colab notebook, which allows you not to have to install everything on your machine (and it’s free)**.
+
+5️⃣ The best way to learn **is to try things on your own**. That’s why we have a challenges section in the colab where we give you some ideas on how you can go further: using another environment, using another model etc.
+
+## Additional readings 📚
+- [Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 1, 2 and 3](http://incompleteideas.net/book/RLbook2020.pdf)
+- [Foundations of Deep RL Series, L1 MDPs, Exact Solution Methods, Max-ent RL by Pieter Abbeel](https://youtu.be/2GwBez0D20A)
+- [Spinning Up RL by OpenAI Part 1: Key concepts of RL](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)
+
+- [Getting Started With OpenAI Gym: The Basic Building Blocks](https://blog.paperspace.com/getting-started-with-openai-gym/)
+
+## How to make the most of this course
+
+To make the most of the course, my advice is to:
+
+- **Participate in Discord** and join a study group.
+- **Read multiple times** the theory part and takes some notes
+- Don’t just do the colab. When you learn something, try to change the environment, change the parameters and read the libraries' documentation. Have fun 🥳
+- Struggling is **a good thing in learning**. It means that you start to build new skills. Deep RL is a complex topic and it takes time to understand. Try different approaches, use our additional readings, and exchange with classmates on discord.
+
+## This is a course built with you 👷🏿‍♀️
+
+We want to improve and update the course iteratively with your feedback. If you have some, please open an issue on the Github Repo: [https://github.com/huggingface/deep-rl-class/issues](https://github.com/huggingface/deep-rl-class/issues)
+
+## Don’t forget to join the Community 📢
+
+We have a discord server where you **can exchange with the community and with us, create study groups to grow each other and more** 
+
+👉🏻 [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Don’t forget to **introduce yourself when you sign up 🤗**
+
+❓If you have other questions, [please check our FAQ](https://github.com/huggingface/deep-rl-class#faq)
+
+Keep learning, stay awesome,

unit2/README.md

@@ -0,0 +1,93 @@
+# Unit 2: Introduction to Q-Learning
+
+In this Unit, we're going to dive deeper into one of the Reinforcement Learning methods: value-based methods and **study our first RL algorithm: Q-Learning**.
+
+We'll also implement our **first RL agent from scratch**: a Q-Learning agent and will train it in two environments:
+
+- Frozen-Lake-v1 ⛄ (non-slippery version): where our agent will need to go from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoiding holes (H).
+- An autonomous taxi 🚕 will need to learn to navigate a city to transport its passengers from point A to point B.
+
+<img src="assets/img/envs.gif" alt="unit 2 environments"/>
+
+You'll then be able to **compare your agent’s results with other classmates thanks to a leaderboard** 🔥.
+
+This Unit is divided into 2 parts:
+- [Part 1](https://huggingface.co/blog/deep-rl-q-part1)
+- [Part 2](https://huggingface.co/blog/deep-rl-q-part2)
+
+<img src="assets/img/two_parts.jpg" alt="Two parts"/>
+
+This course is **self-paced**, you can start whenever you want.
+
+## Required time ⏱️
+The required time for this unit is, approximately:
+- 2-3 hours for the theory
+- 1 hour for the hands-on.
+
+## Start this Unit 🚀
+Here are the steps for this Unit:
+
+1️⃣  If it's not already done, sign up to our Discord Server. This is the place where you **can exchange with the community and with us, create study groups to grow each other and more** 
+
+👉🏻 [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Are you new to Discord? Check our **discord 101 to get the best practices** 👉 https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md
+
+2️⃣ **Introduce yourself on Discord in #introduce-yourself Discord channel 🤗 and check on the left the Reinforcement Learning section.**
+
+- In #rl-announcements we give the last information about the course.
+- #discussions is a place to exchange.
+- #unity-ml-agents is to exchange about everything related to this library.
+- #study-groups, to create study groups with your classmates.
+
+<img src="assets/img/discord_channels.jpg" alt="Discord Channels"/>
+
+3️⃣ 📖 **Read An [Introduction to Q-Learning Part 1](https://huggingface.co/blog/deep-rl-q-part1)**.
+
+4️⃣ 📖 **Read An [Introduction to Q-Learning Part 2](https://huggingface.co/blog/deep-rl-q-part2)**.
+
+5️⃣ 👩‍💻 Then dive on the hands-on, where **you’ll implement our first RL agent from scratch**, a Q-Learning agent, and will train it in two environments:
+1. Frozen Lake v1 ❄️: where our agent will need to **go from the starting state (S) to the goal state (G)** by walking only on frozen tiles (F) and avoiding holes (H).
+2. An autonomous taxi 🚕: where the agent will need **to learn to navigate** a city to **transport its passengers from point A to point B.**
+
+Thanks to a leaderboard, **you'll be able to compare your results with other classmates** and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 2 🏆?
+
+The hands-on 👉 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit2/unit2.ipynb)
+
+The leaderboard 👉 https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard
+
+You can work directly **with the colab notebook, which allows you not to have to install everything on your machine (and it’s free)**.
+
+6️⃣ The best way to learn **is to try things on your own**. That’s why we have a challenges section in the colab where we give you some ideas on how you can go further: using another environment, using another model etc.
+
+## Additional readings 📚
+- [Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 5, 6 and 7](http://incompleteideas.net/book/RLbook2020.pdf)
+- [Foundations of Deep RL Series, L2 Deep Q-Learning by Pieter Abbeel](https://youtu.be/Psrhxy88zww)
+- To divde deeper on Monte Carlo and Temporal Difference Learning:
+  - [Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?](https://stats.stackexchange.com/questions/355820/why-do-temporal-difference-td-methods-have-lower-variance-than-monte-carlo-met)
+  - [When are Monte Carlo methods preferred over temporal difference ones?](https://stats.stackexchange.com/questions/336974/when-are-monte-carlo-methods-preferred-over-temporal-difference-ones)
+ 
+## How to make the most of this course
+
+To make the most of the course, my advice is to:
+
+- **Participate in Discord** and join a study group.
+- **Read multiple times** the theory part and takes some notes
+- Don’t just do the colab. When you learn something, try to change the environment, change the parameters and read the libraries' documentation. Have fun 🥳
+- Struggling is **a good thing in learning**. It means that you start to build new skills. Deep RL is a complex topic and it takes time to understand. Try different approaches, use our additional readings, and exchange with classmates on discord.
+
+## This is a course built with you 👷🏿‍♀️
+
+We want to improve and update the course iteratively with your feedback. If you have some, please open an issue on the Github Repo: [https://github.com/huggingface/deep-rl-class/issues](https://github.com/huggingface/deep-rl-class/issues)
+
+## Don’t forget to join the Community 📢
+
+We have a discord server where you **can exchange with the community and with us, create study groups to grow each other and more** 
+
+👉🏻 [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Don’t forget to **introduce yourself when you sign up 🤗**
+
+❓If you have other questions, [please check our FAQ](https://github.com/huggingface/deep-rl-class#faq)
+
+Keep learning, stay awesome,