diff --git a/DISCORD.Md b/DISCORD.Md
new file mode 100644
index 0000000..bd81db8
--- /dev/null
+++ b/DISCORD.Md
@@ -0,0 +1,33 @@
+## Discord 101
+
+Hey there! My name is Huggy, the dog ๐ถ, and I'm looking forward to train with you during this RL Class! Although I don't know much about bringing sticks (yet), I know one or two things about Discord. So I wrote this guide to help you learn about it!
+
+
+
+Discord is a free chat platform. If you've used Slack, it's quite similar. There is a Hugging Face Community Discord server with 8000 members you can join with a single click [here](http://hf.co/join/discord). ๐ So many humans to play with!
+
+Starting in Discord can be a bit intimidating, so let me take you through it.
+
+* Unlike Slack, you have access to most channels. That means that there might be a bit more noise that what you would like. And I hate loud noise **woof**! ๐ Since channels are gathered by topic, you can mute individual channels or entire categories. For example, the **Audio Lounge** category is about ML for Audio which I don't care about, so I right click the category and then click **Mute**. Ah much better!
+* I still see so many channels **woof**. You can also collapse a category so you don't see channels that are not of interest.
+* So which channels are interesting to me?
+ * They are in the reinforcement learning lounge.
+ * #rl-announcements have announcements about the course.
+ * #discussions is a place where you can talk with fellow humans.
+ * #unity-ml-agents is a place to talk about ML Agents, a library we'll learn during th ecourse.
+ * #study-groups do you want to form a group with fellow humans? This is the place
+
+The HF Community Server has a thriving community of humans interested in many areas, so you can also learn from those. There are paper discussions, events, and many other things. Here are other channels in the community
+ * in #reads people share interesting resources or papers they've found
+ * in #today-im-learning people share what they are learning...today!
+ * in #i-made-this you can show off your amazing projects **woof**.
+
+
+๐ฆด๐Was this useful? There are a couple of tips I can share with you:
+* There are voice channels you can use as well, although most people prefer text chat.
+* You can use markdown style. So if you're writing code, you can use that style. Sadly this does not work as well for links.
+* You can open threads as well!
+
+I hope this is useful! And if you have questions, just ask! See you later! ๐ณ
+
+**Huggy**
diff --git a/README.md b/README.md
index 0f2f378..82b95c1 100644
--- a/README.md
+++ b/README.md
@@ -9,18 +9,22 @@ In this free course, you will:
- ๐ **Participate in challenges** where you will evaluate your agents against other teams.
- ๐๏ธ๐จ **Learn to share your own environments made with Unity and Godot**.
-โก๏ธโก๏ธโก๏ธ Don't forget to sign up here: https://forms.gle/4bbgzs3oVZMjgDed7
+โก๏ธโก๏ธโก๏ธ Don't forget to sign up here: http://eepurl.com/h1pElX
The best way to keep in touch is to **join our discord server to exchange with the community and with us** ๐๐ป https://discord.gg/aYka4Yhff9
+Are you new to Discord? Check our **discord 101 to get the best practices** ๐ https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md
+
And don't forget to share with your friends who want to learn ๐ค!
## The Syllabus ๐๏ธ
+This course is **self-paced** you can start when you want ๐ฅณ.
+
| ๐ Publishing date | ๐ Unit | ๐ฉโ๐ป Hands-on |
|---------------|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
-| May, the 4th | An Introduction to Deep Reinforcement Learning | Train a Deep Reinforcement Learning lander agent to land correctly on the Moon ๐ using Stable-Baselines3 |
-| May, the 11th | Bonus | ๐ it's a suprise ๐ |
-| May, the 18th | Q-Learning | Train an agent to cross a Frozen lake in this new version of the environment. |
+| [Published ๐ฅณ](https://github.com/huggingface/deep-rl-class/tree/main/unit1#unit-1-introduction-to-deep-reinforcement-learning) | [An Introduction to Deep Reinforcement Learning](https://github.com/huggingface/deep-rl-class/tree/main/unit1) | [Train a Deep Reinforcement Learning lander agent to land correctly on the Moon ๐ using Stable-Baselines3](https://github.com/huggingface/deep-rl-class/blob/main/unit1/unit1.ipynb) |
+| May, the 11th | [Bonus](https://discord.com/channels/879548962464493619/968114737655214080/973937495546925056) | |
+| May, the 18th/May, the 20th | [Q-Learning](https://github.com/huggingface/deep-rl-class/blob/main/unit2/README.md) | Train an agent to cross a Frozen lake in this new version of the environment. |
| June, the 1st | Deep Q-Learning and improvements | Train a Deep Q-Learning agent to play Space Invaders |
| | Policy-based methods | ๐๏ธ |
| | Actor-Critic Methods | ๐๏ธ |
@@ -69,4 +73,88 @@ And don't forget to share with your friends who want to learn ๐ค!
- More to come ๐ง
+## Prerequisites
+- Good skills in Python ๐
+- Basics in Deep Learning and Pytorch
+If it's not the case yet, you can check these free resources:
+- Python: https://www.udacity.com/course/introduction-to-python--ud1110
+- Intro to Deep Learning with PyTorch: https://www.udacity.com/course/deep-learning-pytorch--ud188
+- PyTorch in 60min: https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html
+
+## FAQ
+**Is this class free?**
+
+Yes, totally free ๐ฅณ.
+
+
+**Do I need to have a Hugging Face account to follow the course?**
+
+Yes, to push your trained agents during the hands-on, you need an account (it's free) ๐ค.
+
+You can create one here ๐ [https://huggingface.co/join](https://huggingface.co/join)
+
+
+**Whatโs the format of the class?**
+
+The course consists ofย **8 Units.**ย In each of the Units, we'll have:
+
+- **A theory explained part**: an article and a video (based on Deep Reinforcement Learning Course)
+- **A hands-on Google Colab** where you'll learn to use famous Deep RL libraries such as Stable Baselines3, RL Baselines3 Zoo, and RLlib to train your agents in unique environments such as SnowballFight, Huggy the Doggo ๐ถ, and classical ones such as Space Invaders and PyBullet.
+- **Some optional challenges**: train an agent in another environment, and try to beat the results.
+
+It's not a live course video, so you can watch and read each unit when you want ๐ค
+You can check the syllabus here ๐ https://github.com/huggingface/deep-rl-class
+
+
+**What I will do during this course?**
+
+In this free course, you will:
+
+- ๐ Study Deep Reinforcement Learning in theory and practice.
+- ๐งโ๐ป Learn to use famous Deep RL libraries such as Stable Baselines3, RL Baselines3 Zoo, and RLlib.
+- ๐ค Train agents in unique environments such as SnowballFight, Huggy the Doggo ๐ถ, and classical ones such as Space Invaders and PyBullet.
+- ๐พ Publish your trained agents in one line of code to the Hub. But also download powerful agents from the community.
+- ๐ Participate in challenges where you will evaluate your agents against other teams.
+- ๐๏ธ๐จ Learn to share your own environments made with Unity and Godot.
+
+
+**Where do I sign up?**
+
+Here ๐ http://eepurl.com/h1pElX
+
+
+**Where can I find the course?**
+
+**On this repository**, we'll publish every week the links (chapters, hands-ons, videos).
+
+
+**Where can I exchange with my classmates and with you?**
+
+We have a discord server where you **can exchange with the community and with us** ๐๐ป [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9)
+
+Donโt forget to **introduce yourself when you sign up ๐ค**
+
+
+**I have some feedback**
+
+We want to improve and update the course iteratively with your feedback. If you have some, please send a mail to thomas.simonini@huggingface.co
+
+
+**How much background knowledge is needed?**
+
+Some prerequisites:
+
+Good skills in **Python** ๐
+Basics in **Deep Learning and Pytorch**
+
+If it's not the case yet, you can check these free resources:
+
+- Python: [https://www.udacity.com/course/introduction-to-python--ud1110](https://www.udacity.com/course/introduction-to-python--ud1110)
+- Intro to Deep Learning with PyTorch: [https://www.udacity.com/course/deep-learning-pytorch--ud188](https://www.udacity.com/course/deep-learning-pytorch--ud188)
+- PyTorch in 60min: [https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)
+
+
+**Is there a certificate?**
+
+Yes ๐. You'll **need to upload the eight models with the eight hands-on.**
diff --git a/unit1/README.md b/unit1/README.md
new file mode 100644
index 0000000..8c1ddc0
--- /dev/null
+++ b/unit1/README.md
@@ -0,0 +1,76 @@
+# Unit 1: Introduction to Deep Reinforcement Learning
+
+In this Unit, you'll learn the foundations of Deep RL. And **youโll train your first lander agent ๐ to land correctly on the Moon ๐** using Stable-Baselines3 and share it with the community.
+
+
+
+You'll then be able to **compare your agentโs results with other classmates thanks to a leaderboard** ๐ฅ.
+
+This course is **self-paced**, you can start whenever you want.
+
+## Required time โฑ๏ธ
+The required time for this unit is, approximately:
+- 2 hours for the theory
+- 1 hour for the hands-on.
+
+## Start this Unit ๐
+Here are the steps for this Unit:
+
+1๏ธโฃ Sign up to our Discord Server. This is the place where youย **can exchange with the community and with us, create study groups to grow each other and more**ย
+
+๐๐ปย [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Are you new to Discord? Check our **discord 101 to get the best practices** ๐ https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md
+
+2๏ธโฃ **Introduce yourself on Discord in #introduce-yourself Discord channelย ๐ค and check on the left the Reinforcement Learning section.**
+
+- In #rl-announcements we give the last information about the course.
+- #discussions is a place to exchange.
+- #unity-ml-agents is to exchange about everything related to this library.
+- #study-groups, to create study groups with your classmates.
+
+
+
+3๏ธโฃ ๐ **Read An [Introduction to Deep Reinforcement Learning](https://huggingface.co/blog/deep-rl-intro)**, where youโll learn the foundations of Deep RL. You can also watch the video version attached to the article. ๐ https://huggingface.co/blog/deep-rl-intro
+
+4๏ธโฃ ๐ฉโ๐ป Then dive on the hands-on, where **youโll train your first lander agent ๐ to land correctly on the Moon ๐ using Stable-Baselines3 and share it with the community.** Thanks to a leaderboard, **you'll be able to compare your results with other classmates** and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 1 ๐?
+
+The hands-on ๐ [](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit1/unit1.ipynb)
+
+The leaderboard ๐ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard
+
+You can work directly **with the colab notebook, which allows you not to have to install everything on your machine (and itโs free)**.
+
+5๏ธโฃ The best way to learn **is to try things on your own**. Thatโs why we have a challenges section in the colab where we give you some ideas on how you can go further: using another environment, using another model etc.
+
+## Additional readings ๐
+- [Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 1, 2 and 3](http://incompleteideas.net/book/RLbook2020.pdf)
+- [Foundations of Deep RL Series, L1 MDPs, Exact Solution Methods, Max-ent RL by Pieter Abbeel](https://youtu.be/2GwBez0D20A)
+- [Spinning Up RL by OpenAI Part 1: Key concepts of RL](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)
+
+- [Getting Started With OpenAI Gym: The Basic Building Blocks](https://blog.paperspace.com/getting-started-with-openai-gym/)
+
+## How to make the most of this course
+
+To make the most of the course, my advice is to:
+
+- **Participate in Discord** and join a study group.
+- **Read multiple times** the theory part and takes some notes
+- Donโt just do the colab. When you learn something, try to change the environment, change the parameters and read the libraries' documentation. Have fun ๐ฅณ
+- Struggling is **a good thing in learning**. It means that you start to build new skills. Deep RL is a complex topic and it takes time to understand. Try different approaches, use our additional readings, and exchange with classmates on discord.
+
+## This is a course built with you ๐ท๐ฟโโ๏ธ
+
+We want to improve and update the course iteratively with your feedback. If you have some, please open an issue on the Github Repo: [https://github.com/huggingface/deep-rl-class/issues](https://github.com/huggingface/deep-rl-class/issues)
+
+## Donโt forget to join the Community ๐ข
+
+We have a discord server where youย **can exchange with the community and with us, create study groups to grow each other and more**ย
+
+๐๐ปย [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Donโt forget toย **introduce yourself when you sign upย ๐ค**
+
+โIf you have other questions, [please check our FAQ](https://github.com/huggingface/deep-rl-class#faq)
+
+Keep learning, stay awesome,
diff --git a/unit1/assets/img/LunarLander.gif b/unit1/assets/img/LunarLander.gif
new file mode 100644
index 0000000..5caf157
Binary files /dev/null and b/unit1/assets/img/LunarLander.gif differ
diff --git a/unit1/assets/img/discord_channels.jpg b/unit1/assets/img/discord_channels.jpg
new file mode 100644
index 0000000..d76ff68
Binary files /dev/null and b/unit1/assets/img/discord_channels.jpg differ
diff --git a/unit1/unit1.ipynb b/unit1/unit1.ipynb
new file mode 100644
index 0000000..b9ab9f9
--- /dev/null
+++ b/unit1/unit1.ipynb
@@ -0,0 +1,1115 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "njb_ProuHiOe"
+ },
+ "source": [
+ "# Unit 1: Train your first Deep Reinforcement Learning Agent ๐\n",
+ "In this notebook, you'll train your first lander agent to **land correctly on the Moon ๐ share it to the community, and experiment with different configurations**\n",
+ "\n",
+ "โ If you have questions, please post them on #study-group-unit1 discord channel ๐ https://discord.gg/aYka4Yhff9\n",
+ "\n",
+ "\n",
+ "๐ฎ Environment: [LunarLander-v2](https://www.gymlibrary.ml/environments/box2d/lunar_lander/)\n",
+ "\n",
+ "๐ RL-Library: [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/)\n",
+ "\n",
+ "โฌ๏ธ Here is an example of what **you will achieve in just a couple of minutes.** โฌ๏ธ"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 421
+ },
+ "id": "PF46MwbZD00b",
+ "outputId": "8fba2a05-2558-4231-95f3-0840d3d3d108"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ ""
+ ],
+ "text/html": [
+ ""
+ ]
+ },
+ "metadata": {}
+ }
+ ],
+ "source": [
+ "%%html\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4i6tjI2tHQ8j"
+ },
+ "source": [
+ "## Objectives of this notebook ๐\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to use **Gym**, the environment library.\n",
+ "- Be able to use **Stable-Baselines3**, the deep reinforcement learning library.\n",
+ "- Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score ๐ฅ.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ct8eLAabICE-"
+ },
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Class\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6p5HnEefISCB"
+ },
+ "source": [
+ "In this free course, you will:\n",
+ "\n",
+ "- ๐ Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- ๐งโ๐ป Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, and RLlib.\n",
+ "- ๐ค Train **agents in unique environments** \n",
+ "\n",
+ "And more check ๐ the syllabus ๐ https://github.com/huggingface/deep-rl-class\n",
+ "\n",
+ "The best way to keep in touch is to join our discord server to exchange with the community and with us ๐๐ป https://discord.gg/aYka4Yhff9"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Y-mo_6rXIjRi"
+ },
+ "source": [
+ "## Prerequisites ๐๏ธ\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "๐ฒ ๐ [Read the Unit 1 Readme](https://github.com/huggingface/deep-rl-class/blob/main/unit1/README.md) that contains all the information.\n",
+ "\n",
+ "๐ฒ ๐ **The foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by reading first ๐ https://huggingface.co/blog/deep-rl-intro\n",
+ "\n",
+ "๐ฒ ๐ข Sign up to [our Discord Server](https://discord.gg/aYka4Yhff9) and **introduce yourself to #introduce-yourself channel ๐ฅณ**\n",
+ "\n",
+ "๐ฒ ๐ Are you new to Discord? Check our **discord 101 to get the best practices** ๐ https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "aXDZCGHnJ-3h"
+ },
+ "source": [
+ "## A small recap of Deep Reinforcement Learning ๐\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xcQYx9ynaFMD"
+ },
+ "source": [
+ "Let's do a small recap on what we learned in the first Unit:\n",
+ "- Reinforcement Learning is a **computational approach to learning from action**. We build an agent that learns from the environment by **interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.\n",
+ "\n",
+ "- The goal of any RL agent is to **maximize its expected cumulative reward** (also called expected return) because RL is based on the _reward hypothesis_, which is that all goals can be described as the maximization of the expected cumulative reward.\n",
+ "\n",
+ "- The RL process is a **loop that outputs a sequence of state, action, reward, and next state**.\n",
+ "\n",
+ "- To calculate the expected cumulative reward (expected return), **we discount the rewards**: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long-term future reward.\n",
+ "\n",
+ "- To solve an RL problem, you want to **find an optimal policy**; the policy is the \"brain\" of your AI that will tell us what action to take given a state. The optimal one is the one that gives you the actions that max the expected return.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8X2WN2X99BEz"
+ },
+ "source": [
+ "There are **two** ways to find your optimal policy:\n",
+ "- By **training your policy directly**: policy-based methods.\n",
+ "- By **training a value function** that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BPy6dfcm8_WT"
+ },
+ "source": [
+ "- Finally, we spoke about Deep RL because **we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name \"deep.\"**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LzGDDOTSKaF8"
+ },
+ "source": [
+ "# Let's train a Deep Reinforcement Learning lander agent to land correctly on the Moon ๐ and upload it to the Hub.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "YkcELpiYRC5m"
+ },
+ "source": [
+ "### Step 0: Set the GPU ๐ช\n",
+ "- To **faster the agent's training, we'll use a GPU** to do that go to `Runtime > Change Runtime type`\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "38HBd3t1SHJ8"
+ },
+ "source": [
+ "- `Hardware Accelerator > GPU`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "sw0pSDDsR1kq"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install virtual screen libraries and create and run a virtual screen ๐ฅ"
+ ],
+ "metadata": {
+ "id": "LW_lsjc8h_up"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip3 install pyvirtualdisplay\n",
+ "\n",
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "dlx9uWHuhqfh"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jeDAH0h0EBiG"
+ },
+ "source": [
+ "### Step 1: Install dependencies ๐ฝ\n",
+ "The first step is to install the dependencies, weโll install multiple ones:\n",
+ "\n",
+ "- `gym[box2D]`: Contains the LunarLander-v2 environment ๐\n",
+ "- `stable-baselines3[extra]`: The deep reinforcement learning library.\n",
+ "- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face ๐ค Hub.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "9XaULfDZDvrC"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install gym[box2d]\n",
+ "!pip install stable-baselines3[extra]\n",
+ "!pip install huggingface_sb3\n",
+ "!pip install pyglet\n",
+ "!pip install ale-py==0.7.4 # To overcome an issue with gym (https://github.com/DLR-RM/stable-baselines3/issues/875)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "wrgpVFqyENVf"
+ },
+ "source": [
+ "### Step 2: Import the packages ๐ฆ\n",
+ "\n",
+ "One additional library we import is huggingface_hub **to be able to upload and download trained models from the hub**.\n",
+ "\n",
+ "\n",
+ "The Hugging Face Hub ๐ค works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easilly collaborate with others.\n",
+ "\n",
+ "You can see here all the Deep reinforcement Learning models available ๐ https://huggingface.co/models?pipeline_tag=reinforcement-learning&sort=downloads\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "cygWLPGsEQ0m"
+ },
+ "outputs": [],
+ "source": [
+ "import gym\n",
+ "\n",
+ "from huggingface_sb3 import load_from_hub, package_to_hub, push_to_hub\n",
+ "from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
+ "\n",
+ "from stable_baselines3 import PPO\n",
+ "from stable_baselines3.common.evaluation import evaluate_policy\n",
+ "from stable_baselines3.common.env_util import make_vec_env"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 3: Understand what is Gym and how it works? ๐ค\n",
+ "\n",
+ "๐ The library containing our environment is called Gym.\n",
+ "You'll use a lot **Gym in Deep Reinforcement Learning.**\n",
+ "\n",
+ "The Gym library provides two things:\n",
+ "- An interface that allows you to **create RL environments**.\n",
+ "- A **collection of environments** (gym-control, atari, box2D...).\n",
+ "\n",
+ "Let's look at an example, but first let's remember what's the RL Loop.\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "MRqRuRUl8CsB"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "VvCOlJp-_kw4"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "At each step:\n",
+ "- Our Agent receivesย **state S0**ย from theย **Environment**ย โ we receive the first frame of our game (Environment).\n",
+ "- Based on thatย **state S0,**ย the Agent takesย **action A0**ย โ our Agent will move to the right.\n",
+ "- Environment to aย **new**ย **state S1**ย โ new frame.\n",
+ "- The environment gives someย **reward R1**ย to the Agent โ weโre not deadย *(Positive Reward +1)*.\n",
+ "\n",
+ "\n",
+ "With Gym:\n",
+ "\n",
+ "1๏ธโฃ We create our environment using `gym.make()`\n",
+ "\n",
+ "2๏ธโฃ We reset the environment to its initial state with `observation = env.reset()`\n",
+ "\n",
+ "At each step:\n",
+ "\n",
+ "3๏ธโฃ Get an action using our model (in our example we take a random action)\n",
+ "\n",
+ "4๏ธโฃ Using `env.step(action)`, we perform this action in the environment and get\n",
+ "- `obsevation`: The new state (st+1)\n",
+ "- `reward`: The reward we get after executing the action\n",
+ "- `done`: Indicates if the episode terminated\n",
+ "- `info`: A dictionary that provides additional information (depends on the environment).\n",
+ "\n",
+ "If the episode is done:\n",
+ "- We reset the environment to its initial state with `observation = env.reset()`\n",
+ "\n",
+ "**Let's look at an example!** Make sure to read the code\n"
+ ],
+ "metadata": {
+ "id": "-TzNN0bQ_j-3"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import gym\n",
+ "\n",
+ "# First, we create our environment called LunarLander-v2\n",
+ "env = gym.make(\"LunarLander-v2\")\n",
+ "\n",
+ "# Then we reset this environment\n",
+ "observation = env.reset()\n",
+ "\n",
+ "for _ in range(20):\n",
+ " # Take a random action\n",
+ " action = env.action_space.sample()\n",
+ " print(\"Action taken:\", action)\n",
+ "\n",
+ " # Do this action in the environment and get\n",
+ " # next_state, reward, done and info\n",
+ " observation, reward, done, info = env.step(action)\n",
+ " \n",
+ " # If the game is done (in our case we land, crashed or timeout)\n",
+ " if done:\n",
+ " # Reset the environment\n",
+ " print(\"Environment is reset\")\n",
+ " observation = env.reset()"
+ ],
+ "metadata": {
+ "id": "w7vOFlpA_ONz"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "XIrKGGSlENZB"
+ },
+ "source": [
+ "### Step 4: Create the LunarLander environment ๐ and understand how it works\n",
+ "#### [The environment ๐ฎ](https://www.gymlibrary.ml/environments/box2d/lunar_lander/)\n",
+ "In this first tutorial, weโre going to train our agent, a [Lunar Lander](https://www.gymlibrary.ml/environments/box2d/lunar_lander/), **to land correctly on the moon**. To do that, the agent needs to learn **to adapt its speed and position(horizontal, vertical, and angular) to land correctly.**\n",
+ "\n",
+ "\n",
+ "---\n",
+ "\n",
+ "\n",
+ "๐ก A good habit when you start to use an environment is to check its documentation \n",
+ "\n",
+ "๐ https://www.gymlibrary.ml/environments/box2d/lunar_lander/\n",
+ "\n",
+ "---\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "poLBgRocF9aT"
+ },
+ "source": [
+ "Let's see what the Environment looks like:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "ZNPG0g_UGCfh",
+ "outputId": "265ae44f-d404-43d8-9d62-378c3fa61f35"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "_____OBSERVATION SPACE_____ \n",
+ "\n",
+ "Observation Space Shape (8,)\n",
+ "Sample observation [ 0.01816158 -0.56312186 0.08143468 1.8260419 0.51859415 -1.0216727\n",
+ " 0.703369 0.5369312 ]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# We create our environment with gym.make(\"\")\n",
+ "env = gym.make(\"LunarLander-v2\")\n",
+ "env.reset()\n",
+ "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
+ "print(\"Observation Space Shape\", env.observation_space.shape)\n",
+ "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2MXc15qFE0M9"
+ },
+ "source": [
+ "We see with `Observation Space Shape (8,)` that the observation is a vector of size 8, each value is a different information about the lander:\n",
+ "- Horizontal pad coordinate (x)\n",
+ "- Vertical pad coordinate (y)\n",
+ "- Horizontal speed (x)\n",
+ "- Vertical speed (y)\n",
+ "- Angle\n",
+ "- Angular speed\n",
+ "- If the left leg has contact point touched the land\n",
+ "- If the right leg has contact point touched the land\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "We5WqOBGLoSm",
+ "outputId": "c3bab680-a4fc-4e30-eea5-9f8391c2f825"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\n",
+ " _____ACTION SPACE_____ \n",
+ "\n",
+ "Action Space Shape 4\n",
+ "Action Space Sample 0\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
+ "print(\"Action Space Shape\", env.action_space.n)\n",
+ "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MyxXwkI2Magx"
+ },
+ "source": [
+ "The action space (the set of possible actions the agent can take) is discrete with 4 actions available ๐ฎ: \n",
+ "\n",
+ "- Do nothing,\n",
+ "- Fire left orientation engine,\n",
+ "- Fire the main engine,\n",
+ "- Fire right orientation engine.\n",
+ "\n",
+ "Reward function (the function that will gives a reward at each timestep) ๐ฐ:\n",
+ "\n",
+ "- Moving from the top of the screen to the landing pad and zero speed is about 100~140 points.\n",
+ "- Firing main engine is -0.3 each frame\n",
+ "- Each leg ground contact is +10 points\n",
+ "- Episode finishes if the lander crashes (additional - 100 points) or come to rest (+100 points)\n",
+ "- The game is solved if your agent does 200 points."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "dFD9RAFjG8aq"
+ },
+ "source": [
+ "#### Vectorized Environment\n",
+ "- We create a vectorized environment (method for stacking multiple independent environments into a single environment) of 16 environments, this way, **we'll have more diverse experiences during the training.**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "99hqQ_etEy1N"
+ },
+ "outputs": [],
+ "source": [
+ "# Create the environment\n",
+ "env = make_vec_env('LunarLander-v2', n_envs=16)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VgrE86r5E5IK"
+ },
+ "source": [
+ "### Step 5: Create the Model ๐ค\n",
+ "- Now that we studied our environment and we understood the problem: **being able to land correctly the Lunar Lander to the Landing Pad by controlling left, right and main orientation engine**. Let's build the algorithm we're going to use to solve this Problem ๐.\n",
+ "\n",
+ "- To do so, we're going to use our first Deep RL library, [Stable Baselines3 (SB3)](https://stable-baselines3.readthedocs.io/en/master/).\n",
+ "\n",
+ "- SB3 is a set of **reliable implementations of reinforcement learning algorithms in PyTorch**.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "๐ก A good habit when using a new library is to dive first on the documentation: https://stable-baselines3.readthedocs.io/en/master/ and then try some tutorials.\n",
+ "\n",
+ "----"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "3wgg-y0-92Wj"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "HV4yiUM_9_Ka"
+ },
+ "source": [
+ "To solve this problem, we're going to use SB3 **PPO**. [PPO (aka Proximal Policy Optimization) is one of the of the SOTA (state of the art) Deep Reinforcement Learning algorithms that you'll study during this course](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#example%5D).\n",
+ "\n",
+ "PPO is a combination of:\n",
+ "- *Value-based reinforcement learning method*: learning an action-value function that will tell us what's the **most valuable action to take given a state and action**.\n",
+ "- *Policy-based reinforcement learning method*: learning a policy that will **gives us a probability distribution over actions**.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5qL_4HeIOrEJ"
+ },
+ "source": [
+ "Stable-Baselines3 is easy to set up:\n",
+ "\n",
+ "1๏ธโฃ You **create your environment** (in our case it was done above)\n",
+ "\n",
+ "2๏ธโฃ You define the **model you want to use and instantiate this model** `model = PPO(\"MlpPolicy\")`\n",
+ "\n",
+ "3๏ธโฃ You **train the agent** with `model.learn` and define the number of training timesteps\n",
+ "\n",
+ "```\n",
+ "# Create environment\n",
+ "env = gym.make('LunarLander-v2')\n",
+ "\n",
+ "# Instantiate the agent\n",
+ "model = PPO('MlpPolicy', env, verbose=1)\n",
+ "# Train the agent\n",
+ "model.learn(total_timesteps=int(2e5))\n",
+ "```\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "nxI6hT1GE4-A"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO: Define a PPO MlpPolicy architecture\n",
+ "# We use MultiLayerPerceptron (MLPPolicy) because the input is a vector,\n",
+ "# if we had frames as input we would use CnnPolicy\n",
+ "model = "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "QAN7B0_HCVZC"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "543OHYDfcjK4",
+ "outputId": "9babd3e1-0a6d-4d86-da35-f989055e8d82"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Using cuda device\n"
+ ]
+ }
+ ],
+ "source": [
+ "# SOLUTION\n",
+ "# We added some parameters to fasten the training\n",
+ "model = PPO(\n",
+ " policy = 'MlpPolicy',\n",
+ " env = env,\n",
+ " n_steps = 1024,\n",
+ " batch_size = 64,\n",
+ " n_epochs = 4,\n",
+ " gamma = 0.999,\n",
+ " gae_lambda = 0.98,\n",
+ " ent_coef = 0.01,\n",
+ " verbose=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ClJJk88yoBUi"
+ },
+ "source": [
+ "### Step 6: Train the PPO agent ๐\n",
+ "- Let's train our agent for 500,000 timesteps, don't forget to use GPU on Colab. It will take approximately ~10min, but you can use less timesteps if you just want to try it out.\n",
+ "- During the training, take a โ break you deserved it ๐ค"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "qKnYkNiVp89p"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO: Train it for 500,000 timesteps\n",
+ "\n",
+ "# TODO: Specify file name for model and save the model to file\n",
+ "model_name = \"\"\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1bQzQ-QcE3zo"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "poBCy9u_csyR"
+ },
+ "outputs": [],
+ "source": [
+ "# SOLUTION\n",
+ "# Train it for 500,000 timesteps\n",
+ "model.learn(total_timesteps=500000)\n",
+ "# Save the model\n",
+ "model_name = \"ppo-LunarLander-v2\"\n",
+ "model.save(model_name)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BY_HuedOoISR"
+ },
+ "source": [
+ "### Step 7: Evaluate the agent ๐\n",
+ "- Now that's our Lunar Lander agent is trained ๐, we need to **check its performance**.\n",
+ "- Stable-Baselines3 provides a method to do that `evaluate_policy`.\n",
+ "- To fill that part you need to [check the documentation](https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#basic-usage-training-saving-loading)\n",
+ "- In the next step, we'll see **how to automatically evaluate and share your agent to compete in a leaderboard, but for now let's do it ourselves**\n",
+ "\n",
+ "\n",
+ "๐ก When you evaluate your agent, you should not use your training environment but create an evaluation environment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "yRpno0glsADy"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO: Evaluate the agent\n",
+ "# Create a new environment for evaluation\n",
+ "eval_env =\n",
+ "\n",
+ "# Evaluate the model with 10 evaluation episodes and deterministic=True\n",
+ "mean_reward, std_reward = \n",
+ "\n",
+ "# Print the results\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BqPKw3jt_pG5"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "zpz8kHlt_a_m"
+ },
+ "outputs": [],
+ "source": [
+ "#@title\n",
+ "eval_env = gym.make(\"LunarLander-v2\")\n",
+ "mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)\n",
+ "print(f\"mean_reward={mean_reward:.2f} +/- {std_reward}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "reBhoODwcXfr"
+ },
+ "source": [
+ "- In my case, I got a mean reward is `200.20 +/- 20.80` after training for 1 million steps, which means that our lunar lander agent is ready to land on the moon ๐๐ฅณ."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IK_kR78NoNb2"
+ },
+ "source": [
+ "### Step 8: Publish our trained model on the Hub ๐ฅ\n",
+ "Now that we saw we got good results after the training, we can publish our trained model on the hub ๐ค with one line of code.\n",
+ "\n",
+ "๐ The libraries documentation ๐ https://github.com/huggingface/huggingface_sb3/tree/main#hugging-face--x-stable-baselines3-v20\n",
+ "\n",
+ "Here's an example of a Model Card (with Space Invaders):"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Nhded6x9gapA"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Gs-Ew7e1gXN3"
+ },
+ "source": [
+ "By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
+ "\n",
+ "---\n",
+ "The package_to_hub function\n",
+ "---\n",
+ "\n",
+ "This way:\n",
+ "- You can **showcase our work** ๐ฅ\n",
+ "- You can **visualize your agent playing** ๐\n",
+ "- You can **share with the community an agent that others can use** ๐พ\n",
+ "- You can **access a leaderboard ๐ to see how well your agent is performing compared to your classmates** ๐ https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "JquRrWytA6eo"
+ },
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1๏ธโฃ (If it's not already done) create an account to HF โก https://huggingface.co/join\n",
+ "\n",
+ "2๏ธโฃ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens) **with write role**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "RVJYd-YZRXRZ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "- Copy the token \n",
+ "- Run the cell below and past the token"
+ ],
+ "metadata": {
+ "id": "YWxcm2xiRSgA"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GZiFBBlzxzxY"
+ },
+ "outputs": [],
+ "source": [
+ "notebook_login()\n",
+ "!git config --global credential.helper store"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_tsf2uv0g_4p"
+ },
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FGNh9VsZok0i"
+ },
+ "source": [
+ "3๏ธโฃ We're now ready to push our trained agent to the ๐ค Hub ๐ฅ using `package_to_hub()` function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Ay24l6bqFF18"
+ },
+ "source": [
+ "Let's fill the `package_to_hub` function:\n",
+ "- `model`: our trained model.\n",
+ "- `model_name`: the name of the trained model that we defined in `model_save`\n",
+ "- `model_architecture`: the model architecture we used: in our case PPO\n",
+ "- `env_id`: the name of the environment, in our case `LunarLander-v2`\n",
+ "- `eval_env`: the evaluation environment defined in eval_env\n",
+ "- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `(repo_id = {username}/{repo_name})`\n",
+ "\n",
+ "๐ก **A good name is {username}/{model_architecture}-{env_id}**\n",
+ "\n",
+ "- `commit_message`: message of the commit"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import gym\n",
+ "from stable_baselines3.common.vec_env import DummyVecEnv\n",
+ "from stable_baselines3.common.env_util import make_vec_env\n",
+ "\n",
+ "from huggingface_sb3 import package_to_hub\n",
+ "\n",
+ "# Create the evaluation env\n",
+ "eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
+ "\n",
+ "# TODO: Define the name of the environment\n",
+ "env_id = \n",
+ "\n",
+ "# TODO: Define the model architecture we used\n",
+ "model_architecture = \"\"\n",
+ "\n",
+ "## TODO: Define a repo_id\n",
+ "## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ "repo_id = \n",
+ "\n",
+ "## TODO: Define the commit message\n",
+ "commit_message = \"\"\n",
+ "\n",
+ "# method save, evaluate, generate a model card and record a replay video of your agent before pushing the repo to the hub\n",
+ "package_to_hub(model=model, # Our trained model\n",
+ " model_name=model_name, # The name of our trained model \n",
+ " model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
+ " env_id=env_id, # Name of the environment\n",
+ " eval_env=eval_env, # Evaluation Environment\n",
+ " repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ " commit_message=commit_message)\n"
+ ],
+ "metadata": {
+ "id": "JPG7ofdGIHN8"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Avf6gufJBGMw"
+ },
+ "source": [
+ "#### Solution\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "xMkkkukIBQJM"
+ },
+ "outputs": [],
+ "source": [
+ "import gym\n",
+ "\n",
+ "from stable_baselines3 import PPO\n",
+ "from stable_baselines3.common.vec_env import DummyVecEnv\n",
+ "from stable_baselines3.common.env_util import make_vec_env\n",
+ "\n",
+ "from huggingface_sb3 import package_to_hub\n",
+ "\n",
+ "# PLACE the variables you've just defined two cells above\n",
+ "# Define the name of the environment\n",
+ "env_id = \"LunarLander-v2\"\n",
+ "\n",
+ "# TODO: Define the model architecture we used\n",
+ "model_architecture = \"PPO\"\n",
+ "\n",
+ "## Define a repo_id\n",
+ "## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ "## CHANGE WITH YOUR REPO ID\n",
+ "repo_id = \"ThomasSimonini/TEST2ppo-LunarLander-v2\"\n",
+ "\n",
+ "## Define the commit message\n",
+ "commit_message = \"Upload PPO LunarLander-v2 trained agent\"\n",
+ "\n",
+ "# Create the evaluation env\n",
+ "eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
+ "\n",
+ "# PLACE the package_to_hub function you've just filled here\n",
+ "package_to_hub(model=model, # Our trained model\n",
+ " model_name=model_name, # The name of our trained model \n",
+ " model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
+ " env_id=env_id, # Name of the environment\n",
+ " eval_env=eval_env, # Evaluation Environment\n",
+ " repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ " commit_message=commit_message)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Congrats ๐ฅณ you've just trained and uploaded your first Deep Reinforcement Learning agent. The script above should have displayed a link to a model repository such as https://huggingface.co/osanseviero/test_sb3. When you go to this link, you can:\n",
+ "* see a video preview of your agent at the right. \n",
+ "* click \"Files and versions\" to see all the files in the repository.\n",
+ "* click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
+ "* a model card (`README.md` file) which gives a description of the model\n",
+ "\n",
+ "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
+ "\n",
+ "Compare the results of your LunarLander-v2 with your classmates using the leaderboard ๐ ๐ https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard"
+ ],
+ "metadata": {
+ "id": "pN0-W6kkKn44"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BQAwLnYFPk-s"
+ },
+ "source": [
+ "## Some additional challenges ๐\n",
+ "The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
+ "\n",
+ "In the [Leaderboard](https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard) you will find your agents. Can you get to the top?\n",
+ "\n",
+ "Here are some ideas to achieve so:\n",
+ "* Train more steps\n",
+ "* Try different hyperparameters of `PPO`. You can see them at https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#parameters. \n",
+ "* Check the [Stable-Baselines3 documentation](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) and try another models such as DQN.\n",
+ "* **Push your new trained model** on the Hub ๐ฅ\n",
+ "\n",
+ "**Compare the results of your LunarLander-v2 with your classmates** using the [leaderboard](https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard) ๐ ๐ https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard\n",
+ "\n",
+ "Is moon landing too boring to you? Try to **change the environment**, why not using CartPole-v1, MountainCar-v0 or CarRacing-v0? Check how they works [using the gym documentation](https://www.gymlibrary.ml/) and have fun ๐."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9lM95-dvmif8"
+ },
+ "source": [
+ "________________________________________________________________________\n",
+ "Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.**\n",
+ "\n",
+ "If youโre still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
+ "\n",
+ "Take time to really **grasp the material before continuing and try the additional challenges**. Itโs important to master these elements and having a solid foundations.\n",
+ "\n",
+ "Naturally, during the course, weโre going to use and deeper explain again these terms but **itโs better to have a good understanding of them now before diving into the next chapters.**\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "feR90OUSEXq9"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "### This is a course built with you ๐ท๐ฟโโ๏ธ\n",
+ "\n",
+ "We want to improve and update the course iteratively with your feedback. If you have some, please open an issue on the Github Repo: [https://github.com/huggingface/deep-rl-class/issues](https://github.com/huggingface/deep-rl-class/issues)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BjLhT70TEZIn"
+ },
+ "source": [
+ "See you on Unit 2! ๐ฅ\n",
+ "## Keep learning, stay awesome!"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "collapsed_sections": [
+ "QAN7B0_HCVZC",
+ "1bQzQ-QcE3zo",
+ "BqPKw3jt_pG5",
+ "Avf6gufJBGMw"
+ ],
+ "name": "Unit 1: Train your first Deep Reinforcement Learning Agent ๐.ipynb",
+ "provenance": [],
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/unit2/README.md b/unit2/README.md
new file mode 100644
index 0000000..2a3cde9
--- /dev/null
+++ b/unit2/README.md
@@ -0,0 +1,93 @@
+# Unit 2: Introduction to Q-Learning
+
+In this Unit, we're going to dive deeper into one of the Reinforcement Learning methods: value-based methods and **study our first RL algorithm: Q-Learning**.
+
+We'll also implement our **first RL agent from scratch**: a Q-Learning agent and will train it in two environments:
+
+- Frozen-Lake-v1 โ (non-slippery version): where our agent will need to go from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoiding holes (H).
+- An autonomous taxi ๐ will need to learn to navigate a city to transport its passengers from point A to point B.
+
+
+
+You'll then be able to **compare your agentโs results with other classmates thanks to a leaderboard** ๐ฅ.
+
+This Unit is divided into 2 parts:
+- [Part 1](https://huggingface.co/blog/deep-rl-q-part1)
+- [Part 2](https://huggingface.co/blog/deep-rl-q-part2)
+
+
+
+This course is **self-paced**, you can start whenever you want.
+
+## Required time โฑ๏ธ
+The required time for this unit is, approximately:
+- 2-3 hours for the theory
+- 1 hour for the hands-on.
+
+## Start this Unit ๐
+Here are the steps for this Unit:
+
+1๏ธโฃ If it's not already done, sign up to our Discord Server. This is the place where youย **can exchange with the community and with us, create study groups to grow each other and more**ย
+
+๐๐ปย [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Are you new to Discord? Check our **discord 101 to get the best practices** ๐ https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md
+
+2๏ธโฃ **Introduce yourself on Discord in #introduce-yourself Discord channelย ๐ค and check on the left the Reinforcement Learning section.**
+
+- In #rl-announcements we give the last information about the course.
+- #discussions is a place to exchange.
+- #unity-ml-agents is to exchange about everything related to this library.
+- #study-groups, to create study groups with your classmates.
+
+
+
+3๏ธโฃ ๐ **Read An [Introduction to Q-Learning Part 1](https://huggingface.co/blog/deep-rl-q-part1)**.
+
+4๏ธโฃ ๐ **Read An [Introduction to Q-Learning Part 2](https://huggingface.co/blog/deep-rl-q-part2)**.
+
+5๏ธโฃ ๐ฉโ๐ป Then dive on the hands-on, where **youโll implement our first RL agent from scratch**, a Q-Learning agent, and will train it in two environments:
+1. Frozen Lake v1 โ๏ธ: where our agent will need toย **go from the starting state (S) to the goal state (G)**ย by walking only on frozen tiles (F) and avoiding holes (H).
+2. An autonomous taxi ๐: where the agent will needย **to learn to navigate**ย a city toย **transport its passengers from point A to point B.**
+
+Thanks to a leaderboard, **you'll be able to compare your results with other classmates** and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 2 ๐?
+
+The hands-on ๐ [](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit2/unit2.ipynb)
+
+The leaderboard ๐ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard
+
+You can work directly **with the colab notebook, which allows you not to have to install everything on your machine (and itโs free)**.
+
+6๏ธโฃ The best way to learn **is to try things on your own**. Thatโs why we have a challenges section in the colab where we give you some ideas on how you can go further: using another environment, using another model etc.
+
+## Additional readings ๐
+- [Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 5, 6 and 7](http://incompleteideas.net/book/RLbook2020.pdf)
+- [Foundations of Deep RL Series, L2 Deep Q-Learning by Pieter Abbeel](https://youtu.be/Psrhxy88zww)
+- To divde deeper on Monte Carlo and Temporal Difference Learning:
+ - [Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?](https://stats.stackexchange.com/questions/355820/why-do-temporal-difference-td-methods-have-lower-variance-than-monte-carlo-met)
+ - [When are Monte Carlo methods preferred over temporal difference ones?](https://stats.stackexchange.com/questions/336974/when-are-monte-carlo-methods-preferred-over-temporal-difference-ones)
+
+## How to make the most of this course
+
+To make the most of the course, my advice is to:
+
+- **Participate in Discord** and join a study group.
+- **Read multiple times** the theory part and takes some notes
+- Donโt just do the colab. When you learn something, try to change the environment, change the parameters and read the libraries' documentation. Have fun ๐ฅณ
+- Struggling is **a good thing in learning**. It means that you start to build new skills. Deep RL is a complex topic and it takes time to understand. Try different approaches, use our additional readings, and exchange with classmates on discord.
+
+## This is a course built with you ๐ท๐ฟโโ๏ธ
+
+We want to improve and update the course iteratively with your feedback. If you have some, please open an issue on the Github Repo: [https://github.com/huggingface/deep-rl-class/issues](https://github.com/huggingface/deep-rl-class/issues)
+
+## Donโt forget to join the Community ๐ข
+
+We have a discord server where youย **can exchange with the community and with us, create study groups to grow each other and more**ย
+
+๐๐ปย [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Donโt forget toย **introduce yourself when you sign upย ๐ค**
+
+โIf you have other questions, [please check our FAQ](https://github.com/huggingface/deep-rl-class#faq)
+
+Keep learning, stay awesome,
diff --git a/unit2/assets/img/discord_channels.jpg b/unit2/assets/img/discord_channels.jpg
new file mode 100644
index 0000000..d76ff68
Binary files /dev/null and b/unit2/assets/img/discord_channels.jpg differ
diff --git a/unit2/assets/img/envs.gif b/unit2/assets/img/envs.gif
new file mode 100644
index 0000000..9083bb2
Binary files /dev/null and b/unit2/assets/img/envs.gif differ
diff --git a/unit2/assets/img/two_parts.jpg b/unit2/assets/img/two_parts.jpg
new file mode 100644
index 0000000..0445379
Binary files /dev/null and b/unit2/assets/img/two_parts.jpg differ
diff --git a/unit2/unit2.ipynb b/unit2/unit2.ipynb
new file mode 100644
index 0000000..766b190
--- /dev/null
+++ b/unit2/unit2.ipynb
@@ -0,0 +1,1783 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "njb_ProuHiOe"
+ },
+ "source": [
+ "# Unit 2: Q-Learning with FrozenLake-v1 โ and Taxi-v3 ๐\n",
+ "\n",
+ "In this notebook, **you'll code from scratch your first Reinforcement Learning agent** playing FrozenLake โ๏ธ using Q-Learning, share it to the community, and experiment with different configurations\n",
+ "\n",
+ "โ If you have questions, please post them on #study-group-unit2 discord channel ๐ https://discord.gg/aYka4Yhff9\n",
+ "\n",
+ "๐ฎ Environments: \n",
+ "- [FrozenLake-v1](https://www.gymlibrary.ml/environments/toy_text/frozen_lake/)\n",
+ "- [Taxi-v3](https://www.gymlibrary.ml/environments/toy_text/taxi/)\n",
+ "\n",
+ "๐ RL-Library: Python and Numpy\n",
+ "\n",
+ "โฌ๏ธ Here is an example of what **you will achieve in just a couple of minutes.** โฌ๏ธ"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "vRU_vXBrl1Jx"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4i6tjI2tHQ8j"
+ },
+ "source": [
+ "## Objectives of this notebook ๐\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to use **Gym**, the environment library.\n",
+ "- Be able to code from scratch a Q-Learning agent.\n",
+ "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score ๐ฅ.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ct8eLAabICE-"
+ },
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Class\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6p5HnEefISCB"
+ },
+ "source": [
+ "In this free course, you will:\n",
+ "\n",
+ "- ๐ Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- ๐งโ๐ป Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, and RLlib.\n",
+ "- ๐ค Train **agents in unique environments** \n",
+ "\n",
+ "And more check ๐ the syllabus ๐ https://github.com/huggingface/deep-rl-class\n",
+ "\n",
+ "The best way to keep in touch is to join our discord server to exchange with the community and with us ๐๐ป https://discord.gg/aYka4Yhff9"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Y-mo_6rXIjRi"
+ },
+ "source": [
+ "## Prerequisites ๐๏ธ\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "๐ฒ ๐ [Read the Unit 2 Readme](https://github.com/huggingface/deep-rl-class/blob/main/unit2/README.md) that contains all the information.\n",
+ "\n",
+ "๐ฒ ๐ [Read **An Introduction to Q-Learning Part 1**](https://huggingface.co/blog/deep-rl-q-part1) \n",
+ "\n",
+ "๐ฒ ๐ [Read **An Introduction to Q-Learning Part 2**](https://huggingface.co/blog/deep-rl-q-part2) \n",
+ "\n",
+ "๐ฒ ๐ข Sign up to [our Discord Server](https://discord.gg/aYka4Yhff9) and **introduce yourself to #introduce-yourself channel ๐ฅณ**\n",
+ "\n",
+ "๐ฒ ๐ Are you new to Discord? Check our **discord 101 to get the best practices** ๐ https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## A small recap of Q-Learning"
+ ],
+ "metadata": {
+ "id": "f2ONOODsyrMU"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "- The *Q-Learning* **is the RL algorithm that** \n",
+ " - Trains *Q-Function*, an **action-value function** that contains, as internal memory, a *Q-table* **that contains all the state-action pair values.**\n",
+ " \n",
+ " - Given a state and action, our Q-Function **will search into its Q-table the corresponding value.**\n",
+ " \n",
+ "\n",
+ "\n",
+ "\n",
+ "- When the training is done,**we have an optimal Q-Function, so an optimal Q-Table.**\n",
+ " \n",
+ "- And if we **have an optimal Q-function**, we\n",
+ "have an optimal policy,since we **know for each state, what is the best action to take.**\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "But, in the beginning,ย our **Q-Table is useless since it gives arbitrary value for each state-action pairย (most of the time we initialize the Q-Table to 0 values)**. But, as weโllย explore the environment and update our Q-Table it will give us better and better approximations\n",
+ "\n",
+ "\n",
+ "\n",
+ "This is the Q-Learning pseudocode:\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "V68VveLacfxJ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 0: Setup a Virtual Display ๐ป\n",
+ "\n",
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install virtual screen libraries and create and run a virtual screen ๐ฅ"
+ ],
+ "metadata": {
+ "id": "7sUukH74yQ7K"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%%capture\n",
+ "!pip install pyglet==1.5.1 \n",
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip3 install pyvirtualdisplay\n",
+ "\n",
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "f2wrVUvrySkW"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 1: Install dependencies ๐ฝ\n",
+ "The first step is to install the dependencies, weโll install multiple ones:\n",
+ "\n",
+ "- `gym`: Contains the FrozenLake-v1 โ and Taxi-v3 ๐ environments.\n",
+ "- `pygame`: Used for the FrozenLake-v1 and Taxi-v3 UI.\n",
+ "- `numPy`: Used for handling our Q-table.\n",
+ "\n",
+ "The Hugging Face Hub ๐ค works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easilly collaborate with others.\n",
+ "\n",
+ "You can see here all the Deep reinforcement Learning models available ๐ https://huggingface.co/models?other=q-learning\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "mIkOB4Rpw65k"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%%capture\n",
+ "!pip install git+https://github.com/openai/gym.git@a37b956d57bdb024ccd8fef5446ba159e32d4626 # We install gym using git since Taxi-v3 \"rgb_array version\" is not on PyPi release\n",
+ "!pip install pygame\n",
+ "!pip install numpy\n",
+ "\n",
+ "!pip install huggingface_hub\n",
+ "!pip install pickle5\n",
+ "!pip install pyyaml==6.0 # avoid key error metadata\n",
+ "!pip install imageio imageio_ffmpeg"
+ ],
+ "metadata": {
+ "id": "1Ac7wW_5ClJC"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 2: Import the packages ๐ฆ\n",
+ "\n",
+ "In addition to the installed libraries, we also use:\n",
+ "\n",
+ "- `random`: To generate random numbers (that will be useful for Epsilon-Greedy Policy).\n",
+ "- `imageio`: To generate a replay video\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "W-7f-Swax_9x"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import numpy as np\n",
+ "import gym\n",
+ "import random\n",
+ "import imageio\n",
+ "import os\n",
+ "\n",
+ "import pickle5 as pickle"
+ ],
+ "metadata": {
+ "id": "VcNvOAQlysBJ"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We're now ready to code our Q-Learning algorithm ๐ฅ"
+ ],
+ "metadata": {
+ "id": "xp4-bXKIy1mQ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Part 1: Frozen Lake โ (non slippery version)"
+ ],
+ "metadata": {
+ "id": "xya49aNJWVvv"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 1: Create and understand [FrozenLake environment โ]((https://www.gymlibrary.ml/environments/toy_text/frozen_lake/)\n",
+ "---\n",
+ "\n",
+ "๐ก A good habit when you start to use an environment is to check its documentation \n",
+ "\n",
+ "๐ https://www.gymlibrary.ml/environments/toy_text/frozen_lake/\n",
+ "\n",
+ "---\n",
+ "\n",
+ "We're going to train our Q-Learning agent **to navigate from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoid holes (H)**.\n",
+ "\n",
+ "We can have two sizes of environment:\n",
+ "- `map_name=\"4x4\"`: a 4x4 grid version\n",
+ "- `map_name=\"8x8\"`: a 8x8 grid version\n",
+ "\n",
+ "\n",
+ "The environment has two modes:\n",
+ "- `is_slippery=False`: The agent always move in the intended direction due to the non-slippery nature of the frozen lake.\n",
+ "- `is_slippery=True`: The agent may not always move in the intended direction due to the slippery nature of the frozen lake (stochastic)."
+ ],
+ "metadata": {
+ "id": "NAvihuHdy9tw"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "For now let's keep it simple with the 4x4 map and non-slippery"
+ ],
+ "metadata": {
+ "id": "UaW_LHfS0PY2"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Create the FrozenLake-v1 environment using 4x4 map and non-slippery version\n",
+ "env = gym.make()"
+ ],
+ "metadata": {
+ "id": "IzJnb8O3y8up"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Solution"
+ ],
+ "metadata": {
+ "id": "Ji_UrI5l2zzn"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "env = gym.make(\"FrozenLake-v1\", map_name=\"4x4\", is_slippery=False)"
+ ],
+ "metadata": {
+ "id": "jNxUbPMP0akP"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "You can also custom your own grid using:\n",
+ "\n",
+ "```python\n",
+ "desc=[\"SFFF\", \"FHFH\", \"FFFH\", \"HFFG\"]\n",
+ "gym.make('FrozenLake-v1', desc=desc, is_slippery=True)\n",
+ "```\n",
+ "\n",
+ "but we'll use the default environment for now."
+ ],
+ "metadata": {
+ "id": "KASNViqL4tZn"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Let's see what the Environment looks like:\n"
+ ],
+ "metadata": {
+ "id": "SXbTfdeJ1Xi9"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZNPG0g_UGCfh"
+ },
+ "outputs": [],
+ "source": [
+ "# We create our environment with gym.make(\"\")\n",
+ "env.reset()\n",
+ "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
+ "print(\"Observation Space\", env.observation_space)\n",
+ "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2MXc15qFE0M9"
+ },
+ "source": [
+ "We see with `Observation Space Shape Discrete(16)` that the observation is a value representing the **agentโs current position as current_row * nrows + current_col (where both the row and col start at 0)**. \n",
+ "\n",
+ "For example, the goal position in the 4x4 map can be calculated as follows: 3 * 4 + 3 = 15. The number of possible observations is dependent on the size of the map. **For example, the 4x4 map has 16 possible observations.**\n",
+ "\n",
+ "\n",
+ "For instance, at this state = 0\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "nW-hW7n22PI_"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "We5WqOBGLoSm"
+ },
+ "outputs": [],
+ "source": [
+ "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
+ "print(\"Action Space Shape\", env.action_space.n)\n",
+ "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MyxXwkI2Magx"
+ },
+ "source": [
+ "The action space (the set of possible actions the agent can take) is discrete with 4 actions available ๐ฎ:\n",
+ "- 0: GO LEFT\n",
+ "- 1: GO DOWN\n",
+ "- 2: GO RIGHT\n",
+ "- 3: GO UP\n",
+ "\n",
+ "Reward function ๐ฐ:\n",
+ "- Reach goal: +1\n",
+ "- Reach hole: 0\n",
+ "- Reach frozen: 0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 2: Create and Initialize the Q-table ๐๏ธ\n",
+ "(๐ Step 1 of the pseudocode)\n",
+ "\n",
+ "\n",
+ "\n",
+ "It's time to initialize our Q-table! To know how many rows (states) and columns (actions) to use, we need to know the action and observation space. OpenAI Gym provides us a way to do that: `env.action_space.n` and `env.observation_space.n`\n"
+ ],
+ "metadata": {
+ "id": "1pFhWblk3Awr"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "state_space = \n",
+ "print(\"There are \", state_space, \" possible states\")\n",
+ "\n",
+ "action_space = \n",
+ "print(\"There are \", action_space, \" possible actions\")"
+ ],
+ "metadata": {
+ "id": "y3ZCdluj3k0l"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros\n",
+ "def initialize_q_table(state_space, action_space):\n",
+ " Qtable = \n",
+ " return Qtable"
+ ],
+ "metadata": {
+ "id": "rCddoOXM3UQH"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "Qtable_frozenlake = initialize_q_table(state_space, action_space)"
+ ],
+ "metadata": {
+ "id": "9YfvrqRt3jdR"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Solution"
+ ],
+ "metadata": {
+ "id": "67OdoKL63eDD"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "state_space = env.observation_space.n\n",
+ "print(\"There are \", state_space, \" possible states\")\n",
+ "\n",
+ "action_space = env.action_space.n\n",
+ "print(\"There are \", action_space, \" possible actions\")"
+ ],
+ "metadata": {
+ "id": "HuTKv3th3ohG"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros\n",
+ "def initialize_q_table(state_space, action_space):\n",
+ " Qtable = np.zeros((state_space, action_space))\n",
+ " return Qtable"
+ ],
+ "metadata": {
+ "id": "lnrb_nX33fJo"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "Qtable_frozenlake = initialize_q_table(state_space, action_space)"
+ ],
+ "metadata": {
+ "id": "Y0WlgkVO3Jf9"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 3: Define the epsilon-greedy policy ๐ค\n",
+ "\n",
+ "Epsilon-Greedy is the training policy that handles the exploration/exploitation trade-off.\n",
+ "\n",
+ "The idea with Epsilon Greedy:\n",
+ "\n",
+ "- With *probability 1โ-โษ* : **we do exploitation** (aka our agent selects the action with the highest state-action pair value).\n",
+ "\n",
+ "- With *probability ษ*: we do **exploration** (trying random action).\n",
+ "\n",
+ "And as the training goes, we progressively **reduce the epsilon value since we will need less and less exploration and more exploitation.**"
+ ],
+ "metadata": {
+ "id": "flILKhBU3yZ7"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "3kk8TU3w4Ali"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def epsilon_greedy_policy(Qtable, state):\n",
+ " # Randomly generate a number between 0 and 1\n",
+ " random_num = \n",
+ " # if random_num > greater than epsilon --> exploitation\n",
+ " if random_num > epsilon:\n",
+ " # Take the action with the highest value given a state\n",
+ " # np.argmax can be useful here\n",
+ " action = \n",
+ " # else --> exploration\n",
+ " else:\n",
+ " action = # Take a random action\n",
+ " \n",
+ " return action"
+ ],
+ "metadata": {
+ "id": "6Bj7x3in3_Pq"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "#### Solution"
+ ],
+ "metadata": {
+ "id": "8R5ej1fS4P2V"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def epsilon_greedy_policy(Qtable, state):\n",
+ " # Randomly generate a number between 0 and 1\n",
+ " random_int = random.uniform(0,1)\n",
+ " # if random_int > greater than epsilon --> exploitation\n",
+ " if random_int > epsilon:\n",
+ " # Take the action with the highest value given a state\n",
+ " # np.argmax can be useful here\n",
+ " action = np.argmax(Qtable[state])\n",
+ " # else --> exploration\n",
+ " else:\n",
+ " action = env.action_space.sample()\n",
+ " \n",
+ " return action"
+ ],
+ "metadata": {
+ "id": "cYxHuckr4LiG"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 4: Define the greedy policy ๐ค\n",
+ "Remember we have two policies since Q-Learning is an **off-policy** algorithm. This means we're using a **different policy for acting and updating the value function**.\n",
+ "\n",
+ "- Epsilon greedy policy (acting policy)\n",
+ "- Greedy policy (updating policy)\n",
+ "\n",
+ "Greedy policy will also be the final policy we'll have when the Q-learning agent will be trained. The greedy policy is used to select an action from the Q-table."
+ ],
+ "metadata": {
+ "id": "Atll4Z774gri"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "PlmodanK40QA"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def greedy_policy(Qtable, state):\n",
+ " # Exploitation: take the action with the highest state, action value\n",
+ " action = \n",
+ " \n",
+ " return action"
+ ],
+ "metadata": {
+ "id": "E3SCLmLX5bWG"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "#### Solution"
+ ],
+ "metadata": {
+ "id": "B2_-8b8z5k54"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def greedy_policy(Qtable, state):\n",
+ " # Exploitation: take the action with the highest state, action value\n",
+ " action = np.argmax(Qtable[state])\n",
+ " \n",
+ " return action"
+ ],
+ "metadata": {
+ "id": "se2OzWGW5kYJ"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 5: Define the hyperparameters โ๏ธ\n",
+ "The exploration related hyperparamters are some of the most important ones. \n",
+ "\n",
+ "- We need to make sure that our agent **explores enough the state space** in order to learn a good value approximation, in order to do that we need to have progressive decay of the epsilon.\n",
+ "- If you decrease too fast epsilon (too high decay_rate), **you take the risk that your agent is stuck**, since your agent didn't explore enough the state space and hence can't solve the problem."
+ ],
+ "metadata": {
+ "id": "hW80DealcRtu"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Training parameters\n",
+ "n_training_episodes = 10000 # Total training episodes\n",
+ "learning_rate = 0.7 # Learning rate\n",
+ "\n",
+ "# Evaluation parameters\n",
+ "n_eval_episodes = 100 # Total number of test episodes\n",
+ "\n",
+ "# Environment parameters\n",
+ "env_id = \"FrozenLake-v1\" # Name of the environment\n",
+ "max_steps = 99 # Max steps per episode\n",
+ "gamma = 0.95 # Discounting rate\n",
+ "eval_seed = [] # The evaluation seed of the environment\n",
+ "\n",
+ "# Exploration parameters\n",
+ "epsilon = 1.0 # Exploration rate\n",
+ "max_epsilon = 1.0 # Exploration probability at start\n",
+ "min_epsilon = 0.05 # Minimum exploration probability \n",
+ "decay_rate = 0.005 # Exponential decay rate for exploration prob"
+ ],
+ "metadata": {
+ "id": "Y1tWn0tycWZ1"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 6: Create the training loop method"
+ ],
+ "metadata": {
+ "id": "cDb7Tdx8atfL"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n",
+ " for episode in range(n_training_episodes):\n",
+ " # Reduce epsilon (because we need less and less exploration)\n",
+ " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n",
+ " # Reset the environment\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ "\n",
+ " # repeat\n",
+ " for step in range(max_steps):\n",
+ " # Choose the action At using epsilon greedy policy\n",
+ " action = \n",
+ "\n",
+ " # Take action At and observe Rt+1 and St+1\n",
+ " # Take the action (a) and observe the outcome state(s') and reward (r)\n",
+ " new_state, reward, done, info = \n",
+ "\n",
+ " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
+ " Qtable[state][action] = \n",
+ "\n",
+ " # If done, finish the episode\n",
+ " if done:\n",
+ " break\n",
+ " \n",
+ " # Our state is the new state\n",
+ " state = new_state\n",
+ " return Qtable"
+ ],
+ "metadata": {
+ "id": "paOynXy3aoJW"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "#### Solution"
+ ],
+ "metadata": {
+ "id": "Pnpk2ePoem3r"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n",
+ " for episode in range(n_training_episodes):\n",
+ " # Reduce epsilon (because we need less and less exploration)\n",
+ " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n",
+ " # Reset the environment\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ "\n",
+ " # repeat\n",
+ " for step in range(max_steps):\n",
+ " # Choose the action At using epsilon greedy policy\n",
+ " action = epsilon_greedy_policy(Qtable, state)\n",
+ "\n",
+ " # Take action At and observe Rt+1 and St+1\n",
+ " # Take the action (a) and observe the outcome state(s') and reward (r)\n",
+ " new_state, reward, done, info = env.step(action)\n",
+ "\n",
+ " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
+ " Qtable[state][action] = Qtable[state][action] + learning_rate * (reward + gamma * np.max(Qtable[new_state]) - Qtable[state][action]) \n",
+ "\n",
+ " # If done, finish the episode\n",
+ " if done:\n",
+ " break\n",
+ " \n",
+ " # Our state is the new state\n",
+ " state = new_state\n",
+ " return Qtable"
+ ],
+ "metadata": {
+ "id": "IyZaYbUAeolw"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 7: Train the Q-Learning agent ๐"
+ ],
+ "metadata": {
+ "id": "WLwKQ4tUdhGI"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "Qtable_frozenlake = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_frozenlake)"
+ ],
+ "metadata": {
+ "id": "DPBxfjJdTCOH"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 8: Let's see what our Q-Learning table looks like now ๐"
+ ],
+ "metadata": {
+ "id": "yVeEhUCrc30L"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "Qtable_frozenlake"
+ ],
+ "metadata": {
+ "id": "nmfchsTITw4q"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 9: Define the evaluation method ๐"
+ ],
+ "metadata": {
+ "id": "pUrWkxsHccXD"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def evaluate_agent(env, max_steps, n_eval_episodes, Q, seed):\n",
+ " \"\"\"\n",
+ " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n",
+ " :param env: The evaluation environment\n",
+ " :param n_eval_episodes: Number of episode to evaluate the agent\n",
+ " :param Q: The Q-table\n",
+ " :param seed: The evaluation seed array (for taxi-v3)\n",
+ " \"\"\"\n",
+ " episode_rewards = []\n",
+ " for episode in range(n_eval_episodes):\n",
+ " if seed:\n",
+ " state = env.reset(seed=seed[episode])\n",
+ " else:\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ " total_rewards_ep = 0\n",
+ " \n",
+ " for step in range(max_steps):\n",
+ " # Take the action (index) that have the maximum expected future reward given that state\n",
+ " action = np.argmax(Q[state][:])\n",
+ " new_state, reward, done, info = env.step(action)\n",
+ " total_rewards_ep += reward\n",
+ " \n",
+ " if done:\n",
+ " break\n",
+ " state = new_state\n",
+ " episode_rewards.append(total_rewards_ep)\n",
+ " mean_reward = np.mean(episode_rewards)\n",
+ " std_reward = np.std(episode_rewards)\n",
+ "\n",
+ " return mean_reward, std_reward"
+ ],
+ "metadata": {
+ "id": "jNl0_JO2cbkm"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 10: Evaluate our Q-Learning agent ๐\n",
+ "- Normally you should have mean reward of 1.0\n",
+ "- It's relatively easy since the state space is really small (16). What you can try to do is [to replace with the slippery version](https://www.gymlibrary.ml/environments/toy_text/frozen_lake/)."
+ ],
+ "metadata": {
+ "id": "0jJqjaoAnxUo"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Evaluate our Agent\n",
+ "mean_reward, std_reward = evaluate_agent(env, max_steps, n_eval_episodes, Qtable_frozenlake, eval_seed)\n",
+ "print(f\"Mean_reward={mean_reward:.2f} +/- {std_reward:.2f}\")"
+ ],
+ "metadata": {
+ "id": "fAgB7s0HEFMm"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 11: Publish our trained model on the Hub ๐ฅ\n",
+ "Now that we saw we got good results after the training, we can publish our trained model on the hub ๐ค with one line of code.\n",
+ "\n",
+ "Here's an example of a Model Card:"
+ ],
+ "metadata": {
+ "id": "yxaP3bPdg1DV"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "nSjyobJ_n5y_"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent."
+ ],
+ "metadata": {
+ "id": "kv0k1JQjpMq3"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "#### Do not modify this code"
+ ],
+ "metadata": {
+ "id": "QZ5LrR-joIHD"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%%capture\n",
+ "from huggingface_hub import HfApi, HfFolder, Repository\n",
+ "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n",
+ "\n",
+ "from pathlib import Path\n",
+ "import datetime\n",
+ "import json"
+ ],
+ "metadata": {
+ "id": "Jex3i9lZ8ksX"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def record_video(env, Qtable, out_directory, fps=1):\n",
+ " images = [] \n",
+ " done = False\n",
+ " state = env.reset(seed=random.randint(0,500))\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " while not done:\n",
+ " # Take the action (index) that have the maximum expected future reward given that state\n",
+ " action = np.argmax(Qtable[state][:])\n",
+ " state, reward, done, info = env.step(action) # We directly put next_state = state for recording logic\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)"
+ ],
+ "metadata": {
+ "id": "Qo57HBn3W74O"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def push_to_hub(repo_id, \n",
+ " model,\n",
+ " env,\n",
+ " video_fps=1,\n",
+ " local_repo_path=\"hub\",\n",
+ " commit_message=\"Push Q-Learning agent to Hub\",\n",
+ " token= None\n",
+ " ):\n",
+ " _, repo_name = repo_id.split(\"/\")\n",
+ "\n",
+ " eval_env = env\n",
+ " \n",
+ " # Step 1: Clone or create the repo\n",
+ " # Create the repo (or clone its content if it's nonempty)\n",
+ " api = HfApi()\n",
+ " \n",
+ " repo_url = api.create_repo(\n",
+ " repo_id=repo_id,\n",
+ " token=token,\n",
+ " private=False,\n",
+ " exist_ok=True,)\n",
+ " \n",
+ " # Git pull\n",
+ " repo_local_path = Path(local_repo_path) / repo_name\n",
+ " repo = Repository(repo_local_path, clone_from=repo_url, use_auth_token=True)\n",
+ " repo.git_pull()\n",
+ " \n",
+ " repo.lfs_track([\"*.mp4\"])\n",
+ "\n",
+ " # Step 1: Save the model\n",
+ " if env.spec.kwargs.get(\"map_name\"):\n",
+ " model[\"map_name\"] = env.spec.kwargs.get(\"map_name\")\n",
+ " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n",
+ " model[\"slippery\"] = False\n",
+ "\n",
+ " print(model)\n",
+ " \n",
+ " \n",
+ " # Pickle the model\n",
+ " with open(Path(repo_local_path)/'q-learning.pkl', 'wb') as f:\n",
+ " pickle.dump(model, f)\n",
+ " \n",
+ " # Step 2: Evaluate the model and build JSON\n",
+ " mean_reward, std_reward = evaluate_agent(eval_env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])\n",
+ "\n",
+ " # First get datetime\n",
+ " eval_datetime = datetime.datetime.now()\n",
+ " eval_form_datetime = eval_datetime.isoformat()\n",
+ "\n",
+ " evaluate_data = {\n",
+ " \"env_id\": model[\"env_id\"], \n",
+ " \"mean_reward\": mean_reward,\n",
+ " \"n_eval_episodes\": model[\"n_eval_episodes\"],\n",
+ " \"eval_datetime\": eval_form_datetime,\n",
+ " }\n",
+ " # Write a JSON file\n",
+ " with open(Path(repo_local_path) / \"results.json\", \"w\") as outfile:\n",
+ " json.dump(evaluate_data, outfile)\n",
+ "\n",
+ " # Step 3: Create the model card\n",
+ " # Env id\n",
+ " env_name = model[\"env_id\"]\n",
+ " if env.spec.kwargs.get(\"map_name\"):\n",
+ " env_name += \"-\" + env.spec.kwargs.get(\"map_name\")\n",
+ "\n",
+ " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n",
+ " env_name += \"-\" + \"no_slippery\"\n",
+ "\n",
+ " metadata = {}\n",
+ " metadata[\"tags\"] = [\n",
+ " env_name,\n",
+ " \"q-learning\",\n",
+ " \"reinforcement-learning\",\n",
+ " \"custom-implementation\"\n",
+ " ]\n",
+ "\n",
+ " # Add metrics\n",
+ " eval = metadata_eval_result(\n",
+ " model_pretty_name=repo_name,\n",
+ " task_pretty_name=\"reinforcement-learning\",\n",
+ " task_id=\"reinforcement-learning\",\n",
+ " metrics_pretty_name=\"mean_reward\",\n",
+ " metrics_id=\"mean_reward\",\n",
+ " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n",
+ " dataset_pretty_name=env_name,\n",
+ " dataset_id=env_name,\n",
+ " )\n",
+ "\n",
+ " # Merges both dictionaries\n",
+ " metadata = {**metadata, **eval}\n",
+ "\n",
+ " model_card = f\"\"\"\n",
+ " # **Q-Learning** Agent playing **{env_id}**\n",
+ " This is a trained model of a **Q-Learning** agent playing **{env_id}** .\n",
+ " \"\"\"\n",
+ "\n",
+ " model_card += \"\"\"\n",
+ " ## Usage\n",
+ " ```python\n",
+ " \"\"\"\n",
+ "\n",
+ " model_card += f\"\"\"model = load_from_hub(repo_id=\"{repo_id}\", filename=\"q-learning.pkl\")\n",
+ "\n",
+ " # Don't forget to check if you need to add additional attributes (is_slippery=False etc)\n",
+ " env = gym.make(model[\"env_id\"])\n",
+ "\n",
+ " evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])\n",
+ " \"\"\"\n",
+ "\n",
+ " model_card +=\"\"\"\n",
+ " ```\n",
+ " \"\"\"\n",
+ "\n",
+ " readme_path = repo_local_path / \"README.md\"\n",
+ " readme = \"\"\n",
+ " if readme_path.exists():\n",
+ " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n",
+ " readme = f.read()\n",
+ " else:\n",
+ " readme = model_card\n",
+ "\n",
+ " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n",
+ " f.write(readme)\n",
+ "\n",
+ " # Save our metrics to Readme metadata\n",
+ " metadata_save(readme_path, metadata)\n",
+ "\n",
+ " # Step 4: Record a video\n",
+ " video_path = repo_local_path / \"replay.mp4\"\n",
+ " record_video(env, model[\"qtable\"], video_path, video_fps)\n",
+ " \n",
+ " # Push everything to hub\n",
+ " print(f\"Pushing repo {repo_name} to the Hugging Face Hub\")\n",
+ " repo.push_to_hub(commit_message=commit_message)\n",
+ "\n",
+ " print(f\"Your model is pushed to the hub. You can view your model here: {repo_url}\")"
+ ],
+ "metadata": {
+ "id": "pwsNrzB339aF"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### .\n",
+ "By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
+ "\n",
+ "This way:\n",
+ "- You can **showcase our work** ๐ฅ\n",
+ "- You can **visualize your agent playing** ๐\n",
+ "- You can **share with the community an agent that others can use** ๐พ\n",
+ "- You can **access a leaderboard ๐ to see how well your agent is performing compared to your classmates** ๐ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard"
+ ],
+ "metadata": {
+ "id": "81J6cet_ogSS"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1๏ธโฃ (If it's not already done) create an account to HF โก https://huggingface.co/join\n",
+ "\n",
+ "2๏ธโฃ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens) **with write role**"
+ ],
+ "metadata": {
+ "id": "cWnFC0iZooTw"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "hUExHU_5oqPc"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from huggingface_hub import notebook_login\n",
+ "notebook_login()"
+ ],
+ "metadata": {
+ "id": "QB5nIcxR8paT"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "GyWc1x3-o3xG"
+ },
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Gc5AfUeFo3xH"
+ },
+ "source": [
+ "3๏ธโฃ We're now ready to push our trained agent to the ๐ค Hub ๐ฅ using `package_to_hub()` function\n",
+ "\n",
+ "- Let's create **the model dictionnary that contains the hyperparameters and the Q_table**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "model = {\n",
+ " \"env_id\": env_id,\n",
+ " \"max_steps\": max_steps,\n",
+ " \"n_training_episodes\": n_training_episodes,\n",
+ " \"n_eval_episodes\": n_eval_episodes,\n",
+ " \"eval_seed\": eval_seed,\n",
+ "\n",
+ " \"learning_rate\": learning_rate,\n",
+ " \"gamma\": gamma,\n",
+ "\n",
+ " \"epsilon\": epsilon,\n",
+ " \"max_epsilon\": max_epsilon,\n",
+ " \"min_epsilon\": min_epsilon,\n",
+ " \"decay_rate\": decay_rate,\n",
+ "\n",
+ " \"qtable\": Qtable_frozenlake\n",
+ "}"
+ ],
+ "metadata": {
+ "id": "FiMqxqVHg0I4"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9kld-AEso3xH"
+ },
+ "source": [
+ "Let's fill the `package_to_hub` function:\n",
+ "- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `\n",
+ "(repo_id = {username}/{repo_name})`\n",
+ "๐ก **A good name is {username}/q-{env_id}**\n",
+ "- `model`: our model dictionnary containing the hyperparameters and the Qtable.\n",
+ "- `env`: the environment.\n",
+ "- `commit_message`: message of the commit"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "model"
+ ],
+ "metadata": {
+ "id": "5sBo2umnXpPd"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "username = \"\" # FILL THIS\n",
+ "repo_name = \"q-FrozenLake-v1-4x4-noSlippery\"\n",
+ "push_to_hub(\n",
+ " repo_id=f\"{username}/{repo_name}\",\n",
+ " model=model,\n",
+ " env=env)"
+ ],
+ "metadata": {
+ "id": "RpOTtSt83kPZ"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Congrats ๐ฅณ you've just implented from scratch, trained and uploaded your first Reinforcement Learning agent. \n",
+ "FrozenLake-v1 no_slippery is very simple environment, let's try an harder one ๐ฅ."
+ ],
+ "metadata": {
+ "id": "E2875IGsprzq"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Part 2: Taxi-v3 ๐\n",
+ "\n",
+ "### Step 1: Create and understand [Taxi-v3 ๐](https://www.gymlibrary.ml/environments/toy_text/taxi/)\n",
+ "---\n",
+ "\n",
+ "๐ก A good habit when you start to use an environment is to check its documentation \n",
+ "\n",
+ "๐ https://www.gymlibrary.ml/environments/toy_text/taxi/\n",
+ "\n",
+ "---\n",
+ "\n",
+ "In Taxi-v3 ๐, there are four designated locations in the grid world indicated by R(ed), G(reen), Y(ellow), and B(lue). When the episode starts, the taxi starts off at a random square and the passenger is at a random location. The taxi drives to the passengerโs location, picks up the passenger, drives to the passengerโs destination (another one of the four specified locations), and then drops off the passenger. Once the passenger is dropped off, the episode ends."
+ ],
+ "metadata": {
+ "id": "18lN8Bz7yvLt"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "dkGqA8YFmtQf"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "env = gym.make(\"Taxi-v3\")"
+ ],
+ "metadata": {
+ "id": "gL0wpeO8gpej"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "There are **500 discrete states since there are 25 taxi positions, 5 possible locations of the passenger** (including the case when the passenger is in the taxi), and **4 destination locations.**\n"
+ ],
+ "metadata": {
+ "id": "gBOaXgtsrmtT"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "state_space = env.observation_space.n\n",
+ "print(\"There are \", state_space, \" possible states\")"
+ ],
+ "metadata": {
+ "id": "_TPNaGSZrgqA"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "action_space = env.action_space.n\n",
+ "print(\"There are \", action_space, \" possible actions\")"
+ ],
+ "metadata": {
+ "id": "CdeeZuokrhit"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The action space (the set of possible actions the agent can take) is discrete with **6 actions available ๐ฎ**:\n",
+ "- 0: move south\n",
+ "- 1: move north\n",
+ "- 2: move east\n",
+ "- 3: move west\n",
+ "- 4: pickup passenger\n",
+ "- 5: drop off passenger\n",
+ "\n",
+ "Reward function ๐ฐ:\n",
+ "- -1 per step unless other reward is triggered.\n",
+ "- +20 delivering passenger.\n",
+ "- -10 executing โpickupโ and โdrop-offโ actions illegally."
+ ],
+ "metadata": {
+ "id": "R1r50Advrh5Q"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Create our Q table with state_size rows and action_size columns (500x6)\n",
+ "Qtable_taxi = initialize_q_table(state_space, action_space)\n",
+ "print(Qtable_taxi)\n",
+ "print(\"Q-table shape: \", Qtable_taxi .shape)"
+ ],
+ "metadata": {
+ "id": "US3yDXnEtY9I"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 2: Define the hyperparameters โ๏ธ\n",
+ "โ DO NOT MODIFY EVAL_SEED: the eval_seed array **allows us to evaluate your agent with the same taxi starting positions for every classmate**"
+ ],
+ "metadata": {
+ "id": "gUMKPH0_LJyH"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Training parameters\n",
+ "n_training_episodes = 25000 # Total training episodes\n",
+ "learning_rate = 0.7 # Learning rate\n",
+ "\n",
+ "# Evaluation parameters\n",
+ "n_eval_episodes = 100 # Total number of test episodes\n",
+ "\n",
+ "# DO NOT MODIFY EVAL_SEED\n",
+ "eval_seed = [16,54,165,177,191,191,120,80,149,178,48,38,6,125,174,73,50,172,100,148,146,6,25,40,68,148,49,167,9,97,164,176,61,7,54,55,\n",
+ " 161,131,184,51,170,12,120,113,95,126,51,98,36,135,54,82,45,95,89,59,95,124,9,113,58,85,51,134,121,169,105,21,30,11,50,65,12,43,82,145,152,97,106,55,31,85,38,\n",
+ " 112,102,168,123,97,21,83,158,26,80,63,5,81,32,11,28,148] # Evaluation seed, this ensures that all classmates agents are trained on the same taxi starting position\n",
+ " # Each seed has a specific starting state\n",
+ "\n",
+ "# Environment parameters\n",
+ "env_id = \"Taxi-v3\" # Name of the environment\n",
+ "max_steps = 99 # Max steps per episode\n",
+ "gamma = 0.95 # Discounting rate\n",
+ "\n",
+ "# Exploration parameters\n",
+ "epsilon = 1.0 # Exploration rate\n",
+ "max_epsilon = 1.0 # Exploration probability at start\n",
+ "min_epsilon = 0.05 # Minimum exploration probability \n",
+ "decay_rate = 0.005 # Exponential decay rate for exploration prob\n"
+ ],
+ "metadata": {
+ "id": "AB6n__hhg7YS"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 3: Train our Q-Learning agent ๐"
+ ],
+ "metadata": {
+ "id": "1TMORo1VLTsX"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "Qtable_taxi = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_taxi)"
+ ],
+ "metadata": {
+ "id": "MLNwkNDb14h2"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "Qtable_taxi"
+ ],
+ "metadata": {
+ "id": "WwP3Y2z2eS-K"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 4: Create a model dictionary ๐พ and publish our trained model on the Hub ๐ฅ\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "wPdu0SueLVl2"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "model = {\n",
+ " \"env_id\": env_id,\n",
+ " \"max_steps\": max_steps,\n",
+ " \"n_training_episodes\": n_training_episodes,\n",
+ " \"n_eval_episodes\": n_eval_episodes,\n",
+ " \"eval_seed\": eval_seed,\n",
+ "\n",
+ " \"learning_rate\": learning_rate,\n",
+ " \"gamma\": gamma,\n",
+ "\n",
+ " \"epsilon\": epsilon,\n",
+ " \"max_epsilon\": max_epsilon,\n",
+ " \"min_epsilon\": min_epsilon,\n",
+ " \"decay_rate\": decay_rate,\n",
+ "\n",
+ " \"qtable\": Qtable_taxi\n",
+ "}"
+ ],
+ "metadata": {
+ "id": "0a1FpE_3hNYr"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "username = \"\" # FILL THIS\n",
+ "repo_name = \"q-Taxi-v3\"\n",
+ "push_to_hub(\n",
+ " repo_id=f\"{username}/{repo_name}\",\n",
+ " model=model,\n",
+ " env=env)"
+ ],
+ "metadata": {
+ "id": "dhQtiQozhOn1"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Compare the results of your Taxi-v3 with your classmates using the leaderboard ๐ ๐ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard"
+ ],
+ "metadata": {
+ "id": "ZgSdjgbIpRti"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "LyGvq3k4pYv_"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Part 3: Load from Hub ๐ฝ\n",
+ "\n",
+ "What's amazing with Hugging Face Hub ๐ค is that you can easily load powerful models from the community.\n",
+ "\n",
+ "Loading a saved model from the Hub is really easy.\n",
+ "1. You go https://huggingface.co/models?other=q-learning to see the list of all the q-learning saved models.\n",
+ "2. You select one and copy its repo_id"
+ ],
+ "metadata": {
+ "id": "bzgIO70c0bu2"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "qUDSEy8Bn3ps"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "3. Then we just need to use `load_from_hub` with:\n",
+ "- The repo_id\n",
+ "- The filename: the saved model inside the repo."
+ ],
+ "metadata": {
+ "id": "gTth6thRoC6X"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "#### Do not modify this code"
+ ],
+ "metadata": {
+ "id": "EtrfoTaBoNrd"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from urllib.error import HTTPError\n",
+ "\n",
+ "from huggingface_hub import hf_hub_download\n",
+ "\n",
+ "\n",
+ "def load_from_hub(repo_id: str, filename: str) -> str:\n",
+ " \"\"\"\n",
+ " Download a model from Hugging Face Hub.\n",
+ " :param repo_id: id of the model repository from the Hugging Face Hub\n",
+ " :param filename: name of the model zip file from the repository\n",
+ " \"\"\"\n",
+ " try:\n",
+ " from huggingface_hub import cached_download, hf_hub_url\n",
+ " except ImportError:\n",
+ " raise ImportError(\n",
+ " \"You need to install huggingface_hub to use `load_from_hub`. \"\n",
+ " \"See https://pypi.org/project/huggingface-hub/ for installation.\"\n",
+ " )\n",
+ "\n",
+ " # Get the model from the Hub, download and cache the model on your local disk\n",
+ " pickle_model = hf_hub_download(\n",
+ " repo_id=repo_id,\n",
+ " filename=filename\n",
+ " )\n",
+ "\n",
+ " with open(pickle_model, 'rb') as f:\n",
+ " downloaded_model_file = pickle.load(f)\n",
+ " \n",
+ " return downloaded_model_file"
+ ],
+ "metadata": {
+ "id": "Eo8qEzNtCaVI"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### ."
+ ],
+ "metadata": {
+ "id": "b_sM2gNioPZH"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "model = load_from_hub(repo_id=\"ThomasSimonini/q-Taxi-v3\", filename=\"q-learning.pkl\")\n",
+ "\n",
+ "print(model)\n",
+ "env = gym.make(model[\"env_id\"])\n",
+ "\n",
+ "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])"
+ ],
+ "metadata": {
+ "id": "JUm9lz2gCQcU"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "model = load_from_hub(repo_id=\"ThomasSimonini/q-FrozenLake-v1-no-slippery\", filename=\"q-learning.pkl\")\n",
+ "\n",
+ "env = gym.make(model[\"env_id\"], is_slippery=False)\n",
+ "\n",
+ "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])"
+ ],
+ "metadata": {
+ "id": "O7pL8rg1MulN"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Congrats ๐ฅณ, you've just implemented, trained, and uploaded your first Reinforcement Learning agent.\n",
+ "\n",
+ "Understanding Q-Learning is an **important step to understanding value-based methods.**\n",
+ "\n",
+ "In the next Unit with Deep Q-Learning, we'll see that creating and updating a Q-table was a good strategy โ **however, this is not scalable.**\n",
+ "\n",
+ "For instance, imagine you create an agent that learns to play Doom. Doom is a large environment with a huge state space (millions of different states). Creating and updating a Q-table for that environment would not be efficient. That's why we'll study Deep Q-Learning, an algorithm **where we use a neural network that approximates, given a state, the different Q-values for each action.**"
+ ],
+ "metadata": {
+ "id": "p-fW-EU5WejJ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BQAwLnYFPk-s"
+ },
+ "source": [
+ "## Some additional challenges ๐\n",
+ "The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
+ "\n",
+ "In the [Leaderboard](https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
+ "\n",
+ "Here are some ideas to achieve so:\n",
+ "* Train more steps\n",
+ "* Try different hyperparameters by looking at what your classmates have done.\n",
+ "* **Push your new trained model** on the Hub ๐ฅ\n",
+ "\n",
+ "Are walking on ice and driving taxis too boring to you? Try to **change the environment**, why not using FrozenLake-v1 slippery version? Check how they work [using the gym documentation](https://www.gymlibrary.ml/) and have fun ๐."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9lM95-dvmif8"
+ },
+ "source": [
+ "________________________________________________________________________\n",
+ "Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.**\n",
+ "\n",
+ "If youโre still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
+ "\n",
+ "Take time to really **grasp the material before continuing and try the additional challenges**. Itโs important to master these elements and having a solid foundations.\n",
+ "\n",
+ "Naturally, during the course, weโre going to use and deeper explain again these terms but **itโs better to have a good understanding of them now before diving into the next chapters.**\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "feR90OUSEXq9"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "### This is a course built with you ๐ท๐ฟโโ๏ธ\n",
+ "\n",
+ "We want to improve and update the course iteratively with your feedback. If you have some, please open an issue on the Github Repo: [https://github.com/huggingface/deep-rl-class/issues](https://github.com/huggingface/deep-rl-class/issues)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BjLhT70TEZIn"
+ },
+ "source": [
+ "See you on Unit 3! ๐ฅ\n",
+ "## Keep learning, stay awesome!"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "collapsed_sections": [
+ "67OdoKL63eDD",
+ "8R5ej1fS4P2V",
+ "B2_-8b8z5k54",
+ "Pnpk2ePoem3r",
+ "QZ5LrR-joIHD",
+ "EtrfoTaBoNrd"
+ ],
+ "name": "Unit 2: Q-Learning with FrozenLake-v1 and Taxi-v3.ipynb",
+ "provenance": [],
+ "private_outputs": true,
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
+
+You'll then be able to **compare your agentโs results with other classmates thanks to a leaderboard** ๐ฅ.
+
+This course is **self-paced**, you can start whenever you want.
+
+## Required time โฑ๏ธ
+The required time for this unit is, approximately:
+- 2 hours for the theory
+- 1 hour for the hands-on.
+
+## Start this Unit ๐
+Here are the steps for this Unit:
+
+1๏ธโฃ Sign up to our Discord Server. This is the place where youย **can exchange with the community and with us, create study groups to grow each other and more**ย
+
+๐๐ปย [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Are you new to Discord? Check our **discord 101 to get the best practices** ๐ https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md
+
+2๏ธโฃ **Introduce yourself on Discord in #introduce-yourself Discord channelย ๐ค and check on the left the Reinforcement Learning section.**
+
+- In #rl-announcements we give the last information about the course.
+- #discussions is a place to exchange.
+- #unity-ml-agents is to exchange about everything related to this library.
+- #study-groups, to create study groups with your classmates.
+
+
+
+3๏ธโฃ ๐ **Read An [Introduction to Deep Reinforcement Learning](https://huggingface.co/blog/deep-rl-intro)**, where youโll learn the foundations of Deep RL. You can also watch the video version attached to the article. ๐ https://huggingface.co/blog/deep-rl-intro
+
+4๏ธโฃ ๐ฉโ๐ป Then dive on the hands-on, where **youโll train your first lander agent ๐ to land correctly on the Moon ๐ using Stable-Baselines3 and share it with the community.** Thanks to a leaderboard, **you'll be able to compare your results with other classmates** and exchange the best practices to improve your agent's scores Who will win the challenge for Unit 1 ๐?
+
+The hands-on ๐ [](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit1/unit1.ipynb)
+
+The leaderboard ๐ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard
+
+You can work directly **with the colab notebook, which allows you not to have to install everything on your machine (and itโs free)**.
+
+5๏ธโฃ The best way to learn **is to try things on your own**. Thatโs why we have a challenges section in the colab where we give you some ideas on how you can go further: using another environment, using another model etc.
+
+## Additional readings ๐
+- [Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 1, 2 and 3](http://incompleteideas.net/book/RLbook2020.pdf)
+- [Foundations of Deep RL Series, L1 MDPs, Exact Solution Methods, Max-ent RL by Pieter Abbeel](https://youtu.be/2GwBez0D20A)
+- [Spinning Up RL by OpenAI Part 1: Key concepts of RL](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)
+
+- [Getting Started With OpenAI Gym: The Basic Building Blocks](https://blog.paperspace.com/getting-started-with-openai-gym/)
+
+## How to make the most of this course
+
+To make the most of the course, my advice is to:
+
+- **Participate in Discord** and join a study group.
+- **Read multiple times** the theory part and takes some notes
+- Donโt just do the colab. When you learn something, try to change the environment, change the parameters and read the libraries' documentation. Have fun ๐ฅณ
+- Struggling is **a good thing in learning**. It means that you start to build new skills. Deep RL is a complex topic and it takes time to understand. Try different approaches, use our additional readings, and exchange with classmates on discord.
+
+## This is a course built with you ๐ท๐ฟโโ๏ธ
+
+We want to improve and update the course iteratively with your feedback. If you have some, please open an issue on the Github Repo: [https://github.com/huggingface/deep-rl-class/issues](https://github.com/huggingface/deep-rl-class/issues)
+
+## Donโt forget to join the Community ๐ข
+
+We have a discord server where youย **can exchange with the community and with us, create study groups to grow each other and more**ย
+
+๐๐ปย [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Donโt forget toย **introduce yourself when you sign upย ๐ค**
+
+โIf you have other questions, [please check our FAQ](https://github.com/huggingface/deep-rl-class#faq)
+
+Keep learning, stay awesome,
diff --git a/unit1/assets/img/LunarLander.gif b/unit1/assets/img/LunarLander.gif
new file mode 100644
index 0000000..5caf157
Binary files /dev/null and b/unit1/assets/img/LunarLander.gif differ
diff --git a/unit1/assets/img/discord_channels.jpg b/unit1/assets/img/discord_channels.jpg
new file mode 100644
index 0000000..d76ff68
Binary files /dev/null and b/unit1/assets/img/discord_channels.jpg differ
diff --git a/unit1/unit1.ipynb b/unit1/unit1.ipynb
new file mode 100644
index 0000000..b9ab9f9
--- /dev/null
+++ b/unit1/unit1.ipynb
@@ -0,0 +1,1115 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "
+
+You'll then be able to **compare your agentโs results with other classmates thanks to a leaderboard** ๐ฅ.
+
+This Unit is divided into 2 parts:
+- [Part 1](https://huggingface.co/blog/deep-rl-q-part1)
+- [Part 2](https://huggingface.co/blog/deep-rl-q-part2)
+
+
+
+This course is **self-paced**, you can start whenever you want.
+
+## Required time โฑ๏ธ
+The required time for this unit is, approximately:
+- 2-3 hours for the theory
+- 1 hour for the hands-on.
+
+## Start this Unit ๐
+Here are the steps for this Unit:
+
+1๏ธโฃ If it's not already done, sign up to our Discord Server. This is the place where youย **can exchange with the community and with us, create study groups to grow each other and more**ย
+
+๐๐ปย [https://discord.gg/aYka4Yhff9](https://discord.gg/aYka4Yhff9).
+
+Are you new to Discord? Check our **discord 101 to get the best practices** ๐ https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md
+
+2๏ธโฃ **Introduce yourself on Discord in #introduce-yourself Discord channelย ๐ค and check on the left the Reinforcement Learning section.**
+
+- In #rl-announcements we give the last information about the course.
+- #discussions is a place to exchange.
+- #unity-ml-agents is to exchange about everything related to this library.
+- #study-groups, to create study groups with your classmates.
+
+![\"environments\"/](\"https://huggingface.co/blog/assets/70_deep_rl_q_part1/envs.gif\")
"
+ ],
+ "metadata": {
+ "id": "vRU_vXBrl1Jx"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4i6tjI2tHQ8j"
+ },
+ "source": [
+ "## Objectives of this notebook ๐\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to use **Gym**, the environment library.\n",
+ "- Be able to code from scratch a Q-Learning agent.\n",
+ "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score ๐ฅ.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ct8eLAabICE-"
+ },
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Class\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6p5HnEefISCB"
+ },
+ "source": [
+ "In this free course, you will:\n",
+ "\n",
+ "- ๐ Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- ๐งโ๐ป Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, and RLlib.\n",
+ "- ๐ค Train **agents in unique environments** \n",
+ "\n",
+ "And more check ๐ the syllabus ๐ https://github.com/huggingface/deep-rl-class\n",
+ "\n",
+ "The best way to keep in touch is to join our discord server to exchange with the community and with us ๐๐ป https://discord.gg/aYka4Yhff9"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Y-mo_6rXIjRi"
+ },
+ "source": [
+ "## Prerequisites ๐๏ธ\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "๐ฒ ๐ [Read the Unit 2 Readme](https://github.com/huggingface/deep-rl-class/blob/main/unit2/README.md) that contains all the information.\n",
+ "\n",
+ "๐ฒ ๐ [Read **An Introduction to Q-Learning Part 1**](https://huggingface.co/blog/deep-rl-q-part1) \n",
+ "\n",
+ "๐ฒ ๐ [Read **An Introduction to Q-Learning Part 2**](https://huggingface.co/blog/deep-rl-q-part2) \n",
+ "\n",
+ "๐ฒ ๐ข Sign up to [our Discord Server](https://discord.gg/aYka4Yhff9) and **introduce yourself to #introduce-yourself channel ๐ฅณ**\n",
+ "\n",
+ "๐ฒ ๐ Are you new to Discord? Check our **discord 101 to get the best practices** ๐ https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## A small recap of Q-Learning"
+ ],
+ "metadata": {
+ "id": "f2ONOODsyrMU"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "- The *Q-Learning* **is the RL algorithm that** \n",
+ " - Trains *Q-Function*, an **action-value function** that contains, as internal memory, a *Q-table* **that contains all the state-action pair values.**\n",
+ " \n",
+ " - Given a state and action, our Q-Function **will search into its Q-table the corresponding value.**\n",
+ " \n",
+ "\n",
+ "![\"Q](\"https://huggingface.co/blog/assets/70_deep_rl_q_part1/Q-function-2.jpg\")
\n",
+ "\n",
+ "- When the training is done,**we have an optimal Q-Function, so an optimal Q-Table.**\n",
+ " \n",
+ "- And if we **have an optimal Q-function**, we\n",
+ "have an optimal policy,since we **know for each state, what is the best action to take.**\n",
+ "\n",
+ "\n",
+ "![\"Link](\"https://huggingface.co/blog/assets/70_deep_rl_q_part1/link-value-policy.jpg\")
\n",
+ "\n",
+ "But, in the beginning,ย our **Q-Table is useless since it gives arbitrary value for each state-action pairย (most of the time we initialize the Q-Table to 0 values)**. But, as weโllย explore the environment and update our Q-Table it will give us better and better approximations\n",
+ "\n",
+ "![\"Link](\"https://huggingface.co/blog/assets/70_deep_rl_q_part1/Q-learning-1.jpg\")
\n",
+ "\n",
+ "This is the Q-Learning pseudocode:\n",
+ "\n",
+ "![\"Link](\"https://huggingface.co/blog/assets/70_deep_rl_q_part1/Q-learning-2.jpg\")
\n"
+ ],
+ "metadata": {
+ "id": "V68VveLacfxJ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 0: Setup a Virtual Display ๐ป\n",
+ "\n",
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install virtual screen libraries and create and run a virtual screen ๐ฅ"
+ ],
+ "metadata": {
+ "id": "7sUukH74yQ7K"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%%capture\n",
+ "!pip install pyglet==1.5.1 \n",
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip3 install pyvirtualdisplay\n",
+ "\n",
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "f2wrVUvrySkW"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 1: Install dependencies ๐ฝ\n",
+ "The first step is to install the dependencies, weโll install multiple ones:\n",
+ "\n",
+ "- `gym`: Contains the FrozenLake-v1 โ and Taxi-v3 ๐ environments.\n",
+ "- `pygame`: Used for the FrozenLake-v1 and Taxi-v3 UI.\n",
+ "- `numPy`: Used for handling our Q-table.\n",
+ "\n",
+ "The Hugging Face Hub ๐ค works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easilly collaborate with others.\n",
+ "\n",
+ "You can see here all the Deep reinforcement Learning models available ๐ https://huggingface.co/models?other=q-learning\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "mIkOB4Rpw65k"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%%capture\n",
+ "!pip install git+https://github.com/openai/gym.git@a37b956d57bdb024ccd8fef5446ba159e32d4626 # We install gym using git since Taxi-v3 \"rgb_array version\" is not on PyPi release\n",
+ "!pip install pygame\n",
+ "!pip install numpy\n",
+ "\n",
+ "!pip install huggingface_hub\n",
+ "!pip install pickle5\n",
+ "!pip install pyyaml==6.0 # avoid key error metadata\n",
+ "!pip install imageio imageio_ffmpeg"
+ ],
+ "metadata": {
+ "id": "1Ac7wW_5ClJC"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 2: Import the packages ๐ฆ\n",
+ "\n",
+ "In addition to the installed libraries, we also use:\n",
+ "\n",
+ "- `random`: To generate random numbers (that will be useful for Epsilon-Greedy Policy).\n",
+ "- `imageio`: To generate a replay video\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "W-7f-Swax_9x"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import numpy as np\n",
+ "import gym\n",
+ "import random\n",
+ "import imageio\n",
+ "import os\n",
+ "\n",
+ "import pickle5 as pickle"
+ ],
+ "metadata": {
+ "id": "VcNvOAQlysBJ"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We're now ready to code our Q-Learning algorithm ๐ฅ"
+ ],
+ "metadata": {
+ "id": "xp4-bXKIy1mQ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Part 1: Frozen Lake โ (non slippery version)"
+ ],
+ "metadata": {
+ "id": "xya49aNJWVvv"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 1: Create and understand [FrozenLake environment โ]((https://www.gymlibrary.ml/environments/toy_text/frozen_lake/)\n",
+ "---\n",
+ "\n",
+ "๐ก A good habit when you start to use an environment is to check its documentation \n",
+ "\n",
+ "๐ https://www.gymlibrary.ml/environments/toy_text/frozen_lake/\n",
+ "\n",
+ "---\n",
+ "\n",
+ "We're going to train our Q-Learning agent **to navigate from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoid holes (H)**.\n",
+ "\n",
+ "We can have two sizes of environment:\n",
+ "- `map_name=\"4x4\"`: a 4x4 grid version\n",
+ "- `map_name=\"8x8\"`: a 8x8 grid version\n",
+ "\n",
+ "\n",
+ "The environment has two modes:\n",
+ "- `is_slippery=False`: The agent always move in the intended direction due to the non-slippery nature of the frozen lake.\n",
+ "- `is_slippery=True`: The agent may not always move in the intended direction due to the slippery nature of the frozen lake (stochastic)."
+ ],
+ "metadata": {
+ "id": "NAvihuHdy9tw"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "For now let's keep it simple with the 4x4 map and non-slippery"
+ ],
+ "metadata": {
+ "id": "UaW_LHfS0PY2"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Create the FrozenLake-v1 environment using 4x4 map and non-slippery version\n",
+ "env = gym.make()"
+ ],
+ "metadata": {
+ "id": "IzJnb8O3y8up"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Solution"
+ ],
+ "metadata": {
+ "id": "Ji_UrI5l2zzn"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "env = gym.make(\"FrozenLake-v1\", map_name=\"4x4\", is_slippery=False)"
+ ],
+ "metadata": {
+ "id": "jNxUbPMP0akP"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "You can also custom your own grid using:\n",
+ "\n",
+ "```python\n",
+ "desc=[\"SFFF\", \"FHFH\", \"FFFH\", \"HFFG\"]\n",
+ "gym.make('FrozenLake-v1', desc=desc, is_slippery=True)\n",
+ "```\n",
+ "\n",
+ "but we'll use the default environment for now."
+ ],
+ "metadata": {
+ "id": "KASNViqL4tZn"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Let's see what the Environment looks like:\n"
+ ],
+ "metadata": {
+ "id": "SXbTfdeJ1Xi9"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZNPG0g_UGCfh"
+ },
+ "outputs": [],
+ "source": [
+ "# We create our environment with gym.make(\"