diff --git a/README.md b/README.md
index 01c809e..b8172ab 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,19 @@
-# The Hugging Face Deep Reinforcement Learning Class 🤗
+# The Hugging Face Deep Reinforcement Learning Class 🤗
+
+We're launching a **new version (v2.0) of the course starting December the 5th,**
+
+- The syllabus 📚: https://simoninithomas.github.io/deep-rl-course
+
+- The course 📚:
+
+- **Sign up here** ➡️➡️➡️ http://eepurl.com/ic5ZUD
+
+
+
+
+
+
+# The documentation below is for v1.0 (depreciated)
We're launching a **new version (v2.0) of the course starting December the 5th,**
diff --git a/notebooks/bonus-unit1/bonus-unit1.ipynb b/notebooks/bonus-unit1/bonus-unit1.ipynb
new file mode 100644
index 0000000..a96ade7
--- /dev/null
+++ b/notebooks/bonus-unit1/bonus-unit1.ipynb
@@ -0,0 +1,557 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2D3NL_e4crQv"
+ },
+ "source": [
+ "# Bonus Unit 1: Let's train Huggy the Dog 🐶 to fetch a stick"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "\n",
+ "\n",
+ "In this notebook, we'll reinforce what we learned in the first Unit by **teaching Huggy the Dog to fetch the stick and then play with it directly in your browser**\n",
+ "\n",
+ "⬇️ Here is an example of what **you will achieve at the end of the unit.** ⬇️ (launch ▶ to see)"
+ ],
+ "metadata": {
+ "id": "FMYrDriDujzX"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%%html\n",
+ ""
+ ],
+ "metadata": {
+ "id": "PnVhs1yYNyUF"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### The environment 🎮\n",
+ "\n",
+ "- Huggy the Dog, an environment created by [Thomas Simonini](https://twitter.com/ThomasSimonini) based on [Puppo The Corgi](https://blog.unity.com/technology/puppo-the-corgi-cuteness-overload-with-the-unity-ml-agents-toolkit)\n",
+ "\n",
+ "### The library used 📚\n",
+ "\n",
+ "- [MLAgents (Hugging Face version)](https://github.com/huggingface/ml-agents)"
+ ],
+ "metadata": {
+ "id": "x7oR6R-ZIbeS"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
+ ],
+ "metadata": {
+ "id": "60yACvZwO0Cy"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Objectives of this notebook 🏆\n",
+ "\n",
+ "At the end of the notebook, you will:\n",
+ "\n",
+ "- Understand **the state space, action space and reward function used to train Huggy**.\n",
+ "- **Train your own Huggy** to fetch the stick.\n",
+ "- Be able to play **with your trained Huggy directly in your browser**.\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "Oks-ETYdO2Dc"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Course\n",
+ ""
+ ],
+ "metadata": {
+ "id": "mUlVrqnBv2o1"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "In this free course, you will:\n",
+ "\n",
+ "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- 🧑💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
+ "- 🤖 Train **agents in unique environments** \n",
+ "\n",
+ "And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course\n",
+ "\n",
+ "Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
+ "\n",
+ "\n",
+ "The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
+ ],
+ "metadata": {
+ "id": "pAMjaQpHwB_s"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Prerequisites 🏗️\n",
+ "\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by doing Unit 1\n",
+ "\n",
+ "🔲 📚 **Read the introduction to Huggy** by doing Bonus Unit 1"
+ ],
+ "metadata": {
+ "id": "6r7Hl0uywFSO"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Set the GPU 💪\n",
+ "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
+ "\n",
+ ""
+ ],
+ "metadata": {
+ "id": "DssdIjk_8vZE"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "- `Hardware Accelerator > GPU`\n",
+ "\n",
+ ""
+ ],
+ "metadata": {
+ "id": "sTfCXHy68xBv"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "an3ByrXYQ4iK"
+ },
+ "source": [
+ "## Clone the repository and install the dependencies 🔽\n",
+ "\n",
+ "- We need to clone the repository, that **contains the experimental version of the library that allows you to push your trained agent to the Hub.**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "6WNoL04M7rTa"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "# Clone this specific repository (can take 3min)\n",
+ "!git clone https://github.com/huggingface/ml-agents/"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "d8wmVcMk7xKo"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "# Go inside the repository and install the package (can take 3min)\n",
+ "%cd ml-agents\n",
+ "!pip3 install -e ./ml-agents-envs\n",
+ "!pip3 install -e ./ml-agents"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "HRY5ufKUKfhI"
+ },
+ "source": [
+ "## Download and move the environment zip file in `./trained-envs-executables/linux/`\n",
+ "\n",
+ "- Our environment executable is in a zip file.\n",
+ "- We need to download it and place it to `./trained-envs-executables/linux/`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "C9Ls6_6eOKiA"
+ },
+ "outputs": [],
+ "source": [
+ "!mkdir ./trained-envs-executables\n",
+ "!mkdir ./trained-envs-executables/linux"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1zv3M95ZJTWHUVOWT6ckq_cm98nft8gdF' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1zv3M95ZJTWHUVOWT6ckq_cm98nft8gdF\" -O ./trained-envs-executables/linux/Huggy.zip && rm -rf /tmp/cookies.txt"
+ ],
+ "metadata": {
+ "id": "EB-G-80GsxYN"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jsoZGxr1MIXY"
+ },
+ "source": [
+ "Download the file Huggy.zip from https://drive.google.com/uc?export=download&id=1zv3M95ZJTWHUVOWT6ckq_cm98nft8gdF using `wget`. Check out the full solution to download large files from GDrive [here](https://bcrf.biochem.wisc.edu/2021/02/05/download-google-drive-files-using-wget/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "8FPx0an9IAwO"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!unzip -d ./trained-envs-executables/linux/ ./trained-envs-executables/linux/Huggy.zip"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "nyumV5XfPKzu"
+ },
+ "source": [
+ "Make sure your file is accessible "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "EdFsLJ11JvQf"
+ },
+ "outputs": [],
+ "source": [
+ "!chmod -R 755 ./trained-envs-executables/linux/Huggy"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Let's recap how this environment works\n",
+ "\n",
+ "### The State Space: what Huggy \"perceives.\"\n",
+ "\n",
+ "Huggy doesn't \"see\" his environment. Instead, we provide him information about the environment:\n",
+ "\n",
+ "- The target (stick) position\n",
+ "- The relative position between himself and the target\n",
+ "- The orientation of his legs.\n",
+ "\n",
+ "Given all this information, Huggy **can decide which action to take next to fulfill his goal**.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "### The Action Space: what moves Huggy can do\n",
+ "\n",
+ "\n",
+ "**Joint motors drive huggy legs**. It means that to get the target, Huggy needs to **learn to rotate the joint motors of each of his legs correctly so he can move**.\n",
+ "\n",
+ "### The Reward Function\n",
+ "\n",
+ "The reward function is designed so that **Huggy will fulfill his goal** : fetch the stick.\n",
+ "\n",
+ "Remember that one of the foundations of Reinforcement Learning is the *reward hypothesis*: a goal can be described as the **maximization of the expected cumulative reward**.\n",
+ "\n",
+ "Here, our goal is that Huggy **goes towards the stick but without spinning too much**. Hence, our reward function must translate this goal.\n",
+ "\n",
+ "Our reward function:\n",
+ "\n",
+ "\n",
+ "\n",
+ "- *Orientation bonus*: we **reward him for getting close to the target**.\n",
+ "- *Time penalty*: a fixed-time penalty given at every action to **force him to get to the stick as fast as possible**.\n",
+ "- *Rotation penalty*: we penalize Huggy if **he spins too much and turns too quickly**.\n",
+ "- *Getting to the target reward*: we reward Huggy for **reaching the target**."
+ ],
+ "metadata": {
+ "id": "dYKVj8yUvj55"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Check the Huggy config file\n",
+ "\n",
+ "- In ML-Agents, you define the **training hyperparameters into config.yaml files.**\n",
+ "\n",
+ "- For the scope of this notebook, we're not going to modify the hyperparameters, but if you want to try as an experiment, you should also try to modify some other hyperparameters, Unity provides very [good documentation explaining each of them here](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Training-Configuration-File.md)."
+ ],
+ "metadata": {
+ "id": "NAuEq32Mwvtz"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "- **In the case you want to modify the hyperparameters**, in Google Colab notebook, you can click here to open the config.yaml: `/content/ml-agents/config/ppo/Huggy.yaml`\n",
+ "\n",
+ "\n",
+ "We’re now ready to train our agent 🔥."
+ ],
+ "metadata": {
+ "id": "r9wv5NYGw-05"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "f9fI555bO12v"
+ },
+ "source": [
+ "## Train our agent\n",
+ "\n",
+ "To train our agent, we just need to **launch mlagents-learn and select the executable containing the environment.**\n",
+ "\n",
+ "\n",
+ "\n",
+ "With ML Agents, we run a training script. We define four parameters:\n",
+ "\n",
+ "1. `mlagents-learn `: the path where the hyperparameter config file is.\n",
+ "2. `--env`: where the environment executable is.\n",
+ "3. `--run_id`: the name you want to give to your training run id.\n",
+ "4. `--no-graphics`: to not launch the visualization during the training.\n",
+ "\n",
+ "Train the model and use the `--resume` flag to continue training in case of interruption. \n",
+ "\n",
+ "> It will fail first time when you use `--resume`, try running the block again to bypass the error. \n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "The training will take 30 to 45min depending on your machine (don't forget to **set up a GPU**), go take a ☕️you deserve it 🤗."
+ ],
+ "metadata": {
+ "id": "lN32oWF8zPjs"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "bS-Yh1UdHfzy"
+ },
+ "outputs": [],
+ "source": [
+ "!mlagents-learn ./config/ppo/Huggy.yaml --env=./trained-envs-executables/linux/Huggy/Huggy --run-id=\"Huggy\" --no-graphics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5Vue94AzPy1t"
+ },
+ "source": [
+ "## Push the agent to the 🤗 Hub\n",
+ "\n",
+ "- Now that we trained our agent, we’re **ready to push it to the Hub to be able to play with Huggy on your browser🔥.**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
+ "\n",
+ "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
+ "\n",
+ "\n",
+ "\n",
+ "- Copy the token \n",
+ "- Run the cell below and paste the token"
+ ],
+ "metadata": {
+ "id": "izT6FpgNzZ6R"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "rKt2vsYoK56o"
+ },
+ "outputs": [],
+ "source": [
+ "from huggingface_hub import notebook_login\n",
+ "notebook_login()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ],
+ "metadata": {
+ "id": "ew59mK19zjtN"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Xi0y_VASRzJU"
+ },
+ "source": [
+ "Then, we simply need to run `mlagents-push-to-hf`.\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "And we define 4 parameters:\n",
+ "\n",
+ "1. `--run-id`: the name of the training run id.\n",
+ "2. `--local-dir`: where the agent was saved, it’s results/, so in my case results/First Training.\n",
+ "3. `--repo-id`: the name of the Hugging Face repo you want to create or update. It’s always /\n",
+ "If the repo does not exist **it will be created automatically**\n",
+ "4. `--commit-message`: since HF repos are git repository you need to define a commit message."
+ ],
+ "metadata": {
+ "id": "KK4fPfnczunT"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "dGEFAIboLVc6"
+ },
+ "outputs": [],
+ "source": [
+ "!mlagents-push-to-hf --run-id=\"HuggyTraining\" --local-dir=\"./results/Huggy\" --repo-id=\"ThomasSimonini/ppo-Huggy\" --commit-message=\"Huggy\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Else, if everything worked you should have this at the end of the process(but with a different url 😆) :\n",
+ "\n",
+ "\n",
+ "\n",
+ "```\n",
+ "Your model is pushed to the hub. You can view your model here: https://huggingface.co/ThomasSimonini/ppo-Huggy\n",
+ "```\n",
+ "\n",
+ "It’s the link to your model repository. The repository contains a model card that explains how to use the model, your Tensorboard logs and your config file. **What’s awesome is that it’s a git repository, which means you can have different commits, update your repository with a new push, open Pull Requests, etc.**\n",
+ "\n",
+ ""
+ ],
+ "metadata": {
+ "id": "yborB0850FTM"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "But now comes the best: **being able to play with Huggy online 👀.**"
+ ],
+ "metadata": {
+ "id": "5Uaon2cg0NrL"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Play with your Huggy 🐕\n",
+ "\n",
+ "This step is the simplest:\n",
+ "\n",
+ "- Open the game Huggy in your browser: https://huggingface.co/spaces/ThomasSimonini/Huggy\n",
+ "\n",
+ "- Click on Play with my Huggy model\n",
+ "\n",
+ ""
+ ],
+ "metadata": {
+ "id": "VMc4oOsE0QiZ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "1. In step 1, choose your model repository which is the model id (in my case ThomasSimonini/ppo-Huggy).\n",
+ "\n",
+ "2. In step 2, **choose what model you want to replay**:\n",
+ " - I have multiple ones, since we saved a model every 500000 timesteps. \n",
+ " - But since I want the more recent, I choose `Huggy.onnx`\n",
+ "\n",
+ "👉 What’s nice **is to try with different models steps to see the improvement of the agent.**"
+ ],
+ "metadata": {
+ "id": "Djs8c5rR0Z8a"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Congrats on finishing this bonus unit!\n",
+ "\n",
+ "You can now sit and enjoy playing with your Huggy 🐶. And don't **forget to spread the love by sharing Huggy with your friends 🤗**. And if you share about it on social media, **please tag us @huggingface and me @simoninithomas**\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Keep Learning, Stay awesome 🤗"
+ ],
+ "metadata": {
+ "id": "PI6dPWmh064H"
+ }
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "provenance": [],
+ "private_outputs": true
+ },
+ "gpuClass": "standard",
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/notebooks/unit1/unit1.ipynb b/notebooks/unit1/unit1.ipynb
new file mode 100644
index 0000000..3b58d09
--- /dev/null
+++ b/notebooks/unit1/unit1.ipynb
@@ -0,0 +1,1144 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "njb_ProuHiOe"
+ },
+ "source": [
+ "# Unit 1: Train your first Deep Reinforcement Learning Agent 🤖\n",
+ "\n",
+ "\n",
+ "In this notebook, you'll train your **first Deep Reinforcement Learning agent** a Lunar Lander agent that will learn to **land correctly on the Moon 🌕**. Using [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) a Deep Reinforcement Learning library, share them with the community, and experiment with different configurations\n",
+ "\n",
+ "⬇️ Here is an example of what **you will achieve in just a couple of minutes.** ⬇️\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "PF46MwbZD00b"
+ },
+ "outputs": [],
+ "source": [
+ "%%html\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### The environment 🎮\n",
+ "- [LunarLander-v2](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n",
+ "\n",
+ "### The library used 📚\n",
+ "- [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/)"
+ ],
+ "metadata": {
+ "id": "x7oR6R-ZIbeS"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
+ ],
+ "metadata": {
+ "id": "OwEcFHe9RRZW"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4i6tjI2tHQ8j"
+ },
+ "source": [
+ "## Objectives of this notebook 🏆\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to use **Gym**, the environment library.\n",
+ "- Be able to use **Stable-Baselines3**, the deep reinforcement learning library.\n",
+ "- Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score 🔥.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Course\n",
+ ""
+ ],
+ "metadata": {
+ "id": "Ff-nyJdzJPND"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6p5HnEefISCB"
+ },
+ "source": [
+ "In this free course, you will:\n",
+ "\n",
+ "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- 🧑💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
+ "- 🤖 Train **agents in unique environments** \n",
+ "\n",
+ "And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course\n",
+ "\n",
+ "Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
+ "\n",
+ "\n",
+ "The best way to keep in touch and ask questions is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Y-mo_6rXIjRi"
+ },
+ "source": [
+ "## Prerequisites 🏗️\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "🔲 📝 **Done Unit 0** that gives you all the **information about the course and help you to onboard** 🤗 ADD LINK \n",
+ "\n",
+ "🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by doing Unit 1 👉 ADD LINK"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## A small recap of what is Deep Reinforcement Learning 📚\n",
+ ""
+ ],
+ "metadata": {
+ "id": "HoeqMnr5LuYE"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xcQYx9ynaFMD"
+ },
+ "source": [
+ "Let's do a small recap on what we learned in the first Unit:\n",
+ "- Reinforcement Learning is a **computational approach to learning from action**. We build an agent that learns from the environment by **interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.\n",
+ "\n",
+ "- The goal of any RL agent is to **maximize its expected cumulative reward** (also called expected return) because RL is based on the _reward hypothesis_, which is that all goals can be described as the maximization of the expected cumulative reward.\n",
+ "\n",
+ "- The RL process is a **loop that outputs a sequence of state, action, reward, and next state**.\n",
+ "\n",
+ "- To calculate the expected cumulative reward (expected return), **we discount the rewards**: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long-term future reward.\n",
+ "\n",
+ "- To solve an RL problem, you want to **find an optimal policy**; the policy is the \"brain\" of your AI that will tell us what action to take given a state. The optimal one is the one that gives you the actions that max the expected return.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8X2WN2X99BEz"
+ },
+ "source": [
+ "There are **two** ways to find your optimal policy:\n",
+ "- By **training your policy directly**: policy-based methods.\n",
+ "- By **training a value function** that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BPy6dfcm8_WT"
+ },
+ "source": [
+ "- Finally, we spoke about Deep RL because **we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name \"deep.\"**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LzGDDOTSKaF8"
+ },
+ "source": [
+ "# Let's train our first Deep Reinforcement Learning agent and upload it to the Hub 🚀\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Set the GPU 💪\n",
+ "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
+ "\n",
+ ""
+ ],
+ "metadata": {
+ "id": "HqzznTzhNfAC"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "38HBd3t1SHJ8"
+ },
+ "source": [
+ "- `Hardware Accelerator > GPU`\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jeDAH0h0EBiG"
+ },
+ "source": [
+ "## Install dependencies and create a virtual screen 🔽\n",
+ "The first step is to install the dependencies, we’ll install multiple ones.\n",
+ "\n",
+ "- `gym[box2D]`: Contains the LunarLander-v2 environment 🌛 (we use `gym==0.21`)\n",
+ "- `stable-baselines3[extra]`: The deep reinforcement learning library.\n",
+ "- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face 🤗 Hub.\n",
+ "\n",
+ "To make things easier, we created a script to install all these dependencies."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!apt install swig cmake"
+ ],
+ "metadata": {
+ "id": "yQIGLPDkGhgG"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO CHANGE LINK OF THE REQUIREMENTS"
+ ],
+ "metadata": {
+ "id": "32e3NPYgH5ET"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "9XaULfDZDvrC"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -r https://huggingface.co/spaces/ThomasSimonini/temp-space-requirements/raw/main/requirements/requirements-unit1.txt"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install virtual screen libraries and create and run a virtual screen 🖥"
+ ],
+ "metadata": {
+ "id": "BEKeXQJsQCYm"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!sudo apt-get update\n",
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip3 install pyvirtualdisplay"
+ ],
+ "metadata": {
+ "id": "j5f2cGkdP-mb"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To make sure the new installed libraries are used, **sometimes it's required to restart the notebook runtime**. The next cell will force the **runtime to crash, so you'll need to connect again and run the code starting from here**. Thanks for this trick, **we will be able to run our virtual screen.**"
+ ],
+ "metadata": {
+ "id": "TCwBTAwAW9JJ"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "os.kill(os.getpid(), 9)"
+ ],
+ "metadata": {
+ "id": "cYvkbef7XEMi"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "BE5JWP5rQIKf"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "wrgpVFqyENVf"
+ },
+ "source": [
+ "## Import the packages 📦\n",
+ "\n",
+ "One additional library we import is huggingface_hub **to be able to upload and download trained models from the hub**.\n",
+ "\n",
+ "\n",
+ "The Hugging Face Hub 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.\n",
+ "\n",
+ "You can see here all the Deep reinforcement Learning models available 👉 https://huggingface.co/models?pipeline_tag=reinforcement-learning&sort=downloads\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "cygWLPGsEQ0m"
+ },
+ "outputs": [],
+ "source": [
+ "import gym\n",
+ "\n",
+ "from huggingface_sb3 import load_from_hub, package_to_hub, push_to_hub\n",
+ "from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
+ "\n",
+ "from stable_baselines3 import PPO\n",
+ "from stable_baselines3.common.evaluation import evaluate_policy\n",
+ "from stable_baselines3.common.env_util import make_vec_env"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MRqRuRUl8CsB"
+ },
+ "source": [
+ "## Understand what is Gym and how it works 🤖\n",
+ "\n",
+ "🏋 The library containing our environment is called Gym.\n",
+ "**You'll use Gym a lot in Deep Reinforcement Learning.**\n",
+ "\n",
+ "The Gym library provides two things:\n",
+ "- An interface that allows you to **create RL environments**.\n",
+ "- A **collection of environments** (gym-control, atari, box2D...).\n",
+ "\n",
+ "Let's look at an example, but first let's remember what's the RL Loop.\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-TzNN0bQ_j-3"
+ },
+ "source": [
+ "At each step:\n",
+ "- Our Agent receives **state S0** from the **Environment** — we receive the first frame of our game (Environment).\n",
+ "- Based on that **state S0,** the Agent takes **action A0** — our Agent will move to the right.\n",
+ "- Environment to a **new** **state S1** — new frame.\n",
+ "- The environment gives some **reward R1** to the Agent — we’re not dead *(Positive Reward +1)*.\n",
+ "\n",
+ "\n",
+ "With Gym:\n",
+ "\n",
+ "1️⃣ We create our environment using `gym.make()`\n",
+ "\n",
+ "2️⃣ We reset the environment to its initial state with `observation = env.reset()`\n",
+ "\n",
+ "At each step:\n",
+ "\n",
+ "3️⃣ Get an action using our model (in our example we take a random action)\n",
+ "\n",
+ "4️⃣ Using `env.step(action)`, we perform this action in the environment and get\n",
+ "- `observation`: The new state (st+1)\n",
+ "- `reward`: The reward we get after executing the action\n",
+ "- `done`: Indicates if the episode terminated\n",
+ "- `info`: A dictionary that provides additional information (depends on the environment).\n",
+ "\n",
+ "If the episode is done:\n",
+ "- We reset the environment to its initial state with `observation = env.reset()`\n",
+ "\n",
+ "**Let's look at an example!** Make sure to read the code\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "w7vOFlpA_ONz"
+ },
+ "outputs": [],
+ "source": [
+ "import gym\n",
+ "\n",
+ "# First, we create our environment called LunarLander-v2\n",
+ "env = gym.make(\"LunarLander-v2\")\n",
+ "\n",
+ "# Then we reset this environment\n",
+ "observation = env.reset()\n",
+ "\n",
+ "for _ in range(20):\n",
+ " # Take a random action\n",
+ " action = env.action_space.sample()\n",
+ " print(\"Action taken:\", action)\n",
+ "\n",
+ " # Do this action in the environment and get\n",
+ " # next_state, reward, done and info\n",
+ " observation, reward, done, info = env.step(action)\n",
+ " \n",
+ " # If the game is done (in our case we land, crashed or timeout)\n",
+ " if done:\n",
+ " # Reset the environment\n",
+ " print(\"Environment is reset\")\n",
+ " observation = env.reset()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "XIrKGGSlENZB"
+ },
+ "source": [
+ "## Create the LunarLander environment 🌛 and understand how it works\n",
+ "### [The environment 🎮](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n",
+ "In this first tutorial, we’re going to train our agent, a [Lunar Lander](https://www.gymlibrary.dev/environments/box2d/lunar_lander/), **to land correctly on the moon**. To do that, the agent needs to learn **to adapt its speed and position(horizontal, vertical, and angular) to land correctly.**\n",
+ "\n",
+ "\n",
+ "---\n",
+ "\n",
+ "\n",
+ "💡 A good habit when you start to use an environment is to check its documentation \n",
+ "\n",
+ "👉 https://www.gymlibrary.dev/environments/box2d/lunar_lander/\n",
+ "\n",
+ "---\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "poLBgRocF9aT"
+ },
+ "source": [
+ "Let's see what the Environment looks like:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZNPG0g_UGCfh"
+ },
+ "outputs": [],
+ "source": [
+ "# We create our environment with gym.make(\"\")\n",
+ "env = gym.make(\"LunarLander-v2\")\n",
+ "env.reset()\n",
+ "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
+ "print(\"Observation Space Shape\", env.observation_space.shape)\n",
+ "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2MXc15qFE0M9"
+ },
+ "source": [
+ "We see with `Observation Space Shape (8,)` that the observation is a vector of size 8, where each value contains different information about the lander:\n",
+ "- Horizontal pad coordinate (x)\n",
+ "- Vertical pad coordinate (y)\n",
+ "- Horizontal speed (x)\n",
+ "- Vertical speed (y)\n",
+ "- Angle\n",
+ "- Angular speed\n",
+ "- If the left leg has contact point touched the land\n",
+ "- If the right leg has contact point touched the land\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "We5WqOBGLoSm"
+ },
+ "outputs": [],
+ "source": [
+ "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
+ "print(\"Action Space Shape\", env.action_space.n)\n",
+ "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MyxXwkI2Magx"
+ },
+ "source": [
+ "The action space (the set of possible actions the agent can take) is discrete with 4 actions available 🎮: \n",
+ "\n",
+ "- Do nothing,\n",
+ "- Fire left orientation engine,\n",
+ "- Fire the main engine,\n",
+ "- Fire right orientation engine.\n",
+ "\n",
+ "Reward function (the function that will gives a reward at each timestep) 💰:\n",
+ "\n",
+ "- Moving from the top of the screen to the landing pad and zero speed is about 100~140 points.\n",
+ "- Firing main engine is -0.3 each frame\n",
+ "- Each leg ground contact is +10 points\n",
+ "- Episode finishes if the lander crashes (additional - 100 points) or come to rest (+100 points)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "dFD9RAFjG8aq"
+ },
+ "source": [
+ "#### Vectorized Environment\n",
+ "- We create a vectorized environment (method for stacking multiple independent environments into a single environment) of 16 environments, this way, **we'll have more diverse experiences during the training.**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "99hqQ_etEy1N"
+ },
+ "outputs": [],
+ "source": [
+ "# Create the environment\n",
+ "env = make_vec_env('LunarLander-v2', n_envs=16)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VgrE86r5E5IK"
+ },
+ "source": [
+ "## Create the Model 🤖\n",
+ "- Now that we studied our environment and we understood the problem: **being able to land correctly the Lunar Lander to the Landing Pad by controlling left, right and main orientation engine**. Let's build the algorithm we're going to use to solve this Problem 🚀.\n",
+ "\n",
+ "- To do so, we're going to use our first Deep RL library, [Stable Baselines3 (SB3)](https://stable-baselines3.readthedocs.io/en/master/).\n",
+ "\n",
+ "- SB3 is a set of **reliable implementations of reinforcement learning algorithms in PyTorch**.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "💡 A good habit when using a new library is to dive first on the documentation: https://stable-baselines3.readthedocs.io/en/master/ and then try some tutorials.\n",
+ "\n",
+ "----"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "3wgg-y0-92Wj"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "HV4yiUM_9_Ka"
+ },
+ "source": [
+ "To solve this problem, we're going to use SB3 **PPO**. [PPO (aka Proximal Policy Optimization) is one of the of the SOTA (state of the art) Deep Reinforcement Learning algorithms that you'll study during this course](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#example%5D).\n",
+ "\n",
+ "PPO is a combination of:\n",
+ "- *Value-based reinforcement learning method*: learning an action-value function that will tell us what's the **most valuable action to take given a state and action**.\n",
+ "- *Policy-based reinforcement learning method*: learning a policy that will **gives us a probability distribution over actions**.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5qL_4HeIOrEJ"
+ },
+ "source": [
+ "Stable-Baselines3 is easy to set up:\n",
+ "\n",
+ "1️⃣ You **create your environment** (in our case it was done above)\n",
+ "\n",
+ "2️⃣ You define the **model you want to use and instantiate this model** `model = PPO(\"MlpPolicy\")`\n",
+ "\n",
+ "3️⃣ You **train the agent** with `model.learn` and define the number of training timesteps\n",
+ "\n",
+ "```\n",
+ "# Create environment\n",
+ "env = gym.make('LunarLander-v2')\n",
+ "\n",
+ "# Instantiate the agent\n",
+ "model = PPO('MlpPolicy', env, verbose=1)\n",
+ "# Train the agent\n",
+ "model.learn(total_timesteps=int(2e5))\n",
+ "```\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "nxI6hT1GE4-A"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO: Define a PPO MlpPolicy architecture\n",
+ "# We use MultiLayerPerceptron (MLPPolicy) because the input is a vector,\n",
+ "# if we had frames as input we would use CnnPolicy\n",
+ "model = "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "QAN7B0_HCVZC"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "543OHYDfcjK4"
+ },
+ "outputs": [],
+ "source": [
+ "# SOLUTION\n",
+ "# We added some parameters to accelerate the training\n",
+ "model = PPO(\n",
+ " policy = 'MlpPolicy',\n",
+ " env = env,\n",
+ " n_steps = 1024,\n",
+ " batch_size = 64,\n",
+ " n_epochs = 4,\n",
+ " gamma = 0.999,\n",
+ " gae_lambda = 0.98,\n",
+ " ent_coef = 0.01,\n",
+ " verbose=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ClJJk88yoBUi"
+ },
+ "source": [
+ "## Train the PPO agent 🏃\n",
+ "- Let's train our agent for 1,000,000 timesteps, don't forget to use GPU on Colab. It will take approximately ~20min, but you can use less timesteps if you just want to try it out.\n",
+ "- During the training, take a ☕ break you deserved it 🤗"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "qKnYkNiVp89p"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO: Train it for 1,000,000 timesteps\n",
+ "\n",
+ "# TODO: Specify file name for model and save the model to file\n",
+ "model_name = \"\"\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1bQzQ-QcE3zo"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "poBCy9u_csyR"
+ },
+ "outputs": [],
+ "source": [
+ "# SOLUTION\n",
+ "# Train it for 1,000,000 timesteps\n",
+ "model.learn(total_timesteps=1000000)\n",
+ "# Save the model\n",
+ "model_name = \"ppo-LunarLander-v2\"\n",
+ "model.save(model_name)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BY_HuedOoISR"
+ },
+ "source": [
+ "## Evaluate the agent 📈\n",
+ "- Now that our Lunar Lander agent is trained 🚀, we need to **check its performance**.\n",
+ "- Stable-Baselines3 provides a method to do that: `evaluate_policy`.\n",
+ "- To fill that part you need to [check the documentation](https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#basic-usage-training-saving-loading)\n",
+ "- In the next step, we'll see **how to automatically evaluate and share your agent to compete in a leaderboard, but for now let's do it ourselves**\n",
+ "\n",
+ "\n",
+ "💡 When you evaluate your agent, you should not use your training environment but create an evaluation environment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "yRpno0glsADy"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO: Evaluate the agent\n",
+ "# Create a new environment for evaluation\n",
+ "eval_env =\n",
+ "\n",
+ "# Evaluate the model with 10 evaluation episodes and deterministic=True\n",
+ "mean_reward, std_reward = \n",
+ "\n",
+ "# Print the results\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BqPKw3jt_pG5"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "zpz8kHlt_a_m"
+ },
+ "outputs": [],
+ "source": [
+ "#@title\n",
+ "eval_env = gym.make(\"LunarLander-v2\")\n",
+ "mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)\n",
+ "print(f\"mean_reward={mean_reward:.2f} +/- {std_reward}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "reBhoODwcXfr"
+ },
+ "source": [
+ "- In my case, I got a mean reward is `200.20 +/- 20.80` after training for 1 million steps, which means that our lunar lander agent is ready to land on the moon 🌛🥳."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IK_kR78NoNb2"
+ },
+ "source": [
+ "## Publish our trained model on the Hub 🔥\n",
+ "Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
+ "\n",
+ "📚 The libraries documentation 👉 https://github.com/huggingface/huggingface_sb3/tree/main#hugging-face--x-stable-baselines3-v20\n",
+ "\n",
+ "Here's an example of a Model Card (with Space Invaders):"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Gs-Ew7e1gXN3"
+ },
+ "source": [
+ "By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
+ "\n",
+ "This way:\n",
+ "- You can **showcase our work** 🔥\n",
+ "- You can **visualize your agent playing** 👀\n",
+ "- You can **share with the community an agent that others can use** 💾\n",
+ "- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "JquRrWytA6eo"
+ },
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
+ "\n",
+ "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
+ "\n",
+ "\n",
+ "\n",
+ "- Copy the token \n",
+ "- Run the cell below and paste the token"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GZiFBBlzxzxY"
+ },
+ "outputs": [],
+ "source": [
+ "notebook_login()\n",
+ "!git config --global credential.helper store"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_tsf2uv0g_4p"
+ },
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FGNh9VsZok0i"
+ },
+ "source": [
+ "3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using `package_to_hub()` function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Ay24l6bqFF18"
+ },
+ "source": [
+ "Let's fill the `package_to_hub` function:\n",
+ "- `model`: our trained model.\n",
+ "- `model_name`: the name of the trained model that we defined in `model_save`\n",
+ "- `model_architecture`: the model architecture we used: in our case PPO\n",
+ "- `env_id`: the name of the environment, in our case `LunarLander-v2`\n",
+ "- `eval_env`: the evaluation environment defined in eval_env\n",
+ "- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `(repo_id = {username}/{repo_name})`\n",
+ "\n",
+ "💡 **A good name is {username}/{model_architecture}-{env_id}**\n",
+ "\n",
+ "- `commit_message`: message of the commit"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "JPG7ofdGIHN8"
+ },
+ "outputs": [],
+ "source": [
+ "import gym\n",
+ "from stable_baselines3.common.vec_env import DummyVecEnv\n",
+ "from stable_baselines3.common.env_util import make_vec_env\n",
+ "\n",
+ "from huggingface_sb3 import package_to_hub\n",
+ "\n",
+ "## TODO: Define a repo_id\n",
+ "## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ "repo_id = \n",
+ "\n",
+ "# TODO: Define the name of the environment\n",
+ "env_id = \n",
+ "\n",
+ "# Create the evaluation env\n",
+ "eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
+ "\n",
+ "\n",
+ "# TODO: Define the model architecture we used\n",
+ "model_architecture = \"\"\n",
+ "\n",
+ "## TODO: Define the commit message\n",
+ "commit_message = \"\"\n",
+ "\n",
+ "# method save, evaluate, generate a model card and record a replay video of your agent before pushing the repo to the hub\n",
+ "package_to_hub(model=model, # Our trained model\n",
+ " model_name=model_name, # The name of our trained model \n",
+ " model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
+ " env_id=env_id, # Name of the environment\n",
+ " eval_env=eval_env, # Evaluation Environment\n",
+ " repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ " commit_message=commit_message)\n",
+ "\n",
+ "# Note: if after running the package_to_hub function and it gives an issue of rebasing, please run the following code\n",
+ "# cd && git add . && git commit -m \"Add message\" && git pull \n",
+ "# And don't forget to do a \"git push\" at the end to push the change to the hub."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Avf6gufJBGMw"
+ },
+ "source": [
+ "#### Solution\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "xMkkkukIBQJM"
+ },
+ "outputs": [],
+ "source": [
+ "import gym\n",
+ "\n",
+ "from stable_baselines3 import PPO\n",
+ "from stable_baselines3.common.vec_env import DummyVecEnv\n",
+ "from stable_baselines3.common.env_util import make_vec_env\n",
+ "\n",
+ "from huggingface_sb3 import package_to_hub\n",
+ "\n",
+ "# PLACE the variables you've just defined two cells above\n",
+ "# Define the name of the environment\n",
+ "env_id = \"LunarLander-v2\"\n",
+ "\n",
+ "# TODO: Define the model architecture we used\n",
+ "model_architecture = \"PPO\"\n",
+ "\n",
+ "## Define a repo_id\n",
+ "## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ "## CHANGE WITH YOUR REPO ID\n",
+ "repo_id = \"ThomasSimonini/ppo-LunarLander-v2\" # Change with your repo id, you can't push with mine 😄\n",
+ "\n",
+ "## Define the commit message\n",
+ "commit_message = \"Upload PPO LunarLander-v2 trained agent\"\n",
+ "\n",
+ "# Create the evaluation env\n",
+ "eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
+ "\n",
+ "# PLACE the package_to_hub function you've just filled here\n",
+ "package_to_hub(model=model, # Our trained model\n",
+ " model_name=model_name, # The name of our trained model \n",
+ " model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
+ " env_id=env_id, # Name of the environment\n",
+ " eval_env=eval_env, # Evaluation Environment\n",
+ " repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ " commit_message=commit_message)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pN0-W6kkKn44"
+ },
+ "source": [
+ "Congrats 🥳 you've just trained and uploaded your first Deep Reinforcement Learning agent. The script above should have displayed a link to a model repository such as https://huggingface.co/osanseviero/test_sb3. When you go to this link, you can:\n",
+ "* see a video preview of your agent at the right. \n",
+ "* click \"Files and versions\" to see all the files in the repository.\n",
+ "* click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
+ "* a model card (`README.md` file) which gives a description of the model\n",
+ "\n",
+ "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
+ "\n",
+ "Compare the results of your LunarLander-v2 with your classmates using the leaderboard 🏆 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9nWnuQHRfFRa"
+ },
+ "source": [
+ "## Load a saved LunarLander model from the Hub 🤗\n",
+ "Thanks to [ironbar](https://github.com/ironbar) for the contribution.\n",
+ "\n",
+ "Loading a saved model from the Hub is really easy. \n",
+ "\n",
+ "You go https://huggingface.co/models?library=stable-baselines3 to see the list of all the Stable-baselines3 saved models.\n",
+ "1. You select one and copy its repo_id\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "hNPLJF2bfiUw"
+ },
+ "source": [
+ "2. Then we just need to use load_from_hub with:\n",
+ "- The repo_id\n",
+ "- The filename: the saved model inside the repo and its extension (*.zip)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "oj8PSGHJfwz3"
+ },
+ "outputs": [],
+ "source": [
+ "from huggingface_sb3 import load_from_hub\n",
+ "repo_id = \"Classroom-workshop/assignment2-omar\" # The repo_id\n",
+ "filename = \"ppo-LunarLander-v2.zip\" # The model filename.zip\n",
+ "\n",
+ "# When the model was trained on Python 3.8 the pickle protocol is 5\n",
+ "# But Python 3.6, 3.7 use protocol 4\n",
+ "# In order to get compatibility we need to:\n",
+ "# 1. Install pickle5 (we done it at the beginning of the colab)\n",
+ "# 2. Create a custom empty object we pass as parameter to PPO.load()\n",
+ "custom_objects = {\n",
+ " \"learning_rate\": 0.0,\n",
+ " \"lr_schedule\": lambda _: 0.0,\n",
+ " \"clip_range\": lambda _: 0.0,\n",
+ "}\n",
+ "\n",
+ "checkpoint = load_from_hub(repo_id, filename)\n",
+ "model = PPO.load(checkpoint, custom_objects=custom_objects, print_system_info=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Fs0Y-qgPgLUf"
+ },
+ "source": [
+ "Let's evaluate this agent:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "#@title\n",
+ "eval_env = gym.make(\"LunarLander-v2\")\n",
+ "mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)\n",
+ "print(f\"mean_reward={mean_reward:.2f} +/- {std_reward}\")"
+ ],
+ "metadata": {
+ "id": "PAEVwK-aahfx"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BQAwLnYFPk-s"
+ },
+ "source": [
+ "## Some additional challenges 🏆\n",
+ "The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
+ "\n",
+ "In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
+ "\n",
+ "Here are some ideas to achieve so:\n",
+ "* Train more steps\n",
+ "* Try different hyperparameters of `PPO`. You can see them at https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#parameters. \n",
+ "* Check the [Stable-Baselines3 documentation](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) and try another models such as DQN.\n",
+ "* **Push your new trained model** on the Hub 🔥\n",
+ "\n",
+ "**Compare the results of your LunarLander-v2 with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) 🏆\n",
+ "\n",
+ "Is moon landing too boring to you? Try to **change the environment**, why not using MountainCar-v0, CartPole-v1 or CarRacing-v0? Check how they works [using the gym documentation](https://www.gymlibrary.dev/) and have fun 🎉."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9lM95-dvmif8"
+ },
+ "source": [
+ "________________________________________________________________________\n",
+ "Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.**\n",
+ "\n",
+ "If you’re still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
+ "\n",
+ "Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations.\n",
+ "\n",
+ "Naturally, during the course, we’re going to use and deeper explain again these terms but **it’s better to have a good understanding of them now before diving into the next chapters.**\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BjLhT70TEZIn"
+ },
+ "source": [
+ "Next time, in the bonus unit 1, you'll train Huggy the Dog to fetch the stick.\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Keep learning, stay awesome 🤗"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "private_outputs": true,
+ "provenance": [],
+ "collapsed_sections": [
+ "QAN7B0_HCVZC",
+ "1bQzQ-QcE3zo",
+ "BqPKw3jt_pG5",
+ "Avf6gufJBGMw"
+ ]
+ },
+ "gpuClass": "standard",
+ "kernelspec": {
+ "display_name": "Python 3.9.7",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.9.7"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "ed7f8024e43d3b8f5ca3c5e1a8151ab4d136b3ecee1e3fd59e0766ccc55e1b10"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/notebooks/unit2/unit2.ipynb b/notebooks/unit2/unit2.ipynb
new file mode 100644
index 0000000..cf97cdc
--- /dev/null
+++ b/notebooks/unit2/unit2.ipynb
@@ -0,0 +1,1743 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "njb_ProuHiOe"
+ },
+ "source": [
+ "# Unit 2: Q-Learning with FrozenLake-v1 ⛄ and Taxi-v3 🚕\n",
+ "\n",
+ "\n",
+ "\n",
+ "In this notebook, **you'll code from scratch your first Reinforcement Learning agent** playing FrozenLake ❄️ using Q-Learning, share it to the community, and experiment with different configurations.\n",
+ "\n",
+ "\n",
+ "⬇️ Here is an example of what **you will achieve in just a couple of minutes.** ⬇️\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "vRU_vXBrl1Jx"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO: ADD TEXT LIVE INFO"
+ ],
+ "metadata": {
+ "id": "yaBKcncmYku4"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO: ADD IF YOU HAVE QUESTIONS\n"
+ ],
+ "metadata": {
+ "id": "hz5KE5HjYlRh"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "###🎮 Environments: \n",
+ "- [FrozenLake-v1](https://www.gymlibrary.dev/environments/toy_text/frozen_lake/)\n",
+ "- [Taxi-v3](https://www.gymlibrary.dev/environments/toy_text/taxi/)\n",
+ "\n",
+ "###📚 RL-Library: \n",
+ "- Python and Numpy"
+ ],
+ "metadata": {
+ "id": "DPTBOv9HYLZ2"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
+ ],
+ "metadata": {
+ "id": "3iaIxM_TwklQ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4i6tjI2tHQ8j"
+ },
+ "source": [
+ "## Objectives of this notebook 🏆\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to use **Gym**, the environment library.\n",
+ "- Be able to code from scratch a Q-Learning agent.\n",
+ "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Course\n",
+ ""
+ ],
+ "metadata": {
+ "id": "viNzVbVaYvY3"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6p5HnEefISCB"
+ },
+ "source": [
+ "In this free course, you will:\n",
+ "\n",
+ "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- 🧑💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
+ "- 🤖 Train **agents in unique environments** \n",
+ "\n",
+ "And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course\n",
+ "\n",
+ "Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
+ "\n",
+ "\n",
+ "The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Y-mo_6rXIjRi"
+ },
+ "source": [
+ "## Prerequisites 🏗️\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "🔲 📚 **Study Q-Learning by reading Unit 2** 🤗 ADD LINK "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "f2ONOODsyrMU"
+ },
+ "source": [
+ "## A small recap of Q-Learning"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "V68VveLacfxJ"
+ },
+ "source": [
+ "- The *Q-Learning* **is the RL algorithm that** \n",
+ " - Trains *Q-Function*, an **action-value function** that contains, as internal memory, a *Q-table* **that contains all the state-action pair values.**\n",
+ " \n",
+ " - Given a state and action, our Q-Function **will search into its Q-table the corresponding value.**\n",
+ " \n",
+ "\n",
+ "\n",
+ "- When the training is done,**we have an optimal Q-Function, so an optimal Q-Table.**\n",
+ " \n",
+ "- And if we **have an optimal Q-function**, we\n",
+ "have an optimal policy,since we **know for each state, what is the best action to take.**\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "But, in the beginning, our **Q-Table is useless since it gives arbitrary value for each state-action pair (most of the time we initialize the Q-Table to 0 values)**. But, as we’ll explore the environment and update our Q-Table it will give us better and better approximations\n",
+ "\n",
+ "\n",
+ "\n",
+ "This is the Q-Learning pseudocode:\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Let's code our first Reinforcement Learning algorithm 🚀"
+ ],
+ "metadata": {
+ "id": "HEtx8Y8MqKfH"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Install dependencies and create a virtual display 🔽\n",
+ "\n",
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install the librairies and create and run a virtual screen 🖥\n",
+ "\n",
+ "We’ll install multiple ones:\n",
+ "\n",
+ "- `gym`: Contains the FrozenLake-v1 ⛄ and Taxi-v3 🚕 environments. We use `gym==0.24` since it contains a nice Taxi-v3 UI version.\n",
+ "- `pygame`: Used for the FrozenLake-v1 and Taxi-v3 UI.\n",
+ "- `numPy`: Used for handling our Q-table.\n",
+ "\n",
+ "The Hugging Face Hub 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.\n",
+ "\n",
+ "You can see here all the Deep reinforcement Learning models available 👉 https://huggingface.co/models?other=q-learning\n"
+ ],
+ "metadata": {
+ "id": "4gpxC1_kqUYe"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO CHANGE LINK OF THE REQUIREMENTS"
+ ],
+ "metadata": {
+ "id": "32e3NPYgH5ET"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "9XaULfDZDvrC"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -r https://huggingface.co/spaces/ThomasSimonini/temp-space-requirements/raw/main/requirements/requirements-unit2.txt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%capture\n",
+ "!sudo apt-get update\n",
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip3 install pyvirtualdisplay"
+ ],
+ "metadata": {
+ "id": "n71uTX7qqzz2"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "DaY1N4dBrabi"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "W-7f-Swax_9x"
+ },
+ "source": [
+ "## Import the packages 📦\n",
+ "\n",
+ "In addition to the installed libraries, we also use:\n",
+ "\n",
+ "- `random`: To generate random numbers (that will be useful for Epsilon-Greedy Policy).\n",
+ "- `imageio`: To generate a replay video\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "VcNvOAQlysBJ"
+ },
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import gym\n",
+ "import random\n",
+ "import imageio\n",
+ "import os\n",
+ "\n",
+ "import pickle5 as pickle\n",
+ "from tqdm.notebook import tqdm"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xp4-bXKIy1mQ"
+ },
+ "source": [
+ "We're now ready to code our Q-Learning algorithm 🔥"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xya49aNJWVvv"
+ },
+ "source": [
+ "# Part 1: Frozen Lake ⛄ (non slippery version)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "NAvihuHdy9tw"
+ },
+ "source": [
+ "## Create and understand [FrozenLake environment ⛄]((https://www.gymlibrary.dev/environments/toy_text/frozen_lake/)\n",
+ "---\n",
+ "\n",
+ "💡 A good habit when you start to use an environment is to check its documentation \n",
+ "\n",
+ "👉 https://www.gymlibrary.dev/environments/toy_text/frozen_lake/\n",
+ "\n",
+ "---\n",
+ "\n",
+ "We're going to train our Q-Learning agent **to navigate from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoid holes (H)**.\n",
+ "\n",
+ "We can have two sizes of environment:\n",
+ "- `map_name=\"4x4\"`: a 4x4 grid version\n",
+ "- `map_name=\"8x8\"`: a 8x8 grid version\n",
+ "\n",
+ "\n",
+ "The environment has two modes:\n",
+ "- `is_slippery=False`: The agent always move in the intended direction due to the non-slippery nature of the frozen lake.\n",
+ "- `is_slippery=True`: The agent may not always move in the intended direction due to the slippery nature of the frozen lake (stochastic)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UaW_LHfS0PY2"
+ },
+ "source": [
+ "For now let's keep it simple with the 4x4 map and non-slippery"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "IzJnb8O3y8up"
+ },
+ "outputs": [],
+ "source": [
+ "# Create the FrozenLake-v1 environment using 4x4 map and non-slippery version\n",
+ "env = gym.make() # TODO use the correct parameters"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Ji_UrI5l2zzn"
+ },
+ "source": [
+ "### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "jNxUbPMP0akP"
+ },
+ "outputs": [],
+ "source": [
+ "env = gym.make(\"FrozenLake-v1\", map_name=\"4x4\", is_slippery=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "KASNViqL4tZn"
+ },
+ "source": [
+ "You can create your own custom grid like this:\n",
+ "\n",
+ "```python\n",
+ "desc=[\"SFFF\", \"FHFH\", \"FFFH\", \"HFFG\"]\n",
+ "gym.make('FrozenLake-v1', desc=desc, is_slippery=True)\n",
+ "```\n",
+ "\n",
+ "but we'll use the default environment for now."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SXbTfdeJ1Xi9"
+ },
+ "source": [
+ "### Let's see what the Environment looks like:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZNPG0g_UGCfh"
+ },
+ "outputs": [],
+ "source": [
+ "# We create our environment with gym.make(\"\")\n",
+ "env.reset()\n",
+ "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
+ "print(\"Observation Space\", env.observation_space)\n",
+ "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2MXc15qFE0M9"
+ },
+ "source": [
+ "We see with `Observation Space Shape Discrete(16)` that the observation is a value representing the **agent’s current position as current_row * nrows + current_col (where both the row and col start at 0)**. \n",
+ "\n",
+ "For example, the goal position in the 4x4 map can be calculated as follows: 3 * 4 + 3 = 15. The number of possible observations is dependent on the size of the map. **For example, the 4x4 map has 16 possible observations.**\n",
+ "\n",
+ "\n",
+ "For instance, this is what state = 0 looks like:\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "We5WqOBGLoSm"
+ },
+ "outputs": [],
+ "source": [
+ "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
+ "print(\"Action Space Shape\", env.action_space.n)\n",
+ "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MyxXwkI2Magx"
+ },
+ "source": [
+ "The action space (the set of possible actions the agent can take) is discrete with 4 actions available 🎮:\n",
+ "- 0: GO LEFT\n",
+ "- 1: GO DOWN\n",
+ "- 2: GO RIGHT\n",
+ "- 3: GO UP\n",
+ "\n",
+ "Reward function 💰:\n",
+ "- Reach goal: +1\n",
+ "- Reach hole: 0\n",
+ "- Reach frozen: 0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1pFhWblk3Awr"
+ },
+ "source": [
+ "## Create and Initialize the Q-table 🗄️\n",
+ "(👀 Step 1 of the pseudocode)\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "It's time to initialize our Q-table! To know how many rows (states) and columns (actions) to use, we need to know the action and observation space. OpenAI Gym provides us a way to do that: `env.action_space.n` and `env.observation_space.n`\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "y3ZCdluj3k0l"
+ },
+ "outputs": [],
+ "source": [
+ "state_space = \n",
+ "print(\"There are \", state_space, \" possible states\")\n",
+ "\n",
+ "action_space = \n",
+ "print(\"There are \", action_space, \" possible actions\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "rCddoOXM3UQH"
+ },
+ "outputs": [],
+ "source": [
+ "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros\n",
+ "def initialize_q_table(state_space, action_space):\n",
+ " Qtable = \n",
+ " return Qtable"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "9YfvrqRt3jdR"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_frozenlake = initialize_q_table(state_space, action_space)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "67OdoKL63eDD"
+ },
+ "source": [
+ "### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "HuTKv3th3ohG"
+ },
+ "outputs": [],
+ "source": [
+ "state_space = env.observation_space.n\n",
+ "print(\"There are \", state_space, \" possible states\")\n",
+ "\n",
+ "action_space = env.action_space.n\n",
+ "print(\"There are \", action_space, \" possible actions\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "lnrb_nX33fJo"
+ },
+ "outputs": [],
+ "source": [
+ "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros\n",
+ "def initialize_q_table(state_space, action_space):\n",
+ " Qtable = np.zeros((state_space, action_space))\n",
+ " return Qtable"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Y0WlgkVO3Jf9"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_frozenlake = initialize_q_table(state_space, action_space)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "flILKhBU3yZ7"
+ },
+ "source": [
+ "##Define the epsilon-greedy policy 🤖\n",
+ "\n",
+ "Epsilon-Greedy is the training policy that handles the exploration/exploitation trade-off.\n",
+ "\n",
+ "The idea with Epsilon Greedy:\n",
+ "\n",
+ "- With *probability 1 - ɛ* : **we do exploitation** (aka our agent selects the action with the highest state-action pair value).\n",
+ "\n",
+ "- With *probability ɛ*: we do **exploration** (trying random action).\n",
+ "\n",
+ "And as the training goes, we progressively **reduce the epsilon value since we will need less and less exploration and more exploitation.**\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LjZSvhsD7_52"
+ },
+ "source": [
+ "Thanks to Sambit for finding a bug on the epsilon function 🤗"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "6Bj7x3in3_Pq"
+ },
+ "outputs": [],
+ "source": [
+ "def epsilon_greedy_policy(Qtable, state, epsilon):\n",
+ " # Randomly generate a number between 0 and 1\n",
+ " random_num = \n",
+ " # if random_num > greater than epsilon --> exploitation\n",
+ " if random_num > epsilon:\n",
+ " # Take the action with the highest value given a state\n",
+ " # np.argmax can be useful here\n",
+ " action = \n",
+ " # else --> exploration\n",
+ " else:\n",
+ " action = # Take a random action\n",
+ " \n",
+ " return action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8R5ej1fS4P2V"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "cYxHuckr4LiG"
+ },
+ "outputs": [],
+ "source": [
+ "def epsilon_greedy_policy(Qtable, state, epsilon):\n",
+ " # Randomly generate a number between 0 and 1\n",
+ " random_int = random.uniform(0,1)\n",
+ " # if random_int > greater than epsilon --> exploitation\n",
+ " if random_int > epsilon:\n",
+ " # Take the action with the highest value given a state\n",
+ " # np.argmax can be useful here\n",
+ " action = np.argmax(Qtable[state])\n",
+ " # else --> exploration\n",
+ " else:\n",
+ " action = env.action_space.sample()\n",
+ " \n",
+ " return action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Atll4Z774gri"
+ },
+ "source": [
+ "## Define the greedy policy 🤖\n",
+ "Remember we have two policies since Q-Learning is an **off-policy** algorithm. This means we're using a **different policy for acting and updating the value function**.\n",
+ "\n",
+ "- Epsilon greedy policy (acting policy)\n",
+ "- Greedy policy (updating policy)\n",
+ "\n",
+ "Greedy policy will also be the final policy we'll have when the Q-learning agent will be trained. The greedy policy is used to select an action from the Q-table.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "E3SCLmLX5bWG"
+ },
+ "outputs": [],
+ "source": [
+ "def greedy_policy(Qtable, state):\n",
+ " # Exploitation: take the action with the highest state, action value\n",
+ " action = \n",
+ " \n",
+ " return action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "B2_-8b8z5k54"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "se2OzWGW5kYJ"
+ },
+ "outputs": [],
+ "source": [
+ "def greedy_policy(Qtable, state):\n",
+ " # Exploitation: take the action with the highest state, action value\n",
+ " action = np.argmax(Qtable[state])\n",
+ " \n",
+ " return action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "hW80DealcRtu"
+ },
+ "source": [
+ "## Define the hyperparameters ⚙️\n",
+ "The exploration related hyperparamters are some of the most important ones. \n",
+ "\n",
+ "- We need to make sure that our agent **explores enough the state space** in order to learn a good value approximation, in order to do that we need to have progressive decay of the epsilon.\n",
+ "- If you decrease too fast epsilon (too high decay_rate), **you take the risk that your agent is stuck**, since your agent didn't explore enough the state space and hence can't solve the problem."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Y1tWn0tycWZ1"
+ },
+ "outputs": [],
+ "source": [
+ "# Training parameters\n",
+ "n_training_episodes = 10000 # Total training episodes\n",
+ "learning_rate = 0.7 # Learning rate\n",
+ "\n",
+ "# Evaluation parameters\n",
+ "n_eval_episodes = 100 # Total number of test episodes\n",
+ "\n",
+ "# Environment parameters\n",
+ "env_id = \"FrozenLake-v1\" # Name of the environment\n",
+ "max_steps = 99 # Max steps per episode\n",
+ "gamma = 0.95 # Discounting rate\n",
+ "eval_seed = [] # The evaluation seed of the environment\n",
+ "\n",
+ "# Exploration parameters\n",
+ "max_epsilon = 1.0 # Exploration probability at start\n",
+ "min_epsilon = 0.05 # Minimum exploration probability \n",
+ "decay_rate = 0.0005 # Exponential decay rate for exploration prob"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "cDb7Tdx8atfL"
+ },
+ "source": [
+ "## Step 6: Create the training loop method"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "paOynXy3aoJW"
+ },
+ "outputs": [],
+ "source": [
+ "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n",
+ " for episode in range(n_training_episodes):\n",
+ " # Reduce epsilon (because we need less and less exploration)\n",
+ " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n",
+ " # Reset the environment\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ "\n",
+ " # repeat\n",
+ " for step in range(max_steps):\n",
+ " # Choose the action At using epsilon greedy policy\n",
+ " action = \n",
+ "\n",
+ " # Take action At and observe Rt+1 and St+1\n",
+ " # Take the action (a) and observe the outcome state(s') and reward (r)\n",
+ " new_state, reward, done, info = \n",
+ "\n",
+ " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
+ " Qtable[state][action] = \n",
+ "\n",
+ " # If done, finish the episode\n",
+ " if done:\n",
+ " break\n",
+ " \n",
+ " # Our state is the new state\n",
+ " state = new_state\n",
+ " return Qtable"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Pnpk2ePoem3r"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "IyZaYbUAeolw"
+ },
+ "outputs": [],
+ "source": [
+ "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n",
+ " for episode in tqdm(range(n_training_episodes)):\n",
+ " # Reduce epsilon (because we need less and less exploration)\n",
+ " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n",
+ " # Reset the environment\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ "\n",
+ " # repeat\n",
+ " for step in range(max_steps):\n",
+ " # Choose the action At using epsilon greedy policy\n",
+ " action = epsilon_greedy_policy(Qtable, state, epsilon)\n",
+ "\n",
+ " # Take action At and observe Rt+1 and St+1\n",
+ " # Take the action (a) and observe the outcome state(s') and reward (r)\n",
+ " new_state, reward, done, info = env.step(action)\n",
+ "\n",
+ " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
+ " Qtable[state][action] = Qtable[state][action] + learning_rate * (reward + gamma * np.max(Qtable[new_state]) - Qtable[state][action]) \n",
+ "\n",
+ " # If done, finish the episode\n",
+ " if done:\n",
+ " break\n",
+ " \n",
+ " # Our state is the new state\n",
+ " state = new_state\n",
+ " return Qtable"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WLwKQ4tUdhGI"
+ },
+ "source": [
+ "## Train the Q-Learning agent 🏃"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "DPBxfjJdTCOH"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_frozenlake = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_frozenlake)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yVeEhUCrc30L"
+ },
+ "source": [
+ "## Let's see what our Q-Learning table looks like now 👀"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "nmfchsTITw4q"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_frozenlake"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pUrWkxsHccXD"
+ },
+ "source": [
+ "## Define the evaluation method 📝"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "jNl0_JO2cbkm"
+ },
+ "outputs": [],
+ "source": [
+ "def evaluate_agent(env, max_steps, n_eval_episodes, Q, seed):\n",
+ " \"\"\"\n",
+ " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n",
+ " :param env: The evaluation environment\n",
+ " :param n_eval_episodes: Number of episode to evaluate the agent\n",
+ " :param Q: The Q-table\n",
+ " :param seed: The evaluation seed array (for taxi-v3)\n",
+ " \"\"\"\n",
+ " episode_rewards = []\n",
+ " for episode in tqdm(range(n_eval_episodes)):\n",
+ " if seed:\n",
+ " state = env.reset(seed=seed[episode])\n",
+ " else:\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ " total_rewards_ep = 0\n",
+ " \n",
+ " for step in range(max_steps):\n",
+ " # Take the action (index) that have the maximum expected future reward given that state\n",
+ " action = np.argmax(Q[state][:])\n",
+ " new_state, reward, done, info = env.step(action)\n",
+ " total_rewards_ep += reward\n",
+ " \n",
+ " if done:\n",
+ " break\n",
+ " state = new_state\n",
+ " episode_rewards.append(total_rewards_ep)\n",
+ " mean_reward = np.mean(episode_rewards)\n",
+ " std_reward = np.std(episode_rewards)\n",
+ "\n",
+ " return mean_reward, std_reward"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0jJqjaoAnxUo"
+ },
+ "source": [
+ "## Evaluate our Q-Learning agent 📈\n",
+ "- Normally you should have mean reward of 1.0\n",
+ "- It's relatively easy since the state space is really small (16). What you can try to do is [to replace with the slippery version](https://www.gymlibrary.dev/environments/toy_text/frozen_lake/)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "fAgB7s0HEFMm"
+ },
+ "outputs": [],
+ "source": [
+ "# Evaluate our Agent\n",
+ "mean_reward, std_reward = evaluate_agent(env, max_steps, n_eval_episodes, Qtable_frozenlake, eval_seed)\n",
+ "print(f\"Mean_reward={mean_reward:.2f} +/- {std_reward:.2f}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yxaP3bPdg1DV"
+ },
+ "source": [
+ "## Publish our trained model on the Hub 🔥\n",
+ "Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
+ "\n",
+ "Here's an example of a Model Card:\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kv0k1JQjpMq3"
+ },
+ "source": [
+ "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "QZ5LrR-joIHD"
+ },
+ "source": [
+ "#### Do not modify this code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Jex3i9lZ8ksX"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "from huggingface_hub import HfApi, HfFolder, Repository\n",
+ "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n",
+ "\n",
+ "from pathlib import Path\n",
+ "import datetime\n",
+ "import json"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Qo57HBn3W74O"
+ },
+ "outputs": [],
+ "source": [
+ "def record_video(env, Qtable, out_directory, fps=1):\n",
+ " images = [] \n",
+ " done = False\n",
+ " state = env.reset(seed=random.randint(0,500))\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " while not done:\n",
+ " # Take the action (index) that have the maximum expected future reward given that state\n",
+ " action = np.argmax(Qtable[state][:])\n",
+ " state, reward, done, info = env.step(action) # We directly put next_state = state for recording logic\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "pwsNrzB339aF"
+ },
+ "outputs": [],
+ "source": [
+ "def push_to_hub(repo_id, \n",
+ " model,\n",
+ " env,\n",
+ " video_fps=1,\n",
+ " local_repo_path=\"hub\",\n",
+ " commit_message=\"Push Q-Learning agent to Hub\",\n",
+ " token= None\n",
+ " ):\n",
+ " _, repo_name = repo_id.split(\"/\")\n",
+ "\n",
+ " eval_env = env\n",
+ " \n",
+ " # Step 1: Clone or create the repo\n",
+ " # Create the repo (or clone its content if it's nonempty)\n",
+ " api = HfApi()\n",
+ " \n",
+ " repo_url = api.create_repo(\n",
+ " repo_id=repo_id,\n",
+ " token=token,\n",
+ " private=False,\n",
+ " exist_ok=True,)\n",
+ " \n",
+ " # Git pull\n",
+ " repo_local_path = Path(local_repo_path) / repo_name\n",
+ " repo = Repository(repo_local_path, clone_from=repo_url, use_auth_token=True)\n",
+ " repo.git_pull()\n",
+ " \n",
+ " repo.lfs_track([\"*.mp4\"])\n",
+ "\n",
+ " # Step 1: Save the model\n",
+ " if env.spec.kwargs.get(\"map_name\"):\n",
+ " model[\"map_name\"] = env.spec.kwargs.get(\"map_name\")\n",
+ " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n",
+ " model[\"slippery\"] = False\n",
+ "\n",
+ " print(model)\n",
+ " \n",
+ " \n",
+ " # Pickle the model\n",
+ " with open(Path(repo_local_path)/'q-learning.pkl', 'wb') as f:\n",
+ " pickle.dump(model, f)\n",
+ " \n",
+ " # Step 2: Evaluate the model and build JSON\n",
+ " mean_reward, std_reward = evaluate_agent(eval_env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])\n",
+ "\n",
+ " # First get datetime\n",
+ " eval_datetime = datetime.datetime.now()\n",
+ " eval_form_datetime = eval_datetime.isoformat()\n",
+ "\n",
+ " evaluate_data = {\n",
+ " \"env_id\": model[\"env_id\"], \n",
+ " \"mean_reward\": mean_reward,\n",
+ " \"n_eval_episodes\": model[\"n_eval_episodes\"],\n",
+ " \"eval_datetime\": eval_form_datetime,\n",
+ " }\n",
+ " # Write a JSON file\n",
+ " with open(Path(repo_local_path) / \"results.json\", \"w\") as outfile:\n",
+ " json.dump(evaluate_data, outfile)\n",
+ "\n",
+ " # Step 3: Create the model card\n",
+ " # Env id\n",
+ " env_name = model[\"env_id\"]\n",
+ " if env.spec.kwargs.get(\"map_name\"):\n",
+ " env_name += \"-\" + env.spec.kwargs.get(\"map_name\")\n",
+ "\n",
+ " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n",
+ " env_name += \"-\" + \"no_slippery\"\n",
+ "\n",
+ " metadata = {}\n",
+ " metadata[\"tags\"] = [\n",
+ " env_name,\n",
+ " \"q-learning\",\n",
+ " \"reinforcement-learning\",\n",
+ " \"custom-implementation\"\n",
+ " ]\n",
+ "\n",
+ " # Add metrics\n",
+ " eval = metadata_eval_result(\n",
+ " model_pretty_name=repo_name,\n",
+ " task_pretty_name=\"reinforcement-learning\",\n",
+ " task_id=\"reinforcement-learning\",\n",
+ " metrics_pretty_name=\"mean_reward\",\n",
+ " metrics_id=\"mean_reward\",\n",
+ " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n",
+ " dataset_pretty_name=env_name,\n",
+ " dataset_id=env_name,\n",
+ " )\n",
+ "\n",
+ " # Merges both dictionaries\n",
+ " metadata = {**metadata, **eval}\n",
+ "\n",
+ " model_card = f\"\"\"\n",
+ " # **Q-Learning** Agent playing **{env_id}**\n",
+ " This is a trained model of a **Q-Learning** agent playing **{env_id}** .\n",
+ " \"\"\"\n",
+ "\n",
+ " model_card += \"\"\"\n",
+ " ## Usage\n",
+ " ```python\n",
+ " \"\"\"\n",
+ "\n",
+ " model_card += f\"\"\"model = load_from_hub(repo_id=\"{repo_id}\", filename=\"q-learning.pkl\")\n",
+ "\n",
+ " # Don't forget to check if you need to add additional attributes (is_slippery=False etc)\n",
+ " env = gym.make(model[\"env_id\"])\n",
+ "\n",
+ " evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])\n",
+ " \"\"\"\n",
+ "\n",
+ " model_card +=\"\"\"\n",
+ " ```\n",
+ " \"\"\"\n",
+ "\n",
+ " readme_path = repo_local_path / \"README.md\"\n",
+ " readme = \"\"\n",
+ " if readme_path.exists():\n",
+ " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n",
+ " readme = f.read()\n",
+ " else:\n",
+ " readme = model_card\n",
+ "\n",
+ " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n",
+ " f.write(readme)\n",
+ "\n",
+ " # Save our metrics to Readme metadata\n",
+ " metadata_save(readme_path, metadata)\n",
+ "\n",
+ " # Step 4: Record a video\n",
+ " video_path = repo_local_path / \"replay.mp4\"\n",
+ " record_video(env, model[\"qtable\"], video_path, video_fps)\n",
+ " \n",
+ " # Push everything to hub\n",
+ " print(f\"Pushing repo {repo_name} to the Hugging Face Hub\")\n",
+ " repo.push_to_hub(commit_message=commit_message)\n",
+ "\n",
+ " print(f\"Your model is pushed to the hub. You can view your model here: {repo_url}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81J6cet_ogSS"
+ },
+ "source": [
+ "### .\n",
+ "By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
+ "\n",
+ "This way:\n",
+ "- You can **showcase our work** 🔥\n",
+ "- You can **visualize your agent playing** 👀\n",
+ "- You can **share with the community an agent that others can use** 💾\n",
+ "- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "cWnFC0iZooTw"
+ },
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
+ "\n",
+ "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "QB5nIcxR8paT"
+ },
+ "outputs": [],
+ "source": [
+ "from huggingface_hub import notebook_login\n",
+ "notebook_login()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "GyWc1x3-o3xG"
+ },
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Gc5AfUeFo3xH"
+ },
+ "source": [
+ "3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using `package_to_hub()` function\n",
+ "\n",
+ "- Let's create **the model dictionary that contains the hyperparameters and the Q_table**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "FiMqxqVHg0I4"
+ },
+ "outputs": [],
+ "source": [
+ "model = {\n",
+ " \"env_id\": env_id,\n",
+ " \"max_steps\": max_steps,\n",
+ " \"n_training_episodes\": n_training_episodes,\n",
+ " \"n_eval_episodes\": n_eval_episodes,\n",
+ " \"eval_seed\": eval_seed,\n",
+ "\n",
+ " \"learning_rate\": learning_rate,\n",
+ " \"gamma\": gamma,\n",
+ "\n",
+ " \"max_epsilon\": max_epsilon,\n",
+ " \"min_epsilon\": min_epsilon,\n",
+ " \"decay_rate\": decay_rate,\n",
+ "\n",
+ " \"qtable\": Qtable_frozenlake\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9kld-AEso3xH"
+ },
+ "source": [
+ "Let's fill the `package_to_hub` function:\n",
+ "- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `\n",
+ "(repo_id = {username}/{repo_name})`\n",
+ "💡 **A good name is {username}/q-{env_id}**\n",
+ "- `model`: our model dictionary containing the hyperparameters and the Qtable.\n",
+ "- `env`: the environment.\n",
+ "- `commit_message`: message of the commit"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "5sBo2umnXpPd"
+ },
+ "outputs": [],
+ "source": [
+ "model"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "RpOTtSt83kPZ"
+ },
+ "outputs": [],
+ "source": [
+ "username = \"\" # FILL THIS\n",
+ "repo_name = \"q-FrozenLake-v1-4x4-noSlippery\"\n",
+ "push_to_hub(\n",
+ " repo_id=f\"{username}/{repo_name}\",\n",
+ " model=model,\n",
+ " env=env)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "E2875IGsprzq"
+ },
+ "source": [
+ "Congrats 🥳 you've just implemented from scratch, trained and uploaded your first Reinforcement Learning agent. \n",
+ "FrozenLake-v1 no_slippery is very simple environment, let's try an harder one 🔥."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "18lN8Bz7yvLt"
+ },
+ "source": [
+ "# Part 2: Taxi-v3 🚖\n",
+ "\n",
+ "## Create and understand [Taxi-v3 🚕](https://www.gymlibrary.dev/environments/toy_text/taxi/)\n",
+ "---\n",
+ "\n",
+ "💡 A good habit when you start to use an environment is to check its documentation \n",
+ "\n",
+ "👉 https://www.gymlibrary.dev/environments/toy_text/taxi/\n",
+ "\n",
+ "---\n",
+ "\n",
+ "In Taxi-v3 🚕, there are four designated locations in the grid world indicated by R(ed), G(reen), Y(ellow), and B(lue). When the episode starts, the taxi starts off at a random square and the passenger is at a random location. The taxi drives to the passenger’s location, picks up the passenger, drives to the passenger’s destination (another one of the four specified locations), and then drops off the passenger. Once the passenger is dropped off, the episode ends.\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "gL0wpeO8gpej"
+ },
+ "outputs": [],
+ "source": [
+ "env = gym.make(\"Taxi-v3\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gBOaXgtsrmtT"
+ },
+ "source": [
+ "There are **500 discrete states since there are 25 taxi positions, 5 possible locations of the passenger** (including the case when the passenger is in the taxi), and **4 destination locations.**\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "_TPNaGSZrgqA"
+ },
+ "outputs": [],
+ "source": [
+ "state_space = env.observation_space.n\n",
+ "print(\"There are \", state_space, \" possible states\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "CdeeZuokrhit"
+ },
+ "outputs": [],
+ "source": [
+ "action_space = env.action_space.n\n",
+ "print(\"There are \", action_space, \" possible actions\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "R1r50Advrh5Q"
+ },
+ "source": [
+ "The action space (the set of possible actions the agent can take) is discrete with **6 actions available 🎮**:\n",
+ "- 0: move south\n",
+ "- 1: move north\n",
+ "- 2: move east\n",
+ "- 3: move west\n",
+ "- 4: pickup passenger\n",
+ "- 5: drop off passenger\n",
+ "\n",
+ "Reward function 💰:\n",
+ "- -1 per step unless other reward is triggered.\n",
+ "- +20 delivering passenger.\n",
+ "- -10 executing “pickup” and “drop-off” actions illegally."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "US3yDXnEtY9I"
+ },
+ "outputs": [],
+ "source": [
+ "# Create our Q table with state_size rows and action_size columns (500x6)\n",
+ "Qtable_taxi = initialize_q_table(state_space, action_space)\n",
+ "print(Qtable_taxi)\n",
+ "print(\"Q-table shape: \", Qtable_taxi .shape)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gUMKPH0_LJyH"
+ },
+ "source": [
+ "## Define the hyperparameters ⚙️\n",
+ "⚠ DO NOT MODIFY EVAL_SEED: the eval_seed array **allows us to evaluate your agent with the same taxi starting positions for every classmate**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "AB6n__hhg7YS"
+ },
+ "outputs": [],
+ "source": [
+ "# Training parameters\n",
+ "n_training_episodes = 25000 # Total training episodes\n",
+ "learning_rate = 0.7 # Learning rate\n",
+ "\n",
+ "# Evaluation parameters\n",
+ "n_eval_episodes = 100 # Total number of test episodes\n",
+ "\n",
+ "# DO NOT MODIFY EVAL_SEED\n",
+ "eval_seed = [16,54,165,177,191,191,120,80,149,178,48,38,6,125,174,73,50,172,100,148,146,6,25,40,68,148,49,167,9,97,164,176,61,7,54,55,\n",
+ " 161,131,184,51,170,12,120,113,95,126,51,98,36,135,54,82,45,95,89,59,95,124,9,113,58,85,51,134,121,169,105,21,30,11,50,65,12,43,82,145,152,97,106,55,31,85,38,\n",
+ " 112,102,168,123,97,21,83,158,26,80,63,5,81,32,11,28,148] # Evaluation seed, this ensures that all classmates agents are trained on the same taxi starting position\n",
+ " # Each seed has a specific starting state\n",
+ "\n",
+ "# Environment parameters\n",
+ "env_id = \"Taxi-v3\" # Name of the environment\n",
+ "max_steps = 99 # Max steps per episode\n",
+ "gamma = 0.95 # Discounting rate\n",
+ "\n",
+ "# Exploration parameters\n",
+ "max_epsilon = 1.0 # Exploration probability at start\n",
+ "min_epsilon = 0.05 # Minimum exploration probability \n",
+ "decay_rate = 0.005 # Exponential decay rate for exploration prob\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1TMORo1VLTsX"
+ },
+ "source": [
+ "## Train our Q-Learning agent 🏃"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "MLNwkNDb14h2"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_taxi = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_taxi)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "WwP3Y2z2eS-K"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_taxi"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "wPdu0SueLVl2"
+ },
+ "source": [
+ "## Create a model dictionary 💾 and publish our trained model on the Hub 🔥\n",
+ "- We create a model dictionary that will contain all the training hyperparameters for reproducibility and the Q-Table.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "0a1FpE_3hNYr"
+ },
+ "outputs": [],
+ "source": [
+ "model = {\n",
+ " \"env_id\": env_id,\n",
+ " \"max_steps\": max_steps,\n",
+ " \"n_training_episodes\": n_training_episodes,\n",
+ " \"n_eval_episodes\": n_eval_episodes,\n",
+ " \"eval_seed\": eval_seed,\n",
+ "\n",
+ " \"learning_rate\": learning_rate,\n",
+ " \"gamma\": gamma,\n",
+ "\n",
+ " \"max_epsilon\": max_epsilon,\n",
+ " \"min_epsilon\": min_epsilon,\n",
+ " \"decay_rate\": decay_rate,\n",
+ "\n",
+ " \"qtable\": Qtable_taxi\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "dhQtiQozhOn1"
+ },
+ "outputs": [],
+ "source": [
+ "username = \"\" # FILL THIS\n",
+ "repo_name = \"q-Taxi-v3\"\n",
+ "push_to_hub(\n",
+ " repo_id=f\"{username}/{repo_name}\",\n",
+ " model=model,\n",
+ " env=env)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ZgSdjgbIpRti"
+ },
+ "source": [
+ "Now that's on the Hub, you can compare the results of your Taxi-v3 with your classmates using the leaderboard 🏆 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "bzgIO70c0bu2"
+ },
+ "source": [
+ "# Part 3: Load from Hub 🔽\n",
+ "\n",
+ "What's amazing with Hugging Face Hub 🤗 is that you can easily load powerful models from the community.\n",
+ "\n",
+ "Loading a saved model from the Hub is really easy.\n",
+ "1. You go https://huggingface.co/models?other=q-learning to see the list of all the q-learning saved models.\n",
+ "2. You select one and copy its repo_id\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gTth6thRoC6X"
+ },
+ "source": [
+ "3. Then we just need to use `load_from_hub` with:\n",
+ "- The repo_id\n",
+ "- The filename: the saved model inside the repo."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "EtrfoTaBoNrd"
+ },
+ "source": [
+ "#### Do not modify this code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Eo8qEzNtCaVI"
+ },
+ "outputs": [],
+ "source": [
+ "from urllib.error import HTTPError\n",
+ "\n",
+ "from huggingface_hub import hf_hub_download\n",
+ "\n",
+ "\n",
+ "def load_from_hub(repo_id: str, filename: str) -> str:\n",
+ " \"\"\"\n",
+ " Download a model from Hugging Face Hub.\n",
+ " :param repo_id: id of the model repository from the Hugging Face Hub\n",
+ " :param filename: name of the model zip file from the repository\n",
+ " \"\"\"\n",
+ " try:\n",
+ " from huggingface_hub import cached_download, hf_hub_url\n",
+ " except ImportError:\n",
+ " raise ImportError(\n",
+ " \"You need to install huggingface_hub to use `load_from_hub`. \"\n",
+ " \"See https://pypi.org/project/huggingface-hub/ for installation.\"\n",
+ " )\n",
+ "\n",
+ " # Get the model from the Hub, download and cache the model on your local disk\n",
+ " pickle_model = hf_hub_download(\n",
+ " repo_id=repo_id,\n",
+ " filename=filename\n",
+ " )\n",
+ "\n",
+ " with open(pickle_model, 'rb') as f:\n",
+ " downloaded_model_file = pickle.load(f)\n",
+ " \n",
+ " return downloaded_model_file"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "b_sM2gNioPZH"
+ },
+ "source": [
+ "### ."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "JUm9lz2gCQcU"
+ },
+ "outputs": [],
+ "source": [
+ "model = load_from_hub(repo_id=\"ThomasSimonini/q-Taxi-v3\", filename=\"q-learning.pkl\") # Try to use another model\n",
+ "\n",
+ "print(model)\n",
+ "env = gym.make(model[\"env_id\"])\n",
+ "\n",
+ "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "O7pL8rg1MulN"
+ },
+ "outputs": [],
+ "source": [
+ "model = load_from_hub(repo_id=\"ThomasSimonini/q-FrozenLake-v1-no-slippery\", filename=\"q-learning.pkl\") # Try to use another model\n",
+ "\n",
+ "env = gym.make(model[\"env_id\"], is_slippery=False)\n",
+ "\n",
+ "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BQAwLnYFPk-s"
+ },
+ "source": [
+ "## Some additional challenges 🏆\n",
+ "The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
+ "\n",
+ "In the [Leaderboard](https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
+ "\n",
+ "Here are some ideas to achieve so:\n",
+ "* Train more steps\n",
+ "* Try different hyperparameters by looking at what your classmates have done.\n",
+ "* **Push your new trained model** on the Hub 🔥\n",
+ "\n",
+ "Are walking on ice and driving taxis too boring to you? Try to **change the environment**, why not using FrozenLake-v1 slippery version? Check how they work [using the gym documentation](https://www.gymlibrary.dev/) and have fun 🎉."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "p-fW-EU5WejJ"
+ },
+ "source": [
+ "_____________________________________________________________________\n",
+ "Congrats 🥳, you've just implemented, trained, and uploaded your first Reinforcement Learning agent.\n",
+ "\n",
+ "Understanding Q-Learning is an **important step to understanding value-based methods.**\n",
+ "\n",
+ "In the next Unit with Deep Q-Learning, we'll see that creating and updating a Q-table was a good strategy — **however, this is not scalable.**\n",
+ "\n",
+ "For instance, imagine you create an agent that learns to play Doom. \n",
+ "\n",
+ "\n",
+ "\n",
+ "Doom is a large environment with a huge state space (millions of different states). Creating and updating a Q-table for that environment would not be efficient. \n",
+ "\n",
+ "That's why we'll study, in the next unit, Deep Q-Learning, an algorithm **where we use a neural network that approximates, given a state, the different Q-values for each action.**\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BjLhT70TEZIn"
+ },
+ "source": [
+ "See you on [Unit 3](https://github.com/huggingface/deep-rl-class/tree/main/unit2#unit-2-introduction-to-q-learning)! 🔥\n",
+ "TODO CHANGE LINK\n",
+ "## Keep learning, stay awesome 🤗"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "collapsed_sections": [
+ "4i6tjI2tHQ8j",
+ "Y-mo_6rXIjRi",
+ "EtrfoTaBoNrd",
+ "BjLhT70TEZIn"
+ ],
+ "private_outputs": true,
+ "provenance": []
+ },
+ "gpuClass": "standard",
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/notebooks/unit3/unit3.ipynb b/notebooks/unit3/unit3.ipynb
new file mode 100644
index 0000000..019653e
--- /dev/null
+++ b/notebooks/unit3/unit3.ipynb
@@ -0,0 +1,794 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "k7xBVPzoXxOg"
+ },
+ "source": [
+ "# Unit 3: Deep Q-Learning with Atari Games 👾 using RL Baselines3 Zoo\n",
+ "\n",
+ "\n",
+ "\n",
+ "In this notebook, **you'll train a Deep Q-Learning agent** playing Space Invaders using [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), a training framework based on [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.\n",
+ "\n",
+ "We're using the [RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) with no extensions such as Double-DQN, Dueling-DQN, and Prioritized Experience Replay.\n",
+ "\n",
+ "⬇️ Here is an example of what **you will achieve** ⬇️"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "J9S713biXntc"
+ },
+ "outputs": [],
+ "source": [
+ "%%html\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO: ADD TEXT LIVE INFO\n",
+ "\n",
+ "TODO: ADD IF YOU HAVE QUESTIONS\n",
+ "\n",
+ "\n",
+ "###🎮 Environments: \n",
+ "- SpacesInvadersNoFrameskip-v4 \n",
+ "\n",
+ "###📚 RL-Library: \n",
+ "- [RL-Baselines3-Zoo](https://github.com/DLR-RM/rl-baselines3-zoo)"
+ ],
+ "metadata": {
+ "id": "ykJiGevCMVc5"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "wciHGjrFYz9m"
+ },
+ "source": [
+ "## Objectives of this notebook 🏆\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to understand deeper **how RL Baselines3 Zoo works**.\n",
+ "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Course\n",
+ ""
+ ],
+ "metadata": {
+ "id": "TsnP0rjxMn1e"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "nw6fJHIAZd-J"
+ },
+ "source": [
+ "In this free course, you will:\n",
+ "\n",
+ "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- 🧑💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
+ "- 🤖 Train **agents in unique environments** \n",
+ "\n",
+ "And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course\n",
+ "\n",
+ "Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
+ "\n",
+ "\n",
+ "The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0vgANIBBZg1p"
+ },
+ "source": [
+ "## Prerequisites 🏗️\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "🔲 📚 **Study Deep Q-Learning by reading Unit 3** 🤗 ADD LINK "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
+ ],
+ "metadata": {
+ "id": "7kszpGFaRVhq"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "QR0jZtYreSI5"
+ },
+ "source": [
+ "# Let's train a Deep Q-Learning agent playing Atari' Space Invaders 👾 and upload it to the Hub."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Set the GPU 💪\n",
+ "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
+ "\n",
+ ""
+ ],
+ "metadata": {
+ "id": "PU4FVzaoM6fC"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "- `Hardware Accelerator > GPU`\n",
+ "\n",
+ ""
+ ],
+ "metadata": {
+ "id": "KV0NyFdQM9ZG"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Create a virtual display 🔽\n",
+ "\n",
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install the librairies and create and run a virtual screen 🖥"
+ ],
+ "metadata": {
+ "id": "bTpYcVZVMzUI"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "jV6wjQ7Be7p5"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip3 install pyvirtualdisplay"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Additional dependencies for RL Baselines3 Zoo\n",
+ "!apt-get install swig cmake freeglut3-dev "
+ ],
+ "metadata": {
+ "id": "fWyKJCy_NJBX"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!pip install pyglet==1.5.1"
+ ],
+ "metadata": {
+ "id": "C5LwHrISW7Q5"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "ww5PQH1gNLI4"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "mYIMvl5X9NAu"
+ },
+ "source": [
+ "## Clone RL-Baselines3 Zoo Repo 📚\n",
+ "You can now directly install from python package `pip install rl_zoo3` but since we want **the full installation with extra environments and dependencies** we're going to clone `RL-Baselines3-Zoo` repository and install from source."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "eu5ZDPZ09VNQ"
+ },
+ "outputs": [],
+ "source": [
+ "!git clone https://github.com/DLR-RM/rl-baselines3-zoo"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "HCIoSbvbfAQh"
+ },
+ "source": [
+ "## Install dependencies 🔽\n",
+ "We can now install the dependencies RL-Baselines3 Zoo needs (this can take 5min ⏲)\n",
+ "\n",
+ "But we'll also install:\n",
+ "- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face 🤗 Hub."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "s2QsFAk29h-D"
+ },
+ "outputs": [],
+ "source": [
+ "%cd /content/rl-baselines3-zoo/"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "3QaOS7Xj9j1s"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -r requirements.txt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "RLRGKFR39l9s"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install huggingface_sb3"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5iPgzluo9z-u"
+ },
+ "source": [
+ "## Train our Deep Q-Learning Agent to Play Space Invaders 👾\n",
+ "\n",
+ "To train an agent with RL-Baselines3-Zoo, we just need to do two things:\n",
+ "1. We define the hyperparameters in `rl-baselines3-zoo/hyperparams/dqn.yml`\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_VjblFSVDQOj"
+ },
+ "source": [
+ "Here we see that:\n",
+ "- We use the `Atari Wrapper` that preprocess the input (Frame reduction ,grayscale, stack 4 frames)\n",
+ "- We use `CnnPolicy`, since we use Convolutional layers to process the frames\n",
+ "- We train it for 10 million `n_timesteps` \n",
+ "- Memory (Experience Replay) size is 100000, aka the amount of experience steps you saved to train again your agent with.\n",
+ "\n",
+ "💡 My advice is to **reduce the training timesteps to 1M,** which will take about 90 minutes on a P100. `!nvidia-smi` will tell you what GPU you're using. At 10 million steps, this will take about 9 hours, which could likely result in Colab timing out. I recommend running this on your local computer (or somewhere else). Just click on: `File>Download`. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5qTkbWrkECOJ"
+ },
+ "source": [
+ "In terms of hyperparameters optimization, my advice is to focus on these 3 hyperparameters:\n",
+ "- `learning_rate`\n",
+ "- `buffer_size (Experience Memory size)`\n",
+ "- `batch_size`\n",
+ "\n",
+ "As a good practice, you need to **check the documentation to understand what each hyperparameters does**: https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html#parameters\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Hn8bRTHvERRL"
+ },
+ "source": [
+ "2. We run `train.py` and save the models on `logs` folder 📁"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Xr1TVW4xfbz3"
+ },
+ "outputs": [],
+ "source": [
+ "!python train.py --algo ________ --env SpaceInvadersNoFrameskip-v4 -f _________"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SeChoX-3SZfP"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "PuocgdokSab9"
+ },
+ "outputs": [],
+ "source": [
+ "!python train.py --algo dqn --env SpaceInvadersNoFrameskip-v4 -f logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_dLomIiMKQaf"
+ },
+ "source": [
+ "## Let's evaluate our agent 👀\n",
+ "- RL-Baselines3-Zoo provides `enjoy.py` to evaluate our agent.\n",
+ "- Let's evaluate it for 5000 timesteps 🔥"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "co5um_KeKbBJ"
+ },
+ "outputs": [],
+ "source": [
+ "!python enjoy.py --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps _________ --folder logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Q24K1tyWSj7t"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "P_uSmwGRSk0z"
+ },
+ "outputs": [],
+ "source": [
+ "!python enjoy.py --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps 5000 --folder logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "liBeTltiHJtr"
+ },
+ "source": [
+ "## Publish our trained model on the Hub 🚀\n",
+ "Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ezbHS1q3HYVV"
+ },
+ "source": [
+ "By using `rl_zoo3.push_to_hub.py` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
+ "\n",
+ "This way:\n",
+ "- You can **showcase our work** 🔥\n",
+ "- You can **visualize your agent playing** 👀\n",
+ "- You can **share with the community an agent that others can use** 💾\n",
+ "- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "XMSeZRBiHk6X"
+ },
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
+ "\n",
+ "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens) **with write role**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9ToyuaYwHmxG"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9O6FI0F8HnzE"
+ },
+ "source": [
+ "- Copy the token \n",
+ "- Run the cell below and past the token"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Ppu9yePwHrZX"
+ },
+ "outputs": [],
+ "source": [
+ "from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
+ "notebook_login()\n",
+ "!git config --global credential.helper store"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2RVEdunPHs8B"
+ },
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "dSLwdmvhHvjw"
+ },
+ "source": [
+ "3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "PW436XnhHw1H"
+ },
+ "source": [
+ "Let's run push_to_hub.py file to upload our trained agent to the Hub.\n",
+ "\n",
+ "`--repo-name `: The name of the repo\n",
+ "\n",
+ "`-orga`: Your Hugging Face username\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Ygk2sEktTDEw"
+ },
+ "outputs": [],
+ "source": [
+ "!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name _____________________ -orga _____________________ -f logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "otgpa0rhS9wR"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "_HQNlAXuEhci"
+ },
+ "outputs": [],
+ "source": [
+ "!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name dqn-SpaceInvadersNoFrameskip-v4 -orga ThomasSimonini -f logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0D4F5zsTTJ-L"
+ },
+ "source": [
+ "###."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ff89kd2HL1_s"
+ },
+ "source": [
+ "Congrats 🥳 you've just trained and uploaded your first Deep Q-Learning agent using RL-Baselines-3 Zoo. The script above should have displayed a link to a model repository such as https://huggingface.co/ThomasSimonini/dqn-SpaceInvadersNoFrameskip-v4. When you go to this link, you can:\n",
+ "\n",
+ "- See a **video preview of your agent** at the right. \n",
+ "- Click \"Files and versions\" to see all the files in the repository.\n",
+ "- Click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
+ "- A model card (`README.md` file) which gives a description of the model and the hyperparameters you used.\n",
+ "\n",
+ "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
+ "\n",
+ "**Compare the results of your agents with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) 🏆"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fyRKcCYY-dIo"
+ },
+ "source": [
+ "## Load a powerful trained model 🔥\n",
+ "- The Stable-Baselines3 team uploaded **more than 150 trained Deep Reinforcement Learning agents on the Hub**.\n",
+ "\n",
+ "You can find them here: 👉 https://huggingface.co/sb3\n",
+ "\n",
+ "Some examples:\n",
+ "- Asteroids: https://huggingface.co/sb3/dqn-AsteroidsNoFrameskip-v4\n",
+ "- Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4\n",
+ "- Breakout: https://huggingface.co/sb3/dqn-BreakoutNoFrameskip-v4\n",
+ "- Road Runner: https://huggingface.co/sb3/dqn-RoadRunnerNoFrameskip-v4\n",
+ "\n",
+ "Let's load an agent playing Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "B-9QVFIROI5Y"
+ },
+ "outputs": [],
+ "source": [
+ "%%html\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "7ZQNY_r6NJtC"
+ },
+ "source": [
+ "1. We download the model using `rl_zoo3.load_from_hub`, and place it in a new folder that we can call `rl_trained`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "OdBNZHy0NGTR"
+ },
+ "outputs": [],
+ "source": [
+ "# Download model and save it into the logs/ folder\n",
+ "!python -m rl_zoo3.load_from_hub --algo dqn --env BeamRiderNoFrameskip-v4 -orga sb3 -f rl_trained/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LFt6hmWsNdBo"
+ },
+ "source": [
+ "2. Let's evaluate if for 5000 timesteps"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "aOxs0rNuN0uS"
+ },
+ "outputs": [],
+ "source": [
+ "!python enjoy.py --algo dqn --env BeamRiderNoFrameskip-v4 -n 5000 -f rl_trained/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kxMDuDfPON57"
+ },
+ "source": [
+ "Why not trying to train your own **Deep Q-Learning Agent playing BeamRiderNoFrameskip-v4? 🏆.**\n",
+ "\n",
+ "If you want to try, check https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4#hyperparameters **in the model card, you have the hyperparameters of the trained agent.**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xL_ZtUgpOuY6"
+ },
+ "source": [
+ "But finding hyperparameters can be a daunting task. Fortunately, we'll see in the next Unit, how we can **use Optuna for optimizing the Hyperparameters 🔥.**\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-pqaco8W-huW"
+ },
+ "source": [
+ "## Some additional challenges 🏆\n",
+ "The best way to learn **is to try things by your own**!\n",
+ "\n",
+ "In the [Leaderboard](https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
+ "\n",
+ "Here's a list of environments you can try to train your agent with:\n",
+ "- BeamRiderNoFrameskip-v4\n",
+ "- BreakoutNoFrameskip-v4 \n",
+ "- EnduroNoFrameskip-v4\n",
+ "- PongNoFrameskip-v4\n",
+ "\n",
+ "Also, **if you want to learn to implement Deep Q-Learning by yourself**, you definitely should look at CleanRL implementation: https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "paS-XKo4-kmu"
+ },
+ "source": [
+ "________________________________________________________________________\n",
+ "Congrats on finishing this chapter!\n",
+ "\n",
+ "If you’re still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
+ "\n",
+ "Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations.\n",
+ "\n",
+ "In the next unit, **we’re going to learn about [Optuna](https://optuna.org/)**. One of the most critical task in Deep Reinforcement Learning is to find a good set of training hyperparameters. And Optuna is a library that helps you to automate the search.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5WRx7tO7-mvC"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "### This is a course built with you 👷🏿♀️\n",
+ "\n",
+ "Finally, we want to improve and update the course iteratively with your feedback. If you have some, please fill this form 👉 https://forms.gle/3HgA7bEHwAmmLfwh9\n",
+ "\n",
+ "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "See you on [Bonus unit 2](https://github.com/huggingface/deep-rl-class/tree/main/unit2#unit-2-introduction-to-q-learning)! 🔥 TODO CHANGE LINK"
+ ],
+ "metadata": {
+ "id": "Kc3udPT-RcXc"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fS3Xerx0fIMV"
+ },
+ "source": [
+ "### Keep Learning, Stay Awesome 🤗"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "private_outputs": true,
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.6"
+ },
+ "varInspector": {
+ "cols": {
+ "lenName": 16,
+ "lenType": 16,
+ "lenVar": 40
+ },
+ "kernels_config": {
+ "python": {
+ "delete_cmd_postfix": "",
+ "delete_cmd_prefix": "del ",
+ "library": "var_list.py",
+ "varRefreshCmd": "print(var_dic_list())"
+ },
+ "r": {
+ "delete_cmd_postfix": ") ",
+ "delete_cmd_prefix": "rm(",
+ "library": "var_list.r",
+ "varRefreshCmd": "cat(var_dic_list()) "
+ }
+ },
+ "types_to_exclude": [
+ "module",
+ "function",
+ "builtin_function_or_method",
+ "instance",
+ "_Feature"
+ ],
+ "window_display": false
+ },
+ "accelerator": "GPU",
+ "gpuClass": "standard"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/units/en/_toctree.yml b/units/en/_toctree.yml
new file mode 100644
index 0000000..45f4925
--- /dev/null
+++ b/units/en/_toctree.yml
@@ -0,0 +1,46 @@
+- title: Unit 0. Welcome to the course
+ sections:
+ - local: unit0/introduction
+ title: Welcome to the course 🤗
+ - local: unit0/setup
+ title: Setup
+ - local: unit0/discord101
+ title: Discord 101
+- title: Unit 1. Introduction to Deep Reinforcement Learning
+ sections:
+ - local: unit1/introduction
+ title: Introduction
+ - local: unit1/what-is-rl
+ title: What is Reinforcement Learning?
+ - local: unit1/rl-framework
+ title: The Reinforcement Learning Framework
+ - local: unit1/tasks
+ title: The type of tasks
+ - local: unit1/exp-exp-tradeoff
+ title: The Exploration/ Exploitation tradeoff
+ - local: unit1/two-methods
+ title: The two main approaches for solving RL problems
+ - local: unit1/deep-rl
+ title: The “Deep” in Deep Reinforcement Learning
+ - local: unit1/summary
+ title: Summary
+ - local: unit1/hands-on
+ title: Hands-on
+ - local: unit1/quiz
+ title: Quiz
+ - local: unit1/conclusion
+ title: Conclusion
+ - local: unit1/additional-readings
+ title: Additional Readings
+- title: Bonus Unit 1. Introduction to Deep Reinforcement Learning with Huggy
+ sections:
+ - local: unitbonus1/introduction
+ title: Introduction
+ - local: unitbonus1/how-huggy-works
+ title: How Huggy works?
+ - local: unitbonus1/train
+ title: Train Huggy
+ - local: unitbonus1/play
+ title: Play with Huggy
+ - local: unitbonus1/conclusion
+ title: Conclusion
diff --git a/units/en/unit0/discord101.mdx b/units/en/unit0/discord101.mdx
new file mode 100644
index 0000000..f46667f
--- /dev/null
+++ b/units/en/unit0/discord101.mdx
@@ -0,0 +1,33 @@
+# Discord 101 [[discord-101]]
+
+Hey there! My name is Huggy, the dog 🐕, and I'm looking forward to train with you during this RL Course!
+Although I don't know much about bringing sticks (yet), I know one or two things about Discord. So I wrote this guide to help you learn about it!
+
+
+
+Discord is a free chat platform. If you've used Slack, **it's quite similar**. There is a Hugging Face Community Discord server with 18000 members you can join with a single click here. So many humans to play with!
+
+Starting in Discord can be a bit intimidating, so let me take you through it.
+
+When you sign-up to our Discord server, you'll need to specify which topics you're interested in by **clicking #role-assignment at the left**. Here, you can pick different categories. Make sure to **click "Reinforcement Learning"**! :fire:. You'll then get to **introduce yourself in the `#introduction-yourself` channel**.
+
+## So which channels are interesting to me? [[channels]]
+
+They are in the reinforcement learning lounge. **Don't forget to sign up to these channels** by clicking on 🤖 Reinforcement Learning in `role-assigment`.
+- `rl-announcements`: where we give the **lastest information about the course**.
+- `rl-discussions`: where you can **exchange about RL and share information**.
+- `rl-study-group`: where you can **create and join study groups**.
+
+The HF Community Server has a thriving community of human beings interested in many areas, so you can also learn from those. There are paper discussions, events, and many other things.
+
+Was this useful? There are a couple of tips I can share with you:
+
+- There are **voice channels** you can use as well, although most people prefer text chat.
+- You can **use markdown style** for text chats. So if you're writing code, you can use that style. Sadly this does not work as well for links.
+- You can open threads as well! It's a good idea when **it's a long conversation**.
+
+I hope this is useful! And if you have questions, just ask!
+
+See you later!
+
+Huggy 🐶
diff --git a/units/en/unit0/introduction.mdx b/units/en/unit0/introduction.mdx
new file mode 100644
index 0000000..e48f0c4
--- /dev/null
+++ b/units/en/unit0/introduction.mdx
@@ -0,0 +1,126 @@
+# Welcome to the 🤗 Deep Reinforcement Learning Course [[introduction]]
+
+
+
+Welcome to the most fascinating topic in Artificial Intelligence: Deep Reinforcement Learning.
+
+This course will **teach you about Deep Reinforcement Learning from beginner to expert**. It’s completely free and open-source!
+
+In this introduction unit you’ll:
+
+- Learn more about the **course content**.
+- **Define the path** you’re going to take (either self-audit or certification process)
+- Learn more about the **AI vs. AI challenges** you're going to participate to.
+- Learn more **about us**.
+- **Create your Hugging Face account** (it’s free).
+- **Sign-up our Discord server**, the place where you can exchange with your classmates and us (the Hugging Face team).
+
+Let’s get started!
+
+## What to expect? [[expect]]
+
+In this course, you will:
+
+- 📖 Study Deep Reinforcement Learning in **theory and practice.**
+- 🧑💻 Learn to **use famous Deep RL libraries** such as [Stable Baselines3](https://stable-baselines3.readthedocs.io/en/master/), [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), [Sample Factory](https://samplefactory.dev/) and [CleanRL](https://github.com/vwxyzjn/cleanrl).
+- 🤖 **Train agents in unique environments** such as [SnowballFight](https://huggingface.co/spaces/ThomasSimonini/SnowballFight), [Huggy the Doggo 🐶](https://huggingface.co/spaces/ThomasSimonini/Huggy), [MineRL (Minecraft ⛏️)](https://minerl.io/), [VizDoom (Doom)](https://vizdoom.cs.put.edu.pl/) and classical ones such as [Space Invaders](https://www.gymlibrary.dev/environments/atari/) and [PyBullet](https://pybullet.org/wordpress/).
+- 💾 Share your **trained agents with one line of code to the Hub** and also download powerful agents from the community.
+- 🏆 Participate in challenges where you will **evaluate your agents against other teams. You'll also get to play against the agents you'll train.**
+
+And more!
+
+At the end of this course, **you’ll get a solid foundation from the basics to the SOTA (state-of-the-art) methods**.
+
+You can find the syllabus on our website 👉 here
+
+Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**
+
+Sign up 👉 here
+
+
+## What does the course look like? [[course-look-like]]
+The course is composed of:
+
+- *A theory part*: where you learn a **concept in theory (article)**.
+- *A hands-on*: where you’ll learn **to use famous Deep RL libraries** to train your agents in unique environments. These hands-on will be **Google Colab notebooks with companion tutorial videos** if you prefer learning with video format!
+- *Challenges*: you'll get to put your agent to compete against other agents in different challenges. There will also be leaderboards for you to compare the agents' performance.
+
+
+## Two paths: choose your own adventure [[two-paths]]
+
+
+
+You can choose to follow this course either:
+
+- *To get a certificate of completion*: you need to complete 80% of the assignments before the end of March 2023.
+- *As a simple audit*: you can participate in all challenges and do assignments if you want, but you have no deadlines.
+
+Both paths **are completely free**.
+Whatever path you choose, we advise you **to follow the recommended pace to enjoy the course and challenges with your fellow classmates.**
+You don't need to tell us which path you choose. At the end of March, when we verify the assignments **if you get more than 80% of the assignments done, you'll get a certificate.**
+
+
+## How to get most of the course? [[advice]]
+
+To get most of the course, we have some advice:
+
+1. Join or create study groups in Discord : studying in groups is always easier. To do that, you need to join our discord server. If you're new to Discord, no worries! We have some tools that will help you learn about it.
+2. **Do the quizzes and assignments**: the best way to learn is to do and test yourself.
+3. **Define a schedule to stay in sync**: you can use our recommended pace schedule below or create yours.
+
+
+
+## What tools do I need? [[tools]]
+
+You need only 3 things:
+
+- *A computer* with an internet connection.
+- *Google Colab (free version)*: most of our hands-on will use Google Colab, the **free version is enough.**
+- A *Hugging Face Account*: to push and load models. If you don’t have an account yet, you can create one **[here](https://hf.co/join)** (it’s free).
+
+
+
+
+## What is the recommended pace? [[recommended-pace]]
+
+We defined a planning that you can follow to keep up the pace of the course.
+
+
+
+
+
+Each chapter in this course is designed **to be completed in 1 week, with approximately 3-4 hours of work per week**. However, you can take as much time as necessary to complete the course. If you want to dive into a topic more in-depth, we'll provide additional resources to help you achieve that.
+
+
+## Who are we [[who-are-we]]
+About the author:
+
+- Thomas Simonini is a Developer Advocate at Hugging Face 🤗 specializing in Deep Reinforcement Learning. He founded the Deep Reinforcement Learning Course in 2018, which became one of the most used courses in Deep RL.
+
+About the team:
+
+- Omar Sanseviero is a Machine Learning Engineer at Hugging Face where he works in the intersection of ML, Community and Open Source. Previously, Omar worked as a Software Engineer at Google in the teams of Assistant and TensorFlow Graphics. He is from Peru and likes llamas 🦙.
+- Sayak Paul is a Developer Advocate Engineer at Hugging Face. He's interested in the area of representation learning (self-supervision, semi-supervision, model robustness). And he loves watching crime and action thrillers 🔪.
+
+
+## When do the challenges start? [[challenges]]
+
+In this new version of the course, you have two types of challenges:
+- A leaderboard to compare your agent's performance to other classmates'.
+- AI vs. AI challenges where you can train your agent and compete against other classmates' agents.
+
+
+
+These AI vs.AI challenges will be announced **later in December**.
+
+
+## I found a bug, or I want to improve the course [[contribute]]
+
+Contributions are welcomed 🤗
+
+- If you *found a bug 🐛 in a notebook*, please open an issue and **describe the problem**.
+- If you *want to improve the course*, you can open a Pull Request.
+
+## I still have questions [[questions]]
+
+In that case, check our FAQ. And if the question is not in it, ask your question in our discord server #rl-discussions.
diff --git a/units/en/unit0/setup.mdx b/units/en/unit0/setup.mdx
new file mode 100644
index 0000000..4fc55bb
--- /dev/null
+++ b/units/en/unit0/setup.mdx
@@ -0,0 +1,30 @@
+# Setup [[setup]]
+
+After all this information, it's time to get started. We're going to do two things:
+
+1. **Create your Hugging Face account** if it's not already done
+2. **Sign up to Discord and introduce yourself** (don't be shy 🤗)
+
+### Let's create my Hugging Face account
+
+(If it's not already done) create an account to HF here
+
+### Let's join our Discord server
+
+You can now sign up for our Discord Server. This is the place where you **can exchange with the community and with us, create and join study groups to grow each other and more**
+
+👉🏻 Join our discord server here.
+
+When you join, remember to introduce yourself in #introduce-yourself and sign-up for reinforcement channels in #role-assignments.
+
+We have multiple RL-related channels:
+- `rl-announcements`: where we give the last information about the course.
+- `rl-discussions`: where you can exchange about RL and share information.
+- `rl-study-group`: where you can create and join study groups.
+
+If this is your first time using Discord, we wrote a Discord 101 to get the best practices. Check the next section.
+
+Congratulations! **You've just finished the on-boarding**. You're now ready to start to learn Deep Reinforcement Learning. Have fun!
+
+
+### Keep Learning, stay awesome 🤗
diff --git a/units/en/unit1/additional-readings.mdx b/units/en/unit1/additional-readings.mdx
new file mode 100644
index 0000000..b881244
--- /dev/null
+++ b/units/en/unit1/additional-readings.mdx
@@ -0,0 +1,13 @@
+# Additional Readings [[additional-readings]]
+
+These are **optional readings** if you want to go deeper.
+
+## Deep Reinforcement Learning [[deep-rl]]
+
+- [Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 1, 2 and 3](http://incompleteideas.net/book/RLbook2020.pdf)
+- [Foundations of Deep RL Series, L1 MDPs, Exact Solution Methods, Max-ent RL by Pieter Abbeel](https://youtu.be/2GwBez0D20A)
+- [Spinning Up RL by OpenAI Part 1: Key concepts of RL](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)
+
+## Gym [[gym]]
+
+- [Getting Started With OpenAI Gym: The Basic Building Blocks](https://blog.paperspace.com/getting-started-with-openai-gym/)
diff --git a/units/en/unit1/conclusion.mdx b/units/en/unit1/conclusion.mdx
new file mode 100644
index 0000000..de31951
--- /dev/null
+++ b/units/en/unit1/conclusion.mdx
@@ -0,0 +1,16 @@
+# Conclusion [[conclusion]]
+
+Congrats on finishing this unit! **That was the biggest one**, and there was a lot of information. And congrats on finishing the tutorial. You’ve just trained your first Deep RL agents and shared it with the community! 🥳
+
+It's **normal if you still feel confused with some of these elements**. This was the same for me and for all people who studied RL.
+
+**Take time to really grasp the material** before continuing. It’s important to master these elements and having a solid foundations before entering the fun part.
+
+Naturally, during the course, we’re going to use and explain these terms again, but it’s better to understand them before diving into the next units.
+
+In the next (bonus) unit, we’re going to reinforce what we just learned by **training Huggy the Dog to fetch the stick**.
+
+You will be able then to play with him 🤗.
+
+
diff --git a/units/en/unit1/deep-rl.mdx b/units/en/unit1/deep-rl.mdx
new file mode 100644
index 0000000..c0c2247
--- /dev/null
+++ b/units/en/unit1/deep-rl.mdx
@@ -0,0 +1,21 @@
+# The “Deep” in Reinforcement Learning [[deep-rl]]
+
+
+What we've talked about so far is Reinforcement Learning. But where does the "Deep" come into play?
+
+
+Deep Reinforcement Learning introduces **deep neural networks to solve Reinforcement Learning problems** — hence the name “deep”.
+
+For instance, in the next unit, we’ll learn about two value-based algorithms: Q-Learning (classic Reinforcement Learning) and then Deep Q-Learning.
+
+You’ll see the difference is that in the first approach, **we use a traditional algorithm** to create a Q table that helps us find what action to take for each state.
+
+In the second approach, **we will use a Neural Network** (to approximate the Q value).
+
+
+
+Schema inspired by the Q learning notebook by Udacity
+
+
+
+If you are not familiar with Deep Learning you definitely should watch [the FastAI Practical Deep Learning for Coders](https://course.fast.a) (Free).
diff --git a/units/en/unit1/exp-exp-tradeoff.mdx b/units/en/unit1/exp-exp-tradeoff.mdx
new file mode 100644
index 0000000..10798d8
--- /dev/null
+++ b/units/en/unit1/exp-exp-tradeoff.mdx
@@ -0,0 +1,37 @@
+# The Exploration/Exploitation trade-off [[exp-exp-tradeoff]]
+
+Finally, before looking at the different methods to solve Reinforcement Learning problems, we must cover one more very important topic: *the exploration/exploitation trade-off.*
+
+- *Exploration* is exploring the environment by trying random actions in order to **find more information about the environment.**
+- *Exploitation* is **exploiting known information to maximize the reward.**
+
+Remember, the goal of our RL agent is to maximize the expected cumulative reward. However, **we can fall into a common trap**.
+
+Let’s take an example:
+
+
+
+In this game, our mouse can have an **infinite amount of small cheese** (+1 each). But at the top of the maze, there is a gigantic sum of cheese (+1000).
+
+However, if we only focus on exploitation, our agent will never reach the gigantic sum of cheese. Instead, it will only exploit **the nearest source of rewards,** even if this source is small (exploitation).
+
+But if our agent does a little bit of exploration, it can **discover the big reward** (the pile of big cheese).
+
+This is what we call the exploration/exploitation trade-off. We need to balance how much we **explore the environment** and how much we **exploit what we know about the environment.**
+
+Therefore, we must **define a rule that helps to handle this trade-off**. We’ll see the different ways to handle it in the future units.
+
+If it’s still confusing, **think of a real problem: the choice of picking a restaurant:**
+
+
+
+
+Source: Berkley AI Course
+
+
+
+- *Exploitation*: You go every day to the same one that you know is good and **take the risk to miss another better restaurant.**
+- *Exploration*: Try restaurants you never went to before, with the risk of having a bad experience **but the probable opportunity of a fantastic experience.**
+
+To recap:
+
diff --git a/units/en/unit1/hands-on.mdx b/units/en/unit1/hands-on.mdx
new file mode 100644
index 0000000..dfe80c5
--- /dev/null
+++ b/units/en/unit1/hands-on.mdx
@@ -0,0 +1,669 @@
+# Hands on [[hands-on]]
+
+
+
+
+
+Now that you've studied the bases of Reinforcement Learning, you’re ready to train your first agent and share it with the community through the Hub 🔥:
+
+
+
+
+A Lunar Lander agent that will learn to land correctly on the Moon 🌕
+
+
+
+And finally, you'll **upload this trained agent to the Hugging Face Hub 🤗, a free, open platform where people can share ML models, datasets, and demos.**
+
+Thanks to our leaderboard, you'll be able to compare your results with other classmates and exchange the best practices to improve your agent's scores. Who will win the challenge for Unit 1 🏆?
+
+So let's get started! 🚀
+
+**To start the hands-on click on Open In Colab button** 👇 :
+
+[](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/master/notebooks/unit1/unit1.ipynb)
+
+
+# Unit 1: Train your first Deep Reinforcement Learning Agent 🤖
+
+
+In this notebook, you'll train your **first Deep Reinforcement Learning agent** a Lunar Lander agent that will learn to **land correctly on the Moon 🌕**. Using [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) a Deep Reinforcement Learning library, share them with the community, and experiment with different configurations
+
+⬇️ Here is an example of what **you will achieve in just a couple of minutes.** ⬇️
+
+
+
+
+```python
+%%html
+
+```
+
+### The environment 🎮
+
+- [LunarLander-v2](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)
+
+### The library used 📚
+
+- [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/)
+
+We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues).
+
+## Objectives of this notebook 🏆
+
+At the end of the notebook, you will:
+
+- Be able to use **Gym**, the environment library.
+- Be able to use **Stable-Baselines3**, the deep reinforcement learning library.
+- Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score 🔥.
+
+
+
+
+## This notebook is from Deep Reinforcement Learning Course
+
+
+In this free course, you will:
+
+- 📖 Study Deep Reinforcement Learning in **theory and practice**.
+- 🧑💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.
+- 🤖 Train **agents in unique environments**
+
+And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course
+
+Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**
+
+
+The best way to keep in touch and ask questions is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5
+
+## Prerequisites 🏗️
+Before diving into the notebook, you need to:
+
+🔲 📝 **Done Unit 0** that gives you all the **information about the course and help you to onboard** 🤗
+
+🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by doing Unit 1
+
+## A small recap of what is Deep Reinforcement Learning 📚
+
+
+Let's do a small recap on what we learned in the first Unit:
+- Reinforcement Learning is a **computational approach to learning from action**. We build an agent that learns from the environment by **interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.
+
+- The goal of any RL agent is to **maximize its expected cumulative reward** (also called expected return) because RL is based on the _reward hypothesis_, which is that all goals can be described as the maximization of the expected cumulative reward.
+
+- The RL process is a **loop that outputs a sequence of state, action, reward, and next state**.
+
+- To calculate the expected cumulative reward (expected return), **we discount the rewards**: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long-term future reward.
+
+- To solve an RL problem, you want to **find an optimal policy**; the policy is the "brain" of your AI that will tell us what action to take given a state. The optimal one is the one that gives you the actions that max the expected return.
+
+
+There are **two** ways to find your optimal policy:
+- By **training your policy directly**: policy-based methods.
+- By **training a value function** that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods.
+
+- Finally, we spoke about Deep RL because **we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name "deep."**
+
+# Let's train our first Deep Reinforcement Learning agent and upload it to the Hub 🚀
+
+
+## Set the GPU 💪
+- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`
+
+
+
+- `Hardware Accelerator > GPU`
+
+
+
+## Install dependencies and create a virtual screen 🔽
+The first step is to install the dependencies, we’ll install multiple ones.
+
+- `gym[box2D]`: Contains the LunarLander-v2 environment 🌛 (we use `gym==0.21`)
+- `stable-baselines3[extra]`: The deep reinforcement learning library.
+- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face 🤗 Hub.
+
+To make things easier, we created a script to install all these dependencies.
+
+```python
+!apt install swig cmake
+```
+
+TODO CHANGE LINK OF THE REQUIREMENTS
+
+```python
+!pip install -r https://huggingface.co/spaces/ThomasSimonini/temp-space-requirements/raw/main/requirements/requirements-unit1.txt
+```
+
+During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames).
+
+Hence the following cell will install virtual screen libraries and create and run a virtual screen 🖥
+
+```python
+!sudo apt-get update
+!apt install python-opengl
+!apt install ffmpeg
+!apt install xvfb
+!pip3 install pyvirtualdisplay
+```
+
+To make sure the new installed libraries are used, **sometimes it's required to restart the notebook runtime**. The next cell will force the **runtime to crash, so you'll need to connect again and run the code starting from here**. Thanks for this trick, **we will be able to run our virtual screen.**
+
+
+```python
+import os
+os.kill(os.getpid(), 9)
+```
+
+
+```python
+# Virtual display
+from pyvirtualdisplay import Display
+
+virtual_display = Display(visible=0, size=(1400, 900))
+virtual_display.start()
+```
+
+## Import the packages 📦
+
+One additional library we import is huggingface_hub **to be able to upload and download trained models from the hub**.
+
+
+The Hugging Face Hub 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.
+
+You can see here all the Deep reinforcement Learning models available 👉 https://huggingface.co/models?pipeline_tag=reinforcement-learning&sort=downloads
+
+
+
+```python
+import gym
+
+from huggingface_sb3 import load_from_hub, package_to_hub, push_to_hub
+from huggingface_hub import (
+ notebook_login,
+) # To log to our Hugging Face account to be able to upload models to the Hub.
+
+from stable_baselines3 import PPO
+from stable_baselines3.common.evaluation import evaluate_policy
+from stable_baselines3.common.env_util import make_vec_env
+```
+
+## Understand what is Gym and how it works 🤖
+
+🏋 The library containing our environment is called Gym.
+**You'll use Gym a lot in Deep Reinforcement Learning.**
+
+The Gym library provides two things:
+- An interface that allows you to **create RL environments**.
+- A **collection of environments** (gym-control, atari, box2D...).
+
+Let's look at an example, but first let's remember what's the RL Loop.
+
+
+
+At each step:
+- Our Agent receives **state S0** from the **Environment** — we receive the first frame of our game (Environment).
+- Based on that **state S0,** the Agent takes **action A0** — our Agent will move to the right.
+- Environment to a **new** **state S1** — new frame.
+- The environment gives some **reward R1** to the Agent — we’re not dead *(Positive Reward +1)*.
+
+
+With Gym:
+
+1️⃣ We create our environment using `gym.make()`
+
+2️⃣ We reset the environment to its initial state with `observation = env.reset()`
+
+At each step:
+
+3️⃣ Get an action using our model (in our example we take a random action)
+
+4️⃣ Using `env.step(action)`, we perform this action in the environment and get
+- `observation`: The new state (st+1)
+- `reward`: The reward we get after executing the action
+- `done`: Indicates if the episode terminated
+- `info`: A dictionary that provides additional information (depends on the environment).
+
+If the episode is done:
+- We reset the environment to its initial state with `observation = env.reset()`
+
+**Let's look at an example!** Make sure to read the code
+
+
+```python
+import gym
+
+# First, we create our environment called LunarLander-v2
+env = gym.make("LunarLander-v2")
+
+# Then we reset this environment
+observation = env.reset()
+
+for _ in range(20):
+ # Take a random action
+ action = env.action_space.sample()
+ print("Action taken:", action)
+
+ # Do this action in the environment and get
+ # next_state, reward, done and info
+ observation, reward, done, info = env.step(action)
+
+ # If the game is done (in our case we land, crashed or timeout)
+ if done:
+ # Reset the environment
+ print("Environment is reset")
+ observation = env.reset()
+```
+
+## Create the LunarLander environment 🌛 and understand how it works
+
+### The environment 🎮
+
+In this first tutorial, we’re going to train our agent, a [Lunar Lander](https://www.gymlibrary.dev/environments/box2d/lunar_lander/), **to land correctly on the moon**. To do that, the agent needs to learn **to adapt its speed and position(horizontal, vertical, and angular) to land correctly.**
+
+
+---
+
+
+💡 A good habit when you start to use an environment is to check its documentation
+
+👉 https://www.gymlibrary.dev/environments/box2d/lunar_lander/
+
+---
+
+
+Let's see what the Environment looks like:
+
+
+```python
+# We create our environment with gym.make("")
+env = gym.make("LunarLander-v2")
+env.reset()
+print("_____OBSERVATION SPACE_____ \n")
+print("Observation Space Shape", env.observation_space.shape)
+print("Sample observation", env.observation_space.sample()) # Get a random observation
+```
+
+We see with `Observation Space Shape (8,)` that the observation is a vector of size 8, where each value contains different information about the lander:
+- Horizontal pad coordinate (x)
+- Vertical pad coordinate (y)
+- Horizontal speed (x)
+- Vertical speed (y)
+- Angle
+- Angular speed
+- If the left leg has contact point touched the land
+- If the right leg has contact point touched the land
+
+
+```python
+print("\n _____ACTION SPACE_____ \n")
+print("Action Space Shape", env.action_space.n)
+print("Action Space Sample", env.action_space.sample()) # Take a random action
+```
+
+The action space (the set of possible actions the agent can take) is discrete with 4 actions available 🎮:
+
+- Do nothing,
+- Fire left orientation engine,
+- Fire the main engine,
+- Fire right orientation engine.
+
+Reward function (the function that will gives a reward at each timestep) 💰:
+
+- Moving from the top of the screen to the landing pad and zero speed is about 100~140 points.
+- Firing main engine is -0.3 each frame
+- Each leg ground contact is +10 points
+- Episode finishes if the lander crashes (additional - 100 points) or come to rest (+100 points)
+
+#### Vectorized Environment
+
+- We create a vectorized environment (method for stacking multiple independent environments into a single environment) of 16 environments, this way, **we'll have more diverse experiences during the training.**
+
+```python
+# Create the environment
+env = make_vec_env("LunarLander-v2", n_envs=16)
+```
+
+## Create the Model 🤖
+
+- Now that we studied our environment and we understood the problem: **being able to land correctly the Lunar Lander to the Landing Pad by controlling left, right and main orientation engine**. Let's build the algorithm we're going to use to solve this Problem 🚀.
+
+- To do so, we're going to use our first Deep RL library, [Stable Baselines3 (SB3)](https://stable-baselines3.readthedocs.io/en/master/).
+
+- SB3 is a set of **reliable implementations of reinforcement learning algorithms in PyTorch**.
+
+---
+
+💡 A good habit when using a new library is to dive first on the documentation: https://stable-baselines3.readthedocs.io/en/master/ and then try some tutorials.
+
+----
+
+
+
+To solve this problem, we're going to use SB3 **PPO**. [PPO (aka Proximal Policy Optimization) is one of the of the SOTA (state of the art) Deep Reinforcement Learning algorithms that you'll study during this course](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#example%5D).
+
+PPO is a combination of:
+
+- *Value-based reinforcement learning method*: learning an action-value function that will tell us what's the **most valuable action to take given a state and action**.
+- *Policy-based reinforcement learning method*: learning a policy that will **gives us a probability distribution over actions**.
+
+
+Stable-Baselines3 is easy to set up:
+
+1️⃣ You **create your environment** (in our case it was done above)
+
+2️⃣ You define the **model you want to use and instantiate this model** `model = PPO("MlpPolicy")`
+
+3️⃣ You **train the agent** with `model.learn` and define the number of training timesteps
+
+```
+# Create environment
+
+env = gym.make('LunarLander-v2')
+
+# Instantiate the agent
+model = PPO('MlpPolicy', env, verbose=1)
+
+# Train the agent
+model.learn(total_timesteps=int(2e5))
+```
+
+
+
+```python
+# TODO: Define a PPO MlpPolicy architecture
+# We use MultiLayerPerceptron (MLPPolicy) because the input is a vector,
+# if we had frames as input we would use CnnPolicy
+model =
+```
+
+#### Solution
+
+```python
+# SOLUTION
+# We added some parameters to accelerate the training
+model = PPO(
+ policy="MlpPolicy",
+ env=env,
+ n_steps=1024,
+ batch_size=64,
+ n_epochs=4,
+ gamma=0.999,
+ gae_lambda=0.98,
+ ent_coef=0.01,
+ verbose=1,
+)
+```
+
+## Train the PPO agent 🏃
+
+- Let's train our agent for 1,000,000 timesteps, don't forget to use GPU on Colab. It will take approximately ~20min, but you can use less timesteps if you just want to try it out.
+- During the training, take a ☕ break you deserved it 🤗
+
+```python
+# TODO: Train it for 1,000,000 timesteps
+
+# TODO: Specify file name for model and save the model to file
+model_name = ""
+```
+
+#### Solution
+
+```python
+# SOLUTION
+# Train it for 1,000,000 timesteps
+model.learn(total_timesteps=1000000)
+# Save the model
+model_name = "ppo-LunarLander-v2"
+model.save(model_name)
+```
+
+## Evaluate the agent 📈
+
+- Now that our Lunar Lander agent is trained 🚀, we need to **check its performance**.
+- Stable-Baselines3 provides a method to do that: `evaluate_policy`.
+- To fill that part you need to [check the documentation](https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#basic-usage-training-saving-loading)
+- In the next step, we'll see **how to automatically evaluate and share your agent to compete in a leaderboard, but for now let's do it ourselves**
+
+
+💡 When you evaluate your agent, you should not use your training environment but create an evaluation environment.
+
+```python
+# TODO: Evaluate the agent
+# Create a new environment for evaluation
+eval_env =
+
+# Evaluate the model with 10 evaluation episodes and deterministic=True
+mean_reward, std_reward =
+
+# Print the results
+```
+
+#### Solution
+
+```python
+# @title
+eval_env = gym.make("LunarLander-v2")
+mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)
+print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")
+```
+
+- In my case, I got a mean reward is `200.20 +/- 20.80` after training for 1 million steps, which means that our lunar lander agent is ready to land on the moon 🌛🥳.
+
+## Publish our trained model on the Hub 🔥
+Now that we saw we got good results after the training, we can publish our trained model on the Hub 🤗 with one line of code.
+
+📚 The libraries documentation 👉 https://github.com/huggingface/huggingface_sb3/tree/main#hugging-face--x-stable-baselines3-v20
+
+Here's an example of a Model Card (with Space Invaders):
+
+By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.
+
+This way:
+- You can **showcase our work** 🔥
+- You can **visualize your agent playing** 👀
+- You can **share with the community an agent that others can use** 💾
+- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard
+
+
+To be able to share your model with the community there are three more steps to follow:
+
+1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join
+
+2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.
+- Create a new token (https://huggingface.co/settings/tokens) **with write role**
+
+
+
+- Copy the token
+- Run the cell below and paste the token
+
+```python
+notebook_login()
+!git config --global credential.helper store
+```
+
+If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`
+
+3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using `package_to_hub()` function
+
+Let's fill the `package_to_hub` function:
+- `model`: our trained model.
+- `model_name`: the name of the trained model that we defined in `model_save`
+- `model_architecture`: the model architecture we used: in our case PPO
+- `env_id`: the name of the environment, in our case `LunarLander-v2`
+- `eval_env`: the evaluation environment defined in eval_env
+- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `(repo_id = {username}/{repo_name})`
+
+💡 **A good name is `{username}/{model_architecture}-{env_id}` **
+
+- `commit_message`: message of the commit
+
+```python
+import gym
+from stable_baselines3.common.vec_env import DummyVecEnv
+from stable_baselines3.common.env_util import make_vec_env
+
+from huggingface_sb3 import package_to_hub
+
+## TODO: Define a repo_id
+## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2
+repo_id =
+
+# TODO: Define the name of the environment
+env_id =
+
+# Create the evaluation env
+eval_env = DummyVecEnv([lambda: gym.make(env_id)])
+
+
+# TODO: Define the model architecture we used
+model_architecture = ""
+
+## TODO: Define the commit message
+commit_message = ""
+
+# method save, evaluate, generate a model card and record a replay video of your agent before pushing the repo to the hub
+package_to_hub(model=model, # Our trained model
+ model_name=model_name, # The name of our trained model
+ model_architecture=model_architecture, # The model architecture we used: in our case PPO
+ env_id=env_id, # Name of the environment
+ eval_env=eval_env, # Evaluation Environment
+ repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2
+ commit_message=commit_message)
+
+# Note: if after running the package_to_hub function and it gives an issue of rebasing, please run the following code
+# cd && git add . && git commit -m "Add message" && git pull
+# And don't forget to do a "git push" at the end to push the change to the hub.
+```
+
+#### Solution
+
+
+```python
+import gym
+
+from stable_baselines3 import PPO
+from stable_baselines3.common.vec_env import DummyVecEnv
+from stable_baselines3.common.env_util import make_vec_env
+
+from huggingface_sb3 import package_to_hub
+
+# PLACE the variables you've just defined two cells above
+# Define the name of the environment
+env_id = "LunarLander-v2"
+
+# TODO: Define the model architecture we used
+model_architecture = "PPO"
+
+## Define a repo_id
+## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2
+## CHANGE WITH YOUR REPO ID
+repo_id = "ThomasSimonini/ppo-LunarLander-v2" # Change with your repo id, you can't push with mine 😄
+
+## Define the commit message
+commit_message = "Upload PPO LunarLander-v2 trained agent"
+
+# Create the evaluation env
+eval_env = DummyVecEnv([lambda: gym.make(env_id)])
+
+# PLACE the package_to_hub function you've just filled here
+package_to_hub(
+ model=model, # Our trained model
+ model_name=model_name, # The name of our trained model
+ model_architecture=model_architecture, # The model architecture we used: in our case PPO
+ env_id=env_id, # Name of the environment
+ eval_env=eval_env, # Evaluation Environment
+ repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2
+ commit_message=commit_message,
+)
+```
+
+Congrats 🥳 you've just trained and uploaded your first Deep Reinforcement Learning agent. The script above should have displayed a link to a model repository such as https://huggingface.co/osanseviero/test_sb3. When you go to this link, you can:
+* see a video preview of your agent at the right.
+* click "Files and versions" to see all the files in the repository.
+* click "Use in stable-baselines3" to get a code snippet that shows how to load the model.
+* a model card (`README.md` file) which gives a description of the model
+
+Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.
+
+Compare the results of your LunarLander-v2 with your classmates using the leaderboard 🏆 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard
+
+## Load a saved LunarLander model from the Hub 🤗
+Thanks to [ironbar](https://github.com/ironbar) for the contribution.
+
+Loading a saved model from the Hub is really easy.
+
+You go https://huggingface.co/models?library=stable-baselines3 to see the list of all the Stable-baselines3 saved models.
+1. You select one and copy its repo_id
+
+
+
+2. Then we just need to use load_from_hub with:
+- The repo_id
+- The filename: the saved model inside the repo and its extension (*.zip)
+
+```python
+from huggingface_sb3 import load_from_hub
+
+repo_id = "Classroom-workshop/assignment2-omar" # The repo_id
+filename = "ppo-LunarLander-v2.zip" # The model filename.zip
+
+# When the model was trained on Python 3.8 the pickle protocol is 5
+# But Python 3.6, 3.7 use protocol 4
+# In order to get compatibility we need to:
+# 1. Install pickle5 (we done it at the beginning of the colab)
+# 2. Create a custom empty object we pass as parameter to PPO.load()
+custom_objects = {
+ "learning_rate": 0.0,
+ "lr_schedule": lambda _: 0.0,
+ "clip_range": lambda _: 0.0,
+}
+
+checkpoint = load_from_hub(repo_id, filename)
+model = PPO.load(checkpoint, custom_objects=custom_objects, print_system_info=True)
+```
+
+Let's evaluate this agent:
+
+```python
+# @title
+eval_env = gym.make("LunarLander-v2")
+mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)
+print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")
+```
+
+## Some additional challenges 🏆
+The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results!
+
+In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?
+
+Here are some ideas to achieve so:
+* Train more steps
+* Try different hyperparameters of `PPO`. You can see them at https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#parameters.
+* Check the [Stable-Baselines3 documentation](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) and try another models such as DQN.
+* **Push your new trained model** on the Hub 🔥
+
+**Compare the results of your LunarLander-v2 with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) 🏆
+
+Is moon landing too boring to you? Try to **change the environment**, why not using MountainCar-v0, CartPole-v1 or CarRacing-v0? Check how they works [using the gym documentation](https://www.gymlibrary.dev/) and have fun 🎉.
+
+________________________________________________________________________
+Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.**
+
+If you’re still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**
+
+Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations.
+
+Naturally, during the course, we’re going to use and deeper explain again these terms but **it’s better to have a good understanding of them now before diving into the next chapters.**
+
+
+Next time, in the bonus unit 1, you'll train Huggy the Dog to fetch the stick.
+
+
+
+## Keep learning, stay awesome 🤗
diff --git a/units/en/unit1/introduction.mdx b/units/en/unit1/introduction.mdx
new file mode 100644
index 0000000..660ec20
--- /dev/null
+++ b/units/en/unit1/introduction.mdx
@@ -0,0 +1,28 @@
+# Introduction to Deep Reinforcement Learning [[introduction-to-deep-reinforcement-learning]]
+
+
+
+
+Welcome to the most fascinating topic in Artificial Intelligence: **Deep Reinforcement Learning.**
+
+Deep RL is a type of Machine Learning where an agent learns **how to behave** in an environment **by performing actions** and **seeing the results.**
+
+In this first unit, **you'll learn the foundations of Deep Reinforcement Learning.**
+
+
+Then, you'll **train your Deep Reinforcement Learning agent, a lunar lander to land correctly on the Moon** using Stable-Baselines3 , a Deep Reinforcement Learning library.
+
+
+
+
+And finally, you'll **upload this trained agent to the Hugging Face Hub 🤗, a free, open platform where people can share ML models, datasets, and demos.**
+
+It's essential **to master these elements** before diving into implementing Deep Reinforcement Learning agents. The goal of this chapter is to give you solid foundations.
+
+
+After this unit, in a bonus unit, you'll be **able to train Huggy the Dog 🐶 to fetch the stick and play with him 🤗**.
+
+
+
+So let's get started! 🚀
diff --git a/units/en/unit1/quiz.mdx b/units/en/unit1/quiz.mdx
new file mode 100644
index 0000000..5dece14
--- /dev/null
+++ b/units/en/unit1/quiz.mdx
@@ -0,0 +1,168 @@
+# Quiz [[quiz]]
+
+The best way to learn and [to avoid the illusion of competence](https://www.coursera.org/lecture/learning-how-to-learn/illusions-of-competence-BuFzf) **is to test yourself.** This will help you to find **where you need to reinforce your knowledge**.
+
+### Q1: What is Reinforcement Learning?
+
+
+Solution
+
+Reinforcement learning is a **framework for solving control tasks (also called decision problems)** by building agents that learn from the environment by interacting with it through trial and error and **receiving rewards (positive or negative) as unique feedback**.
+
+
+
+
+
+### Q2: Define the RL Loop
+
+
+
+At every step:
+- Our Agent receives ______ from the environment
+- Based on that ______ the Agent takes an ______
+- Our Agent will move to the right
+- The Environment goes to a ______
+- The Environment gives a ______ to the Agent
+
+
+
+
+### Q3: What's the difference between a state and an observation?
+
+
+
+### Q4: A task is an instance of a Reinforcement Learning problem. What are the two types of tasks?
+
+
+
+### Q5: What is the exploration/exploitation tradeoff?
+
+
+Solution
+
+In Reinforcement Learning, we need to **balance how much we explore the environment and how much we exploit what we know about the environment**.
+
+- *Exploration* is exploring the environment by **trying random actions in order to find more information about the environment**.
+
+- *Exploitation* is **exploiting known information to maximize the reward**.
+
+
+
+
+
+
+### Q6: What is a policy?
+
+
+Solution
+
+- The Policy π **is the brain of our Agent**. It’s the function that tells us what action to take given the state we are in. So it defines the agent’s behavior at a given time.
+
+
+
+
+
+
+### Q7: What are value-based methods?
+
+
+Solution
+
+- Value-based methods is one of the main approaches for solving RL problems.
+- In Value-based methods, instead of training a policy function, **we train a value function that maps a state to the expected value of being at that state**.
+
+
+
+
+
+### Q8: What are policy-based methods?
+
+
+Solution
+
+- In *Policy-Based Methods*, we learn a **policy function directly**.
+- This policy function will **map from each state to the best corresponding action at that state**. Or a **probability distribution over the set of possible actions at that state**.
+
+
+
+
+
+
+
+Congrats on finishing this Quiz 🥳, if you missed some elements, take time to read again the chapter to reinforce (😏) your knowledge, but **do not worry**: during the course we'll go over again of these concepts, and you'll **reinforce your theoretical knowledge with hands-on**.
diff --git a/units/en/unit1/rl-framework.mdx b/units/en/unit1/rl-framework.mdx
new file mode 100644
index 0000000..d5d62ab
--- /dev/null
+++ b/units/en/unit1/rl-framework.mdx
@@ -0,0 +1,144 @@
+# The Reinforcement Learning Framework [[the-reinforcement-learning-framework]]
+
+## The RL Process [[the-rl-process]]
+
+
+
+The RL Process: a loop of state, action, reward and next state
+Source: Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto
+
+
+To understand the RL process, let’s imagine an agent learning to play a platform game:
+
+
+
+- Our Agent receives **state \\(S_0\\)** from the **Environment** — we receive the first frame of our game (Environment).
+- Based on that **state \\(S_0\\),** the Agent takes **action \\(A_0\\)** — our Agent will move to the right.
+- Environment goes to a **new** **state \\(S_1\\)** — new frame.
+- The environment gives some **reward \\(R_1\\)** to the Agent — we’re not dead *(Positive Reward +1)*.
+
+This RL loop outputs a sequence of **state, action, reward and next state.**
+
+
+
+The agent's goal is to _maximize_ its cumulative reward, **called the expected return.**
+
+## The reward hypothesis: the central idea of Reinforcement Learning [[reward-hypothesis]]
+
+⇒ Why is the goal of the agent to maximize the expected return?
+
+Because RL is based on the **reward hypothesis**, which is that all goals can be described as the **maximization of the expected return** (expected cumulative reward).
+
+That’s why in Reinforcement Learning, **to have the best behavior,** we aim to learn to take actions that **maximize the expected cumulative reward.**
+
+
+## Markov Property [[markov-property]]
+
+In papers, you’ll see that the RL process is called the **Markov Decision Process** (MDP).
+
+We’ll talk again about the Markov Property in the following units. But if you need to remember something today about it, it's this: the Markov Property implies that our agent needs **only the current state to decide** what action to take and **not the history of all the states and actions** they took before.
+
+## Observations/States Space [[obs-space]]
+
+Observations/States are the **information our agent gets from the environment.** In the case of a video game, it can be a frame (a screenshot). In the case of the trading agent, it can be the value of a certain stock, etc.
+
+There is a differentiation to make between *observation* and *state*, however:
+
+- *State s*: is **a complete description of the state of the world** (there is no hidden information). In a fully observed environment.
+
+
+
+
+In chess game, we receive a state from the environment since we have access to the whole check board information.
+
+
+In a chess game, we have access to the whole board information, so we receive a state from the environment. In other words, the environment is fully observed.
+
+- *Observation o*: is a **partial description of the state.** In a partially observed environment.
+
+
+
+In Super Mario Bros, we only see a part of the level close to the player, so we receive an observation.
+
+
+In Super Mario Bros, we only see a part of the level close to the player, so we receive an observation.
+
+In Super Mario Bros, we are in a partially observed environment. We receive an observation **since we only see a part of the level.**
+
+
+In this course, we use the term "state" to denote both state and observation, but we will make the distinction in implementations.
+
+
+To recap:
+
+
+
+## Action Space [[action-space]]
+
+The Action space is the set of **all possible actions in an environment.**
+
+The actions can come from a *discrete* or *continuous space*:
+
+- *Discrete space*: the number of possible actions is **finite**.
+
+
+
+Again, in Super Mario Bros, we have only 5 possible actions: 4 directions and jumping
+
+
+
+In Super Mario Bros, we have a finite set of actions since we have only 4 directions and jump.
+
+- *Continuous space*: the number of possible actions is **infinite**.
+
+
+
+A Self Driving Car agent has an infinite number of possible actions since it can turn left 20°, 21,1°, 21,2°, honk, turn right 20°…
+
+
+
+To recap:
+
+
+Taking this information into consideration is crucial because it will **have importance when choosing the RL algorithm in the future.**
+
+## Rewards and the discounting [[rewards]]
+
+The reward is fundamental in RL because it’s **the only feedback** for the agent. Thanks to it, our agent knows **if the action taken was good or not.**
+
+The cumulative reward at each time step **t** can be written as:
+
+
+
+The cumulative reward equals to the sum of all rewards of the sequence.
+
+
+
+Which is equivalent to:
+
+
+
+The cumulative reward = rt+1 (rt+k+1 = rt+0+1 = rt+1)+ rt+2 (rt+k+1 = rt+1+1 = rt+2) + ...
+
+
+
+However, in reality, **we can’t just add them like that.** The rewards that come sooner (at the beginning of the game) **are more likely to happen** since they are more predictable than the long-term future reward.
+
+Let’s say your agent is this tiny mouse that can move one tile each time step, and your opponent is the cat (that can move too). The mouse's goal is **to eat the maximum amount of cheese before being eaten by the cat.**
+
+
+
+As we can see in the diagram, **it’s more probable to eat the cheese near us than the cheese close to the cat** (the closer we are to the cat, the more dangerous it is).
+
+Consequently, **the reward near the cat, even if it is bigger (more cheese), will be more discounted** since we’re not really sure we’ll be able to eat it.
+
+To discount the rewards, we proceed like this:
+
+1. We define a discount rate called gamma. **It must be between 0 and 1.** Most of the time between **0.99 and 0.95**.
+- The larger the gamma, the smaller the discount. This means our agent **cares more about the long-term reward.**
+- On the other hand, the smaller the gamma, the bigger the discount. This means our **agent cares more about the short term reward (the nearest cheese).**
+
+2. Then, each reward will be discounted by gamma to the exponent of the time step. As the time step increases, the cat gets closer to us, **so the future reward is less and less likely to happen.**
+
+Our discounted expected cumulative reward is:
+
diff --git a/units/en/unit1/summary.mdx b/units/en/unit1/summary.mdx
new file mode 100644
index 0000000..05c57ff
--- /dev/null
+++ b/units/en/unit1/summary.mdx
@@ -0,0 +1,19 @@
+# Summary [[summary]]
+
+That was a lot of information! Let's summarize:
+
+- Reinforcement Learning is a computational approach of learning from action. We build an agent that learns from the environment **by interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.
+
+- The goal of any RL agent is to maximize its expected cumulative reward (also called expected return) because RL is based on the **reward hypothesis**, which is that **all goals can be described as the maximization of the expected cumulative reward.**
+
+- The RL process is a loop that outputs a sequence of **state, action, reward and next state.**
+
+- To calculate the expected cumulative reward (expected return), we discount the rewards: the rewards that come sooner (at the beginning of the game) **are more probable to happen since they are more predictable than the long term future reward.**
+
+- To solve an RL problem, you want to **find an optimal policy**. The policy is the “brain” of your agent, which will tell us **what action to take given a state.** The optimal policy is the one which **gives you the actions that maximize the expected return.**
+
+- There are two ways to find your optimal policy:
+ 1. By training your policy directly: **policy-based methods.**
+ 2. By training a value function that tells us the expected return the agent will get at each state and use this function to define our policy: **value-based methods.**
+
+- Finally, we speak about Deep RL because we introduce **deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based)** hence the name “deep”.
diff --git a/units/en/unit1/tasks.mdx b/units/en/unit1/tasks.mdx
new file mode 100644
index 0000000..1be4fea
--- /dev/null
+++ b/units/en/unit1/tasks.mdx
@@ -0,0 +1,27 @@
+# Type of tasks [[tasks]]
+
+A task is an **instance** of a Reinforcement Learning problem. We can have two types of tasks: **episodic** and **continuing**.
+
+## Episodic task [[episodic-task]]
+
+In this case, we have a starting point and an ending point **(a terminal state). This creates an episode**: a list of States, Actions, Rewards, and new States.
+
+For instance, think about Super Mario Bros: an episode begin at the launch of a new Mario Level and ending **when you’re killed or you reached the end of the level.**
+
+
+
+Beginning of a new episode.
+
+
+
+
+## Continuing tasks [[continuing-tasks]]
+
+These are tasks that continue forever (no terminal state). In this case, the agent must **learn how to choose the best actions and simultaneously interact with the environment.**
+
+For instance, an agent that does automated stock trading. For this task, there is no starting point and terminal state. **The agent keeps running until we decide to stop it.**
+
+
+
+To recap:
+
diff --git a/units/en/unit1/two-methods.mdx b/units/en/unit1/two-methods.mdx
new file mode 100644
index 0000000..4f3d8a8
--- /dev/null
+++ b/units/en/unit1/two-methods.mdx
@@ -0,0 +1,99 @@
+# Two main approaches for solving RL problems [[two-methods]]
+
+
+Now that we learned the RL framework, how do we solve the RL problem?
+
+
+In other terms, how to build an RL agent that can **select the actions that maximize its expected cumulative reward?**
+
+## The Policy π: the agent’s brain [[policy]]
+
+The Policy **π** is the **brain of our Agent**, it’s the function that tells us what **action to take given the state we are.** So it **defines the agent’s behavior** at a given time.
+
+
+
+Think of policy as the brain of our agent, the function that will tell us the action to take given a state
+
+
+
+
+Think of policy as the brain of our agent, the function that will tells us the action to take given a state
+
+This Policy **is the function we want to learn**, our goal is to find the optimal policy π*, the policy that** maximizes **expected return** when the agent acts according to it. We find this *π through training.**
+
+There are two approaches to train our agent to find this optimal policy π*:
+
+- **Directly,** by teaching the agent to learn which **action to take,** given the current state: **Policy-Based Methods.**
+- Indirectly, **teach the agent to learn which state is more valuable** and then take the action that **leads to the more valuable states**: Value-Based Methods.
+
+## Policy-Based Methods [[policy-based]]
+
+In Policy-Based methods, **we learn a policy function directly.**
+
+This function will define a mapping between each state and the best corresponding action. We can also say that it'll define **a probability distribution over the set of possible actions at that state.**
+
+
+
+As we can see here, the policy (deterministic) directly indicates the action to take for each step.
+
+
+
+
+We have two types of policies:
+
+
+- *Deterministic*: a policy at a given state **will always return the same action.**
+
+
+
+action = policy(state)
+
+
+
+
+
+- *Stochastic*: outputs **a probability distribution over actions.**
+
+
+
+policy(actions | state) = probability distribution over the set of actions given the current state
+
+
+
+
+
+Given an initial state, our stochastic policy will output probability distributions over the possible actions at that state.
+
+
+
+
+If we recap:
+
+
+
+
+
+## Value-based methods [[value-based]]
+
+In value-based methods, instead of training a policy function, we **train a value function** that maps a state to the expected value **of being at that state.**
+
+The value of a state is the **expected discounted return** the agent can get if it **starts in that state, and then act according to our policy.**
+
+“Act according to our policy” just means that our policy is **“going to the state with the highest value”.**
+
+
+
+Here we see that our value function **defined value for each possible state.**
+
+
+
+Thanks to our value function, at each step our policy will select the state with the biggest value defined by the value function: -7, then -6, then -5 (and so on) to attain the goal.
+
+
+
+Thanks to our value function, at each step our policy will select the state with the biggest value defined by the value function: -7, then -6, then -5 (and so on) to attain the goal.
+
+If we recap:
+
+
+
diff --git a/units/en/unit1/what-is-rl.mdx b/units/en/unit1/what-is-rl.mdx
new file mode 100644
index 0000000..410569c
--- /dev/null
+++ b/units/en/unit1/what-is-rl.mdx
@@ -0,0 +1,40 @@
+# What is Reinforcement Learning? [[what-is-reinforcement-learning]]
+
+To understand Reinforcement Learning, let’s start with the big picture.
+
+## The big picture [[the-big-picture]]
+
+The idea behind Reinforcement Learning is that an agent (an AI) will learn from the environment by **interacting with it** (through trial and error) and **receiving rewards** (negative or positive) as feedback for performing actions.
+
+Learning from interactions with the environment **comes from our natural experiences.**
+
+For instance, imagine putting your little brother in front of a video game he never played, giving him a controller, and leaving him alone.
+
+
+
+
+Your brother will interact with the environment (the video game) by pressing the right button (action). He got a coin, that’s a +1 reward. It’s positive, he just understood that in this game **he must get the coins.**
+
+
+
+But then, **he presses right again** and he touches an enemy. He just died, so that's a -1 reward.
+
+
+
+
+By interacting with his environment through trial and error, your little brother understood that **he needed to get coins in this environment but avoid the enemies.**
+
+**Without any supervision**, the child will get better and better at playing the game.
+
+That’s how humans and animals learn, **through interaction.** Reinforcement Learning is just a **computational approach of learning from actions.**
+
+
+### A formal definition [[a-formal-definition]]
+
+If we take now a formal definition:
+
+
+Reinforcement learning is a framework for solving control tasks (also called decision problems) by building agents that learn from the environment by interacting with it through trial and error and receiving rewards (positive or negative) as unique feedback.
+
+
+But how does Reinforcement Learning work?
diff --git a/units/en/unitbonus1/conclusion.mdx b/units/en/unitbonus1/conclusion.mdx
new file mode 100644
index 0000000..57cd254
--- /dev/null
+++ b/units/en/unitbonus1/conclusion.mdx
@@ -0,0 +1,10 @@
+# Conclusion [[conclusion]]
+
+Congrats on finishing this bonus unit!
+
+You can now sit and enjoy playing with your Huggy 🐶. And don't **forget to spread the love by sharing Huggy with your friends 🤗**. And if you share about it on social media, **please tag us @huggingface and me @simoninithomas**
+
+
+
+
+### Keep Learning, Stay Awesome 🤗
diff --git a/units/en/unitbonus1/how-huggy-works.mdx b/units/en/unitbonus1/how-huggy-works.mdx
new file mode 100644
index 0000000..bc1ac4d
--- /dev/null
+++ b/units/en/unitbonus1/how-huggy-works.mdx
@@ -0,0 +1,65 @@
+# How Huggy works [[how-huggy-works]]
+
+Huggy is a Deep Reinforcement Learning environment made by Hugging Face and based on [Puppo the Corgi, a project by the Unity MLAgents team](https://blog.unity.com/technology/puppo-the-corgi-cuteness-overload-with-the-unity-ml-agents-toolkit).
+This environment was created using the [Unity game engine](https://unity.com/) and [MLAgents](https://github.com/Unity-Technologies/ml-agents). ML-Agents is a toolkit for the game engine from Unity that allows us to **create environments using Unity or use pre-made environments to train our agents**.
+
+
+
+In this environment we aim to train Huggy to **fetch the stick we throw at him. It means he needs to move correctly toward the stick**.
+
+## The State Space, what Huggy perceives. [[state-space]]
+Huggy doesn't "see" his environment. Instead, we provide him information about the environment:
+* The target (stick) position
+* The relative position between himself and the target
+* The orientation of his legs.
+
+Given all this information, Huggy can **use his policy to determine which action to take next to fulfill his goal**.
+
+## The Action Space, what moves Huggy can perform [[action-space]]
+
+
+**Joint motors drive Huggy legs**. It means that to get the target, Huggy needs to **learn to rotate the joint motors of each of his legs correctly so he can move**.
+
+## The Reward Function [[reward-function]]
+
+The reward function is designed so that **Huggy will fulfill his goal**: fetch the stick.
+
+Remember that one of the foundations of Reinforcement Learning is the *reward hypothesis*: a goal can be described as the **maximization of the expected cumulative reward**.
+
+Here, our goal is that Huggy **goes towards the stick but without spinning too much**. Hence, our reward function must translate this goal.
+
+Our reward function:
+
+
+
+- *Orientation bonus*: we **reward him for getting close to the target**.
+- *Time penalty*: a fixed-time penalty given at every action to **force him to get to the stick as fast as possible**.
+- *Rotation penalty*: we penalize Huggy if **he spins too much and turns too quickly**.
+- *Getting to the target reward*: we reward Huggy for **reaching the target**.
+
+If you want to see what this reward function looks like mathematically, check [Puppo the Corgi presentation](https://blog.unity.com/technology/puppo-the-corgi-cuteness-overload-with-the-unity-ml-agents-toolkit).
+
+## Train Huggy
+
+Huggy aims **to learn to run correctly and as fast as possible toward the goal**. To do that, at every step and given the environment observation, he needs to decide how to rotate each joint motor of his legs to move correctly (not spinning too much) and towards the goal.
+
+The training loop looks like this:
+
+
+
+
+The training environment looks like this:
+
+
+
+
+It's a place where a **stick is spawned randomly**. When Huggy reaches it, the stick get spawned somewhere else.
+We built **multiple copies of the environment for the training**. This helps speed up the training by providing more diverse experiences.
+
+
+
+Now that you have the big picture of the environment, you're ready to train Huggy to fetch the stick.
+
+To do that, we're going to use [MLAgents](https://github.com/Unity-Technologies/ml-agents). Don't worry if you have never used it before. In this unit we'll use Google Colab to train Huggy, and then you'll be able to load your trained Huggy and play with him directly in the browser.
+
+In a future unit, we will study more in-depth MLAgents and how it works. But for now, we keep things simple by just using the provided implementation.
diff --git a/units/en/unitbonus1/introduction.mdx b/units/en/unitbonus1/introduction.mdx
new file mode 100644
index 0000000..68a57f9
--- /dev/null
+++ b/units/en/unitbonus1/introduction.mdx
@@ -0,0 +1,7 @@
+# Introduction [[introduction]]
+
+In this bonus unit, we'll reinforce what we learned in the first unit by teaching Huggy the Dog to fetch the stick and then [play with him directly in your browser](https://huggingface.co/spaces/ThomasSimonini/Huggy) 🐶
+
+
+
+So let's get started 🚀
diff --git a/units/en/unitbonus1/play.mdx b/units/en/unitbonus1/play.mdx
new file mode 100644
index 0000000..f2154cc
--- /dev/null
+++ b/units/en/unitbonus1/play.mdx
@@ -0,0 +1,19 @@
+# Play with Huggy [[play]]
+
+Now that you've trained Huggy and pushed it to the Hub. **You will be able to play with him ❤️**
+
+For this step it’s simple:
+
+- Open the game Huggy in your browser: https://huggingface.co/spaces/ThomasSimonini/Huggy
+
+- Click on Play with my Huggy model
+
+
+
+1. In step 1, choose your model repository which is the model id (in my case ThomasSimonini/ppo-Huggy).
+
+2. In step 2, **choose what model you want to replay**:
+ - I have multiple one, since we saved a model every 500000 timesteps.
+ - But if I want the more recent I choose Huggy.onnx
+
+👉 What’s nice **is to try with different models step to see the improvement of the agent.**
diff --git a/units/en/unitbonus1/train.mdx b/units/en/unitbonus1/train.mdx
new file mode 100644
index 0000000..b353f86
--- /dev/null
+++ b/units/en/unitbonus1/train.mdx
@@ -0,0 +1,260 @@
+# Let's train and play with Huggy 🐶 [[train]]
+
+
+
+
+
+
+## Let's train Huggy 🐶
+
+**To start to train Huggy, click on Open In Colab button** 👇 :
+
+[](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/master/notebooks/bonus-unit1/bonus-unit1.ipynb)
+
+
+
+In this notebook, we'll reinforce what we learned in the first Unit by **teaching Huggy the Dog to fetch the stick and then play with it directly in your browser**
+
+⬇️ Here is an example of what **you will achieve at the end of the unit.** ⬇️ (launch ▶ to see)
+
+```python
+%%html
+
+```
+
+### The environment 🎮
+
+- Huggy the Dog, an environment created by [Thomas Simonini](https://twitter.com/ThomasSimonini) based on [Puppo The Corgi](https://blog.unity.com/technology/puppo-the-corgi-cuteness-overload-with-the-unity-ml-agents-toolkit)
+
+### The library used 📚
+
+- [MLAgents (Hugging Face version)](https://github.com/huggingface/ml-agents)
+
+We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues).
+
+## Objectives of this notebook 🏆
+
+At the end of the notebook, you will:
+
+- Understand **the state space, action space and reward function used to train Huggy**.
+- **Train your own Huggy** to fetch the stick.
+- Be able to play **with your trained Huggy directly in your browser**.
+
+
+## Prerequisites 🏗️
+
+Before diving into the notebook, you need to:
+
+🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by doing Unit 1
+
+🔲 📚 **Read the introduction to Huggy** by doing Bonus Unit 1
+
+## Set the GPU 💪
+- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`
+
+
+
+- `Hardware Accelerator > GPU`
+
+
+
+## Clone the repository and install the dependencies 🔽
+
+- We need to clone the repository, that **contains the experimental version of the library that allows you to push your trained agent to the Hub.**
+
+```bash
+# Clone this specific repository (can take 3min)
+git clone https://github.com/huggingface/ml-agents/
+```
+
+```bash
+# Go inside the repository and install the package (can take 3min)
+%cd ml-agents
+pip3 install -e ./ml-agents-envs
+pip3 install -e ./ml-agents
+```
+
+## Download and move the environment zip file in `./trained-envs-executables/linux/`
+
+- Our environment executable is in a zip file.
+- We need to download it and place it to `./trained-envs-executables/linux/`
+
+```bash
+mkdir ./trained-envs-executables
+mkdir ./trained-envs-executables/linux
+```
+
+```bash
+wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1zv3M95ZJTWHUVOWT6ckq_cm98nft8gdF' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1zv3M95ZJTWHUVOWT6ckq_cm98nft8gdF" -O ./trained-envs-executables/linux/Huggy.zip && rm -rf /tmp/cookies.txt
+```
+
+Download the file Huggy.zip from https://drive.google.com/uc?export=download&id=1zv3M95ZJTWHUVOWT6ckq_cm98nft8gdF using `wget`. Check out the full solution to download large files from GDrive [here](https://bcrf.biochem.wisc.edu/2021/02/05/download-google-drive-files-using-wget/)
+
+```bash
+%%capture
+unzip -d ./trained-envs-executables/linux/ ./trained-envs-executables/linux/Huggy.zip
+```
+
+Make sure your file is accessible
+
+```bash
+chmod -R 755 ./trained-envs-executables/linux/Huggy
+```
+
+## Let's recap how this environment works
+
+### The State Space: what Huggy perceives.
+
+Huggy doesn't "see" his environment. Instead, we provide him information about the environment:
+
+- The target (stick) position
+- The relative position between himself and the target
+- The orientation of his legs.
+
+Given all this information, Huggy **can decide which action to take next to fulfill his goal**.
+
+
+
+
+### The Action Space: what moves Huggy can do
+
+
+**Joint motors drive huggy legs**. It means that to get the target, Huggy needs to **learn to rotate the joint motors of each of his legs correctly so he can move**.
+
+### The Reward Function
+
+The reward function is designed so that **Huggy will fulfill his goal** : fetch the stick.
+
+Remember that one of the foundations of Reinforcement Learning is the *reward hypothesis*: a goal can be described as the **maximization of the expected cumulative reward**.
+
+Here, our goal is that Huggy **goes towards the stick but without spinning too much**. Hence, our reward function must translate this goal.
+
+Our reward function:
+
+
+
+- *Orientation bonus*: we **reward him for getting close to the target**.
+- *Time penalty*: a fixed-time penalty given at every action to **force him to get to the stick as fast as possible**.
+- *Rotation penalty*: we penalize Huggy if **he spins too much and turns too quickly**.
+- *Getting to the target reward*: we reward Huggy for **reaching the target**.
+
+## Check the Huggy config file
+
+- In ML-Agents, you define the **training hyperparameters into config.yaml files.**
+
+- For the scope of this notebook, we're not going to modify the hyperparameters, but if you want to try as an experiment, you should also try to modify some other hyperparameters, Unity provides very [good documentation explaining each of them here](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Training-Configuration-File.md).
+
+- **In the case you want to modify the hyperparameters**, in Google Colab notebook, you can click here to open the config.yaml: `/content/ml-agents/config/ppo/Huggy.yaml`
+
+
+We’re now ready to train our agent 🔥.
+
+## Train our agent
+
+To train our agent, we just need to **launch mlagents-learn and select the executable containing the environment.**
+
+
+
+With ML Agents, we run a training script. We define four parameters:
+
+1. `mlagents-learn `: the path where the hyperparameter config file is.
+2. `--env`: where the environment executable is.
+3. `--run_id`: the name you want to give to your training run id.
+4. `--no-graphics`: to not launch the visualization during the training.
+
+Train the model and use the `--resume` flag to continue training in case of interruption.
+
+> It will fail first time when you use `--resume`, try running the block again to bypass the error.
+
+
+
+The training will take 30 to 45min depending on your machine (don't forget to **set up a GPU**), go take a ☕️you deserve it 🤗.
+
+```bash
+mlagents-learn ./config/ppo/Huggy.yaml --env=./trained-envs-executables/linux/Huggy/Huggy --run-id="Huggy" --no-graphics
+```
+
+## Push the agent to the 🤗 Hub
+
+- Now that we trained our agent, we’re **ready to push it to the Hub to be able to play with Huggy on your browser🔥.**
+
+To be able to share your model with the community there are three more steps to follow:
+
+1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join
+
+2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.
+- Create a new token (https://huggingface.co/settings/tokens) **with write role**
+
+
+
+- Copy the token
+- Run the cell below and paste the token
+
+```python
+from huggingface_hub import notebook_login
+
+notebook_login()
+```
+
+If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`
+
+Then, we simply need to run `mlagents-push-to-hf`.
+
+
+
+And we define 4 parameters:
+
+1. `--run-id`: the name of the training run id.
+2. `--local-dir`: where the agent was saved, it’s results/, so in my case results/First Training.
+3. `--repo-id`: the name of the Hugging Face repo you want to create or update. It’s always /
+If the repo does not exist **it will be created automatically**
+4. `--commit-message`: since HF repos are git repository you need to define a commit message.
+
+```bash
+mlagents-push-to-hf --run-id="HuggyTraining" --local-dir="./results/Huggy" --repo-id="ThomasSimonini/ppo-Huggy" --commit-message="Huggy"
+```
+
+Else, if everything worked you should have this at the end of the process(but with a different url 😆) :
+
+
+
+```
+Your model is pushed to the hub. You can view your model here: https://huggingface.co/ThomasSimonini/ppo-Huggy
+```
+
+It’s the link to your model repository. The repository contains a model card that explains how to use the model, your Tensorboard logs and your config file. **What’s awesome is that it’s a git repository, which means you can have different commits, update your repository with a new push, open Pull Requests, etc.**
+
+
+
+But now comes the best: **being able to play with Huggy online 👀.**
+
+## Play with your Huggy 🐕
+
+This step is the simplest:
+
+- Open the game Huggy in your browser: https://huggingface.co/spaces/ThomasSimonini/Huggy
+
+- Click on Play with my Huggy model
+
+
+
+1. In step 1, choose your model repository which is the model id (in my case ThomasSimonini/ppo-Huggy).
+
+2. In step 2, **choose what model you want to replay**:
+ - I have multiple ones, since we saved a model every 500000 timesteps.
+ - But since I want the more recent, I choose `Huggy.onnx`
+
+👉 What’s nice **is to try with different models steps to see the improvement of the agent.**
+
+Congrats on finishing this bonus unit!
+
+You can now sit and enjoy playing with your Huggy 🐶. And don't **forget to spread the love by sharing Huggy with your friends 🤗**. And if you share about it on social media, **please tag us @huggingface and me @simoninithomas**
+
+
+
+
+## Keep Learning, Stay awesome 🤗
![\"Bonus](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit2/thumbnail.png\")
\n",
+ "\n",
+ "In this notebook, we'll reinforce what we learned in the first Unit by **teaching Huggy the Dog to fetch the stick and then play with it directly in your browser**\n",
+ "\n",
+ "⬇️ Here is an example of what **you will achieve at the end of the unit.** ⬇️ (launch ▶ to see)"
+ ],
+ "metadata": {
+ "id": "FMYrDriDujzX"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%%html\n",
+ ""
+ ],
+ "metadata": {
+ "id": "PnVhs1yYNyUF"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### The environment 🎮\n",
+ "\n",
+ "- Huggy the Dog, an environment created by [Thomas Simonini](https://twitter.com/ThomasSimonini) based on [Puppo The Corgi](https://blog.unity.com/technology/puppo-the-corgi-cuteness-overload-with-the-unity-ml-agents-toolkit)\n",
+ "\n",
+ "### The library used 📚\n",
+ "\n",
+ "- [MLAgents (Hugging Face version)](https://github.com/huggingface/ml-agents)"
+ ],
+ "metadata": {
+ "id": "x7oR6R-ZIbeS"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
+ ],
+ "metadata": {
+ "id": "60yACvZwO0Cy"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Objectives of this notebook 🏆\n",
+ "\n",
+ "At the end of the notebook, you will:\n",
+ "\n",
+ "- Understand **the state space, action space and reward function used to train Huggy**.\n",
+ "- **Train your own Huggy** to fetch the stick.\n",
+ "- Be able to play **with your trained Huggy directly in your browser**.\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "Oks-ETYdO2Dc"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Course\n",
+ "![\"Huggy\"](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/huggy.jpg\")
\n",
+ "\n",
+ "\n",
+ "### The Action Space: what moves Huggy can do\n",
+ "![\"Huggy](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/huggy-action.jpg\")
\n",
+ "\n",
+ "**Joint motors drive huggy legs**. It means that to get the target, Huggy needs to **learn to rotate the joint motors of each of his legs correctly so he can move**.\n",
+ "\n",
+ "### The Reward Function\n",
+ "\n",
+ "The reward function is designed so that **Huggy will fulfill his goal** : fetch the stick.\n",
+ "\n",
+ "Remember that one of the foundations of Reinforcement Learning is the *reward hypothesis*: a goal can be described as the **maximization of the expected cumulative reward**.\n",
+ "\n",
+ "Here, our goal is that Huggy **goes towards the stick but without spinning too much**. Hence, our reward function must translate this goal.\n",
+ "\n",
+ "Our reward function:\n",
+ "\n",
+ "![\"Huggy](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/reward.jpg\")
\n",
+ "\n",
+ "- *Orientation bonus*: we **reward him for getting close to the target**.\n",
+ "- *Time penalty*: a fixed-time penalty given at every action to **force him to get to the stick as fast as possible**.\n",
+ "- *Rotation penalty*: we penalize Huggy if **he spins too much and turns too quickly**.\n",
+ "- *Getting to the target reward*: we reward Huggy for **reaching the target**."
+ ],
+ "metadata": {
+ "id": "dYKVj8yUvj55"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Check the Huggy config file\n",
+ "\n",
+ "- In ML-Agents, you define the **training hyperparameters into config.yaml files.**\n",
+ "\n",
+ "- For the scope of this notebook, we're not going to modify the hyperparameters, but if you want to try as an experiment, you should also try to modify some other hyperparameters, Unity provides very [good documentation explaining each of them here](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Training-Configuration-File.md)."
+ ],
+ "metadata": {
+ "id": "NAuEq32Mwvtz"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "- **In the case you want to modify the hyperparameters**, in Google Colab notebook, you can click here to open the config.yaml: `/content/ml-agents/config/ppo/Huggy.yaml`\n",
+ "\n",
+ "\n",
+ "We’re now ready to train our agent 🔥."
+ ],
+ "metadata": {
+ "id": "r9wv5NYGw-05"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "f9fI555bO12v"
+ },
+ "source": [
+ "## Train our agent\n",
+ "\n",
+ "To train our agent, we just need to **launch mlagents-learn and select the executable containing the environment.**\n",
+ "\n",
+ "![\"ml](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/mllearn.png\")
\n",
+ "\n",
+ "With ML Agents, we run a training script. We define four parameters:\n",
+ "\n",
+ "1. `mlagents-learn ![\"Create](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/create-token.jpg\")
\n",
+ "\n",
+ "- Copy the token \n",
+ "- Run the cell below and paste the token"
+ ],
+ "metadata": {
+ "id": "izT6FpgNzZ6R"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "rKt2vsYoK56o"
+ },
+ "outputs": [],
+ "source": [
+ "from huggingface_hub import notebook_login\n",
+ "notebook_login()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ],
+ "metadata": {
+ "id": "ew59mK19zjtN"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Xi0y_VASRzJU"
+ },
+ "source": [
+ "Then, we simply need to run `mlagents-push-to-hf`.\n",
+ "\n",
+ "![\"ml](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/mlpush.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "And we define 4 parameters:\n",
+ "\n",
+ "1. `--run-id`: the name of the training run id.\n",
+ "2. `--local-dir`: where the agent was saved, it’s results/![\"ml](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/modelcard.png\")
"
+ ],
+ "metadata": {
+ "id": "yborB0850FTM"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "But now comes the best: **being able to play with Huggy online 👀.**"
+ ],
+ "metadata": {
+ "id": "5Uaon2cg0NrL"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Play with your Huggy 🐕\n",
+ "\n",
+ "This step is the simplest:\n",
+ "\n",
+ "- Open the game Huggy in your browser: https://huggingface.co/spaces/ThomasSimonini/Huggy\n",
+ "\n",
+ "- Click on Play with my Huggy model\n",
+ "\n",
+ "![\"load-huggy\"](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/load-huggy.jpg\")
"
+ ],
+ "metadata": {
+ "id": "VMc4oOsE0QiZ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "1. In step 1, choose your model repository which is the model id (in my case ThomasSimonini/ppo-Huggy).\n",
+ "\n",
+ "2. In step 2, **choose what model you want to replay**:\n",
+ " - I have multiple ones, since we saved a model every 500000 timesteps. \n",
+ " - But since I want the more recent, I choose `Huggy.onnx`\n",
+ "\n",
+ "👉 What’s nice **is to try with different models steps to see the improvement of the agent.**"
+ ],
+ "metadata": {
+ "id": "Djs8c5rR0Z8a"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Congrats on finishing this bonus unit!\n",
+ "\n",
+ "You can now sit and enjoy playing with your Huggy 🐶. And don't **forget to spread the love by sharing Huggy with your friends 🤗**. And if you share about it on social media, **please tag us @huggingface and me @simoninithomas**\n",
+ "\n",
+ "![\"Huggy](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/huggy-cover.jpeg\")
\n",
+ "\n",
+ "\n",
+ "## Keep Learning, Stay awesome 🤗"
+ ],
+ "metadata": {
+ "id": "PI6dPWmh064H"
+ }
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "provenance": [],
+ "private_outputs": true
+ },
+ "gpuClass": "standard",
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/notebooks/unit1/unit1.ipynb b/notebooks/unit1/unit1.ipynb
new file mode 100644
index 0000000..3b58d09
--- /dev/null
+++ b/notebooks/unit1/unit1.ipynb
@@ -0,0 +1,1144 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "njb_ProuHiOe"
+ },
+ "source": [
+ "# Unit 1: Train your first Deep Reinforcement Learning Agent 🤖\n",
+ "\n",
+ "\n",
+ "In this notebook, you'll train your **first Deep Reinforcement Learning agent** a Lunar Lander agent that will learn to **land correctly on the Moon 🌕**. Using [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) a Deep Reinforcement Learning library, share them with the community, and experiment with different configurations\n",
+ "\n",
+ "⬇️ Here is an example of what **you will achieve in just a couple of minutes.** ⬇️\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "PF46MwbZD00b"
+ },
+ "outputs": [],
+ "source": [
+ "%%html\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### The environment 🎮\n",
+ "- [LunarLander-v2](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n",
+ "\n",
+ "### The library used 📚\n",
+ "- [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/)"
+ ],
+ "metadata": {
+ "id": "x7oR6R-ZIbeS"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
+ ],
+ "metadata": {
+ "id": "OwEcFHe9RRZW"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4i6tjI2tHQ8j"
+ },
+ "source": [
+ "## Objectives of this notebook 🏆\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to use **Gym**, the environment library.\n",
+ "- Be able to use **Stable-Baselines3**, the deep reinforcement learning library.\n",
+ "- Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score 🔥.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Course\n",
+ "![\"Deep](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/deep-rl-course-illustration.jpg\")
"
+ ],
+ "metadata": {
+ "id": "Ff-nyJdzJPND"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6p5HnEefISCB"
+ },
+ "source": [
+ "In this free course, you will:\n",
+ "\n",
+ "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- 🧑💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
+ "- 🤖 Train **agents in unique environments** \n",
+ "\n",
+ "And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course\n",
+ "\n",
+ "Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
+ "\n",
+ "\n",
+ "The best way to keep in touch and ask questions is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Y-mo_6rXIjRi"
+ },
+ "source": [
+ "## Prerequisites 🏗️\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "🔲 📝 **Done Unit 0** that gives you all the **information about the course and help you to onboard** 🤗 ADD LINK \n",
+ "\n",
+ "🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by doing Unit 1 👉 ADD LINK"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## A small recap of what is Deep Reinforcement Learning 📚\n",
+ "![\"The](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit1/RL_process_game.jpg\")
"
+ ],
+ "metadata": {
+ "id": "HoeqMnr5LuYE"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xcQYx9ynaFMD"
+ },
+ "source": [
+ "Let's do a small recap on what we learned in the first Unit:\n",
+ "- Reinforcement Learning is a **computational approach to learning from action**. We build an agent that learns from the environment by **interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.\n",
+ "\n",
+ "- The goal of any RL agent is to **maximize its expected cumulative reward** (also called expected return) because RL is based on the _reward hypothesis_, which is that all goals can be described as the maximization of the expected cumulative reward.\n",
+ "\n",
+ "- The RL process is a **loop that outputs a sequence of state, action, reward, and next state**.\n",
+ "\n",
+ "- To calculate the expected cumulative reward (expected return), **we discount the rewards**: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long-term future reward.\n",
+ "\n",
+ "- To solve an RL problem, you want to **find an optimal policy**; the policy is the \"brain\" of your AI that will tell us what action to take given a state. The optimal one is the one that gives you the actions that max the expected return.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8X2WN2X99BEz"
+ },
+ "source": [
+ "There are **two** ways to find your optimal policy:\n",
+ "- By **training your policy directly**: policy-based methods.\n",
+ "- By **training a value function** that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BPy6dfcm8_WT"
+ },
+ "source": [
+ "- Finally, we spoke about Deep RL because **we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name \"deep.\"**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LzGDDOTSKaF8"
+ },
+ "source": [
+ "# Let's train our first Deep Reinforcement Learning agent and upload it to the Hub 🚀\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Set the GPU 💪\n",
+ "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
+ "\n",
+ "![\"GPU](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\")
"
+ ],
+ "metadata": {
+ "id": "HqzznTzhNfAC"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "38HBd3t1SHJ8"
+ },
+ "source": [
+ "- `Hardware Accelerator > GPU`\n",
+ "\n",
+ "![\"GPU](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jeDAH0h0EBiG"
+ },
+ "source": [
+ "## Install dependencies and create a virtual screen 🔽\n",
+ "The first step is to install the dependencies, we’ll install multiple ones.\n",
+ "\n",
+ "- `gym[box2D]`: Contains the LunarLander-v2 environment 🌛 (we use `gym==0.21`)\n",
+ "- `stable-baselines3[extra]`: The deep reinforcement learning library.\n",
+ "- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face 🤗 Hub.\n",
+ "\n",
+ "To make things easier, we created a script to install all these dependencies."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!apt install swig cmake"
+ ],
+ "metadata": {
+ "id": "yQIGLPDkGhgG"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO CHANGE LINK OF THE REQUIREMENTS"
+ ],
+ "metadata": {
+ "id": "32e3NPYgH5ET"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "9XaULfDZDvrC"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -r https://huggingface.co/spaces/ThomasSimonini/temp-space-requirements/raw/main/requirements/requirements-unit1.txt"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install virtual screen libraries and create and run a virtual screen 🖥"
+ ],
+ "metadata": {
+ "id": "BEKeXQJsQCYm"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!sudo apt-get update\n",
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip3 install pyvirtualdisplay"
+ ],
+ "metadata": {
+ "id": "j5f2cGkdP-mb"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To make sure the new installed libraries are used, **sometimes it's required to restart the notebook runtime**. The next cell will force the **runtime to crash, so you'll need to connect again and run the code starting from here**. Thanks for this trick, **we will be able to run our virtual screen.**"
+ ],
+ "metadata": {
+ "id": "TCwBTAwAW9JJ"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import os\n",
+ "os.kill(os.getpid(), 9)"
+ ],
+ "metadata": {
+ "id": "cYvkbef7XEMi"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "BE5JWP5rQIKf"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "wrgpVFqyENVf"
+ },
+ "source": [
+ "## Import the packages 📦\n",
+ "\n",
+ "One additional library we import is huggingface_hub **to be able to upload and download trained models from the hub**.\n",
+ "\n",
+ "\n",
+ "The Hugging Face Hub 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.\n",
+ "\n",
+ "You can see here all the Deep reinforcement Learning models available 👉 https://huggingface.co/models?pipeline_tag=reinforcement-learning&sort=downloads\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "cygWLPGsEQ0m"
+ },
+ "outputs": [],
+ "source": [
+ "import gym\n",
+ "\n",
+ "from huggingface_sb3 import load_from_hub, package_to_hub, push_to_hub\n",
+ "from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
+ "\n",
+ "from stable_baselines3 import PPO\n",
+ "from stable_baselines3.common.evaluation import evaluate_policy\n",
+ "from stable_baselines3.common.env_util import make_vec_env"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MRqRuRUl8CsB"
+ },
+ "source": [
+ "## Understand what is Gym and how it works 🤖\n",
+ "\n",
+ "🏋 The library containing our environment is called Gym.\n",
+ "**You'll use Gym a lot in Deep Reinforcement Learning.**\n",
+ "\n",
+ "The Gym library provides two things:\n",
+ "- An interface that allows you to **create RL environments**.\n",
+ "- A **collection of environments** (gym-control, atari, box2D...).\n",
+ "\n",
+ "Let's look at an example, but first let's remember what's the RL Loop.\n",
+ "\n",
+ "![\"Copy-id\"/](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit1/copy-id.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "hNPLJF2bfiUw"
+ },
+ "source": [
+ "2. Then we just need to use load_from_hub with:\n",
+ "- The repo_id\n",
+ "- The filename: the saved model inside the repo and its extension (*.zip)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "oj8PSGHJfwz3"
+ },
+ "outputs": [],
+ "source": [
+ "from huggingface_sb3 import load_from_hub\n",
+ "repo_id = \"Classroom-workshop/assignment2-omar\" # The repo_id\n",
+ "filename = \"ppo-LunarLander-v2.zip\" # The model filename.zip\n",
+ "\n",
+ "# When the model was trained on Python 3.8 the pickle protocol is 5\n",
+ "# But Python 3.6, 3.7 use protocol 4\n",
+ "# In order to get compatibility we need to:\n",
+ "# 1. Install pickle5 (we done it at the beginning of the colab)\n",
+ "# 2. Create a custom empty object we pass as parameter to PPO.load()\n",
+ "custom_objects = {\n",
+ " \"learning_rate\": 0.0,\n",
+ " \"lr_schedule\": lambda _: 0.0,\n",
+ " \"clip_range\": lambda _: 0.0,\n",
+ "}\n",
+ "\n",
+ "checkpoint = load_from_hub(repo_id, filename)\n",
+ "model = PPO.load(checkpoint, custom_objects=custom_objects, print_system_info=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Fs0Y-qgPgLUf"
+ },
+ "source": [
+ "Let's evaluate this agent:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "#@title\n",
+ "eval_env = gym.make(\"LunarLander-v2\")\n",
+ "mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)\n",
+ "print(f\"mean_reward={mean_reward:.2f} +/- {std_reward}\")"
+ ],
+ "metadata": {
+ "id": "PAEVwK-aahfx"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BQAwLnYFPk-s"
+ },
+ "source": [
+ "## Some additional challenges 🏆\n",
+ "The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
+ "\n",
+ "In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
+ "\n",
+ "Here are some ideas to achieve so:\n",
+ "* Train more steps\n",
+ "* Try different hyperparameters of `PPO`. You can see them at https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#parameters. \n",
+ "* Check the [Stable-Baselines3 documentation](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) and try another models such as DQN.\n",
+ "* **Push your new trained model** on the Hub 🔥\n",
+ "\n",
+ "**Compare the results of your LunarLander-v2 with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) 🏆\n",
+ "\n",
+ "Is moon landing too boring to you? Try to **change the environment**, why not using MountainCar-v0, CartPole-v1 or CarRacing-v0? Check how they works [using the gym documentation](https://www.gymlibrary.dev/) and have fun 🎉."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9lM95-dvmif8"
+ },
+ "source": [
+ "________________________________________________________________________\n",
+ "Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.**\n",
+ "\n",
+ "If you’re still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
+ "\n",
+ "Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations.\n",
+ "\n",
+ "Naturally, during the course, we’re going to use and deeper explain again these terms but **it’s better to have a good understanding of them now before diving into the next chapters.**\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BjLhT70TEZIn"
+ },
+ "source": [
+ "Next time, in the bonus unit 1, you'll train Huggy the Dog to fetch the stick.\n",
+ "\n",
+ "![\"Huggy\"/](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit1/huggy.jpg\")
\n",
+ "\n",
+ "## Keep learning, stay awesome 🤗"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "private_outputs": true,
+ "provenance": [],
+ "collapsed_sections": [
+ "QAN7B0_HCVZC",
+ "1bQzQ-QcE3zo",
+ "BqPKw3jt_pG5",
+ "Avf6gufJBGMw"
+ ]
+ },
+ "gpuClass": "standard",
+ "kernelspec": {
+ "display_name": "Python 3.9.7",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.9.7"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "ed7f8024e43d3b8f5ca3c5e1a8151ab4d136b3ecee1e3fd59e0766ccc55e1b10"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/notebooks/unit2/unit2.ipynb b/notebooks/unit2/unit2.ipynb
new file mode 100644
index 0000000..cf97cdc
--- /dev/null
+++ b/notebooks/unit2/unit2.ipynb
@@ -0,0 +1,1743 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "njb_ProuHiOe"
+ },
+ "source": [
+ "# Unit 2: Q-Learning with FrozenLake-v1 ⛄ and Taxi-v3 🚕\n",
+ "\n",
+ "![\"Unit](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit3/thumbnail.jpg\")
\n",
+ "\n",
+ "In this notebook, **you'll code from scratch your first Reinforcement Learning agent** playing FrozenLake ❄️ using Q-Learning, share it to the community, and experiment with different configurations.\n",
+ "\n",
+ "\n",
+ "⬇️ Here is an example of what **you will achieve in just a couple of minutes.** ⬇️\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "vRU_vXBrl1Jx"
+ },
+ "source": [
+ "![\"Environments\"/](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit3/envs.gif\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO: ADD TEXT LIVE INFO"
+ ],
+ "metadata": {
+ "id": "yaBKcncmYku4"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO: ADD IF YOU HAVE QUESTIONS\n"
+ ],
+ "metadata": {
+ "id": "hz5KE5HjYlRh"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "###🎮 Environments: \n",
+ "- [FrozenLake-v1](https://www.gymlibrary.dev/environments/toy_text/frozen_lake/)\n",
+ "- [Taxi-v3](https://www.gymlibrary.dev/environments/toy_text/taxi/)\n",
+ "\n",
+ "###📚 RL-Library: \n",
+ "- Python and Numpy"
+ ],
+ "metadata": {
+ "id": "DPTBOv9HYLZ2"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
+ ],
+ "metadata": {
+ "id": "3iaIxM_TwklQ"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4i6tjI2tHQ8j"
+ },
+ "source": [
+ "## Objectives of this notebook 🏆\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to use **Gym**, the environment library.\n",
+ "- Be able to code from scratch a Q-Learning agent.\n",
+ "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Course\n",
+ "![\"Q](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit3/Q-function-2.jpg\")
\n",
+ "\n",
+ "- When the training is done,**we have an optimal Q-Function, so an optimal Q-Table.**\n",
+ " \n",
+ "- And if we **have an optimal Q-function**, we\n",
+ "have an optimal policy,since we **know for each state, what is the best action to take.**\n",
+ "\n",
+ "![\"Link](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit3/link-value-policy.jpg\")
\n",
+ "\n",
+ "\n",
+ "But, in the beginning, our **Q-Table is useless since it gives arbitrary value for each state-action pair (most of the time we initialize the Q-Table to 0 values)**. But, as we’ll explore the environment and update our Q-Table it will give us better and better approximations\n",
+ "\n",
+ "![\"q-learning.jpeg\"](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit2/q-learning.jpeg\")
\n",
+ "\n",
+ "This is the Q-Learning pseudocode:\n",
+ "\n",
+ "![\"Q-Learning\"](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit3/Q-learning-2.jpg\")
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Let's code our first Reinforcement Learning algorithm 🚀"
+ ],
+ "metadata": {
+ "id": "HEtx8Y8MqKfH"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Install dependencies and create a virtual display 🔽\n",
+ "\n",
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install the librairies and create and run a virtual screen 🖥\n",
+ "\n",
+ "We’ll install multiple ones:\n",
+ "\n",
+ "- `gym`: Contains the FrozenLake-v1 ⛄ and Taxi-v3 🚕 environments. We use `gym==0.24` since it contains a nice Taxi-v3 UI version.\n",
+ "- `pygame`: Used for the FrozenLake-v1 and Taxi-v3 UI.\n",
+ "- `numPy`: Used for handling our Q-table.\n",
+ "\n",
+ "The Hugging Face Hub 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.\n",
+ "\n",
+ "You can see here all the Deep reinforcement Learning models available 👉 https://huggingface.co/models?other=q-learning\n"
+ ],
+ "metadata": {
+ "id": "4gpxC1_kqUYe"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO CHANGE LINK OF THE REQUIREMENTS"
+ ],
+ "metadata": {
+ "id": "32e3NPYgH5ET"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "9XaULfDZDvrC"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -r https://huggingface.co/spaces/ThomasSimonini/temp-space-requirements/raw/main/requirements/requirements-unit2.txt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "%capture\n",
+ "!sudo apt-get update\n",
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip3 install pyvirtualdisplay"
+ ],
+ "metadata": {
+ "id": "n71uTX7qqzz2"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "DaY1N4dBrabi"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "W-7f-Swax_9x"
+ },
+ "source": [
+ "## Import the packages 📦\n",
+ "\n",
+ "In addition to the installed libraries, we also use:\n",
+ "\n",
+ "- `random`: To generate random numbers (that will be useful for Epsilon-Greedy Policy).\n",
+ "- `imageio`: To generate a replay video\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "VcNvOAQlysBJ"
+ },
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import gym\n",
+ "import random\n",
+ "import imageio\n",
+ "import os\n",
+ "\n",
+ "import pickle5 as pickle\n",
+ "from tqdm.notebook import tqdm"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xp4-bXKIy1mQ"
+ },
+ "source": [
+ "We're now ready to code our Q-Learning algorithm 🔥"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xya49aNJWVvv"
+ },
+ "source": [
+ "# Part 1: Frozen Lake ⛄ (non slippery version)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "NAvihuHdy9tw"
+ },
+ "source": [
+ "## Create and understand [FrozenLake environment ⛄]((https://www.gymlibrary.dev/environments/toy_text/frozen_lake/)\n",
+ "---\n",
+ "\n",
+ "💡 A good habit when you start to use an environment is to check its documentation \n",
+ "\n",
+ "👉 https://www.gymlibrary.dev/environments/toy_text/frozen_lake/\n",
+ "\n",
+ "---\n",
+ "\n",
+ "We're going to train our Q-Learning agent **to navigate from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoid holes (H)**.\n",
+ "\n",
+ "We can have two sizes of environment:\n",
+ "- `map_name=\"4x4\"`: a 4x4 grid version\n",
+ "- `map_name=\"8x8\"`: a 8x8 grid version\n",
+ "\n",
+ "\n",
+ "The environment has two modes:\n",
+ "- `is_slippery=False`: The agent always move in the intended direction due to the non-slippery nature of the frozen lake.\n",
+ "- `is_slippery=True`: The agent may not always move in the intended direction due to the slippery nature of the frozen lake (stochastic)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "UaW_LHfS0PY2"
+ },
+ "source": [
+ "For now let's keep it simple with the 4x4 map and non-slippery"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "IzJnb8O3y8up"
+ },
+ "outputs": [],
+ "source": [
+ "# Create the FrozenLake-v1 environment using 4x4 map and non-slippery version\n",
+ "env = gym.make() # TODO use the correct parameters"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Ji_UrI5l2zzn"
+ },
+ "source": [
+ "### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "jNxUbPMP0akP"
+ },
+ "outputs": [],
+ "source": [
+ "env = gym.make(\"FrozenLake-v1\", map_name=\"4x4\", is_slippery=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "KASNViqL4tZn"
+ },
+ "source": [
+ "You can create your own custom grid like this:\n",
+ "\n",
+ "```python\n",
+ "desc=[\"SFFF\", \"FHFH\", \"FFFH\", \"HFFG\"]\n",
+ "gym.make('FrozenLake-v1', desc=desc, is_slippery=True)\n",
+ "```\n",
+ "\n",
+ "but we'll use the default environment for now."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SXbTfdeJ1Xi9"
+ },
+ "source": [
+ "### Let's see what the Environment looks like:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "ZNPG0g_UGCfh"
+ },
+ "outputs": [],
+ "source": [
+ "# We create our environment with gym.make(\"![\"FrozenLake\"](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit2/frozenlake.png\")
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "We5WqOBGLoSm"
+ },
+ "outputs": [],
+ "source": [
+ "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
+ "print(\"Action Space Shape\", env.action_space.n)\n",
+ "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MyxXwkI2Magx"
+ },
+ "source": [
+ "The action space (the set of possible actions the agent can take) is discrete with 4 actions available 🎮:\n",
+ "- 0: GO LEFT\n",
+ "- 1: GO DOWN\n",
+ "- 2: GO RIGHT\n",
+ "- 3: GO UP\n",
+ "\n",
+ "Reward function 💰:\n",
+ "- Reach goal: +1\n",
+ "- Reach hole: 0\n",
+ "- Reach frozen: 0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1pFhWblk3Awr"
+ },
+ "source": [
+ "## Create and Initialize the Q-table 🗄️\n",
+ "(👀 Step 1 of the pseudocode)\n",
+ "\n",
+ "![\"Q-Learning\"](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit3/Q-learning-4.jpg\")
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LjZSvhsD7_52"
+ },
+ "source": [
+ "Thanks to Sambit for finding a bug on the epsilon function 🤗"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "6Bj7x3in3_Pq"
+ },
+ "outputs": [],
+ "source": [
+ "def epsilon_greedy_policy(Qtable, state, epsilon):\n",
+ " # Randomly generate a number between 0 and 1\n",
+ " random_num = \n",
+ " # if random_num > greater than epsilon --> exploitation\n",
+ " if random_num > epsilon:\n",
+ " # Take the action with the highest value given a state\n",
+ " # np.argmax can be useful here\n",
+ " action = \n",
+ " # else --> exploration\n",
+ " else:\n",
+ " action = # Take a random action\n",
+ " \n",
+ " return action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8R5ej1fS4P2V"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "cYxHuckr4LiG"
+ },
+ "outputs": [],
+ "source": [
+ "def epsilon_greedy_policy(Qtable, state, epsilon):\n",
+ " # Randomly generate a number between 0 and 1\n",
+ " random_int = random.uniform(0,1)\n",
+ " # if random_int > greater than epsilon --> exploitation\n",
+ " if random_int > epsilon:\n",
+ " # Take the action with the highest value given a state\n",
+ " # np.argmax can be useful here\n",
+ " action = np.argmax(Qtable[state])\n",
+ " # else --> exploration\n",
+ " else:\n",
+ " action = env.action_space.sample()\n",
+ " \n",
+ " return action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Atll4Z774gri"
+ },
+ "source": [
+ "## Define the greedy policy 🤖\n",
+ "Remember we have two policies since Q-Learning is an **off-policy** algorithm. This means we're using a **different policy for acting and updating the value function**.\n",
+ "\n",
+ "- Epsilon greedy policy (acting policy)\n",
+ "- Greedy policy (updating policy)\n",
+ "\n",
+ "Greedy policy will also be the final policy we'll have when the Q-learning agent will be trained. The greedy policy is used to select an action from the Q-table.\n",
+ "\n",
+ "![\"Q-Learning\"](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit3/off-on-4.jpg\")
\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "E3SCLmLX5bWG"
+ },
+ "outputs": [],
+ "source": [
+ "def greedy_policy(Qtable, state):\n",
+ " # Exploitation: take the action with the highest state, action value\n",
+ " action = \n",
+ " \n",
+ " return action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "B2_-8b8z5k54"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "se2OzWGW5kYJ"
+ },
+ "outputs": [],
+ "source": [
+ "def greedy_policy(Qtable, state):\n",
+ " # Exploitation: take the action with the highest state, action value\n",
+ " action = np.argmax(Qtable[state])\n",
+ " \n",
+ " return action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "hW80DealcRtu"
+ },
+ "source": [
+ "## Define the hyperparameters ⚙️\n",
+ "The exploration related hyperparamters are some of the most important ones. \n",
+ "\n",
+ "- We need to make sure that our agent **explores enough the state space** in order to learn a good value approximation, in order to do that we need to have progressive decay of the epsilon.\n",
+ "- If you decrease too fast epsilon (too high decay_rate), **you take the risk that your agent is stuck**, since your agent didn't explore enough the state space and hence can't solve the problem."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Y1tWn0tycWZ1"
+ },
+ "outputs": [],
+ "source": [
+ "# Training parameters\n",
+ "n_training_episodes = 10000 # Total training episodes\n",
+ "learning_rate = 0.7 # Learning rate\n",
+ "\n",
+ "# Evaluation parameters\n",
+ "n_eval_episodes = 100 # Total number of test episodes\n",
+ "\n",
+ "# Environment parameters\n",
+ "env_id = \"FrozenLake-v1\" # Name of the environment\n",
+ "max_steps = 99 # Max steps per episode\n",
+ "gamma = 0.95 # Discounting rate\n",
+ "eval_seed = [] # The evaluation seed of the environment\n",
+ "\n",
+ "# Exploration parameters\n",
+ "max_epsilon = 1.0 # Exploration probability at start\n",
+ "min_epsilon = 0.05 # Minimum exploration probability \n",
+ "decay_rate = 0.0005 # Exponential decay rate for exploration prob"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "cDb7Tdx8atfL"
+ },
+ "source": [
+ "## Step 6: Create the training loop method"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "paOynXy3aoJW"
+ },
+ "outputs": [],
+ "source": [
+ "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n",
+ " for episode in range(n_training_episodes):\n",
+ " # Reduce epsilon (because we need less and less exploration)\n",
+ " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n",
+ " # Reset the environment\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ "\n",
+ " # repeat\n",
+ " for step in range(max_steps):\n",
+ " # Choose the action At using epsilon greedy policy\n",
+ " action = \n",
+ "\n",
+ " # Take action At and observe Rt+1 and St+1\n",
+ " # Take the action (a) and observe the outcome state(s') and reward (r)\n",
+ " new_state, reward, done, info = \n",
+ "\n",
+ " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
+ " Qtable[state][action] = \n",
+ "\n",
+ " # If done, finish the episode\n",
+ " if done:\n",
+ " break\n",
+ " \n",
+ " # Our state is the new state\n",
+ " state = new_state\n",
+ " return Qtable"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Pnpk2ePoem3r"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "IyZaYbUAeolw"
+ },
+ "outputs": [],
+ "source": [
+ "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n",
+ " for episode in tqdm(range(n_training_episodes)):\n",
+ " # Reduce epsilon (because we need less and less exploration)\n",
+ " epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n",
+ " # Reset the environment\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ "\n",
+ " # repeat\n",
+ " for step in range(max_steps):\n",
+ " # Choose the action At using epsilon greedy policy\n",
+ " action = epsilon_greedy_policy(Qtable, state, epsilon)\n",
+ "\n",
+ " # Take action At and observe Rt+1 and St+1\n",
+ " # Take the action (a) and observe the outcome state(s') and reward (r)\n",
+ " new_state, reward, done, info = env.step(action)\n",
+ "\n",
+ " # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
+ " Qtable[state][action] = Qtable[state][action] + learning_rate * (reward + gamma * np.max(Qtable[new_state]) - Qtable[state][action]) \n",
+ "\n",
+ " # If done, finish the episode\n",
+ " if done:\n",
+ " break\n",
+ " \n",
+ " # Our state is the new state\n",
+ " state = new_state\n",
+ " return Qtable"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "WLwKQ4tUdhGI"
+ },
+ "source": [
+ "## Train the Q-Learning agent 🏃"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "DPBxfjJdTCOH"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_frozenlake = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_frozenlake)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yVeEhUCrc30L"
+ },
+ "source": [
+ "## Let's see what our Q-Learning table looks like now 👀"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "nmfchsTITw4q"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_frozenlake"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pUrWkxsHccXD"
+ },
+ "source": [
+ "## Define the evaluation method 📝"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "jNl0_JO2cbkm"
+ },
+ "outputs": [],
+ "source": [
+ "def evaluate_agent(env, max_steps, n_eval_episodes, Q, seed):\n",
+ " \"\"\"\n",
+ " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n",
+ " :param env: The evaluation environment\n",
+ " :param n_eval_episodes: Number of episode to evaluate the agent\n",
+ " :param Q: The Q-table\n",
+ " :param seed: The evaluation seed array (for taxi-v3)\n",
+ " \"\"\"\n",
+ " episode_rewards = []\n",
+ " for episode in tqdm(range(n_eval_episodes)):\n",
+ " if seed:\n",
+ " state = env.reset(seed=seed[episode])\n",
+ " else:\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ " total_rewards_ep = 0\n",
+ " \n",
+ " for step in range(max_steps):\n",
+ " # Take the action (index) that have the maximum expected future reward given that state\n",
+ " action = np.argmax(Q[state][:])\n",
+ " new_state, reward, done, info = env.step(action)\n",
+ " total_rewards_ep += reward\n",
+ " \n",
+ " if done:\n",
+ " break\n",
+ " state = new_state\n",
+ " episode_rewards.append(total_rewards_ep)\n",
+ " mean_reward = np.mean(episode_rewards)\n",
+ " std_reward = np.std(episode_rewards)\n",
+ "\n",
+ " return mean_reward, std_reward"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0jJqjaoAnxUo"
+ },
+ "source": [
+ "## Evaluate our Q-Learning agent 📈\n",
+ "- Normally you should have mean reward of 1.0\n",
+ "- It's relatively easy since the state space is really small (16). What you can try to do is [to replace with the slippery version](https://www.gymlibrary.dev/environments/toy_text/frozen_lake/)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "fAgB7s0HEFMm"
+ },
+ "outputs": [],
+ "source": [
+ "# Evaluate our Agent\n",
+ "mean_reward, std_reward = evaluate_agent(env, max_steps, n_eval_episodes, Qtable_frozenlake, eval_seed)\n",
+ "print(f\"Mean_reward={mean_reward:.2f} +/- {std_reward:.2f}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yxaP3bPdg1DV"
+ },
+ "source": [
+ "## Publish our trained model on the Hub 🔥\n",
+ "Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
+ "\n",
+ "Here's an example of a Model Card:\n",
+ "\n",
+ "![\"Model](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit2/modelcard.png\")
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kv0k1JQjpMq3"
+ },
+ "source": [
+ "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "QZ5LrR-joIHD"
+ },
+ "source": [
+ "#### Do not modify this code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Jex3i9lZ8ksX"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "from huggingface_hub import HfApi, HfFolder, Repository\n",
+ "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n",
+ "\n",
+ "from pathlib import Path\n",
+ "import datetime\n",
+ "import json"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Qo57HBn3W74O"
+ },
+ "outputs": [],
+ "source": [
+ "def record_video(env, Qtable, out_directory, fps=1):\n",
+ " images = [] \n",
+ " done = False\n",
+ " state = env.reset(seed=random.randint(0,500))\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " while not done:\n",
+ " # Take the action (index) that have the maximum expected future reward given that state\n",
+ " action = np.argmax(Qtable[state][:])\n",
+ " state, reward, done, info = env.step(action) # We directly put next_state = state for recording logic\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "pwsNrzB339aF"
+ },
+ "outputs": [],
+ "source": [
+ "def push_to_hub(repo_id, \n",
+ " model,\n",
+ " env,\n",
+ " video_fps=1,\n",
+ " local_repo_path=\"hub\",\n",
+ " commit_message=\"Push Q-Learning agent to Hub\",\n",
+ " token= None\n",
+ " ):\n",
+ " _, repo_name = repo_id.split(\"/\")\n",
+ "\n",
+ " eval_env = env\n",
+ " \n",
+ " # Step 1: Clone or create the repo\n",
+ " # Create the repo (or clone its content if it's nonempty)\n",
+ " api = HfApi()\n",
+ " \n",
+ " repo_url = api.create_repo(\n",
+ " repo_id=repo_id,\n",
+ " token=token,\n",
+ " private=False,\n",
+ " exist_ok=True,)\n",
+ " \n",
+ " # Git pull\n",
+ " repo_local_path = Path(local_repo_path) / repo_name\n",
+ " repo = Repository(repo_local_path, clone_from=repo_url, use_auth_token=True)\n",
+ " repo.git_pull()\n",
+ " \n",
+ " repo.lfs_track([\"*.mp4\"])\n",
+ "\n",
+ " # Step 1: Save the model\n",
+ " if env.spec.kwargs.get(\"map_name\"):\n",
+ " model[\"map_name\"] = env.spec.kwargs.get(\"map_name\")\n",
+ " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n",
+ " model[\"slippery\"] = False\n",
+ "\n",
+ " print(model)\n",
+ " \n",
+ " \n",
+ " # Pickle the model\n",
+ " with open(Path(repo_local_path)/'q-learning.pkl', 'wb') as f:\n",
+ " pickle.dump(model, f)\n",
+ " \n",
+ " # Step 2: Evaluate the model and build JSON\n",
+ " mean_reward, std_reward = evaluate_agent(eval_env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])\n",
+ "\n",
+ " # First get datetime\n",
+ " eval_datetime = datetime.datetime.now()\n",
+ " eval_form_datetime = eval_datetime.isoformat()\n",
+ "\n",
+ " evaluate_data = {\n",
+ " \"env_id\": model[\"env_id\"], \n",
+ " \"mean_reward\": mean_reward,\n",
+ " \"n_eval_episodes\": model[\"n_eval_episodes\"],\n",
+ " \"eval_datetime\": eval_form_datetime,\n",
+ " }\n",
+ " # Write a JSON file\n",
+ " with open(Path(repo_local_path) / \"results.json\", \"w\") as outfile:\n",
+ " json.dump(evaluate_data, outfile)\n",
+ "\n",
+ " # Step 3: Create the model card\n",
+ " # Env id\n",
+ " env_name = model[\"env_id\"]\n",
+ " if env.spec.kwargs.get(\"map_name\"):\n",
+ " env_name += \"-\" + env.spec.kwargs.get(\"map_name\")\n",
+ "\n",
+ " if env.spec.kwargs.get(\"is_slippery\", \"\") == False:\n",
+ " env_name += \"-\" + \"no_slippery\"\n",
+ "\n",
+ " metadata = {}\n",
+ " metadata[\"tags\"] = [\n",
+ " env_name,\n",
+ " \"q-learning\",\n",
+ " \"reinforcement-learning\",\n",
+ " \"custom-implementation\"\n",
+ " ]\n",
+ "\n",
+ " # Add metrics\n",
+ " eval = metadata_eval_result(\n",
+ " model_pretty_name=repo_name,\n",
+ " task_pretty_name=\"reinforcement-learning\",\n",
+ " task_id=\"reinforcement-learning\",\n",
+ " metrics_pretty_name=\"mean_reward\",\n",
+ " metrics_id=\"mean_reward\",\n",
+ " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n",
+ " dataset_pretty_name=env_name,\n",
+ " dataset_id=env_name,\n",
+ " )\n",
+ "\n",
+ " # Merges both dictionaries\n",
+ " metadata = {**metadata, **eval}\n",
+ "\n",
+ " model_card = f\"\"\"\n",
+ " # **Q-Learning** Agent playing **{env_id}**\n",
+ " This is a trained model of a **Q-Learning** agent playing **{env_id}** .\n",
+ " \"\"\"\n",
+ "\n",
+ " model_card += \"\"\"\n",
+ " ## Usage\n",
+ " ```python\n",
+ " \"\"\"\n",
+ "\n",
+ " model_card += f\"\"\"model = load_from_hub(repo_id=\"{repo_id}\", filename=\"q-learning.pkl\")\n",
+ "\n",
+ " # Don't forget to check if you need to add additional attributes (is_slippery=False etc)\n",
+ " env = gym.make(model[\"env_id\"])\n",
+ "\n",
+ " evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])\n",
+ " \"\"\"\n",
+ "\n",
+ " model_card +=\"\"\"\n",
+ " ```\n",
+ " \"\"\"\n",
+ "\n",
+ " readme_path = repo_local_path / \"README.md\"\n",
+ " readme = \"\"\n",
+ " if readme_path.exists():\n",
+ " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n",
+ " readme = f.read()\n",
+ " else:\n",
+ " readme = model_card\n",
+ "\n",
+ " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n",
+ " f.write(readme)\n",
+ "\n",
+ " # Save our metrics to Readme metadata\n",
+ " metadata_save(readme_path, metadata)\n",
+ "\n",
+ " # Step 4: Record a video\n",
+ " video_path = repo_local_path / \"replay.mp4\"\n",
+ " record_video(env, model[\"qtable\"], video_path, video_fps)\n",
+ " \n",
+ " # Push everything to hub\n",
+ " print(f\"Pushing repo {repo_name} to the Hugging Face Hub\")\n",
+ " repo.push_to_hub(commit_message=commit_message)\n",
+ "\n",
+ " print(f\"Your model is pushed to the hub. You can view your model here: {repo_url}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "81J6cet_ogSS"
+ },
+ "source": [
+ "### .\n",
+ "By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
+ "\n",
+ "This way:\n",
+ "- You can **showcase our work** 🔥\n",
+ "- You can **visualize your agent playing** 👀\n",
+ "- You can **share with the community an agent that others can use** 💾\n",
+ "- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "cWnFC0iZooTw"
+ },
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
+ "\n",
+ "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
+ "\n",
+ "\n",
+ "![\"Taxi\"](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit2/taxi.png\")
\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "gL0wpeO8gpej"
+ },
+ "outputs": [],
+ "source": [
+ "env = gym.make(\"Taxi-v3\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gBOaXgtsrmtT"
+ },
+ "source": [
+ "There are **500 discrete states since there are 25 taxi positions, 5 possible locations of the passenger** (including the case when the passenger is in the taxi), and **4 destination locations.**\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "_TPNaGSZrgqA"
+ },
+ "outputs": [],
+ "source": [
+ "state_space = env.observation_space.n\n",
+ "print(\"There are \", state_space, \" possible states\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "CdeeZuokrhit"
+ },
+ "outputs": [],
+ "source": [
+ "action_space = env.action_space.n\n",
+ "print(\"There are \", action_space, \" possible actions\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "R1r50Advrh5Q"
+ },
+ "source": [
+ "The action space (the set of possible actions the agent can take) is discrete with **6 actions available 🎮**:\n",
+ "- 0: move south\n",
+ "- 1: move north\n",
+ "- 2: move east\n",
+ "- 3: move west\n",
+ "- 4: pickup passenger\n",
+ "- 5: drop off passenger\n",
+ "\n",
+ "Reward function 💰:\n",
+ "- -1 per step unless other reward is triggered.\n",
+ "- +20 delivering passenger.\n",
+ "- -10 executing “pickup” and “drop-off” actions illegally."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "US3yDXnEtY9I"
+ },
+ "outputs": [],
+ "source": [
+ "# Create our Q table with state_size rows and action_size columns (500x6)\n",
+ "Qtable_taxi = initialize_q_table(state_space, action_space)\n",
+ "print(Qtable_taxi)\n",
+ "print(\"Q-table shape: \", Qtable_taxi .shape)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gUMKPH0_LJyH"
+ },
+ "source": [
+ "## Define the hyperparameters ⚙️\n",
+ "⚠ DO NOT MODIFY EVAL_SEED: the eval_seed array **allows us to evaluate your agent with the same taxi starting positions for every classmate**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "AB6n__hhg7YS"
+ },
+ "outputs": [],
+ "source": [
+ "# Training parameters\n",
+ "n_training_episodes = 25000 # Total training episodes\n",
+ "learning_rate = 0.7 # Learning rate\n",
+ "\n",
+ "# Evaluation parameters\n",
+ "n_eval_episodes = 100 # Total number of test episodes\n",
+ "\n",
+ "# DO NOT MODIFY EVAL_SEED\n",
+ "eval_seed = [16,54,165,177,191,191,120,80,149,178,48,38,6,125,174,73,50,172,100,148,146,6,25,40,68,148,49,167,9,97,164,176,61,7,54,55,\n",
+ " 161,131,184,51,170,12,120,113,95,126,51,98,36,135,54,82,45,95,89,59,95,124,9,113,58,85,51,134,121,169,105,21,30,11,50,65,12,43,82,145,152,97,106,55,31,85,38,\n",
+ " 112,102,168,123,97,21,83,158,26,80,63,5,81,32,11,28,148] # Evaluation seed, this ensures that all classmates agents are trained on the same taxi starting position\n",
+ " # Each seed has a specific starting state\n",
+ "\n",
+ "# Environment parameters\n",
+ "env_id = \"Taxi-v3\" # Name of the environment\n",
+ "max_steps = 99 # Max steps per episode\n",
+ "gamma = 0.95 # Discounting rate\n",
+ "\n",
+ "# Exploration parameters\n",
+ "max_epsilon = 1.0 # Exploration probability at start\n",
+ "min_epsilon = 0.05 # Minimum exploration probability \n",
+ "decay_rate = 0.005 # Exponential decay rate for exploration prob\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1TMORo1VLTsX"
+ },
+ "source": [
+ "## Train our Q-Learning agent 🏃"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "MLNwkNDb14h2"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_taxi = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_taxi)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "WwP3Y2z2eS-K"
+ },
+ "outputs": [],
+ "source": [
+ "Qtable_taxi"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "wPdu0SueLVl2"
+ },
+ "source": [
+ "## Create a model dictionary 💾 and publish our trained model on the Hub 🔥\n",
+ "- We create a model dictionary that will contain all the training hyperparameters for reproducibility and the Q-Table.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "0a1FpE_3hNYr"
+ },
+ "outputs": [],
+ "source": [
+ "model = {\n",
+ " \"env_id\": env_id,\n",
+ " \"max_steps\": max_steps,\n",
+ " \"n_training_episodes\": n_training_episodes,\n",
+ " \"n_eval_episodes\": n_eval_episodes,\n",
+ " \"eval_seed\": eval_seed,\n",
+ "\n",
+ " \"learning_rate\": learning_rate,\n",
+ " \"gamma\": gamma,\n",
+ "\n",
+ " \"max_epsilon\": max_epsilon,\n",
+ " \"min_epsilon\": min_epsilon,\n",
+ " \"decay_rate\": decay_rate,\n",
+ "\n",
+ " \"qtable\": Qtable_taxi\n",
+ "}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "dhQtiQozhOn1"
+ },
+ "outputs": [],
+ "source": [
+ "username = \"\" # FILL THIS\n",
+ "repo_name = \"q-Taxi-v3\"\n",
+ "push_to_hub(\n",
+ " repo_id=f\"{username}/{repo_name}\",\n",
+ " model=model,\n",
+ " env=env)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ZgSdjgbIpRti"
+ },
+ "source": [
+ "Now that's on the Hub, you can compare the results of your Taxi-v3 with your classmates using the leaderboard 🏆 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n",
+ "\n",
+ "![\"Taxi](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit2/taxi-leaderboard.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "bzgIO70c0bu2"
+ },
+ "source": [
+ "# Part 3: Load from Hub 🔽\n",
+ "\n",
+ "What's amazing with Hugging Face Hub 🤗 is that you can easily load powerful models from the community.\n",
+ "\n",
+ "Loading a saved model from the Hub is really easy.\n",
+ "1. You go https://huggingface.co/models?other=q-learning to see the list of all the q-learning saved models.\n",
+ "2. You select one and copy its repo_id\n",
+ "\n",
+ "![\"Copy](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit2/copy-id.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gTth6thRoC6X"
+ },
+ "source": [
+ "3. Then we just need to use `load_from_hub` with:\n",
+ "- The repo_id\n",
+ "- The filename: the saved model inside the repo."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "EtrfoTaBoNrd"
+ },
+ "source": [
+ "#### Do not modify this code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Eo8qEzNtCaVI"
+ },
+ "outputs": [],
+ "source": [
+ "from urllib.error import HTTPError\n",
+ "\n",
+ "from huggingface_hub import hf_hub_download\n",
+ "\n",
+ "\n",
+ "def load_from_hub(repo_id: str, filename: str) -> str:\n",
+ " \"\"\"\n",
+ " Download a model from Hugging Face Hub.\n",
+ " :param repo_id: id of the model repository from the Hugging Face Hub\n",
+ " :param filename: name of the model zip file from the repository\n",
+ " \"\"\"\n",
+ " try:\n",
+ " from huggingface_hub import cached_download, hf_hub_url\n",
+ " except ImportError:\n",
+ " raise ImportError(\n",
+ " \"You need to install huggingface_hub to use `load_from_hub`. \"\n",
+ " \"See https://pypi.org/project/huggingface-hub/ for installation.\"\n",
+ " )\n",
+ "\n",
+ " # Get the model from the Hub, download and cache the model on your local disk\n",
+ " pickle_model = hf_hub_download(\n",
+ " repo_id=repo_id,\n",
+ " filename=filename\n",
+ " )\n",
+ "\n",
+ " with open(pickle_model, 'rb') as f:\n",
+ " downloaded_model_file = pickle.load(f)\n",
+ " \n",
+ " return downloaded_model_file"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "b_sM2gNioPZH"
+ },
+ "source": [
+ "### ."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "JUm9lz2gCQcU"
+ },
+ "outputs": [],
+ "source": [
+ "model = load_from_hub(repo_id=\"ThomasSimonini/q-Taxi-v3\", filename=\"q-learning.pkl\") # Try to use another model\n",
+ "\n",
+ "print(model)\n",
+ "env = gym.make(model[\"env_id\"])\n",
+ "\n",
+ "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "O7pL8rg1MulN"
+ },
+ "outputs": [],
+ "source": [
+ "model = load_from_hub(repo_id=\"ThomasSimonini/q-FrozenLake-v1-no-slippery\", filename=\"q-learning.pkl\") # Try to use another model\n",
+ "\n",
+ "env = gym.make(model[\"env_id\"], is_slippery=False)\n",
+ "\n",
+ "evaluate_agent(env, model[\"max_steps\"], model[\"n_eval_episodes\"], model[\"qtable\"], model[\"eval_seed\"])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BQAwLnYFPk-s"
+ },
+ "source": [
+ "## Some additional challenges 🏆\n",
+ "The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
+ "\n",
+ "In the [Leaderboard](https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
+ "\n",
+ "Here are some ideas to achieve so:\n",
+ "* Train more steps\n",
+ "* Try different hyperparameters by looking at what your classmates have done.\n",
+ "* **Push your new trained model** on the Hub 🔥\n",
+ "\n",
+ "Are walking on ice and driving taxis too boring to you? Try to **change the environment**, why not using FrozenLake-v1 slippery version? Check how they work [using the gym documentation](https://www.gymlibrary.dev/) and have fun 🎉."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "p-fW-EU5WejJ"
+ },
+ "source": [
+ "_____________________________________________________________________\n",
+ "Congrats 🥳, you've just implemented, trained, and uploaded your first Reinforcement Learning agent.\n",
+ "\n",
+ "Understanding Q-Learning is an **important step to understanding value-based methods.**\n",
+ "\n",
+ "In the next Unit with Deep Q-Learning, we'll see that creating and updating a Q-table was a good strategy — **however, this is not scalable.**\n",
+ "\n",
+ "For instance, imagine you create an agent that learns to play Doom. \n",
+ "\n",
+ "![\"Doom\"/](\"https://vizdoom.cs.put.edu.pl/user/pages/01.tutorial/basic.png\")
\n",
+ "\n",
+ "Doom is a large environment with a huge state space (millions of different states). Creating and updating a Q-table for that environment would not be efficient. \n",
+ "\n",
+ "That's why we'll study, in the next unit, Deep Q-Learning, an algorithm **where we use a neural network that approximates, given a state, the different Q-values for each action.**\n",
+ "\n",
+ "![\"Environments\"/](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit4/atari-envs.gif\")
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BjLhT70TEZIn"
+ },
+ "source": [
+ "See you on [Unit 3](https://github.com/huggingface/deep-rl-class/tree/main/unit2#unit-2-introduction-to-q-learning)! 🔥\n",
+ "TODO CHANGE LINK\n",
+ "## Keep learning, stay awesome 🤗"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "collapsed_sections": [
+ "4i6tjI2tHQ8j",
+ "Y-mo_6rXIjRi",
+ "EtrfoTaBoNrd",
+ "BjLhT70TEZIn"
+ ],
+ "private_outputs": true,
+ "provenance": []
+ },
+ "gpuClass": "standard",
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/notebooks/unit3/unit3.ipynb b/notebooks/unit3/unit3.ipynb
new file mode 100644
index 0000000..019653e
--- /dev/null
+++ b/notebooks/unit3/unit3.ipynb
@@ -0,0 +1,794 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "k7xBVPzoXxOg"
+ },
+ "source": [
+ "# Unit 3: Deep Q-Learning with Atari Games 👾 using RL Baselines3 Zoo\n",
+ "\n",
+ "![\"Unit](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit4/thumbnail.jpg\")
\n",
+ "\n",
+ "In this notebook, **you'll train a Deep Q-Learning agent** playing Space Invaders using [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), a training framework based on [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.\n",
+ "\n",
+ "We're using the [RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) with no extensions such as Double-DQN, Dueling-DQN, and Prioritized Experience Replay.\n",
+ "\n",
+ "⬇️ Here is an example of what **you will achieve** ⬇️"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "J9S713biXntc"
+ },
+ "outputs": [],
+ "source": [
+ "%%html\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "TODO: ADD TEXT LIVE INFO\n",
+ "\n",
+ "TODO: ADD IF YOU HAVE QUESTIONS\n",
+ "\n",
+ "\n",
+ "###🎮 Environments: \n",
+ "- SpacesInvadersNoFrameskip-v4 \n",
+ "\n",
+ "###📚 RL-Library: \n",
+ "- [RL-Baselines3-Zoo](https://github.com/DLR-RM/rl-baselines3-zoo)"
+ ],
+ "metadata": {
+ "id": "ykJiGevCMVc5"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "wciHGjrFYz9m"
+ },
+ "source": [
+ "## Objectives of this notebook 🏆\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to understand deeper **how RL Baselines3 Zoo works**.\n",
+ "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Course\n",
+ "![\"DQN](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit3/hyperparameters.png\")
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_VjblFSVDQOj"
+ },
+ "source": [
+ "Here we see that:\n",
+ "- We use the `Atari Wrapper` that preprocess the input (Frame reduction ,grayscale, stack 4 frames)\n",
+ "- We use `CnnPolicy`, since we use Convolutional layers to process the frames\n",
+ "- We train it for 10 million `n_timesteps` \n",
+ "- Memory (Experience Replay) size is 100000, aka the amount of experience steps you saved to train again your agent with.\n",
+ "\n",
+ "💡 My advice is to **reduce the training timesteps to 1M,** which will take about 90 minutes on a P100. `!nvidia-smi` will tell you what GPU you're using. At 10 million steps, this will take about 9 hours, which could likely result in Colab timing out. I recommend running this on your local computer (or somewhere else). Just click on: `File>Download`. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5qTkbWrkECOJ"
+ },
+ "source": [
+ "In terms of hyperparameters optimization, my advice is to focus on these 3 hyperparameters:\n",
+ "- `learning_rate`\n",
+ "- `buffer_size (Experience Memory size)`\n",
+ "- `batch_size`\n",
+ "\n",
+ "As a good practice, you need to **check the documentation to understand what each hyperparameters does**: https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html#parameters\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Hn8bRTHvERRL"
+ },
+ "source": [
+ "2. We run `train.py` and save the models on `logs` folder 📁"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Xr1TVW4xfbz3"
+ },
+ "outputs": [],
+ "source": [
+ "!python train.py --algo ________ --env SpaceInvadersNoFrameskip-v4 -f _________"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SeChoX-3SZfP"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "PuocgdokSab9"
+ },
+ "outputs": [],
+ "source": [
+ "!python train.py --algo dqn --env SpaceInvadersNoFrameskip-v4 -f logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_dLomIiMKQaf"
+ },
+ "source": [
+ "## Let's evaluate our agent 👀\n",
+ "- RL-Baselines3-Zoo provides `enjoy.py` to evaluate our agent.\n",
+ "- Let's evaluate it for 5000 timesteps 🔥"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "co5um_KeKbBJ"
+ },
+ "outputs": [],
+ "source": [
+ "!python enjoy.py --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps _________ --folder logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Q24K1tyWSj7t"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "P_uSmwGRSk0z"
+ },
+ "outputs": [],
+ "source": [
+ "!python enjoy.py --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps 5000 --folder logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "liBeTltiHJtr"
+ },
+ "source": [
+ "## Publish our trained model on the Hub 🚀\n",
+ "Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
+ "\n",
+ "![\"Space](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit3/space-invaders-model.gif\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ezbHS1q3HYVV"
+ },
+ "source": [
+ "By using `rl_zoo3.push_to_hub.py` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
+ "\n",
+ "This way:\n",
+ "- You can **showcase our work** 🔥\n",
+ "- You can **visualize your agent playing** 👀\n",
+ "- You can **share with the community an agent that others can use** 💾\n",
+ "- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "XMSeZRBiHk6X"
+ },
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
+ "\n",
+ "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens) **with write role**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9ToyuaYwHmxG"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9O6FI0F8HnzE"
+ },
+ "source": [
+ "- Copy the token \n",
+ "- Run the cell below and past the token"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Ppu9yePwHrZX"
+ },
+ "outputs": [],
+ "source": [
+ "from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
+ "notebook_login()\n",
+ "!git config --global credential.helper store"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2RVEdunPHs8B"
+ },
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "dSLwdmvhHvjw"
+ },
+ "source": [
+ "3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "PW436XnhHw1H"
+ },
+ "source": [
+ "Let's run push_to_hub.py file to upload our trained agent to the Hub.\n",
+ "\n",
+ "`--repo-name `: The name of the repo\n",
+ "\n",
+ "`-orga`: Your Hugging Face username\n",
+ "\n",
+ "![\"Select](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit3/select-id.png\")
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Ygk2sEktTDEw"
+ },
+ "outputs": [],
+ "source": [
+ "!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name _____________________ -orga _____________________ -f logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "otgpa0rhS9wR"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "_HQNlAXuEhci"
+ },
+ "outputs": [],
+ "source": [
+ "!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name dqn-SpaceInvadersNoFrameskip-v4 -orga ThomasSimonini -f logs/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0D4F5zsTTJ-L"
+ },
+ "source": [
+ "###."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ff89kd2HL1_s"
+ },
+ "source": [
+ "Congrats 🥳 you've just trained and uploaded your first Deep Q-Learning agent using RL-Baselines-3 Zoo. The script above should have displayed a link to a model repository such as https://huggingface.co/ThomasSimonini/dqn-SpaceInvadersNoFrameskip-v4. When you go to this link, you can:\n",
+ "\n",
+ "- See a **video preview of your agent** at the right. \n",
+ "- Click \"Files and versions\" to see all the files in the repository.\n",
+ "- Click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
+ "- A model card (`README.md` file) which gives a description of the model and the hyperparameters you used.\n",
+ "\n",
+ "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
+ "\n",
+ "**Compare the results of your agents with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) 🏆"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fyRKcCYY-dIo"
+ },
+ "source": [
+ "## Load a powerful trained model 🔥\n",
+ "- The Stable-Baselines3 team uploaded **more than 150 trained Deep Reinforcement Learning agents on the Hub**.\n",
+ "\n",
+ "You can find them here: 👉 https://huggingface.co/sb3\n",
+ "\n",
+ "Some examples:\n",
+ "- Asteroids: https://huggingface.co/sb3/dqn-AsteroidsNoFrameskip-v4\n",
+ "- Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4\n",
+ "- Breakout: https://huggingface.co/sb3/dqn-BreakoutNoFrameskip-v4\n",
+ "- Road Runner: https://huggingface.co/sb3/dqn-RoadRunnerNoFrameskip-v4\n",
+ "\n",
+ "Let's load an agent playing Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "B-9QVFIROI5Y"
+ },
+ "outputs": [],
+ "source": [
+ "%%html\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "7ZQNY_r6NJtC"
+ },
+ "source": [
+ "1. We download the model using `rl_zoo3.load_from_hub`, and place it in a new folder that we can call `rl_trained`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "OdBNZHy0NGTR"
+ },
+ "outputs": [],
+ "source": [
+ "# Download model and save it into the logs/ folder\n",
+ "!python -m rl_zoo3.load_from_hub --algo dqn --env BeamRiderNoFrameskip-v4 -orga sb3 -f rl_trained/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LFt6hmWsNdBo"
+ },
+ "source": [
+ "2. Let's evaluate if for 5000 timesteps"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "aOxs0rNuN0uS"
+ },
+ "outputs": [],
+ "source": [
+ "!python enjoy.py --algo dqn --env BeamRiderNoFrameskip-v4 -n 5000 -f rl_trained/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kxMDuDfPON57"
+ },
+ "source": [
+ "Why not trying to train your own **Deep Q-Learning Agent playing BeamRiderNoFrameskip-v4? 🏆.**\n",
+ "\n",
+ "If you want to try, check https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4#hyperparameters **in the model card, you have the hyperparameters of the trained agent.**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xL_ZtUgpOuY6"
+ },
+ "source": [
+ "But finding hyperparameters can be a daunting task. Fortunately, we'll see in the next Unit, how we can **use Optuna for optimizing the Hyperparameters 🔥.**\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-pqaco8W-huW"
+ },
+ "source": [
+ "## Some additional challenges 🏆\n",
+ "The best way to learn **is to try things by your own**!\n",
+ "\n",
+ "In the [Leaderboard](https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
+ "\n",
+ "Here's a list of environments you can try to train your agent with:\n",
+ "- BeamRiderNoFrameskip-v4\n",
+ "- BreakoutNoFrameskip-v4 \n",
+ "- EnduroNoFrameskip-v4\n",
+ "- PongNoFrameskip-v4\n",
+ "\n",
+ "Also, **if you want to learn to implement Deep Q-Learning by yourself**, you definitely should look at CleanRL implementation: https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py\n",
+ "\n",
+ "
+
+Discord is a free chat platform. If you've used Slack, **it's quite similar**. There is a Hugging Face Community Discord server with 18000 members you can join with a single click here. So many humans to play with!
+
+Starting in Discord can be a bit intimidating, so let me take you through it.
+
+When you sign-up to our Discord server, you'll need to specify which topics you're interested in by **clicking #role-assignment at the left**. Here, you can pick different categories. Make sure to **click "Reinforcement Learning"**! :fire:. You'll then get to **introduce yourself in the `#introduction-yourself` channel**.
+
+## So which channels are interesting to me? [[channels]]
+
+They are in the reinforcement learning lounge. **Don't forget to sign up to these channels** by clicking on 🤖 Reinforcement Learning in `role-assigment`.
+- `rl-announcements`: where we give the **lastest information about the course**.
+- `rl-discussions`: where you can **exchange about RL and share information**.
+- `rl-study-group`: where you can **create and join study groups**.
+
+The HF Community Server has a thriving community of human beings interested in many areas, so you can also learn from those. There are paper discussions, events, and many other things.
+
+Was this useful? There are a couple of tips I can share with you:
+
+- There are **voice channels** you can use as well, although most people prefer text chat.
+- You can **use markdown style** for text chats. So if you're writing code, you can use that style. Sadly this does not work as well for links.
+- You can open threads as well! It's a good idea when **it's a long conversation**.
+
+I hope this is useful! And if you have questions, just ask!
+
+See you later!
+
+Huggy 🐶
diff --git a/units/en/unit0/introduction.mdx b/units/en/unit0/introduction.mdx
new file mode 100644
index 0000000..e48f0c4
--- /dev/null
+++ b/units/en/unit0/introduction.mdx
@@ -0,0 +1,126 @@
+# Welcome to the 🤗 Deep Reinforcement Learning Course [[introduction]]
+
+
+
+Welcome to the most fascinating topic in Artificial Intelligence: Deep Reinforcement Learning.
+
+This course will **teach you about Deep Reinforcement Learning from beginner to expert**. It’s completely free and open-source!
+
+In this introduction unit you’ll:
+
+- Learn more about the **course content**.
+- **Define the path** you’re going to take (either self-audit or certification process)
+- Learn more about the **AI vs. AI challenges** you're going to participate to.
+- Learn more **about us**.
+- **Create your Hugging Face account** (it’s free).
+- **Sign-up our Discord server**, the place where you can exchange with your classmates and us (the Hugging Face team).
+
+Let’s get started!
+
+## What to expect? [[expect]]
+
+In this course, you will:
+
+- 📖 Study Deep Reinforcement Learning in **theory and practice.**
+- 🧑💻 Learn to **use famous Deep RL libraries** such as [Stable Baselines3](https://stable-baselines3.readthedocs.io/en/master/), [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), [Sample Factory](https://samplefactory.dev/) and [CleanRL](https://github.com/vwxyzjn/cleanrl).
+- 🤖 **Train agents in unique environments** such as [SnowballFight](https://huggingface.co/spaces/ThomasSimonini/SnowballFight), [Huggy the Doggo 🐶](https://huggingface.co/spaces/ThomasSimonini/Huggy), [MineRL (Minecraft ⛏️)](https://minerl.io/), [VizDoom (Doom)](https://vizdoom.cs.put.edu.pl/) and classical ones such as [Space Invaders](https://www.gymlibrary.dev/environments/atari/) and [PyBullet](https://pybullet.org/wordpress/).
+- 💾 Share your **trained agents with one line of code to the Hub** and also download powerful agents from the community.
+- 🏆 Participate in challenges where you will **evaluate your agents against other teams. You'll also get to play against the agents you'll train.**
+
+And more!
+
+At the end of this course, **you’ll get a solid foundation from the basics to the SOTA (state-of-the-art) methods**.
+
+You can find the syllabus on our website 👉 here
+
+Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**
+
+Sign up 👉 here
+
+
+## What does the course look like? [[course-look-like]]
+The course is composed of:
+
+- *A theory part*: where you learn a **concept in theory (article)**.
+- *A hands-on*: where you’ll learn **to use famous Deep RL libraries** to train your agents in unique environments. These hands-on will be **Google Colab notebooks with companion tutorial videos** if you prefer learning with video format!
+- *Challenges*: you'll get to put your agent to compete against other agents in different challenges. There will also be leaderboards for you to compare the agents' performance.
+
+
+## Two paths: choose your own adventure [[two-paths]]
+
+
+
+You can choose to follow this course either:
+
+- *To get a certificate of completion*: you need to complete 80% of the assignments before the end of March 2023.
+- *As a simple audit*: you can participate in all challenges and do assignments if you want, but you have no deadlines.
+
+Both paths **are completely free**.
+Whatever path you choose, we advise you **to follow the recommended pace to enjoy the course and challenges with your fellow classmates.**
+You don't need to tell us which path you choose. At the end of March, when we verify the assignments **if you get more than 80% of the assignments done, you'll get a certificate.**
+
+
+## How to get most of the course? [[advice]]
+
+To get most of the course, we have some advice:
+
+1. Join or create study groups in Discord : studying in groups is always easier. To do that, you need to join our discord server. If you're new to Discord, no worries! We have some tools that will help you learn about it.
+2. **Do the quizzes and assignments**: the best way to learn is to do and test yourself.
+3. **Define a schedule to stay in sync**: you can use our recommended pace schedule below or create yours.
+
+
+
+## What tools do I need? [[tools]]
+
+You need only 3 things:
+
+- *A computer* with an internet connection.
+- *Google Colab (free version)*: most of our hands-on will use Google Colab, the **free version is enough.**
+- A *Hugging Face Account*: to push and load models. If you don’t have an account yet, you can create one **[here](https://hf.co/join)** (it’s free).
+
+
+
+
+## What is the recommended pace? [[recommended-pace]]
+
+We defined a planning that you can follow to keep up the pace of the course.
+
+
+
+
+
+Each chapter in this course is designed **to be completed in 1 week, with approximately 3-4 hours of work per week**. However, you can take as much time as necessary to complete the course. If you want to dive into a topic more in-depth, we'll provide additional resources to help you achieve that.
+
+
+## Who are we [[who-are-we]]
+About the author:
+
+- Thomas Simonini is a Developer Advocate at Hugging Face 🤗 specializing in Deep Reinforcement Learning. He founded the Deep Reinforcement Learning Course in 2018, which became one of the most used courses in Deep RL.
+
+About the team:
+
+- Omar Sanseviero is a Machine Learning Engineer at Hugging Face where he works in the intersection of ML, Community and Open Source. Previously, Omar worked as a Software Engineer at Google in the teams of Assistant and TensorFlow Graphics. He is from Peru and likes llamas 🦙.
+- Sayak Paul is a Developer Advocate Engineer at Hugging Face. He's interested in the area of representation learning (self-supervision, semi-supervision, model robustness). And he loves watching crime and action thrillers 🔪.
+
+
+## When do the challenges start? [[challenges]]
+
+In this new version of the course, you have two types of challenges:
+- A leaderboard to compare your agent's performance to other classmates'.
+- AI vs. AI challenges where you can train your agent and compete against other classmates' agents.
+
+
+
+These AI vs.AI challenges will be announced **later in December**.
+
+
+## I found a bug, or I want to improve the course [[contribute]]
+
+Contributions are welcomed 🤗
+
+- If you *found a bug 🐛 in a notebook*, please open an issue and **describe the problem**.
+- If you *want to improve the course*, you can open a Pull Request.
+
+## I still have questions [[questions]]
+
+In that case, check our FAQ. And if the question is not in it, ask your question in our discord server #rl-discussions.
diff --git a/units/en/unit0/setup.mdx b/units/en/unit0/setup.mdx
new file mode 100644
index 0000000..4fc55bb
--- /dev/null
+++ b/units/en/unit0/setup.mdx
@@ -0,0 +1,30 @@
+# Setup [[setup]]
+
+After all this information, it's time to get started. We're going to do two things:
+
+1. **Create your Hugging Face account** if it's not already done
+2. **Sign up to Discord and introduce yourself** (don't be shy 🤗)
+
+### Let's create my Hugging Face account
+
+(If it's not already done) create an account to HF here
+
+### Let's join our Discord server
+
+You can now sign up for our Discord Server. This is the place where you **can exchange with the community and with us, create and join study groups to grow each other and more**
+
+👉🏻 Join our discord server here.
+
+When you join, remember to introduce yourself in #introduce-yourself and sign-up for reinforcement channels in #role-assignments.
+
+We have multiple RL-related channels:
+- `rl-announcements`: where we give the last information about the course.
+- `rl-discussions`: where you can exchange about RL and share information.
+- `rl-study-group`: where you can create and join study groups.
+
+If this is your first time using Discord, we wrote a Discord 101 to get the best practices. Check the next section.
+
+Congratulations! **You've just finished the on-boarding**. You're now ready to start to learn Deep Reinforcement Learning. Have fun!
+
+
+### Keep Learning, stay awesome 🤗
diff --git a/units/en/unit1/additional-readings.mdx b/units/en/unit1/additional-readings.mdx
new file mode 100644
index 0000000..b881244
--- /dev/null
+++ b/units/en/unit1/additional-readings.mdx
@@ -0,0 +1,13 @@
+# Additional Readings [[additional-readings]]
+
+These are **optional readings** if you want to go deeper.
+
+## Deep Reinforcement Learning [[deep-rl]]
+
+- [Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 1, 2 and 3](http://incompleteideas.net/book/RLbook2020.pdf)
+- [Foundations of Deep RL Series, L1 MDPs, Exact Solution Methods, Max-ent RL by Pieter Abbeel](https://youtu.be/2GwBez0D20A)
+- [Spinning Up RL by OpenAI Part 1: Key concepts of RL](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)
+
+## Gym [[gym]]
+
+- [Getting Started With OpenAI Gym: The Basic Building Blocks](https://blog.paperspace.com/getting-started-with-openai-gym/)
diff --git a/units/en/unit1/conclusion.mdx b/units/en/unit1/conclusion.mdx
new file mode 100644
index 0000000..de31951
--- /dev/null
+++ b/units/en/unit1/conclusion.mdx
@@ -0,0 +1,16 @@
+# Conclusion [[conclusion]]
+
+Congrats on finishing this unit! **That was the biggest one**, and there was a lot of information. And congrats on finishing the tutorial. You’ve just trained your first Deep RL agents and shared it with the community! 🥳
+
+It's **normal if you still feel confused with some of these elements**. This was the same for me and for all people who studied RL.
+
+**Take time to really grasp the material** before continuing. It’s important to master these elements and having a solid foundations before entering the fun part.
+
+Naturally, during the course, we’re going to use and explain these terms again, but it’s better to understand them before diving into the next units.
+
+In the next (bonus) unit, we’re going to reinforce what we just learned by **training Huggy the Dog to fetch the stick**.
+
+You will be able then to play with him 🤗.
+
+
diff --git a/units/en/unit1/deep-rl.mdx b/units/en/unit1/deep-rl.mdx
new file mode 100644
index 0000000..c0c2247
--- /dev/null
+++ b/units/en/unit1/deep-rl.mdx
@@ -0,0 +1,21 @@
+# The “Deep” in Reinforcement Learning [[deep-rl]]
+
+
+
+
+In this game, our mouse can have an **infinite amount of small cheese** (+1 each). But at the top of the maze, there is a gigantic sum of cheese (+1000).
+
+However, if we only focus on exploitation, our agent will never reach the gigantic sum of cheese. Instead, it will only exploit **the nearest source of rewards,** even if this source is small (exploitation).
+
+But if our agent does a little bit of exploration, it can **discover the big reward** (the pile of big cheese).
+
+This is what we call the exploration/exploitation trade-off. We need to balance how much we **explore the environment** and how much we **exploit what we know about the environment.**
+
+Therefore, we must **define a rule that helps to handle this trade-off**. We’ll see the different ways to handle it in the future units.
+
+If it’s still confusing, **think of a real problem: the choice of picking a restaurant:**
+
+
+
+
diff --git a/units/en/unit1/hands-on.mdx b/units/en/unit1/hands-on.mdx
new file mode 100644
index 0000000..dfe80c5
--- /dev/null
+++ b/units/en/unit1/hands-on.mdx
@@ -0,0 +1,669 @@
+# Hands on [[hands-on]]
+
+
+
+ 
+
+And finally, you'll **upload this trained agent to the Hugging Face Hub 🤗, a free, open platform where people can share ML models, datasets, and demos.**
+
+Thanks to our leaderboard, you'll be able to compare your results with other classmates and exchange the best practices to improve your agent's scores. Who will win the challenge for Unit 1 🏆?
+
+So let's get started! 🚀
+
+**To start the hands-on click on Open In Colab button** 👇 :
+
+[](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/master/notebooks/unit1/unit1.ipynb)
+
+
+# Unit 1: Train your first Deep Reinforcement Learning Agent 🤖
+
+
+In this notebook, you'll train your **first Deep Reinforcement Learning agent** a Lunar Lander agent that will learn to **land correctly on the Moon 🌕**. Using [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) a Deep Reinforcement Learning library, share them with the community, and experiment with different configurations
+
+⬇️ Here is an example of what **you will achieve in just a couple of minutes.** ⬇️
+
+
+
+
+```python
+%%html
+
+```
+
+### The environment 🎮
+
+- [LunarLander-v2](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)
+
+### The library used 📚
+
+- [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/)
+
+We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues).
+
+## Objectives of this notebook 🏆
+
+At the end of the notebook, you will:
+
+- Be able to use **Gym**, the environment library.
+- Be able to use **Stable-Baselines3**, the deep reinforcement learning library.
+- Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score 🔥.
+
+
+
+
+## This notebook is from Deep Reinforcement Learning Course
+
+
+In this free course, you will:
+
+- 📖 Study Deep Reinforcement Learning in **theory and practice**.
+- 🧑💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.
+- 🤖 Train **agents in unique environments**
+
+And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course
+
+Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**
+
+
+The best way to keep in touch and ask questions is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5
+
+## Prerequisites 🏗️
+Before diving into the notebook, you need to:
+
+🔲 📝 **Done Unit 0** that gives you all the **information about the course and help you to onboard** 🤗
+
+🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by doing Unit 1
+
+## A small recap of what is Deep Reinforcement Learning 📚
+
+
+Let's do a small recap on what we learned in the first Unit:
+- Reinforcement Learning is a **computational approach to learning from action**. We build an agent that learns from the environment by **interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.
+
+- The goal of any RL agent is to **maximize its expected cumulative reward** (also called expected return) because RL is based on the _reward hypothesis_, which is that all goals can be described as the maximization of the expected cumulative reward.
+
+- The RL process is a **loop that outputs a sequence of state, action, reward, and next state**.
+
+- To calculate the expected cumulative reward (expected return), **we discount the rewards**: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long-term future reward.
+
+- To solve an RL problem, you want to **find an optimal policy**; the policy is the "brain" of your AI that will tell us what action to take given a state. The optimal one is the one that gives you the actions that max the expected return.
+
+
+There are **two** ways to find your optimal policy:
+- By **training your policy directly**: policy-based methods.
+- By **training a value function** that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods.
+
+- Finally, we spoke about Deep RL because **we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name "deep."**
+
+# Let's train our first Deep Reinforcement Learning agent and upload it to the Hub 🚀
+
+
+## Set the GPU 💪
+- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`
+
+
+
+- `Hardware Accelerator > GPU`
+
+
+
+## Install dependencies and create a virtual screen 🔽
+The first step is to install the dependencies, we’ll install multiple ones.
+
+- `gym[box2D]`: Contains the LunarLander-v2 environment 🌛 (we use `gym==0.21`)
+- `stable-baselines3[extra]`: The deep reinforcement learning library.
+- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face 🤗 Hub.
+
+To make things easier, we created a script to install all these dependencies.
+
+```python
+!apt install swig cmake
+```
+
+TODO CHANGE LINK OF THE REQUIREMENTS
+
+```python
+!pip install -r https://huggingface.co/spaces/ThomasSimonini/temp-space-requirements/raw/main/requirements/requirements-unit1.txt
+```
+
+During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames).
+
+Hence the following cell will install virtual screen libraries and create and run a virtual screen 🖥
+
+```python
+!sudo apt-get update
+!apt install python-opengl
+!apt install ffmpeg
+!apt install xvfb
+!pip3 install pyvirtualdisplay
+```
+
+To make sure the new installed libraries are used, **sometimes it's required to restart the notebook runtime**. The next cell will force the **runtime to crash, so you'll need to connect again and run the code starting from here**. Thanks for this trick, **we will be able to run our virtual screen.**
+
+
+```python
+import os
+os.kill(os.getpid(), 9)
+```
+
+
+```python
+# Virtual display
+from pyvirtualdisplay import Display
+
+virtual_display = Display(visible=0, size=(1400, 900))
+virtual_display.start()
+```
+
+## Import the packages 📦
+
+One additional library we import is huggingface_hub **to be able to upload and download trained models from the hub**.
+
+
+The Hugging Face Hub 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.
+
+You can see here all the Deep reinforcement Learning models available 👉 https://huggingface.co/models?pipeline_tag=reinforcement-learning&sort=downloads
+
+
+
+```python
+import gym
+
+from huggingface_sb3 import load_from_hub, package_to_hub, push_to_hub
+from huggingface_hub import (
+ notebook_login,
+) # To log to our Hugging Face account to be able to upload models to the Hub.
+
+from stable_baselines3 import PPO
+from stable_baselines3.common.evaluation import evaluate_policy
+from stable_baselines3.common.env_util import make_vec_env
+```
+
+## Understand what is Gym and how it works 🤖
+
+🏋 The library containing our environment is called Gym.
+**You'll use Gym a lot in Deep Reinforcement Learning.**
+
+The Gym library provides two things:
+- An interface that allows you to **create RL environments**.
+- A **collection of environments** (gym-control, atari, box2D...).
+
+Let's look at an example, but first let's remember what's the RL Loop.
+
+
+
+- Copy the token
+- Run the cell below and paste the token
+
+```python
+notebook_login()
+!git config --global credential.helper store
+```
+
+If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`
+
+3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using `package_to_hub()` function
+
+Let's fill the `package_to_hub` function:
+- `model`: our trained model.
+- `model_name`: the name of the trained model that we defined in `model_save`
+- `model_architecture`: the model architecture we used: in our case PPO
+- `env_id`: the name of the environment, in our case `LunarLander-v2`
+- `eval_env`: the evaluation environment defined in eval_env
+- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `(repo_id = {username}/{repo_name})`
+
+💡 **A good name is `{username}/{model_architecture}-{env_id}` **
+
+- `commit_message`: message of the commit
+
+```python
+import gym
+from stable_baselines3.common.vec_env import DummyVecEnv
+from stable_baselines3.common.env_util import make_vec_env
+
+from huggingface_sb3 import package_to_hub
+
+## TODO: Define a repo_id
+## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2
+repo_id =
+
+# TODO: Define the name of the environment
+env_id =
+
+# Create the evaluation env
+eval_env = DummyVecEnv([lambda: gym.make(env_id)])
+
+
+# TODO: Define the model architecture we used
+model_architecture = ""
+
+## TODO: Define the commit message
+commit_message = ""
+
+# method save, evaluate, generate a model card and record a replay video of your agent before pushing the repo to the hub
+package_to_hub(model=model, # Our trained model
+ model_name=model_name, # The name of our trained model
+ model_architecture=model_architecture, # The model architecture we used: in our case PPO
+ env_id=env_id, # Name of the environment
+ eval_env=eval_env, # Evaluation Environment
+ repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2
+ commit_message=commit_message)
+
+# Note: if after running the package_to_hub function and it gives an issue of rebasing, please run the following code
+# cd 
+
+2. Then we just need to use load_from_hub with:
+- The repo_id
+- The filename: the saved model inside the repo and its extension (*.zip)
+
+```python
+from huggingface_sb3 import load_from_hub
+
+repo_id = "Classroom-workshop/assignment2-omar" # The repo_id
+filename = "ppo-LunarLander-v2.zip" # The model filename.zip
+
+# When the model was trained on Python 3.8 the pickle protocol is 5
+# But Python 3.6, 3.7 use protocol 4
+# In order to get compatibility we need to:
+# 1. Install pickle5 (we done it at the beginning of the colab)
+# 2. Create a custom empty object we pass as parameter to PPO.load()
+custom_objects = {
+ "learning_rate": 0.0,
+ "lr_schedule": lambda _: 0.0,
+ "clip_range": lambda _: 0.0,
+}
+
+checkpoint = load_from_hub(repo_id, filename)
+model = PPO.load(checkpoint, custom_objects=custom_objects, print_system_info=True)
+```
+
+Let's evaluate this agent:
+
+```python
+# @title
+eval_env = gym.make("LunarLander-v2")
+mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)
+print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")
+```
+
+## Some additional challenges 🏆
+The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results!
+
+In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?
+
+Here are some ideas to achieve so:
+* Train more steps
+* Try different hyperparameters of `PPO`. You can see them at https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#parameters.
+* Check the [Stable-Baselines3 documentation](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) and try another models such as DQN.
+* **Push your new trained model** on the Hub 🔥
+
+**Compare the results of your LunarLander-v2 with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) 🏆
+
+Is moon landing too boring to you? Try to **change the environment**, why not using MountainCar-v0, CartPole-v1 or CarRacing-v0? Check how they works [using the gym documentation](https://www.gymlibrary.dev/) and have fun 🎉.
+
+________________________________________________________________________
+Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.**
+
+If you’re still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**
+
+Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations.
+
+Naturally, during the course, we’re going to use and deeper explain again these terms but **it’s better to have a good understanding of them now before diving into the next chapters.**
+
+
+Next time, in the bonus unit 1, you'll train Huggy the Dog to fetch the stick.
+
+
+
+## Keep learning, stay awesome 🤗
diff --git a/units/en/unit1/introduction.mdx b/units/en/unit1/introduction.mdx
new file mode 100644
index 0000000..660ec20
--- /dev/null
+++ b/units/en/unit1/introduction.mdx
@@ -0,0 +1,28 @@
+# Introduction to Deep Reinforcement Learning [[introduction-to-deep-reinforcement-learning]]
+
+
+
+
+Welcome to the most fascinating topic in Artificial Intelligence: **Deep Reinforcement Learning.**
+
+Deep RL is a type of Machine Learning where an agent learns **how to behave** in an environment **by performing actions** and **seeing the results.**
+
+In this first unit, **you'll learn the foundations of Deep Reinforcement Learning.**
+
+
+Then, you'll **train your Deep Reinforcement Learning agent, a lunar lander to land correctly on the Moon** using Stable-Baselines3 , a Deep Reinforcement Learning library.
+
+
+
+
+At every step:
+- Our Agent receives ______ from the environment
+- Based on that ______ the Agent takes an ______
+- Our Agent will move to the right
+- The Environment goes to a ______
+- The Environment gives a ______ to the Agent
+
+
+
+
+
+
+
+The agent's goal is to _maximize_ its cumulative reward, **called the expected return.**
+
+## The reward hypothesis: the central idea of Reinforcement Learning [[reward-hypothesis]]
+
+⇒ Why is the goal of the agent to maximize the expected return?
+
+Because RL is based on the **reward hypothesis**, which is that all goals can be described as the **maximization of the expected return** (expected cumulative reward).
+
+That’s why in Reinforcement Learning, **to have the best behavior,** we aim to learn to take actions that **maximize the expected cumulative reward.**
+
+
+## Markov Property [[markov-property]]
+
+In papers, you’ll see that the RL process is called the **Markov Decision Process** (MDP).
+
+We’ll talk again about the Markov Property in the following units. But if you need to remember something today about it, it's this: the Markov Property implies that our agent needs **only the current state to decide** what action to take and **not the history of all the states and actions** they took before.
+
+## Observations/States Space [[obs-space]]
+
+Observations/States are the **information our agent gets from the environment.** In the case of a video game, it can be a frame (a screenshot). In the case of the trading agent, it can be the value of a certain stock, etc.
+
+There is a differentiation to make between *observation* and *state*, however:
+
+- *State s*: is **a complete description of the state of the world** (there is no hidden information). In a fully observed environment.
+
+
+
+
+
+
+
+## Action Space [[action-space]]
+
+The Action space is the set of **all possible actions in an environment.**
+
+The actions can come from a *discrete* or *continuous space*:
+
+- *Discrete space*: the number of possible actions is **finite**.
+
+
+
+
+Taking this information into consideration is crucial because it will **have importance when choosing the RL algorithm in the future.**
+
+## Rewards and the discounting [[rewards]]
+
+The reward is fundamental in RL because it’s **the only feedback** for the agent. Thanks to it, our agent knows **if the action taken was good or not.**
+
+The cumulative reward at each time step **t** can be written as:
+
+
+
+
+
+As we can see in the diagram, **it’s more probable to eat the cheese near us than the cheese close to the cat** (the closer we are to the cat, the more dangerous it is).
+
+Consequently, **the reward near the cat, even if it is bigger (more cheese), will be more discounted** since we’re not really sure we’ll be able to eat it.
+
+To discount the rewards, we proceed like this:
+
+1. We define a discount rate called gamma. **It must be between 0 and 1.** Most of the time between **0.99 and 0.95**.
+- The larger the gamma, the smaller the discount. This means our agent **cares more about the long-term reward.**
+- On the other hand, the smaller the gamma, the bigger the discount. This means our **agent cares more about the short term reward (the nearest cheese).**
+
+2. Then, each reward will be discounted by gamma to the exponent of the time step. As the time step increases, the cat gets closer to us, **so the future reward is less and less likely to happen.**
+
+Our discounted expected cumulative reward is:
+
diff --git a/units/en/unit1/summary.mdx b/units/en/unit1/summary.mdx
new file mode 100644
index 0000000..05c57ff
--- /dev/null
+++ b/units/en/unit1/summary.mdx
@@ -0,0 +1,19 @@
+# Summary [[summary]]
+
+That was a lot of information! Let's summarize:
+
+- Reinforcement Learning is a computational approach of learning from action. We build an agent that learns from the environment **by interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.
+
+- The goal of any RL agent is to maximize its expected cumulative reward (also called expected return) because RL is based on the **reward hypothesis**, which is that **all goals can be described as the maximization of the expected cumulative reward.**
+
+- The RL process is a loop that outputs a sequence of **state, action, reward and next state.**
+
+- To calculate the expected cumulative reward (expected return), we discount the rewards: the rewards that come sooner (at the beginning of the game) **are more probable to happen since they are more predictable than the long term future reward.**
+
+- To solve an RL problem, you want to **find an optimal policy**. The policy is the “brain” of your agent, which will tell us **what action to take given a state.** The optimal policy is the one which **gives you the actions that maximize the expected return.**
+
+- There are two ways to find your optimal policy:
+ 1. By training your policy directly: **policy-based methods.**
+ 2. By training a value function that tells us the expected return the agent will get at each state and use this function to define our policy: **value-based methods.**
+
+- Finally, we speak about Deep RL because we introduce **deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based)** hence the name “deep”.
diff --git a/units/en/unit1/tasks.mdx b/units/en/unit1/tasks.mdx
new file mode 100644
index 0000000..1be4fea
--- /dev/null
+++ b/units/en/unit1/tasks.mdx
@@ -0,0 +1,27 @@
+# Type of tasks [[tasks]]
+
+A task is an **instance** of a Reinforcement Learning problem. We can have two types of tasks: **episodic** and **continuing**.
+
+## Episodic task [[episodic-task]]
+
+In this case, we have a starting point and an ending point **(a terminal state). This creates an episode**: a list of States, Actions, Rewards, and new States.
+
+For instance, think about Super Mario Bros: an episode begin at the launch of a new Mario Level and ending **when you’re killed or you reached the end of the level.**
+
+
+
+To recap:
+
diff --git a/units/en/unit1/two-methods.mdx b/units/en/unit1/two-methods.mdx
new file mode 100644
index 0000000..4f3d8a8
--- /dev/null
+++ b/units/en/unit1/two-methods.mdx
@@ -0,0 +1,99 @@
+# Two main approaches for solving RL problems [[two-methods]]
+
+
+
+
+
+- *Stochastic*: outputs **a probability distribution over actions.**
+
+
+
+
+
+
+## Value-based methods [[value-based]]
+
+In value-based methods, instead of training a policy function, we **train a value function** that maps a state to the expected value **of being at that state.**
+
+The value of a state is the **expected discounted return** the agent can get if it **starts in that state, and then act according to our policy.**
+
+“Act according to our policy” just means that our policy is **“going to the state with the highest value”.**
+
+
+
+Here we see that our value function **defined value for each possible state.**
+
+
+
+
diff --git a/units/en/unit1/what-is-rl.mdx b/units/en/unit1/what-is-rl.mdx
new file mode 100644
index 0000000..410569c
--- /dev/null
+++ b/units/en/unit1/what-is-rl.mdx
@@ -0,0 +1,40 @@
+# What is Reinforcement Learning? [[what-is-reinforcement-learning]]
+
+To understand Reinforcement Learning, let’s start with the big picture.
+
+## The big picture [[the-big-picture]]
+
+The idea behind Reinforcement Learning is that an agent (an AI) will learn from the environment by **interacting with it** (through trial and error) and **receiving rewards** (negative or positive) as feedback for performing actions.
+
+Learning from interactions with the environment **comes from our natural experiences.**
+
+For instance, imagine putting your little brother in front of a video game he never played, giving him a controller, and leaving him alone.
+
+
+
+
+Your brother will interact with the environment (the video game) by pressing the right button (action). He got a coin, that’s a +1 reward. It’s positive, he just understood that in this game **he must get the coins.**
+
+
+
+But then, **he presses right again** and he touches an enemy. He just died, so that's a -1 reward.
+
+
+
+
+By interacting with his environment through trial and error, your little brother understood that **he needed to get coins in this environment but avoid the enemies.**
+
+**Without any supervision**, the child will get better and better at playing the game.
+
+That’s how humans and animals learn, **through interaction.** Reinforcement Learning is just a **computational approach of learning from actions.**
+
+
+### A formal definition [[a-formal-definition]]
+
+If we take now a formal definition:
+
+
+
+
+### Keep Learning, Stay Awesome 🤗
diff --git a/units/en/unitbonus1/how-huggy-works.mdx b/units/en/unitbonus1/how-huggy-works.mdx
new file mode 100644
index 0000000..bc1ac4d
--- /dev/null
+++ b/units/en/unitbonus1/how-huggy-works.mdx
@@ -0,0 +1,65 @@
+# How Huggy works [[how-huggy-works]]
+
+Huggy is a Deep Reinforcement Learning environment made by Hugging Face and based on [Puppo the Corgi, a project by the Unity MLAgents team](https://blog.unity.com/technology/puppo-the-corgi-cuteness-overload-with-the-unity-ml-agents-toolkit).
+This environment was created using the [Unity game engine](https://unity.com/) and [MLAgents](https://github.com/Unity-Technologies/ml-agents). ML-Agents is a toolkit for the game engine from Unity that allows us to **create environments using Unity or use pre-made environments to train our agents**.
+
+
+
+In this environment we aim to train Huggy to **fetch the stick we throw at him. It means he needs to move correctly toward the stick**.
+
+## The State Space, what Huggy perceives. [[state-space]]
+Huggy doesn't "see" his environment. Instead, we provide him information about the environment:
+* The target (stick) position
+* The relative position between himself and the target
+* The orientation of his legs.
+
+Given all this information, Huggy can **use his policy to determine which action to take next to fulfill his goal**.
+
+## The Action Space, what moves Huggy can perform [[action-space]]
+
+
+**Joint motors drive Huggy legs**. It means that to get the target, Huggy needs to **learn to rotate the joint motors of each of his legs correctly so he can move**.
+
+## The Reward Function [[reward-function]]
+
+The reward function is designed so that **Huggy will fulfill his goal**: fetch the stick.
+
+Remember that one of the foundations of Reinforcement Learning is the *reward hypothesis*: a goal can be described as the **maximization of the expected cumulative reward**.
+
+Here, our goal is that Huggy **goes towards the stick but without spinning too much**. Hence, our reward function must translate this goal.
+
+Our reward function:
+
+
+
+- *Orientation bonus*: we **reward him for getting close to the target**.
+- *Time penalty*: a fixed-time penalty given at every action to **force him to get to the stick as fast as possible**.
+- *Rotation penalty*: we penalize Huggy if **he spins too much and turns too quickly**.
+- *Getting to the target reward*: we reward Huggy for **reaching the target**.
+
+If you want to see what this reward function looks like mathematically, check [Puppo the Corgi presentation](https://blog.unity.com/technology/puppo-the-corgi-cuteness-overload-with-the-unity-ml-agents-toolkit).
+
+## Train Huggy
+
+Huggy aims **to learn to run correctly and as fast as possible toward the goal**. To do that, at every step and given the environment observation, he needs to decide how to rotate each joint motor of his legs to move correctly (not spinning too much) and towards the goal.
+
+The training loop looks like this:
+
+
+
+
+The training environment looks like this:
+
+
+
+
+It's a place where a **stick is spawned randomly**. When Huggy reaches it, the stick get spawned somewhere else.
+We built **multiple copies of the environment for the training**. This helps speed up the training by providing more diverse experiences.
+
+
+
+Now that you have the big picture of the environment, you're ready to train Huggy to fetch the stick.
+
+To do that, we're going to use [MLAgents](https://github.com/Unity-Technologies/ml-agents). Don't worry if you have never used it before. In this unit we'll use Google Colab to train Huggy, and then you'll be able to load your trained Huggy and play with him directly in the browser.
+
+In a future unit, we will study more in-depth MLAgents and how it works. But for now, we keep things simple by just using the provided implementation.
diff --git a/units/en/unitbonus1/introduction.mdx b/units/en/unitbonus1/introduction.mdx
new file mode 100644
index 0000000..68a57f9
--- /dev/null
+++ b/units/en/unitbonus1/introduction.mdx
@@ -0,0 +1,7 @@
+# Introduction [[introduction]]
+
+In this bonus unit, we'll reinforce what we learned in the first unit by teaching Huggy the Dog to fetch the stick and then [play with him directly in your browser](https://huggingface.co/spaces/ThomasSimonini/Huggy) 🐶
+
+
+
+So let's get started 🚀
diff --git a/units/en/unitbonus1/play.mdx b/units/en/unitbonus1/play.mdx
new file mode 100644
index 0000000..f2154cc
--- /dev/null
+++ b/units/en/unitbonus1/play.mdx
@@ -0,0 +1,19 @@
+# Play with Huggy [[play]]
+
+Now that you've trained Huggy and pushed it to the Hub. **You will be able to play with him ❤️**
+
+For this step it’s simple:
+
+- Open the game Huggy in your browser: https://huggingface.co/spaces/ThomasSimonini/Huggy
+
+- Click on Play with my Huggy model
+
+
+
+1. In step 1, choose your model repository which is the model id (in my case ThomasSimonini/ppo-Huggy).
+
+2. In step 2, **choose what model you want to replay**:
+ - I have multiple one, since we saved a model every 500000 timesteps.
+ - But if I want the more recent I choose Huggy.onnx
+
+👉 What’s nice **is to try with different models step to see the improvement of the agent.**
diff --git a/units/en/unitbonus1/train.mdx b/units/en/unitbonus1/train.mdx
new file mode 100644
index 0000000..b353f86
--- /dev/null
+++ b/units/en/unitbonus1/train.mdx
@@ -0,0 +1,260 @@
+# Let's train and play with Huggy 🐶 [[train]]
+
+
+
+
+ 
+
+With ML Agents, we run a training script. We define four parameters:
+
+1. `mlagents-learn 
+
+And we define 4 parameters:
+
+1. `--run-id`: the name of the training run id.
+2. `--local-dir`: where the agent was saved, it’s results/
+
+But now comes the best: **being able to play with Huggy online 👀.**
+
+## Play with your Huggy 🐕
+
+This step is the simplest:
+
+- Open the game Huggy in your browser: https://huggingface.co/spaces/ThomasSimonini/Huggy
+
+- Click on Play with my Huggy model
+
+