diff --git a/unit1/unit1.ipynb b/unit1/unit1.ipynb
new file mode 100644
index 0000000..55f9255
--- /dev/null
+++ b/unit1/unit1.ipynb
@@ -0,0 +1,1094 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "njb_ProuHiOe"
+ },
+ "source": [
+ "# Unit 1: Train your first Deep Reinforcement Learning Agent ๐\n",
+ "In this notebook, you'll train your first lander agent to ****land correctly on the Moon ๐ share it to the community, and experiment with different configurations**\n",
+ "\n",
+ "โ If you have questions, please post them on #study-group-unit1 discord channel ๐ https://discord.gg/aYka4Yhff9\n",
+ "\n",
+ "\n",
+ "๐ฎ Environment: [LunarLander-v2](https://www.gymlibrary.ml/environments/box2d/lunar_lander/)\n",
+ "\n",
+ "๐ RL-Library: [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/)\n",
+ "\n",
+ "โฌ๏ธ Here is an example of what **you will achieve in just a couple of minutes.** โฌ๏ธ"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 421
+ },
+ "id": "PF46MwbZD00b",
+ "outputId": "8fba2a05-2558-4231-95f3-0840d3d3d108"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ ""
+ ],
+ "text/html": [
+ ""
+ ]
+ },
+ "metadata": {}
+ }
+ ],
+ "source": [
+ "%%html\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4i6tjI2tHQ8j"
+ },
+ "source": [
+ "## Objectives of this notebook ๐\n",
+ "At the end of the notebook, you will:\n",
+ "- Be able to use **Gym**, the environment library.\n",
+ "- Be able to use **Stable-Baselines3**, the deep reinforcement learning library.\n",
+ "- Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score ๐ฅ.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ct8eLAabICE-"
+ },
+ "source": [
+ "## This notebook is from Deep Reinforcement Learning Class\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "6p5HnEefISCB"
+ },
+ "source": [
+ "In this free course, you will:\n",
+ "\n",
+ "- ๐ Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- ๐งโ๐ป Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, and RLlib.\n",
+ "- ๐ค Train **agents in unique environments** \n",
+ "\n",
+ "And more check ๐ the syllabus ๐ https://github.com/huggingface/deep-rl-class\n",
+ "\n",
+ "The best way to keep in touch is to join our discord server to exchange with the community and with us ๐๐ป https://discord.gg/aYka4Yhff9"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Y-mo_6rXIjRi"
+ },
+ "source": [
+ "## Prerequisites ๐๏ธ\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "๐ฒ ๐ [Read the Unit 1 Readme]() that contains all the information.\n",
+ "\n",
+ "๐ฒ ๐ **The foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by reading first ๐ https://huggingface.co/blog/deep-rl-intro\n",
+ "\n",
+ "๐ฒ ๐ข Sign up to [our Discord Server](https://discord.gg/aYka4Yhff9) and **introduce yourself to #introduce-yourself channel ๐ฅณ**\n",
+ "\n",
+ "๐ฒ ๐ Are you new to Discord? Check our **discord 101 to get the best practices** ๐ https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "aXDZCGHnJ-3h"
+ },
+ "source": [
+ "## A small recap of Deep Reinforcement Learning ๐\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xcQYx9ynaFMD"
+ },
+ "source": [
+ "Let's do a small recap on what we learned in the first Unit:\n",
+ "- Reinforcement Learning is a **computational approach to learning from action**. We build an agent that learns from the environment by **interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.\n",
+ "\n",
+ "- The goal of any RL agent is to **maximize its expected cumulative reward** (also called expected return) because RL is based on the _reward hypothesis_, which is that all goals can be described as the maximization of the expected cumulative reward.\n",
+ "\n",
+ "- The RL process is a **loop that outputs a sequence of state, action, reward, and next state**.\n",
+ "\n",
+ "- To calculate the expected cumulative reward (expected return), **we discount the rewards**: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long-term future reward.\n",
+ "\n",
+ "- To solve an RL problem, you want to **find an optimal policy**; the policy is the \"brain\" of your AI that will tell us what action to take given a state. The optimal one is the one that gives you the actions that max the expected return.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8X2WN2X99BEz"
+ },
+ "source": [
+ "There are **two** ways to find your optimal policy:\n",
+ "- By **training your policy directly**: policy-based methods.\n",
+ "- By **training a value function** that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BPy6dfcm8_WT"
+ },
+ "source": [
+ "- Finally, we spoke about Deep RL because **we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name \"deep.\"**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "LzGDDOTSKaF8"
+ },
+ "source": [
+ "# Let's train a Deep Reinforcement Learning lander agent to land correctly on the Moon ๐ and upload it to the Hub.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "YkcELpiYRC5m"
+ },
+ "source": [
+ "### Step 0: Set the GPU ๐ช\n",
+ "- To **faster the agent's training, we'll use a GPU** to do that go to `Runtime > Change Runtime type`\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "38HBd3t1SHJ8"
+ },
+ "source": [
+ "- `Hardware Accelerator > GPU`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "sw0pSDDsR1kq"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install virtual screen libraries and create and run a virtual screen ๐ฅ"
+ ],
+ "metadata": {
+ "id": "LW_lsjc8h_up"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip3 install pyvirtualdisplay\n",
+ "\n",
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "dlx9uWHuhqfh"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jeDAH0h0EBiG"
+ },
+ "source": [
+ "### Step 1: Install dependencies ๐ฝ\n",
+ "The first step is to install the dependencies, weโll install multiple ones:\n",
+ "\n",
+ "- `gym[box2D]`: Contains the LunarLander-v2 environment ๐\n",
+ "- `stable-baselines3[extra]`: The deep reinforcement learning library.\n",
+ "- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face ๐ค Hub.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "9XaULfDZDvrC"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install gym[box2d]\n",
+ "!pip install stable-baselines3[extra]\n",
+ "!pip install huggingface_sb3\n",
+ "!pip install ale-py==0.7.4 # To overcome an issue with gym (https://github.com/DLR-RM/stable-baselines3/issues/875)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "wrgpVFqyENVf"
+ },
+ "source": [
+ "### Step 2: Import the packages ๐ฆ\n",
+ "\n",
+ "One additional library we import is huggingface_hub **to be able to upload and download trained models from the hub**.\n",
+ "\n",
+ "\n",
+ "The Hugging Face Hub ๐ค works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easilly collaborate with others.\n",
+ "\n",
+ "You can see here all the Deep reinforcement Learning models available ๐ https://huggingface.co/models?pipeline_tag=reinforcement-learning&sort=downloads\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "cygWLPGsEQ0m"
+ },
+ "outputs": [],
+ "source": [
+ "import gym\n",
+ "\n",
+ "from huggingface_sb3 import load_from_hub, package_to_hub, push_to_hub\n",
+ "from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
+ "\n",
+ "from stable_baselines3 import PPO\n",
+ "from stable_baselines3.common.evaluation import evaluate_policy\n",
+ "from stable_baselines3.common.env_util import make_vec_env"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Step 3: Understand what is Gym and how it works? ๐ค\n",
+ "\n",
+ "๐ The library containing our environment is called Gym.\n",
+ "You'll use a lot **Gym in Deep Reinforcement Learning.**\n",
+ "\n",
+ "The Gym library provides two things:\n",
+ "- An interface that allows you to **create RL environments**.\n",
+ "- A **collection of environments** (gym-control, atari, box2D...).\n",
+ "\n",
+ "Let's look at an example, but first let's remember what's the RL Loop.\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "MRqRuRUl8CsB"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "VvCOlJp-_kw4"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "At each step:\n",
+ "- Our Agent receivesย **state S0**ย from theย **Environment**ย โ we receive the first frame of our game (Environment).\n",
+ "- Based on thatย **state S0,**ย the Agent takesย **action A0**ย โ our Agent will move to the right.\n",
+ "- Environment to aย **new**ย **state S1**ย โ new frame.\n",
+ "- The environment gives someย **reward R1**ย to the Agent โ weโre not deadย *(Positive Reward +1)*.\n",
+ "\n",
+ "\n",
+ "With Gym:\n",
+ "\n",
+ "1๏ธโฃ We create our environment using `gym.make()`\n",
+ "\n",
+ "2๏ธโฃ We reset the environment to its initial state with `observation = env.reset()`\n",
+ "\n",
+ "At each step:\n",
+ "\n",
+ "3๏ธโฃ Get an action using our model (in our example we take a random action)\n",
+ "\n",
+ "4๏ธโฃ Using `env.step(action)`, we perform this action in the environment and get\n",
+ "- `obsevation`: The new state (st+1)\n",
+ "- `reward`: The reward we get after executing the action\n",
+ "- `done`: Indicates if the episode terminated\n",
+ "- `info`: A dictionary that provides additional information (depends on the environment).\n",
+ "\n",
+ "If the episode is done:\n",
+ "- We reset the environment to its initial state with `observation = env.reset()`\n",
+ "\n",
+ "**Let's look at an example!** Make sure to read the code\n"
+ ],
+ "metadata": {
+ "id": "-TzNN0bQ_j-3"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import gym\n",
+ "\n",
+ "# First, we create our environment called LunarLander-v2\n",
+ "env = gym.make(\"LunarLander-v2\")\n",
+ "\n",
+ "# Then we reset this environment\n",
+ "observation = env.reset()\n",
+ "\n",
+ "for _ in range(20):\n",
+ " # Take a random action\n",
+ " action = env.action_space.sample()\n",
+ " print(\"Action taken:\", action)\n",
+ "\n",
+ " # Do this action in the environment and get\n",
+ " # next_state, reward, done and info\n",
+ " observation, reward, done, info = env.step(action)\n",
+ " \n",
+ " # If the game is done (in our case we land, crashed or timeout)\n",
+ " if done:\n",
+ " # Reset the environment\n",
+ " print(\"Environment is reset\")\n",
+ " observation = env.reset()"
+ ],
+ "metadata": {
+ "id": "w7vOFlpA_ONz"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "XIrKGGSlENZB"
+ },
+ "source": [
+ "### Step 4: Create the LunarLander environment ๐ and understand how it works\n",
+ "#### [The environment ๐ฎ](https://www.gymlibrary.ml/environments/box2d/lunar_lander/)\n",
+ "In this first tutorial, weโre going to train our agent, a [Lunar Lander](https://www.gymlibrary.ml/environments/box2d/lunar_lander/), **to land correctly on the moon**. To do that, the agent needs to learn **to adapt its speed and position(horizontal, vertical, and angular) to land correctly.**\n",
+ "\n",
+ "\n",
+ "---\n",
+ "\n",
+ "\n",
+ "๐ก A good habit when you start to use an environment is to check its documentation \n",
+ "\n",
+ "๐ https://www.gymlibrary.ml/environments/box2d/lunar_lander/\n",
+ "\n",
+ "---\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "poLBgRocF9aT"
+ },
+ "source": [
+ "Let's see what the Environment looks like:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "ZNPG0g_UGCfh",
+ "outputId": "265ae44f-d404-43d8-9d62-378c3fa61f35"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "_____OBSERVATION SPACE_____ \n",
+ "\n",
+ "Observation Space Shape (8,)\n",
+ "Sample observation [ 0.01816158 -0.56312186 0.08143468 1.8260419 0.51859415 -1.0216727\n",
+ " 0.703369 0.5369312 ]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# We create our environment with gym.make(\"\")\n",
+ "env = gym.make(\"LunarLander-v2\")\n",
+ "env.reset()\n",
+ "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
+ "print(\"Observation Space Shape\", env.observation_space.shape)\n",
+ "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2MXc15qFE0M9"
+ },
+ "source": [
+ "We see with `Observation Space Shape (8,)` that the observation is a vector of size 8, each value is a different information about the lander:\n",
+ "- Horizontal pad coordinate (x)\n",
+ "- Vertical pad coordinate (y)\n",
+ "- Horizontal speed (x)\n",
+ "- Vertical speed (y)\n",
+ "- Angle\n",
+ "- Angular speed\n",
+ "- If the left leg has contact point touched the land\n",
+ "- If the right leg has contact point touched the land\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "We5WqOBGLoSm",
+ "outputId": "c3bab680-a4fc-4e30-eea5-9f8391c2f825"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\n",
+ " _____ACTION SPACE_____ \n",
+ "\n",
+ "Action Space Shape 4\n",
+ "Action Space Sample 0\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
+ "print(\"Action Space Shape\", env.action_space.n)\n",
+ "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "MyxXwkI2Magx"
+ },
+ "source": [
+ "The action space (the set of possible actions the agent can take) is discrete with 4 actions available ๐ฎ: \n",
+ "\n",
+ "- Do nothing,\n",
+ "- Fire left orientation engine,\n",
+ "- Fire the main engine,\n",
+ "- Fire right orientation engine.\n",
+ "\n",
+ "Reward function (the function that will gives a reward at each timestep) ๐ฐ:\n",
+ "\n",
+ "- Moving from the top of the screen to the landing pad and zero speed is about 100~140 points.\n",
+ "- Firing main engine is -0.3 each frame\n",
+ "- Each leg ground contact is +10 points\n",
+ "- Episode finishes if the lander crashes (additional - 100 points) or come to rest (+100 points)\n",
+ "- The game is solved if your agent does 200 points."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "dFD9RAFjG8aq"
+ },
+ "source": [
+ "#### Vectorized Environment\n",
+ "- We create a vectorized environment (method for stacking multiple independent environments into a single environment) of 16 environments, this way, **we'll have more diverse experiences during the training.**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "99hqQ_etEy1N"
+ },
+ "outputs": [],
+ "source": [
+ "# Create the environment\n",
+ "env = make_vec_env('LunarLander-v2', n_envs=16)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VgrE86r5E5IK"
+ },
+ "source": [
+ "### Step 5: Create the Model ๐ค\n",
+ "- Now that we studied our environment and we understood the problem: **being able to land correctly the Lunar Lander to the Landing Pad by controlling left, right and main orientation engine**. Let's build the algorithm we're going to use to solve this Problem ๐.\n",
+ "\n",
+ "- To do so, we're going to use our first Deep RL library, [Stable Baselines3 (SB3)](https://stable-baselines3.readthedocs.io/en/master/).\n",
+ "\n",
+ "- SB3 is a set of **reliable implementations of reinforcement learning algorithms in PyTorch**.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "๐ก A good habit when using a new library is to dive first on the documentation: https://stable-baselines3.readthedocs.io/en/master/ and then try some tutorials.\n",
+ "\n",
+ "----"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "3wgg-y0-92Wj"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "HV4yiUM_9_Ka"
+ },
+ "source": [
+ "To solve this problem, we're going to use SB3 **PPO**. [PPO (aka Proximal Policy Optimization) is one of the of the SOTA (state of the art) Deep Reinforcement Learning algorithms that you'll study during this course](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#example%5D).\n",
+ "\n",
+ "PPO is a combination of:\n",
+ "- *Value-based reinforcement learning method*: learning an action-value function that will tell us what's the **most valuable action to take given a state and action**.\n",
+ "- *Policy-based reinforcement learning method*: learning a policy that will **gives us a probability distribution over actions**.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5qL_4HeIOrEJ"
+ },
+ "source": [
+ "Stable-Baselines3 is easy to set up:\n",
+ "\n",
+ "1๏ธโฃ You **create your environment** (in our case it was done above)\n",
+ "\n",
+ "2๏ธโฃ You define the **model you want to use and instantiate this model** `model = PPO(\"MlpPolicy\")`\n",
+ "\n",
+ "3๏ธโฃ You **train the agent** with `model.learn` and define the number of training timesteps\n",
+ "\n",
+ "```\n",
+ "# Create environment\n",
+ "env = gym.make('LunarLander-v2')\n",
+ "\n",
+ "# Instantiate the agent\n",
+ "model = PPO('MlpPolicy', env, verbose=1)\n",
+ "# Train the agent\n",
+ "model.learn(total_timesteps=int(2e5))\n",
+ "```\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "nxI6hT1GE4-A"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO: Define a PPO MlpPolicy architecture\n",
+ "# We use MultiLayerPerceptron (MLPPolicy) because the input is a vector,\n",
+ "# if we had frames as input we would use CnnPolicy\n",
+ "model = "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "QAN7B0_HCVZC"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "543OHYDfcjK4",
+ "outputId": "9babd3e1-0a6d-4d86-da35-f989055e8d82"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Using cuda device\n"
+ ]
+ }
+ ],
+ "source": [
+ "# SOLUTION\n",
+ "# We added some parameters to fasten the training\n",
+ "model = PPO(\n",
+ " policy = 'MlpPolicy',\n",
+ " env = env,\n",
+ " n_steps = 1024,\n",
+ " batch_size = 64,\n",
+ " n_epochs = 4,\n",
+ " gamma = 0.999,\n",
+ " gae_lambda = 0.98,\n",
+ " ent_coef = 0.01,\n",
+ " verbose=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ClJJk88yoBUi"
+ },
+ "source": [
+ "### Step 6: Train the PPO agent ๐\n",
+ "- Let's train our agent for 500,000 timesteps, don't forget to use GPU on Colab. It will take approximately ~10min, but you can use less timesteps if you just want to try it out.\n",
+ "- During the training, take a โ break you deserved it ๐ค"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "qKnYkNiVp89p"
+ },
+ "outputs": [],
+ "source": [
+ "# Train it for 500,000 timesteps"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1bQzQ-QcE3zo"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "poBCy9u_csyR"
+ },
+ "outputs": [],
+ "source": [
+ "# SOLUTION\n",
+ "# Train it for 500,000 timesteps\n",
+ "model.learn(total_timesteps=500000)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BY_HuedOoISR"
+ },
+ "source": [
+ "### Step 7: Evaluate the agent ๐\n",
+ "- Now that's our Lunar Lander agent is trained ๐, we need to **check its performance**.\n",
+ "- Stable-Baselines3 provides a method to do that `evaluate_policy`.\n",
+ "- To fill that part you need to [check the documentation](https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#basic-usage-training-saving-loading)\n",
+ "- In the next step, we'll see **how to automatically evaluate and share your agent to compete in a leaderboard, but for now let's do it ourselves**\n",
+ "\n",
+ "\n",
+ "๐ก When you evaluate your agent, you should not use your training environment but create an evaluation environment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "yRpno0glsADy"
+ },
+ "outputs": [],
+ "source": [
+ "# TODO: Evaluate the agent\n",
+ "# Create a new environment for evaluation\n",
+ "eval_env =\n",
+ "\n",
+ "# Evaluate the model with 10 evaluation episodes and deterministic=True\n",
+ "mean_reward, std_reward = \n",
+ "\n",
+ "# Print the results\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BqPKw3jt_pG5"
+ },
+ "source": [
+ "#### Solution"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "zpz8kHlt_a_m"
+ },
+ "outputs": [],
+ "source": [
+ "#@title\n",
+ "eval_env = gym.make(\"LunarLander-v2\")\n",
+ "mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)\n",
+ "print(f\"mean_reward={mean_reward:.2f} +/- {std_reward}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "reBhoODwcXfr"
+ },
+ "source": [
+ "- In my case, I got a mean reward is `200.20 +/- 20.80` after training for 1 million steps, which means that our lunar lander agent is ready to land on the moon ๐๐ฅณ."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IK_kR78NoNb2"
+ },
+ "source": [
+ "### Step 8: Publish our trained model on the Hub ๐ฅ\n",
+ "Now that we saw we got good results after the training, we can publish our trained model on the hub ๐ค with one line of code.\n",
+ "\n",
+ "๐ The libraries documentation ๐ https://github.com/huggingface/huggingface_sb3/tree/main#hugging-face--x-stable-baselines3-v20\n",
+ "\n",
+ "Here's an example of a Model Card (with Space Invaders):"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Nhded6x9gapA"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Gs-Ew7e1gXN3"
+ },
+ "source": [
+ "By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
+ "\n",
+ "---\n",
+ "The package_to_hub function\n",
+ "---\n",
+ "\n",
+ "This way:\n",
+ "- You can **showcase our work** ๐ฅ\n",
+ "- You can **visualize your agent playing** ๐\n",
+ "- You can **share with the community an agent that others can use** ๐พ\n",
+ "- You can **access a leaderboard ๐ to see how well your agent is performing compared to your classmates** ๐ https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "JquRrWytA6eo"
+ },
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1๏ธโฃ (If it's not already done) create an account to HF โก https://huggingface.co/join\n",
+ "\n",
+ "2๏ธโฃ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens)\n",
+ "- Copy the token \n",
+ "- Run the cell below and past the token"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GZiFBBlzxzxY"
+ },
+ "outputs": [],
+ "source": [
+ "notebook_login()\n",
+ "!git config --global credential.helper store"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_tsf2uv0g_4p"
+ },
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "FGNh9VsZok0i"
+ },
+ "source": [
+ "3๏ธโฃ We're now ready to push our trained agent to the ๐ค Hub ๐ฅ using `package_to_hub()` function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Ay24l6bqFF18"
+ },
+ "source": [
+ "Let's fill the `package_to_hub` function:\n",
+ "- `model`: our trained model.\n",
+ "- `model_name`: the name of the trained model that we defined in `model_save`\n",
+ "- `model_architecture`: the model architecture we used: in our case PPO\n",
+ "- `env_id`: the name of the environment, in our case `LunarLander-v2`\n",
+ "- `eval_env`: the evaluation environment defined in eval_env\n",
+ "- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `(repo_id = {username}/{repo_name})`\n",
+ "\n",
+ "๐ก **A good name is {username}/{model_architecture}-{env_id}**\n",
+ "\n",
+ "- `commit_message`: message of the commit"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import gym\n",
+ "from stable_baselines3.common.vec_env import DummyVecEnv\n",
+ "from stable_baselines3.common.env_util import make_vec_env\n",
+ "\n",
+ "from huggingface_sb3 import package_to_hub\n",
+ "\n",
+ "# Create the evaluation env\n",
+ "eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
+ "\n",
+ "# TODO: Define the name of the environment\n",
+ "env_id = \n",
+ "\n",
+ "# TODO: Define the model architecture we used\n",
+ "model_architecture = \"\"\n",
+ "\n",
+ "## TODO: Define a repo_id\n",
+ "## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ "repo_id = \n",
+ "\n",
+ "## TODO: Define the commit message\n",
+ "commit_message = \"\"\n",
+ "\n",
+ "# method save, evaluate, generate a model card and record a replay video of your agent before pushing the repo to the hub\n",
+ "package_to_hub(model=model, # Our trained model\n",
+ " model_name=model_name, # The name of our trained model \n",
+ " model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
+ " env_id=env_id, # Name of the environment\n",
+ " eval_env=eval_env, # Evaluation Environment\n",
+ " repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ " commit_message=commit_message)\n"
+ ],
+ "metadata": {
+ "id": "JPG7ofdGIHN8"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Avf6gufJBGMw"
+ },
+ "source": [
+ "#### Solution\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "xMkkkukIBQJM"
+ },
+ "outputs": [],
+ "source": [
+ "import gym\n",
+ "\n",
+ "from stable_baselines3 import PPO\n",
+ "from stable_baselines3.common.vec_env import DummyVecEnv\n",
+ "from stable_baselines3.common.env_util import make_vec_env\n",
+ "\n",
+ "from huggingface_sb3 import package_to_hub\n",
+ "\n",
+ "# PLACE the variables you've just defined two cells above\n",
+ "# Define the name of the environment\n",
+ "env_id = \"LunarLander-v2\"\n",
+ "\n",
+ "# Define the name of the trained model that we defined in model_save\n",
+ "model_name = \"ppo-LunarLander-v2\"\n",
+ "\n",
+ "# TODO: Define the model architecture we used\n",
+ "model_architecture = \"PPO\"\n",
+ "\n",
+ "## Define a repo_id\n",
+ "## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ "## CHANGE WITH YOUR REPO ID\n",
+ "repo_id = \"ThomasSimonini/TEST2ppo-LunarLander-v2\"\n",
+ "\n",
+ "## Define the commit message\n",
+ "commit_message = \"Upload PPO LunarLander-v2 trained agent\"\n",
+ "\n",
+ "# Create the evaluation env\n",
+ "eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
+ "\n",
+ "# PLACE the package_to_hub function you've just filled here\n",
+ "package_to_hub(model=model, # Our trained model\n",
+ " model_name=model_name, # The name of our trained model \n",
+ " model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
+ " env_id=env_id, # Name of the environment\n",
+ " eval_env=eval_env, # Evaluation Environment\n",
+ " repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
+ " commit_message=commit_message)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Congrats ๐ฅณ you've just trained and uploaded your first Deep Reinforcement Learning agent. The script above should have displayed a link to a model repository such as https://huggingface.co/osanseviero/test_sb3. When you go to this link, you can:\n",
+ "* see a video preview of your agent at the right. \n",
+ "* click \"Files and versions\" to see all the files in the repository.\n",
+ "* click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
+ "* a model card (`README.md` file) which gives a description of the model\n",
+ "\n",
+ "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
+ "\n",
+ "Compare the results of your LunarLander-v2 with your classmates using the leaderboard ๐ ๐ https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard"
+ ],
+ "metadata": {
+ "id": "pN0-W6kkKn44"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BQAwLnYFPk-s"
+ },
+ "source": [
+ "## Some additional challenges ๐\n",
+ "The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
+ "\n",
+ "In the [Leaderboard](https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard) you will find your agents. Can you get to the top?\n",
+ "\n",
+ "Here are some ideas to achieve so:\n",
+ "* Train more steps\n",
+ "* Try different hyperparameters of `PPO`. You can see them at https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#parameters. \n",
+ "* Check the [Stable-Baselines3 documentation](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) and try another models such as DQN.\n",
+ "* **Push your new trained model** on the Hub ๐ฅ\n",
+ "\n",
+ "**Compare the results of your LunarLander-v2 with your classmates** using the [leaderboard](https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard) ๐ ๐ https://huggingface.co/spaces/ThomasSimonini/Lunar-Lander-Leaderboard\n",
+ "\n",
+ "Is moon landing too boring to you? Try to **change the environment**, why not using CartPole-v1, MountainCar-v0 or CarRacing-v0? Check how they works [using the gym documentation](https://www.gymlibrary.ml/) and have fun ๐."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9lM95-dvmif8"
+ },
+ "source": [
+ "________________________________________________________________________\n",
+ "Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.**\n",
+ "\n",
+ "If youโre still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
+ "\n",
+ "Take time to really **grasp the material before continuing and try the additional challenges**. Itโs important to master these elements and having a solid foundations.\n",
+ "\n",
+ "Naturally, during the course, weโre going to use and deeper explain again these terms but **itโs better to have a good understanding of them now before diving into the next chapters.**\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "feR90OUSEXq9"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "### This is a course built with you ๐ท๐ฟโโ๏ธ\n",
+ "\n",
+ "We want to improve and update the course iteratively with your feedback. If you have some, please open an issue on the Github Repo: [https://github.com/huggingface/deep-rl-class/issues](https://github.com/huggingface/deep-rl-class/issues)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "BjLhT70TEZIn"
+ },
+ "source": [
+ "See you on Unit 2! ๐ฅ\n",
+ "## Keep learning, stay awesome!"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "collapsed_sections": [
+ "QAN7B0_HCVZC",
+ "1bQzQ-QcE3zo",
+ "BqPKw3jt_pG5",
+ "Avf6gufJBGMw"
+ ],
+ "name": "Unit 1: Train your first Deep Reinforcement Learning Agent ๐.ipynb",
+ "provenance": [],
+ "include_colab_link": true
+ },
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}