Updated Unit 2 + added notebook

notebooks/unit2/requirements-unit2.txt

@@ -0,0 +1,10 @@
+gym==0.24
+pygame
+numpy
+
+huggingface_hub
+pickle5
+pyyaml==6.0
+imageio
+imageio_ffmpeg
+pyglet==1.5.1

notebooks/unit2/unit2.ipynb

@@ -25,32 +25,16 @@
         "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit3/envs.gif\" alt=\"Environments\"/>"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "TODO: ADD TEXT LIVE INFO"
-      ],
-      "metadata": {
-        "id": "yaBKcncmYku4"
-      }
-    },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "TODO: ADD IF YOU HAVE QUESTIONS\n"
-      ],
-      "metadata": {
-        "id": "hz5KE5HjYlRh"
-      }
-    },
     {
       "cell_type": "markdown",
       "source": [
         "###🎮 Environments: \n",
+        "\n",
         "- [FrozenLake-v1](https://www.gymlibrary.dev/environments/toy_text/frozen_lake/)\n",
         "- [Taxi-v3](https://www.gymlibrary.dev/environments/toy_text/taxi/)\n",
         "\n",
         "###📚 RL-Library: \n",
+        "\n",
         "- Python and Numpy"
       ],
       "metadata": {
@@ -73,7 +57,9 @@
       },
       "source": [
         "## Objectives of this notebook 🏆\n",
+        "\n",
         "At the end of the notebook, you will:\n",
+        "\n",
         "- Be able to use **Gym**, the environment library.\n",
         "- Be able to code from scratch a Q-Learning agent.\n",
         "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n",
@@ -120,7 +106,7 @@
         "## Prerequisites 🏗️\n",
         "Before diving into the notebook, you need to:\n",
         "\n",
-        "🔲 📚 **Study Q-Learning by reading Unit 2**  🤗 ADD LINK "
+        "🔲 📚 **Study [Q-Learning by reading Unit 2](https://huggingface.co/deep-rl-course/unit2/introduction)**  🤗  "
       ]
     },
     {
@@ -139,6 +125,7 @@
       },
       "source": [
         "- The *Q-Learning* **is the RL algorithm that**  \n",
+        "\n",
         "  - Trains *Q-Function*, an **action-value function** that contains, as internal memory, a *Q-table* **that contains all the state-action pair values.**\n",
         "    \n",
         "  - Given a state and action, our Q-Function **will search into its Q-table the corresponding value.**\n",
@@ -194,15 +181,6 @@
         "id": "4gpxC1_kqUYe"
       }
     },
-    {
-      "cell_type": "markdown",
-      "source": [
-        "TODO CHANGE LINK OF THE REQUIREMENTS"
-      ],
-      "metadata": {
-        "id": "32e3NPYgH5ET"
-      }
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -211,7 +189,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install -r https://huggingface.co/spaces/ThomasSimonini/temp-space-requirements/raw/main/requirements/requirements-unit2.txt"
+        "!pip install -r https://github.com/huggingface/deep-rl-class/tree/main/notebooks/unit2/requirements-unit2.txt"
       ]
     },
     {
@@ -230,6 +208,27 @@
       "execution_count": null,
       "outputs": []
     },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "To make sure the new installed libraries are used, **sometimes it's required to restart the notebook runtime**. The next cell will force the **runtime to crash, so you'll need to connect again and run the code starting from here**. Thanks for this trick, **we will be able to run our virtual screen.**"
+      ],
+      "metadata": {
+        "id": "K6XC13pTfFiD"
+      }
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "import os\n",
+        "os.kill(os.getpid(), 9)"
+      ],
+      "metadata": {
+        "id": "3kuZbWAkfHdg"
+      },
+      "execution_count": null,
+      "outputs": []
+    },
     {
       "cell_type": "code",
       "source": [
@@ -317,11 +316,13 @@
         "We're going to train our Q-Learning agent **to navigate from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoid holes (H)**.\n",
         "\n",
         "We can have two sizes of environment:\n",
+        "\n",
         "- `map_name=\"4x4\"`: a 4x4 grid version\n",
         "- `map_name=\"8x8\"`: a 8x8 grid version\n",
         "\n",
         "\n",
         "The environment has two modes:\n",
+        "\n",
         "- `is_slippery=False`: The agent always move in the intended direction due to the non-slippery nature of the frozen lake.\n",
         "- `is_slippery=True`: The agent may not always move in the intended direction due to the slippery nature of the frozen lake (stochastic)."
       ]
@@ -931,6 +932,7 @@
       },
       "source": [
         "## Evaluate our Q-Learning agent 📈\n",
+        "\n",
         "- Normally you should have mean reward of 1.0\n",
         "- It's relatively easy since the state space is really small (16). What you can try to do is [to replace with the slippery version](https://www.gymlibrary.dev/environments/toy_text/frozen_lake/)."
       ]
@@ -955,6 +957,7 @@
       },
       "source": [
         "## Publish our trained model on the Hub 🔥\n",
+        "\n",
         "Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
         "\n",
         "Here's an example of a Model Card:\n",
@@ -1173,6 +1176,7 @@
       },
       "source": [
         "### .\n",
+        "\n",
         "By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
         "\n",
         "This way:\n",
@@ -1264,9 +1268,10 @@
       },
       "source": [
         "Let's fill the `package_to_hub` function:\n",
+        "\n",
         "- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `\n",
         "(repo_id = {username}/{repo_name})`\n",
-        "💡 **A good name is {username}/q-{env_id}**\n",
+        "💡 A good `repo_id` is `{username}/q-{env_id}`\n",
         "- `model`: our model dictionary containing the hyperparameters and the Qtable.\n",
         "- `env`: the environment.\n",
         "- `commit_message`: message of the commit"
@@ -1326,7 +1331,9 @@
         "\n",
         "---\n",
         "\n",
-        "In Taxi-v3 🚕, there are four designated locations in the grid world indicated by R(ed), G(reen), Y(ellow), and B(lue). When the episode starts, the taxi starts off at a random square and the passenger is at a random location. The taxi drives to the passenger’s location, picks up the passenger, drives to the passenger’s destination (another one of the four specified locations), and then drops off the passenger. Once the passenger is dropped off, the episode ends.\n",
+        "In `Taxi-v3` 🚕, there are four designated locations in the grid world indicated by R(ed), G(reen), Y(ellow), and B(lue). \n",
+        "\n",
+        "When the episode starts, **the taxi starts off at a random square** and the passenger is at a random location. The taxi drives to the passenger’s location, **picks up the passenger**, drives to the passenger’s destination (another one of the four specified locations), and then **drops off the passenger**. Once the passenger is dropped off, the episode ends.\n",
         "\n",
         "\n",
         "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit2/taxi.png\" alt=\"Taxi\">\n"
@@ -1383,6 +1390,7 @@
       },
       "source": [
         "The action space (the set of possible actions the agent can take) is discrete with **6 actions available 🎮**:\n",
+        "\n",
         "- 0: move south\n",
         "- 1: move north\n",
         "- 2: move east\n",
@@ -1391,6 +1399,7 @@
         "- 5: drop off passenger\n",
         "\n",
         "Reward function 💰:\n",
+        "\n",
         "- -1 per step unless other reward is triggered.\n",
         "- +20 delivering passenger.\n",
         "- -10 executing “pickup” and “drop-off” actions illegally."
@@ -1556,7 +1565,8 @@
         "\n",
         "What's amazing with Hugging Face Hub 🤗 is that you can easily load powerful models from the community.\n",
         "\n",
-        "Loading a saved model from the Hub is really easy.\n",
+        "Loading a saved model from the Hub is really easy:\n",
+        "\n",
         "1. You go https://huggingface.co/models?other=q-learning to see the list of all the q-learning saved models.\n",
         "2. You select one and copy its repo_id\n",
         "\n",
@@ -1671,9 +1681,10 @@
         "## Some additional challenges 🏆\n",
         "The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
         "\n",
-        "In the [Leaderboard](https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
+        "In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
         "\n",
         "Here are some ideas to achieve so:\n",
+        "\n",
         "* Train more steps\n",
         "* Try different hyperparameters by looking at what your classmates have done.\n",
         "* **Push your new trained model** on the Hub 🔥\n",
@@ -1711,8 +1722,8 @@
         "id": "BjLhT70TEZIn"
       },
       "source": [
-        "See you on [Unit 3](https://github.com/huggingface/deep-rl-class/tree/main/unit2#unit-2-introduction-to-q-learning)! 🔥\n",
-        "TODO CHANGE LINK\n",
+        "See you on Unit 3! 🔥\n",
+        "\n",
         "## Keep learning, stay awesome 🤗"
       ]
     }