mirror of
https://github.com/huggingface/deep-rl-class.git
synced 2026-04-01 01:30:56 +08:00
1205 lines
3.3 MiB
1205 lines
3.3 MiB
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"colab_type": "text",
|
||
"id": "view-in-github"
|
||
},
|
||
"source": [
|
||
"<a href=\"https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/unit1/unit1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "njb_ProuHiOe"
|
||
},
|
||
"source": [
|
||
"# Unit 1: Train your first Deep Reinforcement Learning Agent 🚀\n",
|
||
"\n",
|
||
"\n",
|
||
"In this notebook, you'll train your first lander agent to **land correctly on the Moon 🌕 share it with the community, and experiment with different configurations**\n",
|
||
"\n",
|
||
"❓ If you have questions, please post them on #study-group-unit1 discord channel 👉 https://discord.gg/aYka4Yhff9\n",
|
||
"\n",
|
||
"\n",
|
||
"🎮 Environment: [LunarLander-v2](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n",
|
||
"\n",
|
||
"📚 RL-Library: [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/)\n",
|
||
"\n",
|
||
"⬇️ Here is an example of what **you will achieve in just a couple of minutes.** ⬇️"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "PF46MwbZD00b"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"%%html\n",
|
||
"<video controls autoplay><source src=\"https://huggingface.co/ThomasSimonini/ppo-LunarLander-v2/resolve/main/replay.mp4\" type=\"video/mp4\"></video>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "4i6tjI2tHQ8j"
|
||
},
|
||
"source": [
|
||
"## Objectives of this notebook 🏆\n",
|
||
"At the end of the notebook, you will:\n",
|
||
"- Be able to use **Gym**, the environment library.\n",
|
||
"- Be able to use **Stable-Baselines3**, the deep reinforcement learning library.\n",
|
||
"- Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score 🔥.\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "ct8eLAabICE-"
|
||
},
|
||
"source": [
|
||
"## This notebook is from Deep Reinforcement Learning Class\n",
|
||
"\n",
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "6p5HnEefISCB"
|
||
},
|
||
"source": [
|
||
"In this free course, you will:\n",
|
||
"\n",
|
||
"- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
|
||
"- 🧑💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, and RLlib.\n",
|
||
"- 🤖 Train **agents in unique environments** \n",
|
||
"\n",
|
||
"And more check 📚 the syllabus 👉 https://github.com/huggingface/deep-rl-class\n",
|
||
"\n",
|
||
"The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/aYka4Yhff9"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Y-mo_6rXIjRi"
|
||
},
|
||
"source": [
|
||
"## Prerequisites 🏗️\n",
|
||
"Before diving into the notebook, you need to:\n",
|
||
"\n",
|
||
"🔲 📚 [Read the Unit 1 Readme](https://github.com/huggingface/deep-rl-class/blob/main/unit1/README.md) that contains all the information.\n",
|
||
"\n",
|
||
"🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by reading first 👉 https://huggingface.co/blog/deep-rl-intro\n",
|
||
"\n",
|
||
"🔲 📢 Sign up to [our Discord Server](https://discord.gg/aYka4Yhff9) and **introduce yourself to #introduce-yourself channel 🥳**\n",
|
||
"\n",
|
||
"🔲 🐕 Are you new to Discord? Check our **discord 101 to get the best practices** 👉 https://github.com/huggingface/deep-rl-class/blob/main/DISCORD.Md\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "aXDZCGHnJ-3h"
|
||
},
|
||
"source": [
|
||
"## A small recap of Deep Reinforcement Learning 📚\n",
|
||
"\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "xcQYx9ynaFMD"
|
||
},
|
||
"source": [
|
||
"Let's do a small recap on what we learned in the first Unit:\n",
|
||
"- Reinforcement Learning is a **computational approach to learning from action**. We build an agent that learns from the environment by **interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.\n",
|
||
"\n",
|
||
"- The goal of any RL agent is to **maximize its expected cumulative reward** (also called expected return) because RL is based on the _reward hypothesis_, which is that all goals can be described as the maximization of the expected cumulative reward.\n",
|
||
"\n",
|
||
"- The RL process is a **loop that outputs a sequence of state, action, reward, and next state**.\n",
|
||
"\n",
|
||
"- To calculate the expected cumulative reward (expected return), **we discount the rewards**: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long-term future reward.\n",
|
||
"\n",
|
||
"- To solve an RL problem, you want to **find an optimal policy**; the policy is the \"brain\" of your AI that will tell us what action to take given a state. The optimal one is the one that gives you the actions that max the expected return.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "8X2WN2X99BEz"
|
||
},
|
||
"source": [
|
||
"There are **two** ways to find your optimal policy:\n",
|
||
"- By **training your policy directly**: policy-based methods.\n",
|
||
"- By **training a value function** that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BPy6dfcm8_WT"
|
||
},
|
||
"source": [
|
||
"- Finally, we spoke about Deep RL because **we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name \"deep.\"**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "LzGDDOTSKaF8"
|
||
},
|
||
"source": [
|
||
"# Let's train a Deep Reinforcement Learning lander agent to land correctly on the Moon 🌕 and upload it to the Hub.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "YkcELpiYRC5m"
|
||
},
|
||
"source": [
|
||
"### Step 0: Set the GPU 💪\n",
|
||
"- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "38HBd3t1SHJ8"
|
||
},
|
||
"source": [
|
||
"- `Hardware Accelerator > GPU`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "sw0pSDDsR1kq"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "LW_lsjc8h_up"
|
||
},
|
||
"source": [
|
||
"During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
|
||
"\n",
|
||
"Hence the following cell will install virtual screen libraries and create and run a virtual screen 🖥\n",
|
||
"\n",
|
||
"If you have this error `FileNotFoundError: [Errno 2] No such file or directory: 'Xvfb': 'Xvfb'` please restart the colab."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "dlx9uWHuhqfh"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!sudo apt-get update\n",
|
||
"!apt install python-opengl\n",
|
||
"!apt install ffmpeg\n",
|
||
"!apt install xvfb\n",
|
||
"!pip3 install pyvirtualdisplay\n",
|
||
"\n",
|
||
"# Virtual display\n",
|
||
"from pyvirtualdisplay import Display\n",
|
||
"\n",
|
||
"virtual_display = Display(visible=0, size=(1400, 900))\n",
|
||
"virtual_display.start()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "jeDAH0h0EBiG"
|
||
},
|
||
"source": [
|
||
"### Step 1: Install dependencies 🔽\n",
|
||
"The first step is to install the dependencies, we’ll install multiple ones:\n",
|
||
"\n",
|
||
"- `gym[box2D]`: Contains the LunarLander-v2 environment 🌛\n",
|
||
"- `stable-baselines3[extra]`: The deep reinforcement learning library.\n",
|
||
"- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face 🤗 Hub.\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "9XaULfDZDvrC"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip install importlib-metadata==4.12.0 # To overcome an issue with importlib-metadata https://stackoverflow.com/questions/73929564/entrypoints-object-has-no-attribute-get-digital-ocean\n",
|
||
"!pip install gym[box2d]\n",
|
||
"!pip install stable-baselines3[extra]\n",
|
||
"!pip install huggingface_sb3\n",
|
||
"!pip install pyglet==1.5.1\n",
|
||
"!pip install ale-py==0.7.4 # To overcome an issue with gym (https://github.com/DLR-RM/stable-baselines3/issues/875)\n",
|
||
"\n",
|
||
"!pip install pickle5"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "wrgpVFqyENVf"
|
||
},
|
||
"source": [
|
||
"### Step 2: Import the packages 📦\n",
|
||
"\n",
|
||
"One additional library we import is huggingface_hub **to be able to upload and download trained models from the hub**.\n",
|
||
"\n",
|
||
"\n",
|
||
"The Hugging Face Hub 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.\n",
|
||
"\n",
|
||
"You can see here all the Deep reinforcement Learning models available 👉 https://huggingface.co/models?pipeline_tag=reinforcement-learning&sort=downloads\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "cygWLPGsEQ0m"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import gym\n",
|
||
"\n",
|
||
"from huggingface_sb3 import load_from_hub, package_to_hub, push_to_hub\n",
|
||
"from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
|
||
"\n",
|
||
"from stable_baselines3 import PPO\n",
|
||
"from stable_baselines3.common.evaluation import evaluate_policy\n",
|
||
"from stable_baselines3.common.env_util import make_vec_env"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "MRqRuRUl8CsB"
|
||
},
|
||
"source": [
|
||
"### Step 3: Understand what is Gym and how it works? 🤖\n",
|
||
"\n",
|
||
"🏋 The library containing our environment is called Gym.\n",
|
||
"**You'll use Gym a lot in Deep Reinforcement Learning.**\n",
|
||
"\n",
|
||
"The Gym library provides two things:\n",
|
||
"- An interface that allows you to **create RL environments**.\n",
|
||
"- A **collection of environments** (gym-control, atari, box2D...).\n",
|
||
"\n",
|
||
"Let's look at an example, but first let's remember what's the RL Loop.\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "VvCOlJp-_kw4"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "-TzNN0bQ_j-3"
|
||
},
|
||
"source": [
|
||
"At each step:\n",
|
||
"- Our Agent receives **state S0** from the **Environment** — we receive the first frame of our game (Environment).\n",
|
||
"- Based on that **state S0,** the Agent takes **action A0** — our Agent will move to the right.\n",
|
||
"- Environment to a **new** **state S1** — new frame.\n",
|
||
"- The environment gives some **reward R1** to the Agent — we’re not dead *(Positive Reward +1)*.\n",
|
||
"\n",
|
||
"\n",
|
||
"With Gym:\n",
|
||
"\n",
|
||
"1️⃣ We create our environment using `gym.make()`\n",
|
||
"\n",
|
||
"2️⃣ We reset the environment to its initial state with `observation = env.reset()`\n",
|
||
"\n",
|
||
"At each step:\n",
|
||
"\n",
|
||
"3️⃣ Get an action using our model (in our example we take a random action)\n",
|
||
"\n",
|
||
"4️⃣ Using `env.step(action)`, we perform this action in the environment and get\n",
|
||
"- `observation`: The new state (st+1)\n",
|
||
"- `reward`: The reward we get after executing the action\n",
|
||
"- `done`: Indicates if the episode terminated\n",
|
||
"- `info`: A dictionary that provides additional information (depends on the environment).\n",
|
||
"\n",
|
||
"If the episode is done:\n",
|
||
"- We reset the environment to its initial state with `observation = env.reset()`\n",
|
||
"\n",
|
||
"**Let's look at an example!** Make sure to read the code\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "w7vOFlpA_ONz"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import gym\n",
|
||
"\n",
|
||
"# First, we create our environment called LunarLander-v2\n",
|
||
"env = gym.make(\"LunarLander-v2\")\n",
|
||
"\n",
|
||
"# Then we reset this environment\n",
|
||
"observation = env.reset()\n",
|
||
"\n",
|
||
"for _ in range(20):\n",
|
||
" # Take a random action\n",
|
||
" action = env.action_space.sample()\n",
|
||
" print(\"Action taken:\", action)\n",
|
||
"\n",
|
||
" # Do this action in the environment and get\n",
|
||
" # next_state, reward, done and info\n",
|
||
" observation, reward, done, info = env.step(action)\n",
|
||
" \n",
|
||
" # If the game is done (in our case we land, crashed or timeout)\n",
|
||
" if done:\n",
|
||
" # Reset the environment\n",
|
||
" print(\"Environment is reset\")\n",
|
||
" observation = env.reset()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "XIrKGGSlENZB"
|
||
},
|
||
"source": [
|
||
"### Step 4: Create the LunarLander environment 🌛 and understand how it works\n",
|
||
"#### [The environment 🎮](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n",
|
||
"In this first tutorial, we’re going to train our agent, a [Lunar Lander](https://www.gymlibrary.dev/environments/box2d/lunar_lander/), **to land correctly on the moon**. To do that, the agent needs to learn **to adapt its speed and position(horizontal, vertical, and angular) to land correctly.**\n",
|
||
"\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"\n",
|
||
"💡 A good habit when you start to use an environment is to check its documentation \n",
|
||
"\n",
|
||
"👉 https://www.gymlibrary.dev/environments/box2d/lunar_lander/\n",
|
||
"\n",
|
||
"---\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "poLBgRocF9aT"
|
||
},
|
||
"source": [
|
||
"Let's see what the Environment looks like:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "ZNPG0g_UGCfh"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# We create our environment with gym.make(\"<name_of_the_environment>\")\n",
|
||
"env = gym.make(\"LunarLander-v2\")\n",
|
||
"env.reset()\n",
|
||
"print(\"_____OBSERVATION SPACE_____ \\n\")\n",
|
||
"print(\"Observation Space Shape\", env.observation_space.shape)\n",
|
||
"print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "2MXc15qFE0M9"
|
||
},
|
||
"source": [
|
||
"We see with `Observation Space Shape (8,)` that the observation is a vector of size 8, where each value contains different information about the lander:\n",
|
||
"- Horizontal pad coordinate (x)\n",
|
||
"- Vertical pad coordinate (y)\n",
|
||
"- Horizontal speed (x)\n",
|
||
"- Vertical speed (y)\n",
|
||
"- Angle\n",
|
||
"- Angular speed\n",
|
||
"- If the left leg has contact point touched the land\n",
|
||
"- If the right leg has contact point touched the land\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "We5WqOBGLoSm"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(\"\\n _____ACTION SPACE_____ \\n\")\n",
|
||
"print(\"Action Space Shape\", env.action_space.n)\n",
|
||
"print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "MyxXwkI2Magx"
|
||
},
|
||
"source": [
|
||
"The action space (the set of possible actions the agent can take) is discrete with 4 actions available 🎮: \n",
|
||
"\n",
|
||
"- Do nothing,\n",
|
||
"- Fire left orientation engine,\n",
|
||
"- Fire the main engine,\n",
|
||
"- Fire right orientation engine.\n",
|
||
"\n",
|
||
"Reward function (the function that will gives a reward at each timestep) 💰:\n",
|
||
"\n",
|
||
"- Moving from the top of the screen to the landing pad and zero speed is about 100~140 points.\n",
|
||
"- Firing main engine is -0.3 each frame\n",
|
||
"- Each leg ground contact is +10 points\n",
|
||
"- Episode finishes if the lander crashes (additional - 100 points) or come to rest (+100 points)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "dFD9RAFjG8aq"
|
||
},
|
||
"source": [
|
||
"#### Vectorized Environment\n",
|
||
"- We create a vectorized environment (method for stacking multiple independent environments into a single environment) of 16 environments, this way, **we'll have more diverse experiences during the training.**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "99hqQ_etEy1N"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create the environment\n",
|
||
"env = make_vec_env('LunarLander-v2', n_envs=16)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "VgrE86r5E5IK"
|
||
},
|
||
"source": [
|
||
"### Step 5: Create the Model 🤖\n",
|
||
"- Now that we studied our environment and we understood the problem: **being able to land correctly the Lunar Lander to the Landing Pad by controlling left, right and main orientation engine**. Let's build the algorithm we're going to use to solve this Problem 🚀.\n",
|
||
"\n",
|
||
"- To do so, we're going to use our first Deep RL library, [Stable Baselines3 (SB3)](https://stable-baselines3.readthedocs.io/en/master/).\n",
|
||
"\n",
|
||
"- SB3 is a set of **reliable implementations of reinforcement learning algorithms in PyTorch**.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"💡 A good habit when using a new library is to dive first on the documentation: https://stable-baselines3.readthedocs.io/en/master/ and then try some tutorials.\n",
|
||
"\n",
|
||
"----"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "3wgg-y0-92Wj"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "HV4yiUM_9_Ka"
|
||
},
|
||
"source": [
|
||
"To solve this problem, we're going to use SB3 **PPO**. [PPO (aka Proximal Policy Optimization) is one of the of the SOTA (state of the art) Deep Reinforcement Learning algorithms that you'll study during this course](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#example%5D).\n",
|
||
"\n",
|
||
"PPO is a combination of:\n",
|
||
"- *Value-based reinforcement learning method*: learning an action-value function that will tell us what's the **most valuable action to take given a state and action**.\n",
|
||
"- *Policy-based reinforcement learning method*: learning a policy that will **gives us a probability distribution over actions**.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "5qL_4HeIOrEJ"
|
||
},
|
||
"source": [
|
||
"Stable-Baselines3 is easy to set up:\n",
|
||
"\n",
|
||
"1️⃣ You **create your environment** (in our case it was done above)\n",
|
||
"\n",
|
||
"2️⃣ You define the **model you want to use and instantiate this model** `model = PPO(\"MlpPolicy\")`\n",
|
||
"\n",
|
||
"3️⃣ You **train the agent** with `model.learn` and define the number of training timesteps\n",
|
||
"\n",
|
||
"```\n",
|
||
"# Create environment\n",
|
||
"env = gym.make('LunarLander-v2')\n",
|
||
"\n",
|
||
"# Instantiate the agent\n",
|
||
"model = PPO('MlpPolicy', env, verbose=1)\n",
|
||
"# Train the agent\n",
|
||
"model.learn(total_timesteps=int(2e5))\n",
|
||
"```\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "nxI6hT1GE4-A"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# TODO: Define a PPO MlpPolicy architecture\n",
|
||
"# We use MultiLayerPerceptron (MLPPolicy) because the input is a vector,\n",
|
||
"# if we had frames as input we would use CnnPolicy\n",
|
||
"model = "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "QAN7B0_HCVZC"
|
||
},
|
||
"source": [
|
||
"#### Solution"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "543OHYDfcjK4"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# SOLUTION\n",
|
||
"# We added some parameters to accelerate the training\n",
|
||
"model = PPO(\n",
|
||
" policy = 'MlpPolicy',\n",
|
||
" env = env,\n",
|
||
" n_steps = 1024,\n",
|
||
" batch_size = 64,\n",
|
||
" n_epochs = 4,\n",
|
||
" gamma = 0.999,\n",
|
||
" gae_lambda = 0.98,\n",
|
||
" ent_coef = 0.01,\n",
|
||
" verbose=1)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "ClJJk88yoBUi"
|
||
},
|
||
"source": [
|
||
"### Step 6: Train the PPO agent 🏃\n",
|
||
"- Let's train our agent for 500,000 timesteps, don't forget to use GPU on Colab. It will take approximately ~10min, but you can use less timesteps if you just want to try it out.\n",
|
||
"- During the training, take a ☕ break you deserved it 🤗"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "qKnYkNiVp89p"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# TODO: Train it for 500,000 timesteps\n",
|
||
"\n",
|
||
"# TODO: Specify file name for model and save the model to file\n",
|
||
"model_name = \"\"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "1bQzQ-QcE3zo"
|
||
},
|
||
"source": [
|
||
"#### Solution"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "poBCy9u_csyR"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# SOLUTION\n",
|
||
"# Train it for 500,000 timesteps\n",
|
||
"model.learn(total_timesteps=500000)\n",
|
||
"# Save the model\n",
|
||
"model_name = \"ppo-LunarLander-v2\"\n",
|
||
"model.save(model_name)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BY_HuedOoISR"
|
||
},
|
||
"source": [
|
||
"### Step 7: Evaluate the agent 📈\n",
|
||
"- Now that our Lunar Lander agent is trained 🚀, we need to **check its performance**.\n",
|
||
"- Stable-Baselines3 provides a method to do that: `evaluate_policy`.\n",
|
||
"- To fill that part you need to [check the documentation](https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#basic-usage-training-saving-loading)\n",
|
||
"- In the next step, we'll see **how to automatically evaluate and share your agent to compete in a leaderboard, but for now let's do it ourselves**\n",
|
||
"\n",
|
||
"\n",
|
||
"💡 When you evaluate your agent, you should not use your training environment but create an evaluation environment."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "yRpno0glsADy"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# TODO: Evaluate the agent\n",
|
||
"# Create a new environment for evaluation\n",
|
||
"eval_env =\n",
|
||
"\n",
|
||
"# Evaluate the model with 10 evaluation episodes and deterministic=True\n",
|
||
"mean_reward, std_reward = \n",
|
||
"\n",
|
||
"# Print the results\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BqPKw3jt_pG5"
|
||
},
|
||
"source": [
|
||
"#### Solution"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "zpz8kHlt_a_m"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"#@title\n",
|
||
"eval_env = gym.make(\"LunarLander-v2\")\n",
|
||
"mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)\n",
|
||
"print(f\"mean_reward={mean_reward:.2f} +/- {std_reward}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "reBhoODwcXfr"
|
||
},
|
||
"source": [
|
||
"- In my case, I got a mean reward is `200.20 +/- 20.80` after training for 1 million steps, which means that our lunar lander agent is ready to land on the moon 🌛🥳."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "IK_kR78NoNb2"
|
||
},
|
||
"source": [
|
||
"### Step 8: Publish our trained model on the Hub 🔥\n",
|
||
"Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
|
||
"\n",
|
||
"📚 The libraries documentation 👉 https://github.com/huggingface/huggingface_sb3/tree/main#hugging-face--x-stable-baselines3-v20\n",
|
||
"\n",
|
||
"Here's an example of a Model Card (with Space Invaders):"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Nhded6x9gapA"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Gs-Ew7e1gXN3"
|
||
},
|
||
"source": [
|
||
"By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
|
||
"\n",
|
||
"---\n",
|
||
"The package_to_hub function\n",
|
||
"---\n",
|
||
"\n",
|
||
"This way:\n",
|
||
"- You can **showcase our work** 🔥\n",
|
||
"- You can **visualize your agent playing** 👀\n",
|
||
"- You can **share with the community an agent that others can use** 💾\n",
|
||
"- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "JquRrWytA6eo"
|
||
},
|
||
"source": [
|
||
"To be able to share your model with the community there are three more steps to follow:\n",
|
||
"\n",
|
||
"1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
|
||
"\n",
|
||
"2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
|
||
"- Create a new token (https://huggingface.co/settings/tokens) **with write role**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "RVJYd-YZRXRZ"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "YWxcm2xiRSgA"
|
||
},
|
||
"source": [
|
||
"- Copy the token \n",
|
||
"- Run the cell below and paste the token"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "GZiFBBlzxzxY"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"notebook_login()\n",
|
||
"!git config --global credential.helper store"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "_tsf2uv0g_4p"
|
||
},
|
||
"source": [
|
||
"If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "FGNh9VsZok0i"
|
||
},
|
||
"source": [
|
||
"3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using `package_to_hub()` function"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Ay24l6bqFF18"
|
||
},
|
||
"source": [
|
||
"Let's fill the `package_to_hub` function:\n",
|
||
"- `model`: our trained model.\n",
|
||
"- `model_name`: the name of the trained model that we defined in `model_save`\n",
|
||
"- `model_architecture`: the model architecture we used: in our case PPO\n",
|
||
"- `env_id`: the name of the environment, in our case `LunarLander-v2`\n",
|
||
"- `eval_env`: the evaluation environment defined in eval_env\n",
|
||
"- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `(repo_id = {username}/{repo_name})`\n",
|
||
"\n",
|
||
"💡 **A good name is {username}/{model_architecture}-{env_id}**\n",
|
||
"\n",
|
||
"- `commit_message`: message of the commit"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "JPG7ofdGIHN8"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import gym\n",
|
||
"from stable_baselines3.common.vec_env import DummyVecEnv\n",
|
||
"from stable_baselines3.common.env_util import make_vec_env\n",
|
||
"\n",
|
||
"from huggingface_sb3 import package_to_hub\n",
|
||
"\n",
|
||
"## TODO: Define a repo_id\n",
|
||
"## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
|
||
"repo_id = \n",
|
||
"\n",
|
||
"# TODO: Define the name of the environment\n",
|
||
"env_id = \n",
|
||
"\n",
|
||
"# Create the evaluation env\n",
|
||
"eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
|
||
"\n",
|
||
"\n",
|
||
"# TODO: Define the model architecture we used\n",
|
||
"model_architecture = \"\"\n",
|
||
"\n",
|
||
"## TODO: Define the commit message\n",
|
||
"commit_message = \"\"\n",
|
||
"\n",
|
||
"# method save, evaluate, generate a model card and record a replay video of your agent before pushing the repo to the hub\n",
|
||
"package_to_hub(model=model, # Our trained model\n",
|
||
" model_name=model_name, # The name of our trained model \n",
|
||
" model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
|
||
" env_id=env_id, # Name of the environment\n",
|
||
" eval_env=eval_env, # Evaluation Environment\n",
|
||
" repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
|
||
" commit_message=commit_message)\n",
|
||
"\n",
|
||
"# Note: if after running the package_to_hub function and it gives an issue of rebasing, please run the following code\n",
|
||
"# cd <path_to_repo> && git add . && git commit -m \"Add message\" && git pull \n",
|
||
"# And don't forget to do a \"git push\" at the end to push the change to the hub."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Avf6gufJBGMw"
|
||
},
|
||
"source": [
|
||
"#### Solution\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "xMkkkukIBQJM"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import gym\n",
|
||
"\n",
|
||
"from stable_baselines3 import PPO\n",
|
||
"from stable_baselines3.common.vec_env import DummyVecEnv\n",
|
||
"from stable_baselines3.common.env_util import make_vec_env\n",
|
||
"\n",
|
||
"from huggingface_sb3 import package_to_hub\n",
|
||
"\n",
|
||
"# PLACE the variables you've just defined two cells above\n",
|
||
"# Define the name of the environment\n",
|
||
"env_id = \"LunarLander-v2\"\n",
|
||
"\n",
|
||
"# TODO: Define the model architecture we used\n",
|
||
"model_architecture = \"PPO\"\n",
|
||
"\n",
|
||
"## Define a repo_id\n",
|
||
"## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
|
||
"## CHANGE WITH YOUR REPO ID\n",
|
||
"repo_id = \"ThomasSimonini/ppo-LunarLander-v2\"\n",
|
||
"\n",
|
||
"## Define the commit message\n",
|
||
"commit_message = \"Upload PPO LunarLander-v2 trained agent\"\n",
|
||
"\n",
|
||
"# Create the evaluation env\n",
|
||
"eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
|
||
"\n",
|
||
"# PLACE the package_to_hub function you've just filled here\n",
|
||
"package_to_hub(model=model, # Our trained model\n",
|
||
" model_name=model_name, # The name of our trained model \n",
|
||
" model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
|
||
" env_id=env_id, # Name of the environment\n",
|
||
" eval_env=eval_env, # Evaluation Environment\n",
|
||
" repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
|
||
" commit_message=commit_message)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "pN0-W6kkKn44"
|
||
},
|
||
"source": [
|
||
"Congrats 🥳 you've just trained and uploaded your first Deep Reinforcement Learning agent. The script above should have displayed a link to a model repository such as https://huggingface.co/osanseviero/test_sb3. When you go to this link, you can:\n",
|
||
"* see a video preview of your agent at the right. \n",
|
||
"* click \"Files and versions\" to see all the files in the repository.\n",
|
||
"* click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
|
||
"* a model card (`README.md` file) which gives a description of the model\n",
|
||
"\n",
|
||
"Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
|
||
"\n",
|
||
"Compare the results of your LunarLander-v2 with your classmates using the leaderboard 🏆 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "9nWnuQHRfFRa"
|
||
},
|
||
"source": [
|
||
"### Step 9: Load a saved LunarLander model from the Hub 🤗\n",
|
||
"Thanks to [ironbar](https://github.com/ironbar) for the contribution.\n",
|
||
"\n",
|
||
"Loading a saved model from the Hub is really easy. \n",
|
||
"\n",
|
||
"You go https://huggingface.co/models?library=stable-baselines3 to see the list of all the Stable-baselines3 saved models.\n",
|
||
"1. You select one and copy its repo_id"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "qDwKbhJ0ffuG"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "hNPLJF2bfiUw"
|
||
},
|
||
"source": [
|
||
"2. Then we just need to use load_from_hub with:\n",
|
||
"- The repo_id\n",
|
||
"- The filename: the saved model inside the repo and its extension (*.zip)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "oj8PSGHJfwz3"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from huggingface_sb3 import load_from_hub\n",
|
||
"repo_id = \"\" # The repo_id\n",
|
||
"filename = \"\" # The model filename.zip\n",
|
||
"\n",
|
||
"# When the model was trained on Python 3.8 the pickle protocol is 5\n",
|
||
"# But Python 3.6, 3.7 use protocol 4\n",
|
||
"# In order to get compatibility we need to:\n",
|
||
"# 1. Install pickle5 (we done it at the beginning of the colab)\n",
|
||
"# 2. Create a custom empty object we pass as parameter to PPO.load()\n",
|
||
"custom_objects = {\n",
|
||
" \"learning_rate\": 0.0,\n",
|
||
" \"lr_schedule\": lambda _: 0.0,\n",
|
||
" \"clip_range\": lambda _: 0.0,\n",
|
||
"}\n",
|
||
"\n",
|
||
"checkpoint = load_from_hub(repo_id, filename)\n",
|
||
"model = PPO.load(checkpoint, custom_objects=custom_objects, print_system_info=True)\n",
|
||
"\n",
|
||
"# Evaluate this model\n",
|
||
"eval_env = gym.make(\"LunarLander-v2\")\n",
|
||
"mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)\n",
|
||
"print(f\"mean_reward={mean_reward:.2f} +/- {std_reward}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Fs0Y-qgPgLUf"
|
||
},
|
||
"source": [
|
||
"Let's watch our agent performing 🎥 (Google Colab only) 👀\n",
|
||
"We're going to use [colabgymrender package by Ryan Rudes](https://github.com/ryanrudes) that records our agent performing in the environment and outputs a video."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "stUaYyj8gKXE"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip install gym pyvirtualdisplay > /dev/null 2>&1\n",
|
||
"!apt-get install -y xvfb python-opengl ffmpeg > /dev/null 2>&1\n",
|
||
"!pip install -U colabgymrender"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "XAq_Fosen9Z7"
|
||
},
|
||
"source": [
|
||
"If you have a runtime bug `RuntimeError: imageio.ffmpeg.download() has been deprecated. Use 'pip install imageio-ffmpeg' instead.'` here please install this package and click on restart button after the installation"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "3wmUjb5LoCiK"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip install imageio==2.4.1"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "ickoCCH8gW1-"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from colabgymrender.recorder import Recorder\n",
|
||
"\n",
|
||
"directory = './video'\n",
|
||
"env = Recorder(eval_env, directory)\n",
|
||
"\n",
|
||
"obs = env.reset()\n",
|
||
"done = False\n",
|
||
"while not done:\n",
|
||
" action, _state = model.predict(obs)\n",
|
||
" obs, reward, done, info = env.step(action)\n",
|
||
"env.play()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BQAwLnYFPk-s"
|
||
},
|
||
"source": [
|
||
"## Some additional challenges 🏆\n",
|
||
"The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
|
||
"\n",
|
||
"In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
|
||
"\n",
|
||
"Here are some ideas to achieve so:\n",
|
||
"* Train more steps\n",
|
||
"* Try different hyperparameters of `PPO`. You can see them at https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#parameters. \n",
|
||
"* Check the [Stable-Baselines3 documentation](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) and try another models such as DQN.\n",
|
||
"* **Push your new trained model** on the Hub 🔥\n",
|
||
"\n",
|
||
"**Compare the results of your LunarLander-v2 with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) 🏆\n",
|
||
"\n",
|
||
"Is moon landing too boring to you? Try to **change the environment**, why not using CartPole-v1, MountainCar-v0 or CarRacing-v0? Check how they works [using the gym documentation](https://www.gymlibrary.dev/) and have fun 🎉."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "9lM95-dvmif8"
|
||
},
|
||
"source": [
|
||
"________________________________________________________________________\n",
|
||
"Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.**\n",
|
||
"\n",
|
||
"If you’re still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
|
||
"\n",
|
||
"Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations.\n",
|
||
"\n",
|
||
"Naturally, during the course, we’re going to use and deeper explain again these terms but **it’s better to have a good understanding of them now before diving into the next chapters.**\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "feR90OUSEXq9"
|
||
},
|
||
"source": [
|
||
"\n",
|
||
"\n",
|
||
"### This is a course built with you 👷🏿♀️\n",
|
||
"\n",
|
||
"We want to improve and update the course iteratively with your feedback. If you have some, please fill this form 👉 https://forms.gle/3HgA7bEHwAmmLfwh9\n",
|
||
"\n",
|
||
"If you found some issues in this notebook, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues).\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BjLhT70TEZIn"
|
||
},
|
||
"source": [
|
||
"See you on [Unit 2](https://github.com/huggingface/deep-rl-class/tree/main/unit2#unit-2-introduction-to-q-learning)! 🔥\n",
|
||
"## Keep learning, stay awesome 🤗"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"accelerator": "GPU",
|
||
"colab": {
|
||
"collapsed_sections": [],
|
||
"include_colab_link": true,
|
||
"name": "Copie de Unit 1: Train your first Deep Reinforcement Learning Agent 🚀.ipynb",
|
||
"private_outputs": true,
|
||
"provenance": []
|
||
},
|
||
"gpuClass": "standard",
|
||
"kernelspec": {
|
||
"display_name": "Python 3.9.7",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"name": "python",
|
||
"version": "3.9.7"
|
||
},
|
||
"vscode": {
|
||
"interpreter": {
|
||
"hash": "ed7f8024e43d3b8f5ca3c5e1a8151ab4d136b3ecee1e3fd59e0766ccc55e1b10"
|
||
}
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 0
|
||
}
|