diff --git a/README.md b/README.md index ee3f53c..d9cd483 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # [The Hugging Face Deep Reinforcement Learning Course 🤗 (v2.0)](https://huggingface.co/deep-rl-course/unit0/introduction) -This repository contains the Deep Reinforcement Learning Course mdx files and notebooks. The website is here: https://huggingface.co/deep-rl-course/unit0/introduction?fw=pt +If you like the course, don't hesitate to **⭐ star this repository. This helps us 🤗**. + +This repository contains the Deep Reinforcement Learning Course mdx files and notebooks. **The website is here**: https://huggingface.co/deep-rl-course/unit0/introduction?fw=pt - The syllabus 📚: https://simoninithomas.github.io/deep-rl-course diff --git a/notebooks/unit1/unit1.ipynb b/notebooks/unit1/unit1.ipynb index ec814b0..0e7b4f1 100644 --- a/notebooks/unit1/unit1.ipynb +++ b/notebooks/unit1/unit1.ipynb @@ -1099,7 +1099,7 @@ "\n", "Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations.\n", "\n", - "Naturally, during the course, we’re going to use and deeper explain again these terms but **it’s better to have a good understanding of them now before diving into the next chapters.**\n" + "Naturally, during the course, we’re going to dive deeper into these concepts but **it’s better to have a good understanding of them now before diving into the next chapters.**\n\n" ] }, { diff --git a/notebooks/unit2/unit2.ipynb b/notebooks/unit2/unit2.ipynb index 90baea8..13a86f0 100644 --- a/notebooks/unit2/unit2.ipynb +++ b/notebooks/unit2/unit2.ipynb @@ -511,7 +511,7 @@ }, "outputs": [], "source": [ - "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros\n", + "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros. np.zeros needs a tuple (a,b)\n", "def initialize_q_table(state_space, action_space):\n", " Qtable = \n", " return Qtable" diff --git a/notebooks/unit3/unit3.ipynb b/notebooks/unit3/unit3.ipynb index e776208..5c21dca 100644 --- a/notebooks/unit3/unit3.ipynb +++ b/notebooks/unit3/unit3.ipynb @@ -301,7 +301,7 @@ "## Train our Deep Q-Learning Agent to Play Space Invaders 👾\n", "\n", "To train an agent with RL-Baselines3-Zoo, we just need to do two things:\n", - "1. We define the hyperparameters in `rl-baselines3-zoo/hyperparams/dqn.yml`\n", + "1. We define the hyperparameters in `/content/rl-baselines3-zoo/hyperparams/dqn.yml`\n", "\n", "\"DQN\n" ] diff --git a/notebooks/unit6/unit6.ipynb b/notebooks/unit6/unit6.ipynb index 0ca054e..28f8a5c 100644 --- a/notebooks/unit6/unit6.ipynb +++ b/notebooks/unit6/unit6.ipynb @@ -431,7 +431,7 @@ "source": [ "env = make_vec_env(env_id, n_envs=4)\n", "\n", - "env = VecNormalize(env, norm_obs=True, norm_reward=False, clip_obs=10.)" + "env = VecNormalize(env, norm_obs=True, norm_reward=True, clip_obs=10.)" ], "metadata": { "id": "2O67mqgC-hol" diff --git a/notebooks/unit8/unit8_part1.ipynb b/notebooks/unit8/unit8_part1.ipynb new file mode 100644 index 0000000..a1862c8 --- /dev/null +++ b/notebooks/unit8/unit8_part1.ipynb @@ -0,0 +1,1357 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "view-in-github", + "colab_type": "text" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-cf5-oDPjwf8" + }, + "source": [ + "# Unit 8: Proximal Policy Gradient (PPO) with PyTorch 🤖\n", + "\n", + "\"Unit\n", + "\n", + "\n", + "In this notebook, you'll learn to **code your PPO agent from scratch with PyTorch using CleanRL implementation as model**.\n", + "\n", + "To test its robustness, we're going to train it in:\n", + "\n", + "- [LunarLander-v2 🚀](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2Fl6Rxt0lc0O" + }, + "source": [ + "⬇️ Here is an example of what you will achieve. ⬇️" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DbKfCj5ilgqT" + }, + "outputs": [], + "source": [ + "%%html\n", + "" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YcOFdWpnlxNf" + }, + "source": [ + "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)." + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Objectives of this notebook 🏆\n", + "\n", + "At the end of the notebook, you will:\n", + "\n", + "- Be able to **code your PPO agent from scratch using PyTorch**.\n", + "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n", + "\n", + "\n" + ], + "metadata": { + "id": "T6lIPYFghhYL" + } + }, + { + "cell_type": "markdown", + "source": [ + "## This notebook is from the Deep Reinforcement Learning Course\n", + "\"Deep\n", + "\n", + "In this free course, you will:\n", + "\n", + "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n", + "- 🧑‍💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n", + "- 🤖 Train **agents in unique environments** \n", + "\n", + "Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n", + "\n", + "\n", + "The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5" + ], + "metadata": { + "id": "Wp-rD6Fuhq31" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Prerequisites 🏗️\n", + "Before diving into the notebook, you need to:\n", + "\n", + "🔲 📚 Study [PPO by reading Unit 8](https://huggingface.co/deep-rl-course/unit8/introduction) 🤗 " + ], + "metadata": { + "id": "rasqqGQlhujA" + } + }, + { + "cell_type": "markdown", + "source": [ + "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model, we don't ask for a minimal result but we **advise you to try different hyperparameters settings to get better results**.\n", + "\n", + "If you don't find your model, **go to the bottom of the page and click on the refresh button**\n", + "\n", + "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" + ], + "metadata": { + "id": "PUFfMGOih3CW" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Set the GPU 💪\n", + "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n", + "\n", + "\"GPU" + ], + "metadata": { + "id": "PU4FVzaoM6fC" + } + }, + { + "cell_type": "markdown", + "source": [ + "- `Hardware Accelerator > GPU`\n", + "\n", + "\"GPU" + ], + "metadata": { + "id": "KV0NyFdQM9ZG" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Create a virtual display 🔽\n", + "\n", + "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n", + "\n", + "Hence the following cell will install the librairies and create and run a virtual screen 🖥" + ], + "metadata": { + "id": "bTpYcVZVMzUI" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "jV6wjQ7Be7p5" + }, + "outputs": [], + "source": [ + "%%capture\n", + "!apt install python-opengl\n", + "!apt install ffmpeg\n", + "!apt install xvfb\n", + "!pip install pyglet==1.5\n", + "!pip3 install pyvirtualdisplay" + ] + }, + { + "cell_type": "code", + "source": [ + "# Virtual display\n", + "from pyvirtualdisplay import Display\n", + "\n", + "virtual_display = Display(visible=0, size=(1400, 900))\n", + "virtual_display.start()" + ], + "metadata": { + "id": "ww5PQH1gNLI4" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ncIgfNf3mOtc" + }, + "source": [ + "## Install dependencies 🔽\n", + "For this exercise, we use `gym==0.21`\n" + ] + }, + { + "cell_type": "code", + "source": [ + "!pip install gym==0.21\n", + "!pip install imageio-ffmpeg\n", + "!pip install huggingface_hub\n", + "!pip install box2d" + ], + "metadata": { + "id": "9xZQFTPcsKUK" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "oDkUufewmq6v" + }, + "source": [ + "## Let's code PPO from scratch with Costa Huang tutorial\n", + "- For the core implementation of PPO we're going to use the excellent [Costa Huang](https://costa.sh/) tutorial.\n", + "- In addition to the tutorial, to go deeper you can read the 37 core implementation details: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/\n", + "\n", + "👉 The video tutorial: https://youtu.be/MEt6rrxH8W4" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "aNgEL1_uvhaq" + }, + "outputs": [], + "source": [ + "from IPython.display import HTML\n", + "\n", + "HTML('')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f34ILn7AvTbt" + }, + "source": [ + "- The best is to code first on the cell below, this way, if you kill the machine **you don't loose the implementation**." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_bE708C6mhE7" + }, + "outputs": [], + "source": [ + "### Your code here:" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mk-a9CmNuS2W" + }, + "source": [ + "## Add the Hugging Face Integration 🤗\n", + "- In order to push our model to the Hub, we need to define a function `package_to_hub`" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TPi1Nme-oGWd" + }, + "source": [ + "- Add dependencies we need to push our model to the Hub" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Sj8bz-AmoNVj" + }, + "outputs": [], + "source": [ + "from huggingface_hub import HfApi, upload_folder\n", + "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n", + "\n", + "from pathlib import Path\n", + "import datetime\n", + "import tempfile\n", + "import json\n", + "import shutil\n", + "import imageio\n", + "\n", + "from wasabi import Printer\n", + "msg = Printer()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5rDr8-lWn0zi" + }, + "source": [ + "- Add new argument in `parse_args()` function to define the repo-id where we want to push the model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "iHQiqQEFn0QH" + }, + "outputs": [], + "source": [ + "# Adding HuggingFace argument\n", + "parser.add_argument(\"--repo-id\", type=str, default=\"ThomasSimonini/ppo-CartPole-v1\", help=\"id of the model repository from the Hugging Face Hub {username/repo_name}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "blLZMiBAoUVT" + }, + "source": [ + "- Next, we add the methods needed to push the model to the Hub\n", + "\n", + "- These methods will:\n", + " - `_evalutate_agent()`: evaluate the agent.\n", + " - `_generate_model_card()`: generate the model card of your agent.\n", + " - `_record_video()`: record a video of your agent." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "WlLcz4L9odXs" + }, + "outputs": [], + "source": [ + "def package_to_hub(repo_id, \n", + " model,\n", + " hyperparameters,\n", + " eval_env,\n", + " video_fps=30,\n", + " commit_message=\"Push agent to the Hub\",\n", + " token= None,\n", + " logs=None\n", + " ):\n", + " \"\"\"\n", + " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n", + " This method does the complete pipeline:\n", + " - It evaluates the model\n", + " - It generates the model card\n", + " - It generates a replay video of the agent\n", + " - It pushes everything to the hub\n", + " :param repo_id: id of the model repository from the Hugging Face Hub\n", + " :param model: trained model\n", + " :param eval_env: environment used to evaluate the agent\n", + " :param fps: number of fps for rendering the video\n", + " :param commit_message: commit message\n", + " :param logs: directory on local machine of tensorboard logs you'd like to upload\n", + " \"\"\"\n", + " msg.info(\n", + " \"This function will save, evaluate, generate a video of your agent, \"\n", + " \"create a model card and push everything to the hub. \"\n", + " \"It might take up to 1min. \\n \"\n", + " \"This is a work in progress: if you encounter a bug, please open an issue.\"\n", + " )\n", + " # Step 1: Clone or create the repo\n", + " repo_url = HfApi().create_repo(\n", + " repo_id=repo_id,\n", + " token=token,\n", + " private=False,\n", + " exist_ok=True,\n", + " )\n", + " \n", + " with tempfile.TemporaryDirectory() as tmpdirname:\n", + " tmpdirname = Path(tmpdirname)\n", + "\n", + " # Step 2: Save the model\n", + " torch.save(model.state_dict(), tmpdirname / \"model.pt\")\n", + " \n", + " # Step 3: Evaluate the model and build JSON\n", + " mean_reward, std_reward = _evaluate_agent(eval_env, \n", + " 10, \n", + " model)\n", + "\n", + " # First get datetime\n", + " eval_datetime = datetime.datetime.now()\n", + " eval_form_datetime = eval_datetime.isoformat()\n", + "\n", + " evaluate_data = {\n", + " \"env_id\": hyperparameters.env_id, \n", + " \"mean_reward\": mean_reward,\n", + " \"std_reward\": std_reward,\n", + " \"n_evaluation_episodes\": 10,\n", + " \"eval_datetime\": eval_form_datetime,\n", + " }\n", + " \n", + " # Write a JSON file\n", + " with open(tmpdirname / \"results.json\", \"w\") as outfile:\n", + " json.dump(evaluate_data, outfile)\n", + "\n", + " # Step 4: Generate a video\n", + " video_path = tmpdirname / \"replay.mp4\"\n", + " record_video(eval_env, model, video_path, video_fps)\n", + " \n", + " # Step 5: Generate the model card\n", + " generated_model_card, metadata = _generate_model_card(\"PPO\", hyperparameters.env_id, mean_reward, std_reward, hyperparameters)\n", + " _save_model_card(tmpdirname, generated_model_card, metadata)\n", + "\n", + " # Step 6: Add logs if needed\n", + " if logs:\n", + " _add_logdir(tmpdirname, Path(logs))\n", + " \n", + " msg.info(f\"Pushing repo {repo_id} to the Hugging Face Hub\")\n", + " \n", + " repo_url = upload_folder(\n", + " repo_id=repo_id,\n", + " folder_path=tmpdirname,\n", + " path_in_repo=\"\",\n", + " commit_message=commit_message,\n", + " token=token,\n", + " )\n", + "\n", + " msg.info(f\"Your model is pushed to the Hub. You can view your model here: {repo_url}\")\n", + " return repo_url\n", + "\n", + "\n", + "def _evaluate_agent(env, n_eval_episodes, policy):\n", + " \"\"\"\n", + " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n", + " :param env: The evaluation environment\n", + " :param n_eval_episodes: Number of episode to evaluate the agent\n", + " :param policy: The agent\n", + " \"\"\"\n", + " episode_rewards = []\n", + " for episode in range(n_eval_episodes):\n", + " state = env.reset()\n", + " step = 0\n", + " done = False\n", + " total_rewards_ep = 0\n", + " \n", + " while done is False:\n", + " state = torch.Tensor(state).to(device)\n", + " action, _, _, _ = policy.get_action_and_value(state)\n", + " new_state, reward, done, info = env.step(action.cpu().numpy())\n", + " total_rewards_ep += reward \n", + " if done:\n", + " break\n", + " state = new_state\n", + " episode_rewards.append(total_rewards_ep)\n", + " mean_reward = np.mean(episode_rewards)\n", + " std_reward = np.std(episode_rewards)\n", + "\n", + " return mean_reward, std_reward\n", + "\n", + "\n", + "def record_video(env, policy, out_directory, fps=30):\n", + " images = [] \n", + " done = False\n", + " state = env.reset()\n", + " img = env.render(mode='rgb_array')\n", + " images.append(img)\n", + " while not done:\n", + " state = torch.Tensor(state).to(device)\n", + " # Take the action (index) that have the maximum expected future reward given that state\n", + " action, _, _, _ = policy.get_action_and_value(state)\n", + " state, reward, done, info = env.step(action.cpu().numpy()) # We directly put next_state = state for recording logic\n", + " img = env.render(mode='rgb_array')\n", + " images.append(img)\n", + " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)\n", + "\n", + "\n", + "def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters):\n", + " \"\"\"\n", + " Generate the model card for the Hub\n", + " :param model_name: name of the model\n", + " :env_id: name of the environment\n", + " :mean_reward: mean reward of the agent\n", + " :std_reward: standard deviation of the mean reward of the agent\n", + " :hyperparameters: training arguments\n", + " \"\"\"\n", + " # Step 1: Select the tags\n", + " metadata = generate_metadata(model_name, env_id, mean_reward, std_reward)\n", + "\n", + " # Transform the hyperparams namespace to string\n", + " converted_dict = vars(hyperparameters)\n", + " converted_str = str(converted_dict)\n", + " converted_str = converted_str.split(\", \")\n", + " converted_str = '\\n'.join(converted_str)\n", + " \n", + " # Step 2: Generate the model card\n", + " model_card = f\"\"\"\n", + " # PPO Agent Playing {env_id}\n", + "\n", + " This is a trained model of a PPO agent playing {env_id}.\n", + " \n", + " # Hyperparameters\n", + " ```python\n", + " {converted_str}\n", + " ```\n", + " \"\"\"\n", + " return model_card, metadata\n", + "\n", + "\n", + "def generate_metadata(model_name, env_id, mean_reward, std_reward):\n", + " \"\"\"\n", + " Define the tags for the model card\n", + " :param model_name: name of the model\n", + " :param env_id: name of the environment\n", + " :mean_reward: mean reward of the agent\n", + " :std_reward: standard deviation of the mean reward of the agent\n", + " \"\"\"\n", + " metadata = {}\n", + " metadata[\"tags\"] = [\n", + " env_id,\n", + " \"ppo\",\n", + " \"deep-reinforcement-learning\",\n", + " \"reinforcement-learning\",\n", + " \"custom-implementation\",\n", + " \"deep-rl-course\"\n", + " ]\n", + "\n", + " # Add metrics\n", + " eval = metadata_eval_result(\n", + " model_pretty_name=model_name,\n", + " task_pretty_name=\"reinforcement-learning\",\n", + " task_id=\"reinforcement-learning\",\n", + " metrics_pretty_name=\"mean_reward\",\n", + " metrics_id=\"mean_reward\",\n", + " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n", + " dataset_pretty_name=env_id,\n", + " dataset_id=env_id,\n", + " )\n", + "\n", + " # Merges both dictionaries\n", + " metadata = {**metadata, **eval}\n", + "\n", + " return metadata\n", + "\n", + "\n", + "def _save_model_card(local_path, generated_model_card, metadata):\n", + " \"\"\"Saves a model card for the repository.\n", + " :param local_path: repository directory\n", + " :param generated_model_card: model card generated by _generate_model_card()\n", + " :param metadata: metadata\n", + " \"\"\"\n", + " readme_path = local_path / \"README.md\"\n", + " readme = \"\"\n", + " if readme_path.exists():\n", + " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n", + " readme = f.read()\n", + " else:\n", + " readme = generated_model_card\n", + "\n", + " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n", + " f.write(readme)\n", + "\n", + " # Save our metrics to Readme metadata\n", + " metadata_save(readme_path, metadata)\n", + "\n", + "\n", + "def _add_logdir(local_path: Path, logdir: Path):\n", + " \"\"\"Adds a logdir to the repository.\n", + " :param local_path: repository directory\n", + " :param logdir: logdir directory\n", + " \"\"\"\n", + " if logdir.exists() and logdir.is_dir():\n", + " # Add the logdir to the repository under new dir called logs\n", + " repo_logdir = local_path / \"logs\"\n", + " \n", + " # Delete current logs if they exist\n", + " if repo_logdir.exists():\n", + " shutil.rmtree(repo_logdir)\n", + "\n", + " # Copy logdir into repo logdir\n", + " shutil.copytree(logdir, repo_logdir)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TqX8z8_rooD6" + }, + "source": [ + "- Finally, we call this function at the end of the PPO training" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "I8V1vNiTo2hL" + }, + "outputs": [], + "source": [ + "# Create the evaluation environment\n", + "eval_env = gym.make(args.env_id)\n", + "\n", + "package_to_hub(repo_id = args.repo_id,\n", + " model = agent, # The model we want to save\n", + " hyperparameters = args,\n", + " eval_env = gym.make(args.env_id),\n", + " logs= f\"runs/{run_name}\",\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "muCCzed4o5TC" + }, + "source": [ + "- Here's what look the ppo.py final file" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "LviRdtXgo7kF" + }, + "outputs": [], + "source": [ + "# docs and experiment results can be found at https://docs.cleanrl.dev/rl-algorithms/ppo/#ppopy\n", + "\n", + "import argparse\n", + "import os\n", + "import random\n", + "import time\n", + "from distutils.util import strtobool\n", + "\n", + "import gym\n", + "import numpy as np\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.optim as optim\n", + "from torch.distributions.categorical import Categorical\n", + "from torch.utils.tensorboard import SummaryWriter\n", + "\n", + "from huggingface_hub import HfApi, upload_folder\n", + "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n", + "\n", + "from pathlib import Path\n", + "import datetime\n", + "import tempfile\n", + "import json\n", + "import shutil\n", + "import imageio\n", + "\n", + "from wasabi import Printer\n", + "msg = Printer()\n", + "\n", + "def parse_args():\n", + " # fmt: off\n", + " parser = argparse.ArgumentParser()\n", + " parser.add_argument(\"--exp-name\", type=str, default=os.path.basename(__file__).rstrip(\".py\"),\n", + " help=\"the name of this experiment\")\n", + " parser.add_argument(\"--seed\", type=int, default=1,\n", + " help=\"seed of the experiment\")\n", + " parser.add_argument(\"--torch-deterministic\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", + " help=\"if toggled, `torch.backends.cudnn.deterministic=False`\")\n", + " parser.add_argument(\"--cuda\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", + " help=\"if toggled, cuda will be enabled by default\")\n", + " parser.add_argument(\"--track\", type=lambda x: bool(strtobool(x)), default=False, nargs=\"?\", const=True,\n", + " help=\"if toggled, this experiment will be tracked with Weights and Biases\")\n", + " parser.add_argument(\"--wandb-project-name\", type=str, default=\"cleanRL\",\n", + " help=\"the wandb's project name\")\n", + " parser.add_argument(\"--wandb-entity\", type=str, default=None,\n", + " help=\"the entity (team) of wandb's project\")\n", + " parser.add_argument(\"--capture-video\", type=lambda x: bool(strtobool(x)), default=False, nargs=\"?\", const=True,\n", + " help=\"weather to capture videos of the agent performances (check out `videos` folder)\")\n", + "\n", + " # Algorithm specific arguments\n", + " parser.add_argument(\"--env-id\", type=str, default=\"CartPole-v1\",\n", + " help=\"the id of the environment\")\n", + " parser.add_argument(\"--total-timesteps\", type=int, default=50000,\n", + " help=\"total timesteps of the experiments\")\n", + " parser.add_argument(\"--learning-rate\", type=float, default=2.5e-4,\n", + " help=\"the learning rate of the optimizer\")\n", + " parser.add_argument(\"--num-envs\", type=int, default=4,\n", + " help=\"the number of parallel game environments\")\n", + " parser.add_argument(\"--num-steps\", type=int, default=128,\n", + " help=\"the number of steps to run in each environment per policy rollout\")\n", + " parser.add_argument(\"--anneal-lr\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", + " help=\"Toggle learning rate annealing for policy and value networks\")\n", + " parser.add_argument(\"--gae\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", + " help=\"Use GAE for advantage computation\")\n", + " parser.add_argument(\"--gamma\", type=float, default=0.99,\n", + " help=\"the discount factor gamma\")\n", + " parser.add_argument(\"--gae-lambda\", type=float, default=0.95,\n", + " help=\"the lambda for the general advantage estimation\")\n", + " parser.add_argument(\"--num-minibatches\", type=int, default=4,\n", + " help=\"the number of mini-batches\")\n", + " parser.add_argument(\"--update-epochs\", type=int, default=4,\n", + " help=\"the K epochs to update the policy\")\n", + " parser.add_argument(\"--norm-adv\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", + " help=\"Toggles advantages normalization\")\n", + " parser.add_argument(\"--clip-coef\", type=float, default=0.2,\n", + " help=\"the surrogate clipping coefficient\")\n", + " parser.add_argument(\"--clip-vloss\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", + " help=\"Toggles whether or not to use a clipped loss for the value function, as per the paper.\")\n", + " parser.add_argument(\"--ent-coef\", type=float, default=0.01,\n", + " help=\"coefficient of the entropy\")\n", + " parser.add_argument(\"--vf-coef\", type=float, default=0.5,\n", + " help=\"coefficient of the value function\")\n", + " parser.add_argument(\"--max-grad-norm\", type=float, default=0.5,\n", + " help=\"the maximum norm for the gradient clipping\")\n", + " parser.add_argument(\"--target-kl\", type=float, default=None,\n", + " help=\"the target KL divergence threshold\")\n", + " \n", + " # Adding HuggingFace argument\n", + " parser.add_argument(\"--repo-id\", type=str, default=\"ThomasSimonini/ppo-CartPole-v1\", help=\"id of the model repository from the Hugging Face Hub {username/repo_name}\")\n", + "\n", + " args = parser.parse_args()\n", + " args.batch_size = int(args.num_envs * args.num_steps)\n", + " args.minibatch_size = int(args.batch_size // args.num_minibatches)\n", + " # fmt: on\n", + " return args\n", + "\n", + "def package_to_hub(repo_id, \n", + " model,\n", + " hyperparameters,\n", + " eval_env,\n", + " video_fps=30,\n", + " commit_message=\"Push agent to the Hub\",\n", + " token= None,\n", + " logs=None\n", + " ):\n", + " \"\"\"\n", + " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n", + " This method does the complete pipeline:\n", + " - It evaluates the model\n", + " - It generates the model card\n", + " - It generates a replay video of the agent\n", + " - It pushes everything to the hub\n", + " :param repo_id: id of the model repository from the Hugging Face Hub\n", + " :param model: trained model\n", + " :param eval_env: environment used to evaluate the agent\n", + " :param fps: number of fps for rendering the video\n", + " :param commit_message: commit message\n", + " :param logs: directory on local machine of tensorboard logs you'd like to upload\n", + " \"\"\"\n", + " msg.info(\n", + " \"This function will save, evaluate, generate a video of your agent, \"\n", + " \"create a model card and push everything to the hub. \"\n", + " \"It might take up to 1min. \\n \"\n", + " \"This is a work in progress: if you encounter a bug, please open an issue.\"\n", + " )\n", + " # Step 1: Clone or create the repo\n", + " repo_url = HfApi().create_repo(\n", + " repo_id=repo_id,\n", + " token=token,\n", + " private=False,\n", + " exist_ok=True,\n", + " )\n", + " \n", + " with tempfile.TemporaryDirectory() as tmpdirname:\n", + " tmpdirname = Path(tmpdirname)\n", + "\n", + " # Step 2: Save the model\n", + " torch.save(model.state_dict(), tmpdirname / \"model.pt\")\n", + " \n", + " # Step 3: Evaluate the model and build JSON\n", + " mean_reward, std_reward = _evaluate_agent(eval_env, \n", + " 10, \n", + " model)\n", + "\n", + " # First get datetime\n", + " eval_datetime = datetime.datetime.now()\n", + " eval_form_datetime = eval_datetime.isoformat()\n", + "\n", + " evaluate_data = {\n", + " \"env_id\": hyperparameters.env_id, \n", + " \"mean_reward\": mean_reward,\n", + " \"std_reward\": std_reward,\n", + " \"n_evaluation_episodes\": 10,\n", + " \"eval_datetime\": eval_form_datetime,\n", + " }\n", + " \n", + " # Write a JSON file\n", + " with open(tmpdirname / \"results.json\", \"w\") as outfile:\n", + " json.dump(evaluate_data, outfile)\n", + "\n", + " # Step 4: Generate a video\n", + " video_path = tmpdirname / \"replay.mp4\"\n", + " record_video(eval_env, model, video_path, video_fps)\n", + " \n", + " # Step 5: Generate the model card\n", + " generated_model_card, metadata = _generate_model_card(\"PPO\", hyperparameters.env_id, mean_reward, std_reward, hyperparameters)\n", + " _save_model_card(tmpdirname, generated_model_card, metadata)\n", + "\n", + " # Step 6: Add logs if needed\n", + " if logs:\n", + " _add_logdir(tmpdirname, Path(logs))\n", + " \n", + " msg.info(f\"Pushing repo {repo_id} to the Hugging Face Hub\")\n", + " \n", + " repo_url = upload_folder(\n", + " repo_id=repo_id,\n", + " folder_path=tmpdirname,\n", + " path_in_repo=\"\",\n", + " commit_message=commit_message,\n", + " token=token,\n", + " )\n", + "\n", + " msg.info(f\"Your model is pushed to the Hub. You can view your model here: {repo_url}\")\n", + " return repo_url\n", + "\n", + "def _evaluate_agent(env, n_eval_episodes, policy):\n", + " \"\"\"\n", + " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n", + " :param env: The evaluation environment\n", + " :param n_eval_episodes: Number of episode to evaluate the agent\n", + " :param policy: The agent\n", + " \"\"\"\n", + " episode_rewards = []\n", + " for episode in range(n_eval_episodes):\n", + " state = env.reset()\n", + " step = 0\n", + " done = False\n", + " total_rewards_ep = 0\n", + " \n", + " while done is False:\n", + " state = torch.Tensor(state).to(device)\n", + " action, _, _, _ = policy.get_action_and_value(state)\n", + " new_state, reward, done, info = env.step(action.cpu().numpy())\n", + " total_rewards_ep += reward \n", + " if done:\n", + " break\n", + " state = new_state\n", + " episode_rewards.append(total_rewards_ep)\n", + " mean_reward = np.mean(episode_rewards)\n", + " std_reward = np.std(episode_rewards)\n", + "\n", + " return mean_reward, std_reward\n", + "\n", + "\n", + "def record_video(env, policy, out_directory, fps=30):\n", + " images = [] \n", + " done = False\n", + " state = env.reset()\n", + " img = env.render(mode='rgb_array')\n", + " images.append(img)\n", + " while not done:\n", + " state = torch.Tensor(state).to(device)\n", + " # Take the action (index) that have the maximum expected future reward given that state\n", + " action, _, _, _ = policy.get_action_and_value(state)\n", + " state, reward, done, info = env.step(action.cpu().numpy()) # We directly put next_state = state for recording logic\n", + " img = env.render(mode='rgb_array')\n", + " images.append(img)\n", + " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)\n", + "\n", + "\n", + "def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters):\n", + " \"\"\"\n", + " Generate the model card for the Hub\n", + " :param model_name: name of the model\n", + " :env_id: name of the environment\n", + " :mean_reward: mean reward of the agent\n", + " :std_reward: standard deviation of the mean reward of the agent\n", + " :hyperparameters: training arguments\n", + " \"\"\"\n", + " # Step 1: Select the tags\n", + " metadata = generate_metadata(model_name, env_id, mean_reward, std_reward)\n", + "\n", + " # Transform the hyperparams namespace to string\n", + " converted_dict = vars(hyperparameters)\n", + " converted_str = str(converted_dict)\n", + " converted_str = converted_str.split(\", \")\n", + " converted_str = '\\n'.join(converted_str)\n", + " \n", + " # Step 2: Generate the model card\n", + " model_card = f\"\"\"\n", + " # PPO Agent Playing {env_id}\n", + "\n", + " This is a trained model of a PPO agent playing {env_id}.\n", + " \n", + " # Hyperparameters\n", + " ```python\n", + " {converted_str}\n", + " ```\n", + " \"\"\"\n", + " return model_card, metadata\n", + "\n", + "def generate_metadata(model_name, env_id, mean_reward, std_reward):\n", + " \"\"\"\n", + " Define the tags for the model card\n", + " :param model_name: name of the model\n", + " :param env_id: name of the environment\n", + " :mean_reward: mean reward of the agent\n", + " :std_reward: standard deviation of the mean reward of the agent\n", + " \"\"\"\n", + " metadata = {}\n", + " metadata[\"tags\"] = [\n", + " env_id,\n", + " \"ppo\",\n", + " \"deep-reinforcement-learning\",\n", + " \"reinforcement-learning\",\n", + " \"custom-implementation\",\n", + " \"deep-rl-course\"\n", + " ]\n", + "\n", + " # Add metrics\n", + " eval = metadata_eval_result(\n", + " model_pretty_name=model_name,\n", + " task_pretty_name=\"reinforcement-learning\",\n", + " task_id=\"reinforcement-learning\",\n", + " metrics_pretty_name=\"mean_reward\",\n", + " metrics_id=\"mean_reward\",\n", + " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n", + " dataset_pretty_name=env_id,\n", + " dataset_id=env_id,\n", + " )\n", + "\n", + " # Merges both dictionaries\n", + " metadata = {**metadata, **eval}\n", + "\n", + " return metadata\n", + "\n", + "def _save_model_card(local_path, generated_model_card, metadata):\n", + " \"\"\"Saves a model card for the repository.\n", + " :param local_path: repository directory\n", + " :param generated_model_card: model card generated by _generate_model_card()\n", + " :param metadata: metadata\n", + " \"\"\"\n", + " readme_path = local_path / \"README.md\"\n", + " readme = \"\"\n", + " if readme_path.exists():\n", + " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n", + " readme = f.read()\n", + " else:\n", + " readme = generated_model_card\n", + "\n", + " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n", + " f.write(readme)\n", + "\n", + " # Save our metrics to Readme metadata\n", + " metadata_save(readme_path, metadata)\n", + "\n", + "def _add_logdir(local_path: Path, logdir: Path):\n", + " \"\"\"Adds a logdir to the repository.\n", + " :param local_path: repository directory\n", + " :param logdir: logdir directory\n", + " \"\"\"\n", + " if logdir.exists() and logdir.is_dir():\n", + " # Add the logdir to the repository under new dir called logs\n", + " repo_logdir = local_path / \"logs\"\n", + " \n", + " # Delete current logs if they exist\n", + " if repo_logdir.exists():\n", + " shutil.rmtree(repo_logdir)\n", + "\n", + " # Copy logdir into repo logdir\n", + " shutil.copytree(logdir, repo_logdir)\n", + "\n", + "def make_env(env_id, seed, idx, capture_video, run_name):\n", + " def thunk():\n", + " env = gym.make(env_id)\n", + " env = gym.wrappers.RecordEpisodeStatistics(env)\n", + " if capture_video:\n", + " if idx == 0:\n", + " env = gym.wrappers.RecordVideo(env, f\"videos/{run_name}\")\n", + " env.seed(seed)\n", + " env.action_space.seed(seed)\n", + " env.observation_space.seed(seed)\n", + " return env\n", + "\n", + " return thunk\n", + "\n", + "\n", + "def layer_init(layer, std=np.sqrt(2), bias_const=0.0):\n", + " torch.nn.init.orthogonal_(layer.weight, std)\n", + " torch.nn.init.constant_(layer.bias, bias_const)\n", + " return layer\n", + "\n", + "\n", + "class Agent(nn.Module):\n", + " def __init__(self, envs):\n", + " super().__init__()\n", + " self.critic = nn.Sequential(\n", + " layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),\n", + " nn.Tanh(),\n", + " layer_init(nn.Linear(64, 64)),\n", + " nn.Tanh(),\n", + " layer_init(nn.Linear(64, 1), std=1.0),\n", + " )\n", + " self.actor = nn.Sequential(\n", + " layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),\n", + " nn.Tanh(),\n", + " layer_init(nn.Linear(64, 64)),\n", + " nn.Tanh(),\n", + " layer_init(nn.Linear(64, envs.single_action_space.n), std=0.01),\n", + " )\n", + "\n", + " def get_value(self, x):\n", + " return self.critic(x)\n", + "\n", + " def get_action_and_value(self, x, action=None):\n", + " logits = self.actor(x)\n", + " probs = Categorical(logits=logits)\n", + " if action is None:\n", + " action = probs.sample()\n", + " return action, probs.log_prob(action), probs.entropy(), self.critic(x)\n", + "\n", + "\n", + "if __name__ == \"__main__\":\n", + " args = parse_args()\n", + " run_name = f\"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}\"\n", + " if args.track:\n", + " import wandb\n", + "\n", + " wandb.init(\n", + " project=args.wandb_project_name,\n", + " entity=args.wandb_entity,\n", + " sync_tensorboard=True,\n", + " config=vars(args),\n", + " name=run_name,\n", + " monitor_gym=True,\n", + " save_code=True,\n", + " )\n", + " writer = SummaryWriter(f\"runs/{run_name}\")\n", + " writer.add_text(\n", + " \"hyperparameters\",\n", + " \"|param|value|\\n|-|-|\\n%s\" % (\"\\n\".join([f\"|{key}|{value}|\" for key, value in vars(args).items()])),\n", + " )\n", + "\n", + " # TRY NOT TO MODIFY: seeding\n", + " random.seed(args.seed)\n", + " np.random.seed(args.seed)\n", + " torch.manual_seed(args.seed)\n", + " torch.backends.cudnn.deterministic = args.torch_deterministic\n", + "\n", + " device = torch.device(\"cuda\" if torch.cuda.is_available() and args.cuda else \"cpu\")\n", + "\n", + " # env setup\n", + " envs = gym.vector.SyncVectorEnv(\n", + " [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)]\n", + " )\n", + " assert isinstance(envs.single_action_space, gym.spaces.Discrete), \"only discrete action space is supported\"\n", + "\n", + " agent = Agent(envs).to(device)\n", + " optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5)\n", + "\n", + " # ALGO Logic: Storage setup\n", + " obs = torch.zeros((args.num_steps, args.num_envs) + envs.single_observation_space.shape).to(device)\n", + " actions = torch.zeros((args.num_steps, args.num_envs) + envs.single_action_space.shape).to(device)\n", + " logprobs = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", + " rewards = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", + " dones = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", + " values = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", + "\n", + " # TRY NOT TO MODIFY: start the game\n", + " global_step = 0\n", + " start_time = time.time()\n", + " next_obs = torch.Tensor(envs.reset()).to(device)\n", + " next_done = torch.zeros(args.num_envs).to(device)\n", + " num_updates = args.total_timesteps // args.batch_size\n", + "\n", + " for update in range(1, num_updates + 1):\n", + " # Annealing the rate if instructed to do so.\n", + " if args.anneal_lr:\n", + " frac = 1.0 - (update - 1.0) / num_updates\n", + " lrnow = frac * args.learning_rate\n", + " optimizer.param_groups[0][\"lr\"] = lrnow\n", + "\n", + " for step in range(0, args.num_steps):\n", + " global_step += 1 * args.num_envs\n", + " obs[step] = next_obs\n", + " dones[step] = next_done\n", + "\n", + " # ALGO LOGIC: action logic\n", + " with torch.no_grad():\n", + " action, logprob, _, value = agent.get_action_and_value(next_obs)\n", + " values[step] = value.flatten()\n", + " actions[step] = action\n", + " logprobs[step] = logprob\n", + "\n", + " # TRY NOT TO MODIFY: execute the game and log data.\n", + " next_obs, reward, done, info = envs.step(action.cpu().numpy())\n", + " rewards[step] = torch.tensor(reward).to(device).view(-1)\n", + " next_obs, next_done = torch.Tensor(next_obs).to(device), torch.Tensor(done).to(device)\n", + "\n", + " for item in info:\n", + " if \"episode\" in item.keys():\n", + " print(f\"global_step={global_step}, episodic_return={item['episode']['r']}\")\n", + " writer.add_scalar(\"charts/episodic_return\", item[\"episode\"][\"r\"], global_step)\n", + " writer.add_scalar(\"charts/episodic_length\", item[\"episode\"][\"l\"], global_step)\n", + " break\n", + "\n", + " # bootstrap value if not done\n", + " with torch.no_grad():\n", + " next_value = agent.get_value(next_obs).reshape(1, -1)\n", + " if args.gae:\n", + " advantages = torch.zeros_like(rewards).to(device)\n", + " lastgaelam = 0\n", + " for t in reversed(range(args.num_steps)):\n", + " if t == args.num_steps - 1:\n", + " nextnonterminal = 1.0 - next_done\n", + " nextvalues = next_value\n", + " else:\n", + " nextnonterminal = 1.0 - dones[t + 1]\n", + " nextvalues = values[t + 1]\n", + " delta = rewards[t] + args.gamma * nextvalues * nextnonterminal - values[t]\n", + " advantages[t] = lastgaelam = delta + args.gamma * args.gae_lambda * nextnonterminal * lastgaelam\n", + " returns = advantages + values\n", + " else:\n", + " returns = torch.zeros_like(rewards).to(device)\n", + " for t in reversed(range(args.num_steps)):\n", + " if t == args.num_steps - 1:\n", + " nextnonterminal = 1.0 - next_done\n", + " next_return = next_value\n", + " else:\n", + " nextnonterminal = 1.0 - dones[t + 1]\n", + " next_return = returns[t + 1]\n", + " returns[t] = rewards[t] + args.gamma * nextnonterminal * next_return\n", + " advantages = returns - values\n", + "\n", + " # flatten the batch\n", + " b_obs = obs.reshape((-1,) + envs.single_observation_space.shape)\n", + " b_logprobs = logprobs.reshape(-1)\n", + " b_actions = actions.reshape((-1,) + envs.single_action_space.shape)\n", + " b_advantages = advantages.reshape(-1)\n", + " b_returns = returns.reshape(-1)\n", + " b_values = values.reshape(-1)\n", + "\n", + " # Optimizing the policy and value network\n", + " b_inds = np.arange(args.batch_size)\n", + " clipfracs = []\n", + " for epoch in range(args.update_epochs):\n", + " np.random.shuffle(b_inds)\n", + " for start in range(0, args.batch_size, args.minibatch_size):\n", + " end = start + args.minibatch_size\n", + " mb_inds = b_inds[start:end]\n", + "\n", + " _, newlogprob, entropy, newvalue = agent.get_action_and_value(b_obs[mb_inds], b_actions.long()[mb_inds])\n", + " logratio = newlogprob - b_logprobs[mb_inds]\n", + " ratio = logratio.exp()\n", + "\n", + " with torch.no_grad():\n", + " # calculate approx_kl http://joschu.net/blog/kl-approx.html\n", + " old_approx_kl = (-logratio).mean()\n", + " approx_kl = ((ratio - 1) - logratio).mean()\n", + " clipfracs += [((ratio - 1.0).abs() > args.clip_coef).float().mean().item()]\n", + "\n", + " mb_advantages = b_advantages[mb_inds]\n", + " if args.norm_adv:\n", + " mb_advantages = (mb_advantages - mb_advantages.mean()) / (mb_advantages.std() + 1e-8)\n", + "\n", + " # Policy loss\n", + " pg_loss1 = -mb_advantages * ratio\n", + " pg_loss2 = -mb_advantages * torch.clamp(ratio, 1 - args.clip_coef, 1 + args.clip_coef)\n", + " pg_loss = torch.max(pg_loss1, pg_loss2).mean()\n", + "\n", + " # Value loss\n", + " newvalue = newvalue.view(-1)\n", + " if args.clip_vloss:\n", + " v_loss_unclipped = (newvalue - b_returns[mb_inds]) ** 2\n", + " v_clipped = b_values[mb_inds] + torch.clamp(\n", + " newvalue - b_values[mb_inds],\n", + " -args.clip_coef,\n", + " args.clip_coef,\n", + " )\n", + " v_loss_clipped = (v_clipped - b_returns[mb_inds]) ** 2\n", + " v_loss_max = torch.max(v_loss_unclipped, v_loss_clipped)\n", + " v_loss = 0.5 * v_loss_max.mean()\n", + " else:\n", + " v_loss = 0.5 * ((newvalue - b_returns[mb_inds]) ** 2).mean()\n", + "\n", + " entropy_loss = entropy.mean()\n", + " loss = pg_loss - args.ent_coef * entropy_loss + v_loss * args.vf_coef\n", + "\n", + " optimizer.zero_grad()\n", + " loss.backward()\n", + " nn.utils.clip_grad_norm_(agent.parameters(), args.max_grad_norm)\n", + " optimizer.step()\n", + "\n", + " if args.target_kl is not None:\n", + " if approx_kl > args.target_kl:\n", + " break\n", + "\n", + " y_pred, y_true = b_values.cpu().numpy(), b_returns.cpu().numpy()\n", + " var_y = np.var(y_true)\n", + " explained_var = np.nan if var_y == 0 else 1 - np.var(y_true - y_pred) / var_y\n", + "\n", + " # TRY NOT TO MODIFY: record rewards for plotting purposes\n", + " writer.add_scalar(\"charts/learning_rate\", optimizer.param_groups[0][\"lr\"], global_step)\n", + " writer.add_scalar(\"losses/value_loss\", v_loss.item(), global_step)\n", + " writer.add_scalar(\"losses/policy_loss\", pg_loss.item(), global_step)\n", + " writer.add_scalar(\"losses/entropy\", entropy_loss.item(), global_step)\n", + " writer.add_scalar(\"losses/old_approx_kl\", old_approx_kl.item(), global_step)\n", + " writer.add_scalar(\"losses/approx_kl\", approx_kl.item(), global_step)\n", + " writer.add_scalar(\"losses/clipfrac\", np.mean(clipfracs), global_step)\n", + " writer.add_scalar(\"losses/explained_variance\", explained_var, global_step)\n", + " print(\"SPS:\", int(global_step / (time.time() - start_time)))\n", + " writer.add_scalar(\"charts/SPS\", int(global_step / (time.time() - start_time)), global_step)\n", + "\n", + " envs.close()\n", + " writer.close()\n", + "\n", + " # Create the evaluation environment\n", + " eval_env = gym.make(args.env_id)\n", + "\n", + " package_to_hub(repo_id = args.repo_id,\n", + " model = agent, # The model we want to save\n", + " hyperparameters = args,\n", + " eval_env = gym.make(args.env_id),\n", + " logs= f\"runs/{run_name}\",\n", + " )\n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JquRrWytA6eo" + }, + "source": [ + "To be able to share your model with the community there are three more steps to follow:\n", + "\n", + "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n", + "\n", + "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n", + "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n", + "\n", + "\"Create\n", + "\n", + "- Copy the token \n", + "- Run the cell below and paste the token" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "GZiFBBlzxzxY" + }, + "outputs": [], + "source": [ + "from huggingface_hub import notebook_login\n", + "notebook_login()\n", + "!git config --global credential.helper store" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_tsf2uv0g_4p" + }, + "source": [ + "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jRqkGvk7pFQ6" + }, + "source": [ + "## Let's start the training 🔥\n", + "- Now that you've coded from scratch PPO and added the Hugging Face Integration, we're ready to start the training 🔥" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0tmEArP8ug2l" + }, + "source": [ + "- First, you need to copy all your code to a file you create called `ppo.py`" + ] + }, + { + "cell_type": "markdown", + "source": [ + "\"PPO\"/" + ], + "metadata": { + "id": "Sq0My0LOjPYR" + } + }, + { + "cell_type": "markdown", + "source": [ + "\"PPO\"/" + ], + "metadata": { + "id": "A8C-Q5ZyjUe3" + } + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VrS80GmMu_j5" + }, + "source": [ + "- Now we just need to run this python script using `python .py` with the additional parameters we defined with `argparse`\n", + "\n", + "- You should modify more hyperparameters otherwise the training will not be super stable." + ] + }, + { + "cell_type": "code", + "source": [ + "!python ppo.py --env-id=\"LunarLander-v2\" --repo-id=\"YOUR_REPO_ID\" --total-timesteps=50000" + ], + "metadata": { + "id": "KXLih6mKseBs" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eVsVJ5AdqLE7" + }, + "source": [ + "## Some additional challenges 🏆\n", + "The best way to learn **is to try things by your own**! Why not trying another environment?\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nYdl758GqLXT" + }, + "source": [ + "See you on Unit 8, part 2 where we going to train agents to play Doom 🔥\n", + "## Keep learning, stay awesome 🤗" + ] + } + ], + "metadata": { + "colab": { + "private_outputs": true, + "provenance": [], + "include_colab_link": true + }, + "gpuClass": "standard", + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + }, + "accelerator": "GPU" + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/notebooks/unit8/unit8_part2.ipynb b/notebooks/unit8/unit8_part2.ipynb new file mode 100644 index 0000000..51819df --- /dev/null +++ b/notebooks/unit8/unit8_part2.ipynb @@ -0,0 +1,680 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "view-in-github", + "colab_type": "text" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OVx1gdg9wt9t" + }, + "source": [ + "# Unit 8 Part 2: Advanced Deep Reinforcement Learning. Using Sample Factory to play Doom from pixels\n", + "\n", + "\"Thumbnail\"/\n", + "\n", + "In this notebook, we will learn how to train a Deep Neural Network to collect objects in a 3D environment based on the game of Doom, a video of the resulting policy is shown below. We train this policy using [Sample Factory](https://www.samplefactory.dev/), an asynchronous implementation of the PPO algorithm.\n", + "\n", + "Please note the following points:\n", + "\n", + "* [Sample Factory](https://www.samplefactory.dev/) is an advanced RL framework and **only functions on Linux and Mac** (not Windows).\n", + "\n", + "* The framework performs best on a **GPU machine with many CPU cores**, where it can achieve speeds of 100k interactions per second. The resources available on a standard Colab notebook **limit the performance of this library**. So the speed in this setting **does not reflect the real-world performance**.\n", + "* Benchmarks for Sample Factory are available in a number of settings, check out the [examples](https://github.com/alex-petrenko/sample-factory/tree/master/sf_examples) if you want to find out more.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "I6_67HfI1CKg" + }, + "outputs": [], + "source": [ + "from IPython.display import HTML\n", + "\n", + "HTML(''''''\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DgHRAsYEXdyw" + }, + "source": [ + "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model:\n", + "\n", + "- `doom_health_gathering_supreme` get a result of >= 5.\n", + "\n", + "To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward**\n", + "\n", + "If you don't find your model, **go to the bottom of the page and click on the refresh button**\n", + "\n", + "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PU4FVzaoM6fC" + }, + "source": [ + "## Set the GPU 💪\n", + "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n", + "\n", + "\"GPU" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KV0NyFdQM9ZG" + }, + "source": [ + "- `Hardware Accelerator > GPU`\n", + "\n", + "\"GPU" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-fSy5HzUcMWB" + }, + "source": [ + "Before starting to train our agent, let's **study the library and environments we're going to use**.\n", + "\n", + "## Sample Factory\n", + "\n", + "[Sample Factory](https://www.samplefactory.dev/) is one of the **fastest RL libraries focused on very efficient synchronous and asynchronous implementations of policy gradients (PPO)**.\n", + "\n", + "Sample Factory is thoroughly **tested, used by many researchers and practitioners**, and is actively maintained. Our implementation is known to **reach SOTA performance in a variety of domains while minimizing RL experiment training time and hardware requirements**.\n", + "\n", + "\"Sample\n", + "\n", + "\n", + "\n", + "### Key features\n", + "\n", + "- Highly optimized algorithm [architecture](https://www.samplefactory.dev/06-architecture/overview/) for maximum learning throughput\n", + "- [Synchronous and asynchronous](https://www.samplefactory.dev/07-advanced-topics/sync-async/) training regimes\n", + "- [Serial (single-process) mode](https://www.samplefactory.dev/07-advanced-topics/serial-mode/) for easy debugging\n", + "- Optimal performance in both CPU-based and [GPU-accelerated environments](https://www.samplefactory.dev/09-environment-integrations/isaacgym/)\n", + "- Single- & multi-agent training, self-play, supports [training multiple policies](https://www.samplefactory.dev/07-advanced-topics/multi-policy-training/) at once on one or many GPUs\n", + "- Population-Based Training ([PBT](https://www.samplefactory.dev/07-advanced-topics/pbt/))\n", + "- Discrete, continuous, hybrid action spaces\n", + "- Vector-based, image-based, dictionary observation spaces\n", + "- Automatically creates a model architecture by parsing action/observation space specification. Supports [custom model architectures](https://www.samplefactory.dev/03-customization/custom-models/)\n", + "- Designed to be imported into other projects, [custom environments](https://www.samplefactory.dev/03-customization/custom-environments/) are first-class citizens\n", + "- Detailed [WandB and Tensorboard summaries](https://www.samplefactory.dev/05-monitoring/metrics-reference/), [custom metrics](https://www.samplefactory.dev/05-monitoring/custom-metrics/)\n", + "- [HuggingFace 🤗 integration](https://www.samplefactory.dev/10-huggingface/huggingface/) (upload trained models and metrics to the Hub)\n", + "- [Multiple](https://www.samplefactory.dev/09-environment-integrations/mujoco/) [example](https://www.samplefactory.dev/09-environment-integrations/atari/) [environment](https://www.samplefactory.dev/09-environment-integrations/vizdoom/) [integrations](https://www.samplefactory.dev/09-environment-integrations/dmlab/) with tuned parameters and trained models\n", + "\n", + "All of the above policies are available on the 🤗 hub. Search for the tag [sample-factory](https://huggingface.co/models?library=sample-factory&sort=downloads)\n", + "\n", + "### How sample-factory works\n", + "\n", + "Sample-factory is one of the **most highly optimized RL implementations available to the community**. \n", + "\n", + "It works by **spawning multiple processes that run rollout workers, inference workers and a learner worker**. \n", + "\n", + "The *workers* **communicate through shared memory, which lowers the communication cost between processes**. \n", + "\n", + "The *rollout workers* interact with the environment and send observations to the *inference workers*. \n", + "\n", + "The *inferences workers* query a fixed version of the policy and **send actions back to the rollout worker**. \n", + "\n", + "After *k* steps the rollout works send a trajectory of experience to the learner worker, **which it uses to update the agent’s policy network**.\n", + "\n", + "\"Sample" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nB68Eb9UgC94" + }, + "source": [ + "### Actor Critic models in Sample-factory\n", + "\n", + "Actor Critic models in Sample Factory are composed of three components:\n", + "\n", + "- **Encoder** - Process input observations (images, vectors) and map them to a vector. This is the part of the model you will most likely want to customize.\n", + "- **Core** - Intergrate vectors from one or more encoders, can optionally include a single- or multi-layer LSTM/GRU in a memory-based agent.\n", + "- **Decoder** - Apply additional layers to the output of the model core before computing the policy and value outputs.\n", + "\n", + "The library has been designed to automatically support any observation and action spaces. Users can easily add their custom models. You can find out more in the [documentation](https://www.samplefactory.dev/03-customization/custom-models/#actor-critic-models-in-sample-factory)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ez5UhUtYcWXF" + }, + "source": [ + "## ViZDoom\n", + "\n", + "[ViZDoom](https://vizdoom.cs.put.edu.pl/) is an **open-source python interface for the Doom Engine**. \n", + "\n", + "The library was created in 2016 by Marek Wydmuch, Michal Kempka at the Institute of Computing Science, Poznan University of Technology, Poland. \n", + "\n", + "The library enables the **training of agents directly from the screen pixels in a number of scenarios**, including team deathmatch, shown in the video below. Because the ViZDoom environment is based on a game the was created in the 90s, it can be run on modern hardware at accelerated speeds, **allowing us to learn complex AI behaviors fairly quickly**.\n", + "\n", + "The library includes feature such as:\n", + "\n", + "- Multi-platform (Linux, macOS, Windows),\n", + "- API for Python and C++,\n", + "- [OpenAI Gym](https://www.gymlibrary.dev/) environment wrappers\n", + "- Easy-to-create custom scenarios (visual editors, scripting language, and examples available),\n", + "- Async and sync single-player and multiplayer modes,\n", + "- Lightweight (few MBs) and fast (up to 7000 fps in sync mode, single-threaded),\n", + "- Customizable resolution and rendering parameters,\n", + "- Access to the depth buffer (3D vision),\n", + "- Automatic labeling of game objects visible in the frame,\n", + "- Access to the audio buffer\n", + "- Access to the list of actors/objects and map geometry,\n", + "- Off-screen rendering and episode recording,\n", + "- Time scaling in async mode." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wAMwza0d5QVj" + }, + "source": [ + "## We first need to install some dependencies that are required for the ViZDoom environment\n", + "\n", + "Now that our Colab runtime is set up, we can start by installing the dependencies required to run ViZDoom on linux. \n", + "\n", + "If you are following on your machine on Mac, you will want to follow the installation instructions on the [github page](https://github.com/Farama-Foundation/ViZDoom/blob/master/doc/Quickstart.md#-quickstart-for-macos-and-anaconda3-python-36)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "RJMxkaldwIVx" + }, + "outputs": [], + "source": [ + "%%capture\n", + "%%bash\n", + "# Install ViZDoom deps from \n", + "# https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md#-linux\n", + "\n", + "apt-get install build-essential zlib1g-dev libsdl2-dev libjpeg-dev \\\n", + "nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \\\n", + "libopenal-dev timidity libwildmidi-dev unzip ffmpeg\n", + "\n", + "# Boost libraries\n", + "apt-get install libboost-all-dev\n", + "\n", + "# Lua binding dependencies\n", + "apt-get install liblua5.1-dev" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JT4att2c57MW" + }, + "source": [ + "## Then we can install Sample Factory and ViZDoom\n", + "- This can take 7min" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bbqfPZnIsvA6" + }, + "outputs": [], + "source": [ + "# install python libraries\n", + "# thanks toinsson\n", + "!pip install sample-factory==2.0.2\n", + "!pip install faster-fifo==1.4.2\n", + "!pip install vizdoom" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1jizouGpghUZ" + }, + "source": [ + "## Setting up the Doom Environment in sample-factory" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bCgZbeiavcDU" + }, + "outputs": [], + "source": [ + "import functools\n", + "\n", + "from sample_factory.algo.utils.context import global_model_factory\n", + "from sample_factory.cfg.arguments import parse_full_cfg, parse_sf_args\n", + "from sample_factory.envs.env_utils import register_env\n", + "from sample_factory.train import run_rl\n", + "\n", + "from sf_examples.vizdoom.doom.doom_model import make_vizdoom_encoder\n", + "from sf_examples.vizdoom.doom.doom_params import add_doom_env_args, doom_override_defaults\n", + "from sf_examples.vizdoom.doom.doom_utils import DOOM_ENVS, make_doom_env_from_spec\n", + "\n", + "\n", + "# Registers all the ViZDoom environments\n", + "def register_vizdoom_envs():\n", + " for env_spec in DOOM_ENVS:\n", + " make_env_func = functools.partial(make_doom_env_from_spec, env_spec)\n", + " register_env(env_spec.name, make_env_func)\n", + "\n", + "# Sample Factory allows the registration of a custom Neural Network architecture\n", + "# See https://github.com/alex-petrenko/sample-factory/blob/master/sf_examples/vizdoom/doom/doom_model.py for more details\n", + "def register_vizdoom_models():\n", + " global_model_factory().register_encoder_factory(make_vizdoom_encoder)\n", + "\n", + "\n", + "def register_vizdoom_components():\n", + " register_vizdoom_envs()\n", + " register_vizdoom_models()\n", + "\n", + "# parse the command line args and create a config\n", + "def parse_vizdoom_cfg(argv=None, evaluation=False):\n", + " parser, _ = parse_sf_args(argv=argv, evaluation=evaluation)\n", + " # parameters specific to Doom envs\n", + " add_doom_env_args(parser)\n", + " # override Doom default values for algo parameters\n", + " doom_override_defaults(parser)\n", + " # second parsing pass yields the final configuration\n", + " final_cfg = parse_full_cfg(parser, argv)\n", + " return final_cfg" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sgRy6wnrgnij" + }, + "source": [ + "Now that the setup if complete, we can train the agent. We have chosen here to learn a ViZDoom task called `Health Gathering Supreme`.\n", + "\n", + "### The scenario: Health Gathering Supreme\n", + "\n", + "\"Health-Gathering-Supreme\"/\n", + "\n", + "\n", + "\n", + "The objective of this scenario is to **teach the agent how to survive without knowing what makes him survive**. Agent know only that **life is precious** and death is bad so **it must learn what prolongs his existence and that his health is connected with it**.\n", + "\n", + "Map is a rectangle containing walls and with a green, acidic floor which **hurts the player periodically**. Initially there are some medkits spread uniformly over the map. A new medkit falls from the skies every now and then. **Medkits heal some portions of player's health** - to survive agent needs to pick them up. Episode finishes after player's death or on timeout.\n", + "\n", + "Further configuration:\n", + "- Living_reward = 1\n", + "- 3 available buttons: turn left, turn right, move forward\n", + "- 1 available game variable: HEALTH\n", + "- death penalty = 100\n", + "\n", + "You can find out more about the scenarios available in ViZDoom [here](https://github.com/Farama-Foundation/ViZDoom/tree/master/scenarios). \n", + "\n", + "There are also a number of more complex scenarios that have been create for ViZDoom, such as the ones detailed on [this github page](https://github.com/edbeeching/3d_control_deep_rl).\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "siHZZ34DiZEp" + }, + "source": [ + "## Training the agent\n", + "- We're going to train the agent for 4000000 steps it will take approximately 20min" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "y_TeicMvyKHP" + }, + "outputs": [], + "source": [ + "## Start the training, this should take around 15 minutes\n", + "register_vizdoom_components()\n", + "\n", + "# The scenario we train on today is health gathering\n", + "# other scenarios include \"doom_basic\", \"doom_two_colors_easy\", \"doom_dm\", \"doom_dwango5\", \"doom_my_way_home\", \"doom_deadly_corridor\", \"doom_defend_the_center\", \"doom_defend_the_line\"\n", + "env = \"doom_health_gathering_supreme\"\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=8\", \"--num_envs_per_worker=4\", \"--train_for_env_steps=4000000\"])\n", + "\n", + "status = run_rl(cfg)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5L0nBS9e_jqC" + }, + "source": [ + "## Let's take a look at the performance of the trained policy and output a video of the agent." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "MGSA4Kg5_i0j" + }, + "outputs": [], + "source": [ + "from sample_factory.enjoy import enjoy\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\"], evaluation=True)\n", + "status = enjoy(cfg)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Lj5L1x0WLxwB" + }, + "source": [ + "## Now lets visualize the performance of the agent" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "WsXhBY7JNOdJ" + }, + "outputs": [], + "source": [ + "from base64 import b64encode\n", + "from IPython.display import HTML\n", + "\n", + "mp4 = open('/content/train_dir/default_experiment/replay.mp4','rb').read()\n", + "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", + "HTML(\"\"\"\n", + "\n", + "\"\"\" % data_url)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "The agent has learned something, but its performance could be better. We would clearly need to train for longer. But let's upload this model to the Hub." + ], + "metadata": { + "id": "2A4pf_1VwPqR" + } + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CSQVWF0kNuy9" + }, + "source": [ + "## Now lets upload your checkpoint and video to the Hugging Face Hub\n", + "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JquRrWytA6eo" + }, + "source": [ + "To be able to share your model with the community there are three more steps to follow:\n", + "\n", + "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n", + "\n", + "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n", + "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n", + "\n", + "\"Create\n", + "\n", + "- Copy the token \n", + "- Run the cell below and paste the token" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_tsf2uv0g_4p" + }, + "source": [ + "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "GoQm_jYSOts0" + }, + "outputs": [], + "source": [ + "from huggingface_hub import notebook_login\n", + "notebook_login()\n", + "!git config --global credential.helper store" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "sEawW_i0OvJV" + }, + "outputs": [], + "source": [ + "from sample_factory.enjoy import enjoy\n", + "\n", + "hf_username = \"ThomasSimonini\" # insert your HuggingFace username here\n", + "\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--max_num_frames=100000\", \"--push_to_hub\", f\"--hf_repository={hf_username}/rl_course_vizdoom_health_gathering_supreme\"], evaluation=True)\n", + "status = enjoy(cfg)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Let's load another model\n", + "\n", + "\n" + ], + "metadata": { + "id": "9PzeXx-qxVvw" + } + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mHZAWSgL5F7P" + }, + "source": [ + "This agent's performance was good, but can do better! Let's download and visualize an agent trained for 10B timesteps from the hub." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Ud6DwAUl5S-l" + }, + "outputs": [], + "source": [ + "#download the agent from the hub\n", + "!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_health_gathering_supreme_2222 -d ./train_dir\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qoUJhL6x6sY5" + }, + "outputs": [], + "source": [ + "!ls train_dir/doom_health_gathering_supreme_2222" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lZskc8LG8qr8" + }, + "outputs": [], + "source": [ + "env = \"doom_health_gathering_supreme\"\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--experiment=doom_health_gathering_supreme_2222\", \"--train_dir=train_dir\"], evaluation=True)\n", + "status = enjoy(cfg)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BtzXBoj65Wmq" + }, + "outputs": [], + "source": [ + "mp4 = open('/content/train_dir/doom_health_gathering_supreme_2222/replay.mp4','rb').read()\n", + "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", + "HTML(\"\"\"\n", + "\n", + "\"\"\" % data_url)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Some additional challenges 🏆: Doom Deathmatch\n", + "\n", + "Training an agent to play a Doom deathmatch **takes many hours on a more beefy machine than is available in Colab**. \n", + "\n", + "Fortunately, we have have **already trained an agent in this scenario and it is available in the 🤗 Hub!** Let’s download the model and visualize the agent’s performance." + ], + "metadata": { + "id": "ie5YWC3NyKO8" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "fq3WFeus81iI" + }, + "outputs": [], + "source": [ + "# Download the agent from the hub\n", + "!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_deathmatch_bots_2222 -d ./train_dir" + ] + }, + { + "cell_type": "markdown", + "source": [ + "Given the agent plays for a long time the video generation can take **10 minutes**." + ], + "metadata": { + "id": "7AX_LwxR2FQ0" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0hq6XL__85Bv" + }, + "outputs": [], + "source": [ + "\n", + "from sample_factory.enjoy import enjoy\n", + "register_vizdoom_components()\n", + "env = \"doom_deathmatch_bots\"\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=1\", \"--experiment=doom_deathmatch_bots_2222\", \"--train_dir=train_dir\"], evaluation=True)\n", + "status = enjoy(cfg)\n", + "mp4 = open('/content/train_dir/doom_deathmatch_bots_2222/replay.mp4','rb').read()\n", + "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", + "HTML(\"\"\"\n", + "\n", + "\"\"\" % data_url)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "You **can try to train your agent in this environment** using the code above, but not on colab.\n", + "**Good luck 🤞**" + ], + "metadata": { + "id": "N6mEC-4zyihx" + } + }, + { + "cell_type": "markdown", + "source": [ + "If you prefer an easier scenario, **why not try training in another ViZDoom scenario such as `doom_deadly_corridor` or `doom_defend_the_center`.**\n", + "\n", + "\n", + "\n", + "---\n", + "\n", + "\n", + "This concludes the last unit. But we are not finished yet! 🤗 The following **bonus section include some of the most interesting, advanced and cutting edge work in Deep Reinforcement Learning**.\n", + "\n", + "## Keep learning, stay awesome 🤗" + ], + "metadata": { + "id": "YnDAngN6zeeI" + } + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "provenance": [], + "collapsed_sections": [ + "PU4FVzaoM6fC", + "nB68Eb9UgC94", + "ez5UhUtYcWXF", + "sgRy6wnrgnij" + ], + "private_outputs": true, + "include_colab_link": true + }, + "gpuClass": "standard", + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/unit1/README.md b/unit1/README.md index 32cbb03..a6d2acb 100644 --- a/unit1/README.md +++ b/unit1/README.md @@ -1,4 +1,8 @@ -# Unit 1: Introduction to Deep Reinforcement Learning 🚀 +# DEPRECIATED THE NEW UNIT 1 IS HERE: https://huggingface.co/deep-rl-course/unit1/introduction +**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit1/introduction + + +# Unit 1: Introduction to Deep Reinforcement Learning 🚀 (DEPRECIATED) ![cover](assets/img/thumbnail.png) diff --git a/unit1/unit1.ipynb b/unit1/unit1.ipynb index a333c08..49eb1b0 100644 --- a/unit1/unit1.ipynb +++ b/unit1/unit1.ipynb @@ -16,6 +16,11 @@ "id": "njb_ProuHiOe" }, "source": [ + "# DEPRECIATED NOTEBOOK, THE NEW UNIT 1 IS HERE: https://huggingface.co/deep-rl-course/unit1/introduction", + "\n", + "**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit1/introduction", + "\n", + "\n", "# Unit 1: Train your first Deep Reinforcement Learning Agent 🚀\n", "![Cover](https://github.com/huggingface/deep-rl-class/blob/main/unit1/assets/img/thumbnail.png?raw=true)\n", "\n", diff --git a/unit2/README.md b/unit2/README.md index 677e655..ff6682d 100644 --- a/unit2/README.md +++ b/unit2/README.md @@ -1,3 +1,7 @@ +# DEPRECIATED UNIT, THE NEW UNIT 2 IS HERE: https://huggingface.co/deep-rl-course/unit2/introduction +**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit2/introduction" + + # Unit 2: Introduction to Q-Learning In this Unit, we're going to dive deeper into one of the Reinforcement Learning methods: value-based methods and **study our first RL algorithm: Q-Learning**. @@ -70,7 +74,7 @@ You can work directly **with the colab notebook, which allows you not to have to - To dive deeper on Monte Carlo and Temporal Difference Learning: - [Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?](https://stats.stackexchange.com/questions/355820/why-do-temporal-difference-td-methods-have-lower-variance-than-monte-carlo-met) - [When are Monte Carlo methods preferred over temporal difference ones?](https://stats.stackexchange.com/questions/336974/when-are-monte-carlo-methods-preferred-over-temporal-difference-ones) - + ## How to make the most of this course To make the most of the course, my advice is to: diff --git a/unit2/unit2.ipynb b/unit2/unit2.ipynb index 803db1c..c6119cb 100644 --- a/unit2/unit2.ipynb +++ b/unit2/unit2.ipynb @@ -16,6 +16,11 @@ "id": "njb_ProuHiOe" }, "source": [ + "# DEPRECIATED NOTEBOOK, THE NEW UNIT 2 IS HERE: https://huggingface.co/deep-rl-course/unit2/introduction", + "\n", + "**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit2/introduction", + "\n", + "\n", "# Unit 2: Q-Learning with FrozenLake-v1 ⛄ and Taxi-v3 🚕\n", "\n", "In this notebook, **you'll code from scratch your first Reinforcement Learning agent** playing FrozenLake ❄️ using Q-Learning, share it to the community, and experiment with different configurations\n", diff --git a/unit3/README.md b/unit3/README.md index 07f17c3..897d8f5 100644 --- a/unit3/README.md +++ b/unit3/README.md @@ -1,6 +1,10 @@ +# DEPRECIATED THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unit3/introduction +**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit3/introduction + + # Unit 3: Deep Q-Learning with Atari Games 👾 -In this Unit, **we'll study our first Deep Reinforcement Learning agent**: Deep Q-Learning. +In this Unit, **we'll study our first Deep Reinforcement Learning agent**: Deep Q-Learning. And **we'll train it to play Space Invaders and other Atari environments using [RL-Zoo](https://github.com/DLR-RM/rl-baselines3-zoo)**, a training framework for RL using Stable-Baselines that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results, and recording videos. diff --git a/unit3/unit3.ipynb b/unit3/unit3.ipynb index 9657cce..fead1ad 100644 --- a/unit3/unit3.ipynb +++ b/unit3/unit3.ipynb @@ -16,6 +16,11 @@ "id": "k7xBVPzoXxOg" }, "source": [ + "# DEPRECIATED NOTEBOOK, THE NEW UNIT 3 IS HERE: https://huggingface.co/deep-rl-course/unit3/introduction", + "\n", + "**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit3/introduction", + "\n", + "\n", "# Unit 3: Deep Q-Learning with Atari Games 👾 using RL Baselines3 Zoo\n", "\n", "In this notebook, **you'll train a Deep Q-Learning agent** playing Space Invaders using [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), a training framework based on [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.\n", diff --git a/unit4/README.md b/unit4/README.md index 2de68c8..9bf4fca 100644 --- a/unit4/README.md +++ b/unit4/README.md @@ -1,3 +1,7 @@ +# DEPRECIATED THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unit5/introduction +**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit5/introduction + + # Unit 4: An Introduction to Unity MLAgents with Hugging Face 🤗 ![cover](https://miro.medium.com/max/1400/1*8DV9EFl-vdijvcTHilHuEw.png) diff --git a/unit4/unit4.ipynb b/unit4/unit4.ipynb index 79e53aa..9232ad6 100644 --- a/unit4/unit4.ipynb +++ b/unit4/unit4.ipynb @@ -16,6 +16,11 @@ "id": "2D3NL_e4crQv" }, "source": [ + "# DEPRECIATED NOTEBOOK, THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unit5/introduction", + "\n", + "**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit5/introduction", + "\n", + "\n", "# Unit 4: Let's learn about Unity ML-Agents with Hugging Face 🤗\n", "\n" ] @@ -561,4 +566,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} \ No newline at end of file +} diff --git a/unit5/README.md b/unit5/README.md index 2b50dbe..757f089 100644 --- a/unit5/README.md +++ b/unit5/README.md @@ -1,6 +1,10 @@ -# Unit 5: Policy Gradient with PyTorch +# DEPRECIATED THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unit4/introduction +**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit4/introduction -In this Unit, **we'll study Policy Gradient Methods**. + +# Unit 5: Policy Gradient with PyTorch + +In this Unit, **we'll study Policy Gradient Methods**. And we'll **implement Reinforce (a policy gradient method) from scratch using PyTorch**. Before testing its robustness using CartPole-v1, PixelCopter, and Pong. diff --git a/unit5/unit5.ipynb b/unit5/unit5.ipynb index b23e2e5..85de7b5 100644 --- a/unit5/unit5.ipynb +++ b/unit5/unit5.ipynb @@ -16,6 +16,11 @@ "id": "CjRWziAVU2lZ" }, "source": [ + "# DEPRECIATED NOTEBOOK, THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unit4/introduction", + "\n", + "**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit4/introduction", + "\n", + "\n", "# Unit 5: Code your first Deep Reinforcement Learning Algorithm with PyTorch: Reinforce. And test its robustness 💪\n", "In this notebook, you'll code your first Deep Reinforcement Learning algorithm from scratch: Reinforce (also called Monte Carlo Policy Gradient).\n", "\n", diff --git a/unit7/README.md b/unit7/README.md index ded4fdc..d09bbdb 100644 --- a/unit7/README.md +++ b/unit7/README.md @@ -1,3 +1,7 @@ +# DEPRECIATED THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unit6/introduction +**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit6/introduction + + # Unit 7: Advantage Actor Critic (A2C) using Robotics Simulations with PyBullet 🤖 One of the major industries that use Reinforcement Learning is robotics. Unfortunately, **having access to robot equipment is very expensive**. Fortunately, some simulations exist to train Robots: @@ -32,7 +36,7 @@ Thanks to a leaderboard, you'll be able to compare your results with other class The leaderboard 👉 https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard ## Additional readings 📚 -- [Making Sense of the Bias / Variance Trade-off in (Deep) Reinforcement Learning](https://blog.mlreview.com/making-sense-of-the-bias-variance-trade-off-in-deep-reinforcement-learning-79cf1e83d565) +- [Making Sense of the Bias / Variance Trade-off in (Deep) Reinforcement Learning](https://blog.mlreview.com/making-sense-of-the-bias-variance-trade-off-in-deep-reinforcement-learning-79cf1e83d565) - [Bias-variance Tradeoff in Reinforcement Learning](https://www.endtoend.ai/blog/bias-variance-tradeoff-in-reinforcement-learning/) - [Foundations of Deep RL Series, L3 Policy Gradients and Advantage Estimation by Pieter Abbeel](https://youtu.be/AKbX1Zvo7r8) diff --git a/unit7/unit7.ipynb b/unit7/unit7.ipynb index 31e9f2e..f87fa02 100644 --- a/unit7/unit7.ipynb +++ b/unit7/unit7.ipynb @@ -34,6 +34,11 @@ { "cell_type": "markdown", "source": [ + "# DEPRECIATED NOTEBOOK, THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unit6/introduction", + "\n", + "**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit6/introduction", + "\n", + "\n", "# Unit 7: Advantage Actor Critic (A2C) using Robotics Simulations with PyBullet 🤖\n", "In this small notebook you'll learn to use A2C with PyBullet. And train an agent to walk. More precisely a spider (they say Ant but come on... it's a spider 😆) 🕸️\n", "\n", @@ -533,4 +538,4 @@ } } ] -} \ No newline at end of file +} diff --git a/unit8/README.md b/unit8/README.md index 61664da..1280536 100644 --- a/unit8/README.md +++ b/unit8/README.md @@ -1,3 +1,7 @@ +# DEPRECIATED THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unit8/introduction +**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit8/introduction + + # Unit 8: Proximal Policy Optimization (PPO) with PyTorch Today we'll learn about Proximal Policy Optimization (PPO), an architecture that improves our agent's training stability by avoiding too large policy updates. To do that, we use a ratio that will indicates the difference between our current and old policy and clip this ratio from a specific range $[1 - \epsilon, 1 + \epsilon]$. Doing this will ensure that our policy update will not be too large and that the training is more stable. @@ -29,7 +33,7 @@ Thanks to a leaderboard, you'll be able to compare your results with other class The leaderboard 👉 https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-Learning-Leaderboard ## Additional readings 📚 -- [Towards Delivering a Coherent Self-Contained Explanation of Proximal Policy Optimization by Daniel Bick](https://fse.studenttheses.ub.rug.nl/25709/1/mAI_2021_BickD.pdf) +- [Towards Delivering a Coherent Self-Contained Explanation of Proximal Policy Optimization by Daniel Bick](https://fse.studenttheses.ub.rug.nl/25709/1/mAI_2021_BickD.pdf) - [What is the way to understand Proximal Policy Optimization Algorithm in RL?](https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl) - [Foundations of Deep RL Series, L4 TRPO and PPO by Pieter Abbeel](https://youtu.be/KjWF8VIMGiY) - [OpenAI PPO Blogpost](https://openai.com/blog/openai-baselines-ppo/) diff --git a/unit8/unit8.ipynb b/unit8/unit8.ipynb index 2bbd7e1..5a177dd 100644 --- a/unit8/unit8.ipynb +++ b/unit8/unit8.ipynb @@ -16,6 +16,11 @@ "id": "-cf5-oDPjwf8" }, "source": [ + "# DEPRECIATED NOTEBOOK, THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unit8/introduction", + "\n", + "**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unit8/introduction", + "\n", + "\n", "# Unit 8: Proximal Policy Gradient (PPO) with PyTorch 🤖\n", "\n", "In this unit, you'll learn to **code your PPO agent from scratch with PyTorch**.\n", diff --git a/unit9/README.md b/unit9/README.md index 051fe94..7770e3c 100644 --- a/unit9/README.md +++ b/unit9/README.md @@ -1,8 +1,13 @@ +# DEPRECIATED THE NEW VERSION OF THIS UNIT IS HERE: https://huggingface.co/deep-rl-course/unitbonus3/decision-transformers +**Everything under is depreciated** 👇, the new version of the course is here: https://huggingface.co/deep-rl-course/unitbonus3/decision-transformers + + + # Unit 9: Decision Transformers and offline Reinforcement Learning 🤖 ![cover](assets/img/thumbnail.gif) -In this Unit, you'll learn what is Decision Transformer and Offline Reinforcement Learning. And then, you’ll train your first Offline Decision Transformer model from scratch to make a half-cheetah run. +In this Unit, you'll learn what is Decision Transformer and Offline Reinforcement Learning. And then, you’ll train your first Offline Decision Transformer model from scratch to make a half-cheetah run. This course is **self-paced**, you can start whenever you want. @@ -18,12 +23,12 @@ Here are the steps for this Unit: 2️⃣ 👩‍💻 Then dive on the first hands-on. 👩‍💻 The hands-on 👉 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1K3UuajwoPY1MzRKNkONNRS3gS5DxZ-qF?usp=sharing) - + 3️⃣ 📖 Read [Train your first Decision Transformer](https://huggingface.co/blog/train-decision-transformers) -4️⃣ 👩‍💻 Then dive on the hands-on, where **you’ll train your first Offline Decision Transformer model from scratch to make a half-cheetah run**. +4️⃣ 👩‍💻 Then dive on the hands-on, where **you’ll train your first Offline Decision Transformer model from scratch to make a half-cheetah run**. 👩‍💻 The hands-on 👉 https://github.com/huggingface/blog/blob/main/notebooks/101_train-decision-transformers.ipynb - + ## How to make the most of this course To make the most of the course, my advice is to: diff --git a/units/en/_toctree.yml b/units/en/_toctree.yml index 99b3d94..040e10a 100644 --- a/units/en/_toctree.yml +++ b/units/en/_toctree.yml @@ -178,6 +178,52 @@ title: Conclusion - local: unit7/additional-readings title: Additional Readings +- title: Unit 8. Part 1 Proximal Policy Optimization (PPO) + sections: + - local: unit8/introduction + title: Introduction + - local: unit8/intuition-behind-ppo + title: The intuition behind PPO + - local: unit8/clipped-surrogate-objective + title: Introducing the Clipped Surrogate Objective Function + - local: unit8/visualize + title: Visualize the Clipped Surrogate Objective Function + - local: unit8/hands-on-cleanrl + title: PPO with CleanRL + - local: unit8/conclusion + title: Conclusion + - local: unit8/additional-readings + title: Additional Readings +- title: Unit 8. Part 2 Proximal Policy Optimization (PPO) with Doom + sections: + - local: unit8/introduction-sf + title: Introduction + - local: unit8/hands-on-sf + title: PPO with Sample Factory and Doom + - local: unit8/conclusion-sf + title: Conclusion +- title: Bonus Unit 3. Advanced Topics in Reinforcement Learning + sections: + - local: unitbonus3/introduction + title: Introduction + - local: unitbonus3/model-based + title: Model-Based Reinforcement Learning + - local: unitbonus3/offline-online + title: Offline vs. Online Reinforcement Learning + - local: unitbonus3/rlhf + title: Reinforcement Learning from Human Feedback + - local: unitbonus3/decision-transformers + title: Decision Transformers and Offline RL + - local: unitbonus3/language-models + title: Language models in RL + - local: unitbonus3/curriculum-learning + title: (Automatic) Curriculum Learning for RL + - local: unitbonus3/envs-to-try + title: Interesting environments to try + - local: unitbonus3/godotrl + title: An Introduction to Godot RL + - local: unitbonus3/rl-documentation + title: Brief introduction to RL documentation - title: Certification and congratulations sections: - local: communication/conclusion diff --git a/units/en/unit0/discord101.mdx b/units/en/unit0/discord101.mdx index 9904168..e3bda2c 100644 --- a/units/en/unit0/discord101.mdx +++ b/units/en/unit0/discord101.mdx @@ -9,11 +9,11 @@ Discord is a free chat platform. If you've used Slack, **it's quite similar**. T Starting in Discord can be a bit intimidating, so let me take you through it. -When you sign-up to our Discord server, you'll need to specify which topics you're interested in by **clicking #role-assignment at the left**. +When you sign-up to our Discord server, you'll need to specify which topics you're interested in by **clicking #role-assignment at the left**. Discord -In #role-assignment, you can pick different categories. Make sure to **click "Reinforcement Learning"**. You'll then get to **introduce yourself in the `#introduction-yourself` channel**. +In #role-assignment, you can pick different categories. Make sure to **click "Reinforcement Learning"**. You'll then get to **introduce yourself in the `#introduce-yourself` channel**. Discord diff --git a/units/en/unit1/hands-on.mdx b/units/en/unit1/hands-on.mdx index c2dc4cd..5c181a0 100644 --- a/units/en/unit1/hands-on.mdx +++ b/units/en/unit1/hands-on.mdx @@ -22,6 +22,8 @@ To validate this hands-on for the [certification process](https://huggingface.co To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward** +**If you don't find your model, go to the bottom of the page and click on the refresh button.** + For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process And you can check your progress here 👉 https://huggingface.co/spaces/ThomasSimonini/Check-my-progress-Deep-RL-Course @@ -43,13 +45,6 @@ You can either do this hands-on by reading the notebook or following it with the In this notebook, you'll train your **first Deep Reinforcement Learning agent** a Lunar Lander agent that will learn to **land correctly on the Moon 🌕**. Using [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) a Deep Reinforcement Learning library, share them with the community, and experiment with different configurations -⬇️ Here is an example of what **you will achieve in just a couple of minutes.** ⬇️ - -```python -%%html - -``` - ### The environment 🎮 - [LunarLander-v2](https://www.gymlibrary.dev/environments/box2d/lunar_lander/) @@ -92,7 +87,7 @@ Before diving into the notebook, you need to: 🔲 📝 **Read Unit 0** that gives you all the **information about the course and help you to onboard** 🤗 -🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by doing Unit 1 +🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** by reading Unit 1 ## A small recap of what is Deep Reinforcement Learning 📚 The RL process @@ -664,8 +659,7 @@ If you’re still feel confused with all these elements...it's totally normal! * Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations. -Naturally, during the course, we’re going to use and deeper explain again these terms but **it’s better to have a good understanding of them now before diving into the next chapters.** - +Naturally, during the course, we’re going to dive deeper into these concepts but **it’s better to have a good understanding of them now before diving into the next chapters.** Next time, in the bonus unit 1, you'll train Huggy the Dog to fetch the stick. diff --git a/units/en/unit1/introduction.mdx b/units/en/unit1/introduction.mdx index f8017cd..e1ebd76 100644 --- a/units/en/unit1/introduction.mdx +++ b/units/en/unit1/introduction.mdx @@ -22,6 +22,6 @@ It's essential **to master these elements** before diving into implementing Dee After this unit, in a bonus unit, you'll be **able to train Huggy the Dog 🐶 to fetch the stick and play with him 🤗**. -Huggy +