diff --git a/notebooks/unit8_part1.ipynb b/notebooks/unit8_part1.ipynb
new file mode 100644
index 0000000..a1862c8
--- /dev/null
+++ b/notebooks/unit8_part1.ipynb
@@ -0,0 +1,1357 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github",
+ "colab_type": "text"
+ },
+ "source": [
+ "![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-cf5-oDPjwf8"
+ },
+ "source": [
+ "# Unit 8: Proximal Policy Gradient (PPO) with PyTorch π€\n",
+ "\n",
+ "![\"Unit](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit9/thumbnail.png\")
\n",
+ "\n",
+ "\n",
+ "In this notebook, you'll learn to **code your PPO agent from scratch with PyTorch using CleanRL implementation as model**.\n",
+ "\n",
+ "To test its robustness, we're going to train it in:\n",
+ "\n",
+ "- [LunarLander-v2 π](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2Fl6Rxt0lc0O"
+ },
+ "source": [
+ "β¬οΈ Here is an example of what you will achieve. β¬οΈ"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "DbKfCj5ilgqT"
+ },
+ "outputs": [],
+ "source": [
+ "%%html\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "YcOFdWpnlxNf"
+ },
+ "source": [
+ "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Objectives of this notebook π\n",
+ "\n",
+ "At the end of the notebook, you will:\n",
+ "\n",
+ "- Be able to **code your PPO agent from scratch using PyTorch**.\n",
+ "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score π₯.\n",
+ "\n",
+ "\n"
+ ],
+ "metadata": {
+ "id": "T6lIPYFghhYL"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## This notebook is from the Deep Reinforcement Learning Course\n",
+ "![\"Deep](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/deep-rl-course-illustration.jpg\")
\n",
+ "\n",
+ "In this free course, you will:\n",
+ "\n",
+ "- π Study Deep Reinforcement Learning in **theory and practice**.\n",
+ "- π§βπ» Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
+ "- π€ Train **agents in unique environments** \n",
+ "\n",
+ "Donβt forget to **sign up to the course** (we are collecting your email to be able toΒ **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
+ "\n",
+ "\n",
+ "The best way to keep in touch is to join our discord server to exchange with the community and with us ππ» https://discord.gg/ydHrjt3WP5"
+ ],
+ "metadata": {
+ "id": "Wp-rD6Fuhq31"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Prerequisites ποΈ\n",
+ "Before diving into the notebook, you need to:\n",
+ "\n",
+ "π² π Study [PPO by reading Unit 8](https://huggingface.co/deep-rl-course/unit8/introduction) π€ "
+ ],
+ "metadata": {
+ "id": "rasqqGQlhujA"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model, we don't ask for a minimal result but we **advise you to try different hyperparameters settings to get better results**.\n",
+ "\n",
+ "If you don't find your model, **go to the bottom of the page and click on the refresh button**\n",
+ "\n",
+ "For more information about the certification process, check this section π https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process"
+ ],
+ "metadata": {
+ "id": "PUFfMGOih3CW"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Set the GPU πͺ\n",
+ "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
+ "\n",
+ "![\"GPU](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\")
"
+ ],
+ "metadata": {
+ "id": "PU4FVzaoM6fC"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "- `Hardware Accelerator > GPU`\n",
+ "\n",
+ "![\"GPU](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\")
"
+ ],
+ "metadata": {
+ "id": "KV0NyFdQM9ZG"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Create a virtual display π½\n",
+ "\n",
+ "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
+ "\n",
+ "Hence the following cell will install the librairies and create and run a virtual screen π₯"
+ ],
+ "metadata": {
+ "id": "bTpYcVZVMzUI"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "jV6wjQ7Be7p5"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!apt install python-opengl\n",
+ "!apt install ffmpeg\n",
+ "!apt install xvfb\n",
+ "!pip install pyglet==1.5\n",
+ "!pip3 install pyvirtualdisplay"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Virtual display\n",
+ "from pyvirtualdisplay import Display\n",
+ "\n",
+ "virtual_display = Display(visible=0, size=(1400, 900))\n",
+ "virtual_display.start()"
+ ],
+ "metadata": {
+ "id": "ww5PQH1gNLI4"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ncIgfNf3mOtc"
+ },
+ "source": [
+ "## Install dependencies π½\n",
+ "For this exercise, we use `gym==0.21`\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!pip install gym==0.21\n",
+ "!pip install imageio-ffmpeg\n",
+ "!pip install huggingface_hub\n",
+ "!pip install box2d"
+ ],
+ "metadata": {
+ "id": "9xZQFTPcsKUK"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "oDkUufewmq6v"
+ },
+ "source": [
+ "## Let's code PPO from scratch with Costa Huang tutorial\n",
+ "- For the core implementation of PPO we're going to use the excellent [Costa Huang](https://costa.sh/) tutorial.\n",
+ "- In addition to the tutorial, to go deeper you can read the 37 core implementation details: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/\n",
+ "\n",
+ "π The video tutorial: https://youtu.be/MEt6rrxH8W4"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "aNgEL1_uvhaq"
+ },
+ "outputs": [],
+ "source": [
+ "from IPython.display import HTML\n",
+ "\n",
+ "HTML('')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "f34ILn7AvTbt"
+ },
+ "source": [
+ "- The best is to code first on the cell below, this way, if you kill the machine **you don't loose the implementation**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "_bE708C6mhE7"
+ },
+ "outputs": [],
+ "source": [
+ "### Your code here:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "mk-a9CmNuS2W"
+ },
+ "source": [
+ "## Add the Hugging Face Integration π€\n",
+ "- In order to push our model to the Hub, we need to define a function `package_to_hub`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "TPi1Nme-oGWd"
+ },
+ "source": [
+ "- Add dependencies we need to push our model to the Hub"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Sj8bz-AmoNVj"
+ },
+ "outputs": [],
+ "source": [
+ "from huggingface_hub import HfApi, upload_folder\n",
+ "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n",
+ "\n",
+ "from pathlib import Path\n",
+ "import datetime\n",
+ "import tempfile\n",
+ "import json\n",
+ "import shutil\n",
+ "import imageio\n",
+ "\n",
+ "from wasabi import Printer\n",
+ "msg = Printer()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5rDr8-lWn0zi"
+ },
+ "source": [
+ "- Add new argument in `parse_args()` function to define the repo-id where we want to push the model."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "iHQiqQEFn0QH"
+ },
+ "outputs": [],
+ "source": [
+ "# Adding HuggingFace argument\n",
+ "parser.add_argument(\"--repo-id\", type=str, default=\"ThomasSimonini/ppo-CartPole-v1\", help=\"id of the model repository from the Hugging Face Hub {username/repo_name}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "blLZMiBAoUVT"
+ },
+ "source": [
+ "- Next, we add the methods needed to push the model to the Hub\n",
+ "\n",
+ "- These methods will:\n",
+ " - `_evalutate_agent()`: evaluate the agent.\n",
+ " - `_generate_model_card()`: generate the model card of your agent.\n",
+ " - `_record_video()`: record a video of your agent."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "WlLcz4L9odXs"
+ },
+ "outputs": [],
+ "source": [
+ "def package_to_hub(repo_id, \n",
+ " model,\n",
+ " hyperparameters,\n",
+ " eval_env,\n",
+ " video_fps=30,\n",
+ " commit_message=\"Push agent to the Hub\",\n",
+ " token= None,\n",
+ " logs=None\n",
+ " ):\n",
+ " \"\"\"\n",
+ " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n",
+ " This method does the complete pipeline:\n",
+ " - It evaluates the model\n",
+ " - It generates the model card\n",
+ " - It generates a replay video of the agent\n",
+ " - It pushes everything to the hub\n",
+ " :param repo_id: id of the model repository from the Hugging Face Hub\n",
+ " :param model: trained model\n",
+ " :param eval_env: environment used to evaluate the agent\n",
+ " :param fps: number of fps for rendering the video\n",
+ " :param commit_message: commit message\n",
+ " :param logs: directory on local machine of tensorboard logs you'd like to upload\n",
+ " \"\"\"\n",
+ " msg.info(\n",
+ " \"This function will save, evaluate, generate a video of your agent, \"\n",
+ " \"create a model card and push everything to the hub. \"\n",
+ " \"It might take up to 1min. \\n \"\n",
+ " \"This is a work in progress: if you encounter a bug, please open an issue.\"\n",
+ " )\n",
+ " # Step 1: Clone or create the repo\n",
+ " repo_url = HfApi().create_repo(\n",
+ " repo_id=repo_id,\n",
+ " token=token,\n",
+ " private=False,\n",
+ " exist_ok=True,\n",
+ " )\n",
+ " \n",
+ " with tempfile.TemporaryDirectory() as tmpdirname:\n",
+ " tmpdirname = Path(tmpdirname)\n",
+ "\n",
+ " # Step 2: Save the model\n",
+ " torch.save(model.state_dict(), tmpdirname / \"model.pt\")\n",
+ " \n",
+ " # Step 3: Evaluate the model and build JSON\n",
+ " mean_reward, std_reward = _evaluate_agent(eval_env, \n",
+ " 10, \n",
+ " model)\n",
+ "\n",
+ " # First get datetime\n",
+ " eval_datetime = datetime.datetime.now()\n",
+ " eval_form_datetime = eval_datetime.isoformat()\n",
+ "\n",
+ " evaluate_data = {\n",
+ " \"env_id\": hyperparameters.env_id, \n",
+ " \"mean_reward\": mean_reward,\n",
+ " \"std_reward\": std_reward,\n",
+ " \"n_evaluation_episodes\": 10,\n",
+ " \"eval_datetime\": eval_form_datetime,\n",
+ " }\n",
+ " \n",
+ " # Write a JSON file\n",
+ " with open(tmpdirname / \"results.json\", \"w\") as outfile:\n",
+ " json.dump(evaluate_data, outfile)\n",
+ "\n",
+ " # Step 4: Generate a video\n",
+ " video_path = tmpdirname / \"replay.mp4\"\n",
+ " record_video(eval_env, model, video_path, video_fps)\n",
+ " \n",
+ " # Step 5: Generate the model card\n",
+ " generated_model_card, metadata = _generate_model_card(\"PPO\", hyperparameters.env_id, mean_reward, std_reward, hyperparameters)\n",
+ " _save_model_card(tmpdirname, generated_model_card, metadata)\n",
+ "\n",
+ " # Step 6: Add logs if needed\n",
+ " if logs:\n",
+ " _add_logdir(tmpdirname, Path(logs))\n",
+ " \n",
+ " msg.info(f\"Pushing repo {repo_id} to the Hugging Face Hub\")\n",
+ " \n",
+ " repo_url = upload_folder(\n",
+ " repo_id=repo_id,\n",
+ " folder_path=tmpdirname,\n",
+ " path_in_repo=\"\",\n",
+ " commit_message=commit_message,\n",
+ " token=token,\n",
+ " )\n",
+ "\n",
+ " msg.info(f\"Your model is pushed to the Hub. You can view your model here: {repo_url}\")\n",
+ " return repo_url\n",
+ "\n",
+ "\n",
+ "def _evaluate_agent(env, n_eval_episodes, policy):\n",
+ " \"\"\"\n",
+ " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n",
+ " :param env: The evaluation environment\n",
+ " :param n_eval_episodes: Number of episode to evaluate the agent\n",
+ " :param policy: The agent\n",
+ " \"\"\"\n",
+ " episode_rewards = []\n",
+ " for episode in range(n_eval_episodes):\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ " total_rewards_ep = 0\n",
+ " \n",
+ " while done is False:\n",
+ " state = torch.Tensor(state).to(device)\n",
+ " action, _, _, _ = policy.get_action_and_value(state)\n",
+ " new_state, reward, done, info = env.step(action.cpu().numpy())\n",
+ " total_rewards_ep += reward \n",
+ " if done:\n",
+ " break\n",
+ " state = new_state\n",
+ " episode_rewards.append(total_rewards_ep)\n",
+ " mean_reward = np.mean(episode_rewards)\n",
+ " std_reward = np.std(episode_rewards)\n",
+ "\n",
+ " return mean_reward, std_reward\n",
+ "\n",
+ "\n",
+ "def record_video(env, policy, out_directory, fps=30):\n",
+ " images = [] \n",
+ " done = False\n",
+ " state = env.reset()\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " while not done:\n",
+ " state = torch.Tensor(state).to(device)\n",
+ " # Take the action (index) that have the maximum expected future reward given that state\n",
+ " action, _, _, _ = policy.get_action_and_value(state)\n",
+ " state, reward, done, info = env.step(action.cpu().numpy()) # We directly put next_state = state for recording logic\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)\n",
+ "\n",
+ "\n",
+ "def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters):\n",
+ " \"\"\"\n",
+ " Generate the model card for the Hub\n",
+ " :param model_name: name of the model\n",
+ " :env_id: name of the environment\n",
+ " :mean_reward: mean reward of the agent\n",
+ " :std_reward: standard deviation of the mean reward of the agent\n",
+ " :hyperparameters: training arguments\n",
+ " \"\"\"\n",
+ " # Step 1: Select the tags\n",
+ " metadata = generate_metadata(model_name, env_id, mean_reward, std_reward)\n",
+ "\n",
+ " # Transform the hyperparams namespace to string\n",
+ " converted_dict = vars(hyperparameters)\n",
+ " converted_str = str(converted_dict)\n",
+ " converted_str = converted_str.split(\", \")\n",
+ " converted_str = '\\n'.join(converted_str)\n",
+ " \n",
+ " # Step 2: Generate the model card\n",
+ " model_card = f\"\"\"\n",
+ " # PPO Agent Playing {env_id}\n",
+ "\n",
+ " This is a trained model of a PPO agent playing {env_id}.\n",
+ " \n",
+ " # Hyperparameters\n",
+ " ```python\n",
+ " {converted_str}\n",
+ " ```\n",
+ " \"\"\"\n",
+ " return model_card, metadata\n",
+ "\n",
+ "\n",
+ "def generate_metadata(model_name, env_id, mean_reward, std_reward):\n",
+ " \"\"\"\n",
+ " Define the tags for the model card\n",
+ " :param model_name: name of the model\n",
+ " :param env_id: name of the environment\n",
+ " :mean_reward: mean reward of the agent\n",
+ " :std_reward: standard deviation of the mean reward of the agent\n",
+ " \"\"\"\n",
+ " metadata = {}\n",
+ " metadata[\"tags\"] = [\n",
+ " env_id,\n",
+ " \"ppo\",\n",
+ " \"deep-reinforcement-learning\",\n",
+ " \"reinforcement-learning\",\n",
+ " \"custom-implementation\",\n",
+ " \"deep-rl-course\"\n",
+ " ]\n",
+ "\n",
+ " # Add metrics\n",
+ " eval = metadata_eval_result(\n",
+ " model_pretty_name=model_name,\n",
+ " task_pretty_name=\"reinforcement-learning\",\n",
+ " task_id=\"reinforcement-learning\",\n",
+ " metrics_pretty_name=\"mean_reward\",\n",
+ " metrics_id=\"mean_reward\",\n",
+ " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n",
+ " dataset_pretty_name=env_id,\n",
+ " dataset_id=env_id,\n",
+ " )\n",
+ "\n",
+ " # Merges both dictionaries\n",
+ " metadata = {**metadata, **eval}\n",
+ "\n",
+ " return metadata\n",
+ "\n",
+ "\n",
+ "def _save_model_card(local_path, generated_model_card, metadata):\n",
+ " \"\"\"Saves a model card for the repository.\n",
+ " :param local_path: repository directory\n",
+ " :param generated_model_card: model card generated by _generate_model_card()\n",
+ " :param metadata: metadata\n",
+ " \"\"\"\n",
+ " readme_path = local_path / \"README.md\"\n",
+ " readme = \"\"\n",
+ " if readme_path.exists():\n",
+ " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n",
+ " readme = f.read()\n",
+ " else:\n",
+ " readme = generated_model_card\n",
+ "\n",
+ " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n",
+ " f.write(readme)\n",
+ "\n",
+ " # Save our metrics to Readme metadata\n",
+ " metadata_save(readme_path, metadata)\n",
+ "\n",
+ "\n",
+ "def _add_logdir(local_path: Path, logdir: Path):\n",
+ " \"\"\"Adds a logdir to the repository.\n",
+ " :param local_path: repository directory\n",
+ " :param logdir: logdir directory\n",
+ " \"\"\"\n",
+ " if logdir.exists() and logdir.is_dir():\n",
+ " # Add the logdir to the repository under new dir called logs\n",
+ " repo_logdir = local_path / \"logs\"\n",
+ " \n",
+ " # Delete current logs if they exist\n",
+ " if repo_logdir.exists():\n",
+ " shutil.rmtree(repo_logdir)\n",
+ "\n",
+ " # Copy logdir into repo logdir\n",
+ " shutil.copytree(logdir, repo_logdir)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "TqX8z8_rooD6"
+ },
+ "source": [
+ "- Finally, we call this function at the end of the PPO training"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "I8V1vNiTo2hL"
+ },
+ "outputs": [],
+ "source": [
+ "# Create the evaluation environment\n",
+ "eval_env = gym.make(args.env_id)\n",
+ "\n",
+ "package_to_hub(repo_id = args.repo_id,\n",
+ " model = agent, # The model we want to save\n",
+ " hyperparameters = args,\n",
+ " eval_env = gym.make(args.env_id),\n",
+ " logs= f\"runs/{run_name}\",\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "muCCzed4o5TC"
+ },
+ "source": [
+ "- Here's what look the ppo.py final file"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "LviRdtXgo7kF"
+ },
+ "outputs": [],
+ "source": [
+ "# docs and experiment results can be found at https://docs.cleanrl.dev/rl-algorithms/ppo/#ppopy\n",
+ "\n",
+ "import argparse\n",
+ "import os\n",
+ "import random\n",
+ "import time\n",
+ "from distutils.util import strtobool\n",
+ "\n",
+ "import gym\n",
+ "import numpy as np\n",
+ "import torch\n",
+ "import torch.nn as nn\n",
+ "import torch.optim as optim\n",
+ "from torch.distributions.categorical import Categorical\n",
+ "from torch.utils.tensorboard import SummaryWriter\n",
+ "\n",
+ "from huggingface_hub import HfApi, upload_folder\n",
+ "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n",
+ "\n",
+ "from pathlib import Path\n",
+ "import datetime\n",
+ "import tempfile\n",
+ "import json\n",
+ "import shutil\n",
+ "import imageio\n",
+ "\n",
+ "from wasabi import Printer\n",
+ "msg = Printer()\n",
+ "\n",
+ "def parse_args():\n",
+ " # fmt: off\n",
+ " parser = argparse.ArgumentParser()\n",
+ " parser.add_argument(\"--exp-name\", type=str, default=os.path.basename(__file__).rstrip(\".py\"),\n",
+ " help=\"the name of this experiment\")\n",
+ " parser.add_argument(\"--seed\", type=int, default=1,\n",
+ " help=\"seed of the experiment\")\n",
+ " parser.add_argument(\"--torch-deterministic\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n",
+ " help=\"if toggled, `torch.backends.cudnn.deterministic=False`\")\n",
+ " parser.add_argument(\"--cuda\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n",
+ " help=\"if toggled, cuda will be enabled by default\")\n",
+ " parser.add_argument(\"--track\", type=lambda x: bool(strtobool(x)), default=False, nargs=\"?\", const=True,\n",
+ " help=\"if toggled, this experiment will be tracked with Weights and Biases\")\n",
+ " parser.add_argument(\"--wandb-project-name\", type=str, default=\"cleanRL\",\n",
+ " help=\"the wandb's project name\")\n",
+ " parser.add_argument(\"--wandb-entity\", type=str, default=None,\n",
+ " help=\"the entity (team) of wandb's project\")\n",
+ " parser.add_argument(\"--capture-video\", type=lambda x: bool(strtobool(x)), default=False, nargs=\"?\", const=True,\n",
+ " help=\"weather to capture videos of the agent performances (check out `videos` folder)\")\n",
+ "\n",
+ " # Algorithm specific arguments\n",
+ " parser.add_argument(\"--env-id\", type=str, default=\"CartPole-v1\",\n",
+ " help=\"the id of the environment\")\n",
+ " parser.add_argument(\"--total-timesteps\", type=int, default=50000,\n",
+ " help=\"total timesteps of the experiments\")\n",
+ " parser.add_argument(\"--learning-rate\", type=float, default=2.5e-4,\n",
+ " help=\"the learning rate of the optimizer\")\n",
+ " parser.add_argument(\"--num-envs\", type=int, default=4,\n",
+ " help=\"the number of parallel game environments\")\n",
+ " parser.add_argument(\"--num-steps\", type=int, default=128,\n",
+ " help=\"the number of steps to run in each environment per policy rollout\")\n",
+ " parser.add_argument(\"--anneal-lr\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n",
+ " help=\"Toggle learning rate annealing for policy and value networks\")\n",
+ " parser.add_argument(\"--gae\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n",
+ " help=\"Use GAE for advantage computation\")\n",
+ " parser.add_argument(\"--gamma\", type=float, default=0.99,\n",
+ " help=\"the discount factor gamma\")\n",
+ " parser.add_argument(\"--gae-lambda\", type=float, default=0.95,\n",
+ " help=\"the lambda for the general advantage estimation\")\n",
+ " parser.add_argument(\"--num-minibatches\", type=int, default=4,\n",
+ " help=\"the number of mini-batches\")\n",
+ " parser.add_argument(\"--update-epochs\", type=int, default=4,\n",
+ " help=\"the K epochs to update the policy\")\n",
+ " parser.add_argument(\"--norm-adv\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n",
+ " help=\"Toggles advantages normalization\")\n",
+ " parser.add_argument(\"--clip-coef\", type=float, default=0.2,\n",
+ " help=\"the surrogate clipping coefficient\")\n",
+ " parser.add_argument(\"--clip-vloss\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n",
+ " help=\"Toggles whether or not to use a clipped loss for the value function, as per the paper.\")\n",
+ " parser.add_argument(\"--ent-coef\", type=float, default=0.01,\n",
+ " help=\"coefficient of the entropy\")\n",
+ " parser.add_argument(\"--vf-coef\", type=float, default=0.5,\n",
+ " help=\"coefficient of the value function\")\n",
+ " parser.add_argument(\"--max-grad-norm\", type=float, default=0.5,\n",
+ " help=\"the maximum norm for the gradient clipping\")\n",
+ " parser.add_argument(\"--target-kl\", type=float, default=None,\n",
+ " help=\"the target KL divergence threshold\")\n",
+ " \n",
+ " # Adding HuggingFace argument\n",
+ " parser.add_argument(\"--repo-id\", type=str, default=\"ThomasSimonini/ppo-CartPole-v1\", help=\"id of the model repository from the Hugging Face Hub {username/repo_name}\")\n",
+ "\n",
+ " args = parser.parse_args()\n",
+ " args.batch_size = int(args.num_envs * args.num_steps)\n",
+ " args.minibatch_size = int(args.batch_size // args.num_minibatches)\n",
+ " # fmt: on\n",
+ " return args\n",
+ "\n",
+ "def package_to_hub(repo_id, \n",
+ " model,\n",
+ " hyperparameters,\n",
+ " eval_env,\n",
+ " video_fps=30,\n",
+ " commit_message=\"Push agent to the Hub\",\n",
+ " token= None,\n",
+ " logs=None\n",
+ " ):\n",
+ " \"\"\"\n",
+ " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n",
+ " This method does the complete pipeline:\n",
+ " - It evaluates the model\n",
+ " - It generates the model card\n",
+ " - It generates a replay video of the agent\n",
+ " - It pushes everything to the hub\n",
+ " :param repo_id: id of the model repository from the Hugging Face Hub\n",
+ " :param model: trained model\n",
+ " :param eval_env: environment used to evaluate the agent\n",
+ " :param fps: number of fps for rendering the video\n",
+ " :param commit_message: commit message\n",
+ " :param logs: directory on local machine of tensorboard logs you'd like to upload\n",
+ " \"\"\"\n",
+ " msg.info(\n",
+ " \"This function will save, evaluate, generate a video of your agent, \"\n",
+ " \"create a model card and push everything to the hub. \"\n",
+ " \"It might take up to 1min. \\n \"\n",
+ " \"This is a work in progress: if you encounter a bug, please open an issue.\"\n",
+ " )\n",
+ " # Step 1: Clone or create the repo\n",
+ " repo_url = HfApi().create_repo(\n",
+ " repo_id=repo_id,\n",
+ " token=token,\n",
+ " private=False,\n",
+ " exist_ok=True,\n",
+ " )\n",
+ " \n",
+ " with tempfile.TemporaryDirectory() as tmpdirname:\n",
+ " tmpdirname = Path(tmpdirname)\n",
+ "\n",
+ " # Step 2: Save the model\n",
+ " torch.save(model.state_dict(), tmpdirname / \"model.pt\")\n",
+ " \n",
+ " # Step 3: Evaluate the model and build JSON\n",
+ " mean_reward, std_reward = _evaluate_agent(eval_env, \n",
+ " 10, \n",
+ " model)\n",
+ "\n",
+ " # First get datetime\n",
+ " eval_datetime = datetime.datetime.now()\n",
+ " eval_form_datetime = eval_datetime.isoformat()\n",
+ "\n",
+ " evaluate_data = {\n",
+ " \"env_id\": hyperparameters.env_id, \n",
+ " \"mean_reward\": mean_reward,\n",
+ " \"std_reward\": std_reward,\n",
+ " \"n_evaluation_episodes\": 10,\n",
+ " \"eval_datetime\": eval_form_datetime,\n",
+ " }\n",
+ " \n",
+ " # Write a JSON file\n",
+ " with open(tmpdirname / \"results.json\", \"w\") as outfile:\n",
+ " json.dump(evaluate_data, outfile)\n",
+ "\n",
+ " # Step 4: Generate a video\n",
+ " video_path = tmpdirname / \"replay.mp4\"\n",
+ " record_video(eval_env, model, video_path, video_fps)\n",
+ " \n",
+ " # Step 5: Generate the model card\n",
+ " generated_model_card, metadata = _generate_model_card(\"PPO\", hyperparameters.env_id, mean_reward, std_reward, hyperparameters)\n",
+ " _save_model_card(tmpdirname, generated_model_card, metadata)\n",
+ "\n",
+ " # Step 6: Add logs if needed\n",
+ " if logs:\n",
+ " _add_logdir(tmpdirname, Path(logs))\n",
+ " \n",
+ " msg.info(f\"Pushing repo {repo_id} to the Hugging Face Hub\")\n",
+ " \n",
+ " repo_url = upload_folder(\n",
+ " repo_id=repo_id,\n",
+ " folder_path=tmpdirname,\n",
+ " path_in_repo=\"\",\n",
+ " commit_message=commit_message,\n",
+ " token=token,\n",
+ " )\n",
+ "\n",
+ " msg.info(f\"Your model is pushed to the Hub. You can view your model here: {repo_url}\")\n",
+ " return repo_url\n",
+ "\n",
+ "def _evaluate_agent(env, n_eval_episodes, policy):\n",
+ " \"\"\"\n",
+ " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n",
+ " :param env: The evaluation environment\n",
+ " :param n_eval_episodes: Number of episode to evaluate the agent\n",
+ " :param policy: The agent\n",
+ " \"\"\"\n",
+ " episode_rewards = []\n",
+ " for episode in range(n_eval_episodes):\n",
+ " state = env.reset()\n",
+ " step = 0\n",
+ " done = False\n",
+ " total_rewards_ep = 0\n",
+ " \n",
+ " while done is False:\n",
+ " state = torch.Tensor(state).to(device)\n",
+ " action, _, _, _ = policy.get_action_and_value(state)\n",
+ " new_state, reward, done, info = env.step(action.cpu().numpy())\n",
+ " total_rewards_ep += reward \n",
+ " if done:\n",
+ " break\n",
+ " state = new_state\n",
+ " episode_rewards.append(total_rewards_ep)\n",
+ " mean_reward = np.mean(episode_rewards)\n",
+ " std_reward = np.std(episode_rewards)\n",
+ "\n",
+ " return mean_reward, std_reward\n",
+ "\n",
+ "\n",
+ "def record_video(env, policy, out_directory, fps=30):\n",
+ " images = [] \n",
+ " done = False\n",
+ " state = env.reset()\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " while not done:\n",
+ " state = torch.Tensor(state).to(device)\n",
+ " # Take the action (index) that have the maximum expected future reward given that state\n",
+ " action, _, _, _ = policy.get_action_and_value(state)\n",
+ " state, reward, done, info = env.step(action.cpu().numpy()) # We directly put next_state = state for recording logic\n",
+ " img = env.render(mode='rgb_array')\n",
+ " images.append(img)\n",
+ " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)\n",
+ "\n",
+ "\n",
+ "def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters):\n",
+ " \"\"\"\n",
+ " Generate the model card for the Hub\n",
+ " :param model_name: name of the model\n",
+ " :env_id: name of the environment\n",
+ " :mean_reward: mean reward of the agent\n",
+ " :std_reward: standard deviation of the mean reward of the agent\n",
+ " :hyperparameters: training arguments\n",
+ " \"\"\"\n",
+ " # Step 1: Select the tags\n",
+ " metadata = generate_metadata(model_name, env_id, mean_reward, std_reward)\n",
+ "\n",
+ " # Transform the hyperparams namespace to string\n",
+ " converted_dict = vars(hyperparameters)\n",
+ " converted_str = str(converted_dict)\n",
+ " converted_str = converted_str.split(\", \")\n",
+ " converted_str = '\\n'.join(converted_str)\n",
+ " \n",
+ " # Step 2: Generate the model card\n",
+ " model_card = f\"\"\"\n",
+ " # PPO Agent Playing {env_id}\n",
+ "\n",
+ " This is a trained model of a PPO agent playing {env_id}.\n",
+ " \n",
+ " # Hyperparameters\n",
+ " ```python\n",
+ " {converted_str}\n",
+ " ```\n",
+ " \"\"\"\n",
+ " return model_card, metadata\n",
+ "\n",
+ "def generate_metadata(model_name, env_id, mean_reward, std_reward):\n",
+ " \"\"\"\n",
+ " Define the tags for the model card\n",
+ " :param model_name: name of the model\n",
+ " :param env_id: name of the environment\n",
+ " :mean_reward: mean reward of the agent\n",
+ " :std_reward: standard deviation of the mean reward of the agent\n",
+ " \"\"\"\n",
+ " metadata = {}\n",
+ " metadata[\"tags\"] = [\n",
+ " env_id,\n",
+ " \"ppo\",\n",
+ " \"deep-reinforcement-learning\",\n",
+ " \"reinforcement-learning\",\n",
+ " \"custom-implementation\",\n",
+ " \"deep-rl-course\"\n",
+ " ]\n",
+ "\n",
+ " # Add metrics\n",
+ " eval = metadata_eval_result(\n",
+ " model_pretty_name=model_name,\n",
+ " task_pretty_name=\"reinforcement-learning\",\n",
+ " task_id=\"reinforcement-learning\",\n",
+ " metrics_pretty_name=\"mean_reward\",\n",
+ " metrics_id=\"mean_reward\",\n",
+ " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n",
+ " dataset_pretty_name=env_id,\n",
+ " dataset_id=env_id,\n",
+ " )\n",
+ "\n",
+ " # Merges both dictionaries\n",
+ " metadata = {**metadata, **eval}\n",
+ "\n",
+ " return metadata\n",
+ "\n",
+ "def _save_model_card(local_path, generated_model_card, metadata):\n",
+ " \"\"\"Saves a model card for the repository.\n",
+ " :param local_path: repository directory\n",
+ " :param generated_model_card: model card generated by _generate_model_card()\n",
+ " :param metadata: metadata\n",
+ " \"\"\"\n",
+ " readme_path = local_path / \"README.md\"\n",
+ " readme = \"\"\n",
+ " if readme_path.exists():\n",
+ " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n",
+ " readme = f.read()\n",
+ " else:\n",
+ " readme = generated_model_card\n",
+ "\n",
+ " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n",
+ " f.write(readme)\n",
+ "\n",
+ " # Save our metrics to Readme metadata\n",
+ " metadata_save(readme_path, metadata)\n",
+ "\n",
+ "def _add_logdir(local_path: Path, logdir: Path):\n",
+ " \"\"\"Adds a logdir to the repository.\n",
+ " :param local_path: repository directory\n",
+ " :param logdir: logdir directory\n",
+ " \"\"\"\n",
+ " if logdir.exists() and logdir.is_dir():\n",
+ " # Add the logdir to the repository under new dir called logs\n",
+ " repo_logdir = local_path / \"logs\"\n",
+ " \n",
+ " # Delete current logs if they exist\n",
+ " if repo_logdir.exists():\n",
+ " shutil.rmtree(repo_logdir)\n",
+ "\n",
+ " # Copy logdir into repo logdir\n",
+ " shutil.copytree(logdir, repo_logdir)\n",
+ "\n",
+ "def make_env(env_id, seed, idx, capture_video, run_name):\n",
+ " def thunk():\n",
+ " env = gym.make(env_id)\n",
+ " env = gym.wrappers.RecordEpisodeStatistics(env)\n",
+ " if capture_video:\n",
+ " if idx == 0:\n",
+ " env = gym.wrappers.RecordVideo(env, f\"videos/{run_name}\")\n",
+ " env.seed(seed)\n",
+ " env.action_space.seed(seed)\n",
+ " env.observation_space.seed(seed)\n",
+ " return env\n",
+ "\n",
+ " return thunk\n",
+ "\n",
+ "\n",
+ "def layer_init(layer, std=np.sqrt(2), bias_const=0.0):\n",
+ " torch.nn.init.orthogonal_(layer.weight, std)\n",
+ " torch.nn.init.constant_(layer.bias, bias_const)\n",
+ " return layer\n",
+ "\n",
+ "\n",
+ "class Agent(nn.Module):\n",
+ " def __init__(self, envs):\n",
+ " super().__init__()\n",
+ " self.critic = nn.Sequential(\n",
+ " layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),\n",
+ " nn.Tanh(),\n",
+ " layer_init(nn.Linear(64, 64)),\n",
+ " nn.Tanh(),\n",
+ " layer_init(nn.Linear(64, 1), std=1.0),\n",
+ " )\n",
+ " self.actor = nn.Sequential(\n",
+ " layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),\n",
+ " nn.Tanh(),\n",
+ " layer_init(nn.Linear(64, 64)),\n",
+ " nn.Tanh(),\n",
+ " layer_init(nn.Linear(64, envs.single_action_space.n), std=0.01),\n",
+ " )\n",
+ "\n",
+ " def get_value(self, x):\n",
+ " return self.critic(x)\n",
+ "\n",
+ " def get_action_and_value(self, x, action=None):\n",
+ " logits = self.actor(x)\n",
+ " probs = Categorical(logits=logits)\n",
+ " if action is None:\n",
+ " action = probs.sample()\n",
+ " return action, probs.log_prob(action), probs.entropy(), self.critic(x)\n",
+ "\n",
+ "\n",
+ "if __name__ == \"__main__\":\n",
+ " args = parse_args()\n",
+ " run_name = f\"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}\"\n",
+ " if args.track:\n",
+ " import wandb\n",
+ "\n",
+ " wandb.init(\n",
+ " project=args.wandb_project_name,\n",
+ " entity=args.wandb_entity,\n",
+ " sync_tensorboard=True,\n",
+ " config=vars(args),\n",
+ " name=run_name,\n",
+ " monitor_gym=True,\n",
+ " save_code=True,\n",
+ " )\n",
+ " writer = SummaryWriter(f\"runs/{run_name}\")\n",
+ " writer.add_text(\n",
+ " \"hyperparameters\",\n",
+ " \"|param|value|\\n|-|-|\\n%s\" % (\"\\n\".join([f\"|{key}|{value}|\" for key, value in vars(args).items()])),\n",
+ " )\n",
+ "\n",
+ " # TRY NOT TO MODIFY: seeding\n",
+ " random.seed(args.seed)\n",
+ " np.random.seed(args.seed)\n",
+ " torch.manual_seed(args.seed)\n",
+ " torch.backends.cudnn.deterministic = args.torch_deterministic\n",
+ "\n",
+ " device = torch.device(\"cuda\" if torch.cuda.is_available() and args.cuda else \"cpu\")\n",
+ "\n",
+ " # env setup\n",
+ " envs = gym.vector.SyncVectorEnv(\n",
+ " [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)]\n",
+ " )\n",
+ " assert isinstance(envs.single_action_space, gym.spaces.Discrete), \"only discrete action space is supported\"\n",
+ "\n",
+ " agent = Agent(envs).to(device)\n",
+ " optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5)\n",
+ "\n",
+ " # ALGO Logic: Storage setup\n",
+ " obs = torch.zeros((args.num_steps, args.num_envs) + envs.single_observation_space.shape).to(device)\n",
+ " actions = torch.zeros((args.num_steps, args.num_envs) + envs.single_action_space.shape).to(device)\n",
+ " logprobs = torch.zeros((args.num_steps, args.num_envs)).to(device)\n",
+ " rewards = torch.zeros((args.num_steps, args.num_envs)).to(device)\n",
+ " dones = torch.zeros((args.num_steps, args.num_envs)).to(device)\n",
+ " values = torch.zeros((args.num_steps, args.num_envs)).to(device)\n",
+ "\n",
+ " # TRY NOT TO MODIFY: start the game\n",
+ " global_step = 0\n",
+ " start_time = time.time()\n",
+ " next_obs = torch.Tensor(envs.reset()).to(device)\n",
+ " next_done = torch.zeros(args.num_envs).to(device)\n",
+ " num_updates = args.total_timesteps // args.batch_size\n",
+ "\n",
+ " for update in range(1, num_updates + 1):\n",
+ " # Annealing the rate if instructed to do so.\n",
+ " if args.anneal_lr:\n",
+ " frac = 1.0 - (update - 1.0) / num_updates\n",
+ " lrnow = frac * args.learning_rate\n",
+ " optimizer.param_groups[0][\"lr\"] = lrnow\n",
+ "\n",
+ " for step in range(0, args.num_steps):\n",
+ " global_step += 1 * args.num_envs\n",
+ " obs[step] = next_obs\n",
+ " dones[step] = next_done\n",
+ "\n",
+ " # ALGO LOGIC: action logic\n",
+ " with torch.no_grad():\n",
+ " action, logprob, _, value = agent.get_action_and_value(next_obs)\n",
+ " values[step] = value.flatten()\n",
+ " actions[step] = action\n",
+ " logprobs[step] = logprob\n",
+ "\n",
+ " # TRY NOT TO MODIFY: execute the game and log data.\n",
+ " next_obs, reward, done, info = envs.step(action.cpu().numpy())\n",
+ " rewards[step] = torch.tensor(reward).to(device).view(-1)\n",
+ " next_obs, next_done = torch.Tensor(next_obs).to(device), torch.Tensor(done).to(device)\n",
+ "\n",
+ " for item in info:\n",
+ " if \"episode\" in item.keys():\n",
+ " print(f\"global_step={global_step}, episodic_return={item['episode']['r']}\")\n",
+ " writer.add_scalar(\"charts/episodic_return\", item[\"episode\"][\"r\"], global_step)\n",
+ " writer.add_scalar(\"charts/episodic_length\", item[\"episode\"][\"l\"], global_step)\n",
+ " break\n",
+ "\n",
+ " # bootstrap value if not done\n",
+ " with torch.no_grad():\n",
+ " next_value = agent.get_value(next_obs).reshape(1, -1)\n",
+ " if args.gae:\n",
+ " advantages = torch.zeros_like(rewards).to(device)\n",
+ " lastgaelam = 0\n",
+ " for t in reversed(range(args.num_steps)):\n",
+ " if t == args.num_steps - 1:\n",
+ " nextnonterminal = 1.0 - next_done\n",
+ " nextvalues = next_value\n",
+ " else:\n",
+ " nextnonterminal = 1.0 - dones[t + 1]\n",
+ " nextvalues = values[t + 1]\n",
+ " delta = rewards[t] + args.gamma * nextvalues * nextnonterminal - values[t]\n",
+ " advantages[t] = lastgaelam = delta + args.gamma * args.gae_lambda * nextnonterminal * lastgaelam\n",
+ " returns = advantages + values\n",
+ " else:\n",
+ " returns = torch.zeros_like(rewards).to(device)\n",
+ " for t in reversed(range(args.num_steps)):\n",
+ " if t == args.num_steps - 1:\n",
+ " nextnonterminal = 1.0 - next_done\n",
+ " next_return = next_value\n",
+ " else:\n",
+ " nextnonterminal = 1.0 - dones[t + 1]\n",
+ " next_return = returns[t + 1]\n",
+ " returns[t] = rewards[t] + args.gamma * nextnonterminal * next_return\n",
+ " advantages = returns - values\n",
+ "\n",
+ " # flatten the batch\n",
+ " b_obs = obs.reshape((-1,) + envs.single_observation_space.shape)\n",
+ " b_logprobs = logprobs.reshape(-1)\n",
+ " b_actions = actions.reshape((-1,) + envs.single_action_space.shape)\n",
+ " b_advantages = advantages.reshape(-1)\n",
+ " b_returns = returns.reshape(-1)\n",
+ " b_values = values.reshape(-1)\n",
+ "\n",
+ " # Optimizing the policy and value network\n",
+ " b_inds = np.arange(args.batch_size)\n",
+ " clipfracs = []\n",
+ " for epoch in range(args.update_epochs):\n",
+ " np.random.shuffle(b_inds)\n",
+ " for start in range(0, args.batch_size, args.minibatch_size):\n",
+ " end = start + args.minibatch_size\n",
+ " mb_inds = b_inds[start:end]\n",
+ "\n",
+ " _, newlogprob, entropy, newvalue = agent.get_action_and_value(b_obs[mb_inds], b_actions.long()[mb_inds])\n",
+ " logratio = newlogprob - b_logprobs[mb_inds]\n",
+ " ratio = logratio.exp()\n",
+ "\n",
+ " with torch.no_grad():\n",
+ " # calculate approx_kl http://joschu.net/blog/kl-approx.html\n",
+ " old_approx_kl = (-logratio).mean()\n",
+ " approx_kl = ((ratio - 1) - logratio).mean()\n",
+ " clipfracs += [((ratio - 1.0).abs() > args.clip_coef).float().mean().item()]\n",
+ "\n",
+ " mb_advantages = b_advantages[mb_inds]\n",
+ " if args.norm_adv:\n",
+ " mb_advantages = (mb_advantages - mb_advantages.mean()) / (mb_advantages.std() + 1e-8)\n",
+ "\n",
+ " # Policy loss\n",
+ " pg_loss1 = -mb_advantages * ratio\n",
+ " pg_loss2 = -mb_advantages * torch.clamp(ratio, 1 - args.clip_coef, 1 + args.clip_coef)\n",
+ " pg_loss = torch.max(pg_loss1, pg_loss2).mean()\n",
+ "\n",
+ " # Value loss\n",
+ " newvalue = newvalue.view(-1)\n",
+ " if args.clip_vloss:\n",
+ " v_loss_unclipped = (newvalue - b_returns[mb_inds]) ** 2\n",
+ " v_clipped = b_values[mb_inds] + torch.clamp(\n",
+ " newvalue - b_values[mb_inds],\n",
+ " -args.clip_coef,\n",
+ " args.clip_coef,\n",
+ " )\n",
+ " v_loss_clipped = (v_clipped - b_returns[mb_inds]) ** 2\n",
+ " v_loss_max = torch.max(v_loss_unclipped, v_loss_clipped)\n",
+ " v_loss = 0.5 * v_loss_max.mean()\n",
+ " else:\n",
+ " v_loss = 0.5 * ((newvalue - b_returns[mb_inds]) ** 2).mean()\n",
+ "\n",
+ " entropy_loss = entropy.mean()\n",
+ " loss = pg_loss - args.ent_coef * entropy_loss + v_loss * args.vf_coef\n",
+ "\n",
+ " optimizer.zero_grad()\n",
+ " loss.backward()\n",
+ " nn.utils.clip_grad_norm_(agent.parameters(), args.max_grad_norm)\n",
+ " optimizer.step()\n",
+ "\n",
+ " if args.target_kl is not None:\n",
+ " if approx_kl > args.target_kl:\n",
+ " break\n",
+ "\n",
+ " y_pred, y_true = b_values.cpu().numpy(), b_returns.cpu().numpy()\n",
+ " var_y = np.var(y_true)\n",
+ " explained_var = np.nan if var_y == 0 else 1 - np.var(y_true - y_pred) / var_y\n",
+ "\n",
+ " # TRY NOT TO MODIFY: record rewards for plotting purposes\n",
+ " writer.add_scalar(\"charts/learning_rate\", optimizer.param_groups[0][\"lr\"], global_step)\n",
+ " writer.add_scalar(\"losses/value_loss\", v_loss.item(), global_step)\n",
+ " writer.add_scalar(\"losses/policy_loss\", pg_loss.item(), global_step)\n",
+ " writer.add_scalar(\"losses/entropy\", entropy_loss.item(), global_step)\n",
+ " writer.add_scalar(\"losses/old_approx_kl\", old_approx_kl.item(), global_step)\n",
+ " writer.add_scalar(\"losses/approx_kl\", approx_kl.item(), global_step)\n",
+ " writer.add_scalar(\"losses/clipfrac\", np.mean(clipfracs), global_step)\n",
+ " writer.add_scalar(\"losses/explained_variance\", explained_var, global_step)\n",
+ " print(\"SPS:\", int(global_step / (time.time() - start_time)))\n",
+ " writer.add_scalar(\"charts/SPS\", int(global_step / (time.time() - start_time)), global_step)\n",
+ "\n",
+ " envs.close()\n",
+ " writer.close()\n",
+ "\n",
+ " # Create the evaluation environment\n",
+ " eval_env = gym.make(args.env_id)\n",
+ "\n",
+ " package_to_hub(repo_id = args.repo_id,\n",
+ " model = agent, # The model we want to save\n",
+ " hyperparameters = args,\n",
+ " eval_env = gym.make(args.env_id),\n",
+ " logs= f\"runs/{run_name}\",\n",
+ " )\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "JquRrWytA6eo"
+ },
+ "source": [
+ "To be able to share your model with the community there are three more steps to follow:\n",
+ "\n",
+ "1οΈβ£ (If it's not already done) create an account to HF β‘ https://huggingface.co/join\n",
+ "\n",
+ "2οΈβ£ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
+ "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
+ "\n",
+ "![\"Create](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/create-token.jpg\")
\n",
+ "\n",
+ "- Copy the token \n",
+ "- Run the cell below and paste the token"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "GZiFBBlzxzxY"
+ },
+ "outputs": [],
+ "source": [
+ "from huggingface_hub import notebook_login\n",
+ "notebook_login()\n",
+ "!git config --global credential.helper store"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "_tsf2uv0g_4p"
+ },
+ "source": [
+ "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jRqkGvk7pFQ6"
+ },
+ "source": [
+ "## Let's start the training π₯\n",
+ "- Now that you've coded from scratch PPO and added the Hugging Face Integration, we're ready to start the training π₯"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0tmEArP8ug2l"
+ },
+ "source": [
+ "- First, you need to copy all your code to a file you create called `ppo.py`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "![\"PPO\"/](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit9/step1.png\")
"
+ ],
+ "metadata": {
+ "id": "Sq0My0LOjPYR"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "![\"PPO\"/](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit9/step2.png\")
"
+ ],
+ "metadata": {
+ "id": "A8C-Q5ZyjUe3"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VrS80GmMu_j5"
+ },
+ "source": [
+ "- Now we just need to run this python script using `python .py` with the additional parameters we defined with `argparse`\n",
+ "\n",
+ "- You should modify more hyperparameters otherwise the training will not be super stable."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!python ppo.py --env-id=\"LunarLander-v2\" --repo-id=\"YOUR_REPO_ID\" --total-timesteps=50000"
+ ],
+ "metadata": {
+ "id": "KXLih6mKseBs"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "eVsVJ5AdqLE7"
+ },
+ "source": [
+ "## Some additional challenges π\n",
+ "The best way to learn **is to try things by your own**! Why not trying another environment?\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "nYdl758GqLXT"
+ },
+ "source": [
+ "See you on Unit 8, part 2 where we going to train agents to play Doom π₯\n",
+ "## Keep learning, stay awesome π€"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "private_outputs": true,
+ "provenance": [],
+ "include_colab_link": true
+ },
+ "gpuClass": "standard",
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python"
+ },
+ "accelerator": "GPU"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file