From 83046bbf6c5ba7ba0403f98a10c43e380d90d359 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 17 Feb 2023 15:33:24 +0100 Subject: [PATCH 01/14] Update introduction.mdx --- units/en/unit8/introduction.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/units/en/unit8/introduction.mdx b/units/en/unit8/introduction.mdx index 7c74578..7657ec1 100644 --- a/units/en/unit8/introduction.mdx +++ b/units/en/unit8/introduction.mdx @@ -17,7 +17,7 @@ This Unit is in two parts:
Environment -
This is the environments you're going to use to train your agents: VizDoom and GodotRL environments
+
This is the environments you're going to use to train your agents: VizDoom environments
Sounds exciting? Let's get started! 🚀 From c91e6dd546d4a4ae67c5f66fca6650cfff5c0f9e Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 17 Feb 2023 15:39:25 +0100 Subject: [PATCH 02/14] Create introduction-sf --- units/en/unit8/introduction-sf | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 units/en/unit8/introduction-sf diff --git a/units/en/unit8/introduction-sf b/units/en/unit8/introduction-sf new file mode 100644 index 0000000..b49aafe --- /dev/null +++ b/units/en/unit8/introduction-sf @@ -0,0 +1,10 @@ +# Introduction to PPO with Sample-Factory + +thumbnail + +In this second part of Unit 8, we'll get deeper into PPO optimization by using [Sample-Factory](https://samplefactory.dev/), an asynchronous implementation of the PPO algorithm, to train our agent playing [vizdoom](https://vizdoom.cs.put.edu.pl/) (an open source version of Doom). +During the notebook, you'll train your agent to play Health Gathering level, where our agent needs to collect health packs to avoid dying. And after that, you'll be able to train your agent to play more complex versions of the levels, such as Deathmatch. + +Environment + +Sounds exciting? Let's get started! 🚀 From 770333c20b16c1a16fb13b97e833cfd6263bb667 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 17 Feb 2023 15:40:17 +0100 Subject: [PATCH 03/14] Update _toctree.yml --- units/en/_toctree.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/units/en/_toctree.yml b/units/en/_toctree.yml index a8c93b8..59e6c95 100644 --- a/units/en/_toctree.yml +++ b/units/en/_toctree.yml @@ -194,6 +194,10 @@ title: Conclusion - local: unit8/additional-readings title: Additional Readings +- title: Unit 8. Part 2 Proximal Policy Optimization (PPO) with Doom + sections: + - local: unit8/introduction-sf + title: Introduction - title: What's next? New Units Publishing Schedule sections: - local: communication/publishing-schedule From dd3a20fd5768731a2559ee3ad90e5f96019db76a Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 17 Feb 2023 15:46:14 +0100 Subject: [PATCH 04/14] Rename introduction-sf to introduction-sf.mdx --- units/en/unit8/{introduction-sf => introduction-sf.mdx} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename units/en/unit8/{introduction-sf => introduction-sf.mdx} (100%) diff --git a/units/en/unit8/introduction-sf b/units/en/unit8/introduction-sf.mdx similarity index 100% rename from units/en/unit8/introduction-sf rename to units/en/unit8/introduction-sf.mdx From 6684987502fc8fe525b86efa275a6778cf871edd Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 17 Feb 2023 15:46:49 +0100 Subject: [PATCH 05/14] Update introduction-sf.mdx --- units/en/unit8/introduction-sf.mdx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/units/en/unit8/introduction-sf.mdx b/units/en/unit8/introduction-sf.mdx index b49aafe..2fd45f4 100644 --- a/units/en/unit8/introduction-sf.mdx +++ b/units/en/unit8/introduction-sf.mdx @@ -2,8 +2,9 @@ thumbnail -In this second part of Unit 8, we'll get deeper into PPO optimization by using [Sample-Factory](https://samplefactory.dev/), an asynchronous implementation of the PPO algorithm, to train our agent playing [vizdoom](https://vizdoom.cs.put.edu.pl/) (an open source version of Doom). -During the notebook, you'll train your agent to play Health Gathering level, where our agent needs to collect health packs to avoid dying. And after that, you'll be able to train your agent to play more complex versions of the levels, such as Deathmatch. +In this second part of Unit 8, we'll get deeper into PPO optimization by using [Sample-Factory](https://samplefactory.dev/), an **asynchronous implementation of the PPO algorithm**, to train our agent playing [vizdoom](https://vizdoom.cs.put.edu.pl/) (an open source version of Doom). + +During the notebook, **you'll train your agent to play Health Gathering level**, where our agent needs to collect health packs to avoid dying. And after that, you'll be able to **train your agent to play more complex versions of the levels, such as Deathmatch**. Environment From 60afe9d8fa56b872d2a57177980bb069ddb73ec3 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 17 Feb 2023 15:50:44 +0100 Subject: [PATCH 06/14] Create conclusion-sf.mdx --- units/en/unit8/conclusion-sf.mdx | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 units/en/unit8/conclusion-sf.mdx diff --git a/units/en/unit8/conclusion-sf.mdx b/units/en/unit8/conclusion-sf.mdx new file mode 100644 index 0000000..645d361 --- /dev/null +++ b/units/en/unit8/conclusion-sf.mdx @@ -0,0 +1,11 @@ +# Conclusion + +That's all for today. Congrats on finishing this Unit and the tutorial! + +Now that you've successfully trained your Doom agent, why not try deathmatch? We will provide you with some advice in the hands-on. But remember, that's a much more complex level than the one you've just trained. **But it's a nice experiment, and I advise you to try it.** + +If you do it, don't hesitate to share your model in the `#rl-i-made-this` channel in our [discord server](https://www.hf.co/join/discord). + +See you the last Unit 🔥, + +## Keep Learning, Stay awesome 🤗 From d5981c90446bf83f3d228565c7fc3ecf48e203eb Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 17 Feb 2023 15:51:07 +0100 Subject: [PATCH 07/14] Create hands-on-sf.mdx --- units/en/unit8/hands-on-sf.mdx | 1 + 1 file changed, 1 insertion(+) create mode 100644 units/en/unit8/hands-on-sf.mdx diff --git a/units/en/unit8/hands-on-sf.mdx b/units/en/unit8/hands-on-sf.mdx new file mode 100644 index 0000000..a93b860 --- /dev/null +++ b/units/en/unit8/hands-on-sf.mdx @@ -0,0 +1 @@ +# Hands-on Doom From 49b3e0b741b217ee2f20c37a9da9ebc567d18776 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Fri, 17 Feb 2023 15:52:11 +0100 Subject: [PATCH 08/14] Update _toctree.yml --- units/en/_toctree.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/units/en/_toctree.yml b/units/en/_toctree.yml index 59e6c95..2c1e2fd 100644 --- a/units/en/_toctree.yml +++ b/units/en/_toctree.yml @@ -198,6 +198,10 @@ sections: - local: unit8/introduction-sf title: Introduction + - local: unit8/hands-on-sf + title: PPO with Sample Factory and Doom + - local: unit8/conclusion-sf + title: Conclusion - title: What's next? New Units Publishing Schedule sections: - local: communication/publishing-schedule From 7220220848ffb2015900c7af804ab4ab4834e388 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Tue, 21 Feb 2023 07:15:41 +0100 Subject: [PATCH 09/14] Add Ed author --- units/en/unit8/introduction-sf.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/units/en/unit8/introduction-sf.mdx b/units/en/unit8/introduction-sf.mdx index 2fd45f4..486b416 100644 --- a/units/en/unit8/introduction-sf.mdx +++ b/units/en/unit8/introduction-sf.mdx @@ -4,8 +4,10 @@ In this second part of Unit 8, we'll get deeper into PPO optimization by using [Sample-Factory](https://samplefactory.dev/), an **asynchronous implementation of the PPO algorithm**, to train our agent playing [vizdoom](https://vizdoom.cs.put.edu.pl/) (an open source version of Doom). -During the notebook, **you'll train your agent to play Health Gathering level**, where our agent needs to collect health packs to avoid dying. And after that, you'll be able to **train your agent to play more complex versions of the levels, such as Deathmatch**. +During the notebook, **you'll train your agent to play Health Gathering level**, where our agent must collect health packs to avoid dying. And after that, you can **train your agent to play more complex versions of the levels, such as Deathmatch**. Environment Sounds exciting? Let's get started! 🚀 + +The hands-on is made by [Edward Beeching](https://twitter.com/edwardbeeching), a Machine Learning Research Scientist at Hugging Face. He worked on Godot Reinforcement Learning Agents, an open-source interface for developing environments and agents in the Godot Game Engine. From 012313d5b1ddfae501273ea3dfeaeef087495a0f Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Tue, 21 Feb 2023 07:17:01 +0100 Subject: [PATCH 10/14] Update conclusion-sf.mdx --- units/en/unit8/conclusion-sf.mdx | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/units/en/unit8/conclusion-sf.mdx b/units/en/unit8/conclusion-sf.mdx index 645d361..ac9893f 100644 --- a/units/en/unit8/conclusion-sf.mdx +++ b/units/en/unit8/conclusion-sf.mdx @@ -2,10 +2,12 @@ That's all for today. Congrats on finishing this Unit and the tutorial! -Now that you've successfully trained your Doom agent, why not try deathmatch? We will provide you with some advice in the hands-on. But remember, that's a much more complex level than the one you've just trained. **But it's a nice experiment, and I advise you to try it.** +Now that you've successfully trained your Doom agent, why not try deathmatch? But remember, that's a much more complex level than the one you've just trained. **But it's a nice experiment, and I advise you to try it.** If you do it, don't hesitate to share your model in the `#rl-i-made-this` channel in our [discord server](https://www.hf.co/join/discord). -See you the last Unit 🔥, +This concludes the last unit. But we are not finished yet! 🤗 The following **bonus unit include some of the most interesting, advanced and cutting edge work in Deep Reinforcement Learning**. + +See you next time 🔥, ## Keep Learning, Stay awesome 🤗 From 444305a270385dcfc9733ac2ab94f4a64da89d6d Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Tue, 21 Feb 2023 16:05:39 +0100 Subject: [PATCH 11/14] Add colab --- notebooks/unit8/unit8_part2.ipynb | 678 ++++++++++++++++++++++++++++++ 1 file changed, 678 insertions(+) create mode 100644 notebooks/unit8/unit8_part2.ipynb diff --git a/notebooks/unit8/unit8_part2.ipynb b/notebooks/unit8/unit8_part2.ipynb new file mode 100644 index 0000000..b36924a --- /dev/null +++ b/notebooks/unit8/unit8_part2.ipynb @@ -0,0 +1,678 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "view-in-github", + "colab_type": "text" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OVx1gdg9wt9t" + }, + "source": [ + "# Unit 8 Part 2: Advanced Deep Reinforcement Learning. Using Sample Factory to play Doom from pixels\n", + "\n", + "\"Thumbnail\"/\n", + "\n", + "In this notebook, we will learn how to train a Deep Neural Network to collect objects in a 3D environment based on the game of Doom, a video of the resulting policy is shown below. We train this policy using [Sample Factory](https://www.samplefactory.dev/), an asynchronous implementation of the PPO algorithm.\n", + "\n", + "Please note the following points:\n", + "\n", + "* [Sample Factory](https://www.samplefactory.dev/) is an advanced RL framework and **only functions on Linux and Mac** (not Windows).\n", + "\n", + "* The framework performs best on a **GPU machine with many CPU cores**, where it can achieve speeds of 100k interactions per second. The resources available on a standard Colab notebook **limit the performance of this library**. So the speed in this setting **does not reflect the real-world performance**.\n", + "* Benchmarks for Sample Factory are available in a number of settings, check out the [examples](https://github.com/alex-petrenko/sample-factory/tree/master/sf_examples) if you want to find out more.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "I6_67HfI1CKg" + }, + "outputs": [], + "source": [ + "from IPython.display import HTML\n", + "\n", + "HTML(''''''\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DgHRAsYEXdyw" + }, + "source": [ + "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model:\n", + "\n", + "- `doom_health_gathering_supreme` get a result of >= 5.\n", + "\n", + "To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward**\n", + "\n", + "If you don't find your model, **go to the bottom of the page and click on the refresh button**\n", + "\n", + "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PU4FVzaoM6fC" + }, + "source": [ + "## Set the GPU 💪\n", + "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n", + "\n", + "\"GPU" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KV0NyFdQM9ZG" + }, + "source": [ + "- `Hardware Accelerator > GPU`\n", + "\n", + "\"GPU" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-fSy5HzUcMWB" + }, + "source": [ + "Before starting to train our agent, let's **study the library and environments we're going to use**.\n", + "\n", + "## Sample Factory\n", + "\n", + "[Sample Factory](https://www.samplefactory.dev/) is one of the **fastest RL libraries focused on very efficient synchronous and asynchronous implementations of policy gradients (PPO)**.\n", + "\n", + "Sample Factory is thoroughly **tested, used by many researchers and practitioners**, and is actively maintained. Our implementation is known to **reach SOTA performance in a variety of domains while minimizing RL experiment training time and hardware requirements**.\n", + "\n", + "\"Sample\n", + "\n", + "\n", + "\n", + "### Key features\n", + "\n", + "- Highly optimized algorithm [architecture](https://www.samplefactory.dev/06-architecture/overview/) for maximum learning throughput\n", + "- [Synchronous and asynchronous](https://www.samplefactory.dev/07-advanced-topics/sync-async/) training regimes\n", + "- [Serial (single-process) mode](https://www.samplefactory.dev/07-advanced-topics/serial-mode/) for easy debugging\n", + "- Optimal performance in both CPU-based and [GPU-accelerated environments](https://www.samplefactory.dev/09-environment-integrations/isaacgym/)\n", + "- Single- & multi-agent training, self-play, supports [training multiple policies](https://www.samplefactory.dev/07-advanced-topics/multi-policy-training/) at once on one or many GPUs\n", + "- Population-Based Training ([PBT](https://www.samplefactory.dev/07-advanced-topics/pbt/))\n", + "- Discrete, continuous, hybrid action spaces\n", + "- Vector-based, image-based, dictionary observation spaces\n", + "- Automatically creates a model architecture by parsing action/observation space specification. Supports [custom model architectures](https://www.samplefactory.dev/03-customization/custom-models/)\n", + "- Designed to be imported into other projects, [custom environments](https://www.samplefactory.dev/03-customization/custom-environments/) are first-class citizens\n", + "- Detailed [WandB and Tensorboard summaries](https://www.samplefactory.dev/05-monitoring/metrics-reference/), [custom metrics](https://www.samplefactory.dev/05-monitoring/custom-metrics/)\n", + "- [HuggingFace 🤗 integration](https://www.samplefactory.dev/10-huggingface/huggingface/) (upload trained models and metrics to the Hub)\n", + "- [Multiple](https://www.samplefactory.dev/09-environment-integrations/mujoco/) [example](https://www.samplefactory.dev/09-environment-integrations/atari/) [environment](https://www.samplefactory.dev/09-environment-integrations/vizdoom/) [integrations](https://www.samplefactory.dev/09-environment-integrations/dmlab/) with tuned parameters and trained models\n", + "\n", + "All of the above policies are available on the 🤗 hub. Search for the tag [sample-factory](https://huggingface.co/models?library=sample-factory&sort=downloads)\n", + "\n", + "### How sample-factory works\n", + "\n", + "Sample-factory is one of the **most highly optimized RL implementations available to the community**. \n", + "\n", + "It works by **spawning multiple processes that run rollout workers, inference workers and a learner worker**. \n", + "\n", + "The *workers* **communicate through shared memory, which lowers the communication cost between processes**. \n", + "\n", + "The *rollout workers* interact with the environment and send observations to the *inference workers*. \n", + "\n", + "The *inferences workers* query a fixed version of the policy and **send actions back to the rollout worker**. \n", + "\n", + "After *k* steps the rollout works send a trajectory of experience to the learner worker, **which it uses to update the agent’s policy network**.\n", + "\n", + "\"Sample" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nB68Eb9UgC94" + }, + "source": [ + "### Actor Critic models in Sample-factory\n", + "\n", + "Actor Critic models in Sample Factory are composed of three components:\n", + "\n", + "- **Encoder** - Process input observations (images, vectors) and map them to a vector. This is the part of the model you will most likely want to customize.\n", + "- **Core** - Intergrate vectors from one or more encoders, can optionally include a single- or multi-layer LSTM/GRU in a memory-based agent.\n", + "- **Decoder** - Apply additional layers to the output of the model core before computing the policy and value outputs.\n", + "\n", + "The library has been designed to automatically support any observation and action spaces. Users can easily add their custom models. You can find out more in the [documentation](https://www.samplefactory.dev/03-customization/custom-models/#actor-critic-models-in-sample-factory)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ez5UhUtYcWXF" + }, + "source": [ + "## ViZDoom\n", + "\n", + "[ViZDoom](https://vizdoom.cs.put.edu.pl/) is an **open-source python interface for the Doom Engine**. \n", + "\n", + "The library was created in 2016 by Marek Wydmuch, Michal Kempka at the Institute of Computing Science, Poznan University of Technology, Poland. \n", + "\n", + "The library enables the **training of agents directly from the screen pixels in a number of scenarios**, including team deathmatch, shown in the video below. Because the ViZDoom environment is based on a game the was created in the 90s, it can be run on modern hardware at accelerated speeds, **allowing us to learn complex AI behaviors fairly quickly**.\n", + "\n", + "The library includes feature such as:\n", + "\n", + "- Multi-platform (Linux, macOS, Windows),\n", + "- API for Python and C++,\n", + "- [OpenAI Gym](https://www.gymlibrary.dev/) environment wrappers\n", + "- Easy-to-create custom scenarios (visual editors, scripting language, and examples available),\n", + "- Async and sync single-player and multiplayer modes,\n", + "- Lightweight (few MBs) and fast (up to 7000 fps in sync mode, single-threaded),\n", + "- Customizable resolution and rendering parameters,\n", + "- Access to the depth buffer (3D vision),\n", + "- Automatic labeling of game objects visible in the frame,\n", + "- Access to the audio buffer\n", + "- Access to the list of actors/objects and map geometry,\n", + "- Off-screen rendering and episode recording,\n", + "- Time scaling in async mode." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wAMwza0d5QVj" + }, + "source": [ + "## We first need to install some dependencies that are required for the ViZDoom environment\n", + "\n", + "Now that our Colab runtime is set up, we can start by installing the dependencies required to run ViZDoom on linux. \n", + "\n", + "If you are following on your machine on Mac, you will want to follow the installation instructions on the [github page](https://github.com/Farama-Foundation/ViZDoom/blob/master/doc/Quickstart.md#-quickstart-for-macos-and-anaconda3-python-36)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "RJMxkaldwIVx" + }, + "outputs": [], + "source": [ + "%%capture\n", + "%%bash\n", + "# Install ViZDoom deps from \n", + "# https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md#-linux\n", + "\n", + "apt-get install build-essential zlib1g-dev libsdl2-dev libjpeg-dev \\\n", + "nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \\\n", + "libopenal-dev timidity libwildmidi-dev unzip ffmpeg\n", + "\n", + "# Boost libraries\n", + "apt-get install libboost-all-dev\n", + "\n", + "# Lua binding dependencies\n", + "apt-get install liblua5.1-dev" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JT4att2c57MW" + }, + "source": [ + "## Then we can install Sample Factory and ViZDoom\n", + "- This can take 7min" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bbqfPZnIsvA6" + }, + "outputs": [], + "source": [ + "# install python libraries\n", + "!pip install sample-factory\n", + "!pip install vizdoom" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1jizouGpghUZ" + }, + "source": [ + "## Setting up the Doom Environment in sample-factory" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bCgZbeiavcDU" + }, + "outputs": [], + "source": [ + "import functools\n", + "\n", + "from sample_factory.algo.utils.context import global_model_factory\n", + "from sample_factory.cfg.arguments import parse_full_cfg, parse_sf_args\n", + "from sample_factory.envs.env_utils import register_env\n", + "from sample_factory.train import run_rl\n", + "\n", + "from sf_examples.vizdoom.doom.doom_model import make_vizdoom_encoder\n", + "from sf_examples.vizdoom.doom.doom_params import add_doom_env_args, doom_override_defaults\n", + "from sf_examples.vizdoom.doom.doom_utils import DOOM_ENVS, make_doom_env_from_spec\n", + "\n", + "\n", + "# Registers all the ViZDoom environments\n", + "def register_vizdoom_envs():\n", + " for env_spec in DOOM_ENVS:\n", + " make_env_func = functools.partial(make_doom_env_from_spec, env_spec)\n", + " register_env(env_spec.name, make_env_func)\n", + "\n", + "# Sample Factory allows the registration of a custom Neural Network architecture\n", + "# See https://github.com/alex-petrenko/sample-factory/blob/master/sf_examples/vizdoom/doom/doom_model.py for more details\n", + "def register_vizdoom_models():\n", + " global_model_factory().register_encoder_factory(make_vizdoom_encoder)\n", + "\n", + "\n", + "def register_vizdoom_components():\n", + " register_vizdoom_envs()\n", + " register_vizdoom_models()\n", + "\n", + "# parse the command line args and create a config\n", + "def parse_vizdoom_cfg(argv=None, evaluation=False):\n", + " parser, _ = parse_sf_args(argv=argv, evaluation=evaluation)\n", + " # parameters specific to Doom envs\n", + " add_doom_env_args(parser)\n", + " # override Doom default values for algo parameters\n", + " doom_override_defaults(parser)\n", + " # second parsing pass yields the final configuration\n", + " final_cfg = parse_full_cfg(parser, argv)\n", + " return final_cfg" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sgRy6wnrgnij" + }, + "source": [ + "Now that the setup if complete, we can train the agent. We have chosen here to learn a ViZDoom task called `Health Gathering Supreme`.\n", + "\n", + "### The scenario: Health Gathering Supreme\n", + "\n", + "\"Health-Gathering-Supreme\"/\n", + "\n", + "\n", + "\n", + "The objective of this scenario is to **teach the agent how to survive without knowing what makes him survive**. Agent know only that **life is precious** and death is bad so **it must learn what prolongs his existence and that his health is connected with it**.\n", + "\n", + "Map is a rectangle containing walls and with a green, acidic floor which **hurts the player periodically**. Initially there are some medkits spread uniformly over the map. A new medkit falls from the skies every now and then. **Medkits heal some portions of player's health** - to survive agent needs to pick them up. Episode finishes after player's death or on timeout.\n", + "\n", + "Further configuration:\n", + "- Living_reward = 1\n", + "- 3 available buttons: turn left, turn right, move forward\n", + "- 1 available game variable: HEALTH\n", + "- death penalty = 100\n", + "\n", + "You can find out more about the scenarios available in ViZDoom [here](https://github.com/Farama-Foundation/ViZDoom/tree/master/scenarios). \n", + "\n", + "There are also a number of more complex scenarios that have been create for ViZDoom, such as the ones detailed on [this github page](https://github.com/edbeeching/3d_control_deep_rl).\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "siHZZ34DiZEp" + }, + "source": [ + "## Training the agent\n", + "- We're going to train the agent for 4000000 steps it will take approximately 20min" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "y_TeicMvyKHP" + }, + "outputs": [], + "source": [ + "## Start the training, this should take around 15 minutes\n", + "register_vizdoom_components()\n", + "\n", + "# The scenario we train on today is health gathering\n", + "# other scenarios include \"doom_basic\", \"doom_two_colors_easy\", \"doom_dm\", \"doom_dwango5\", \"doom_my_way_home\", \"doom_deadly_corridor\", \"doom_defend_the_center\", \"doom_defend_the_line\"\n", + "env = \"doom_health_gathering_supreme\"\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=8\", \"--num_envs_per_worker=4\", \"--train_for_env_steps=4000000\"])\n", + "\n", + "status = run_rl(cfg)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5L0nBS9e_jqC" + }, + "source": [ + "## Let's take a look at the performance of the trained policy and output a video of the agent." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "MGSA4Kg5_i0j" + }, + "outputs": [], + "source": [ + "from sample_factory.enjoy import enjoy\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\"], evaluation=True)\n", + "status = enjoy(cfg)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Lj5L1x0WLxwB" + }, + "source": [ + "## Now lets visualize the performance of the agent" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "WsXhBY7JNOdJ" + }, + "outputs": [], + "source": [ + "from base64 import b64encode\n", + "from IPython.display import HTML\n", + "\n", + "mp4 = open('/content/train_dir/default_experiment/replay.mp4','rb').read()\n", + "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", + "HTML(\"\"\"\n", + "\n", + "\"\"\" % data_url)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "The agent has learned something, but its performance could be better. We would clearly need to train for longer. But let's upload this model to the Hub." + ], + "metadata": { + "id": "2A4pf_1VwPqR" + } + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CSQVWF0kNuy9" + }, + "source": [ + "## Now lets upload your checkpoint and video to the Hugging Face Hub\n", + "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JquRrWytA6eo" + }, + "source": [ + "To be able to share your model with the community there are three more steps to follow:\n", + "\n", + "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n", + "\n", + "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n", + "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n", + "\n", + "\"Create\n", + "\n", + "- Copy the token \n", + "- Run the cell below and paste the token" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_tsf2uv0g_4p" + }, + "source": [ + "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "GoQm_jYSOts0" + }, + "outputs": [], + "source": [ + "from huggingface_hub import notebook_login\n", + "notebook_login()\n", + "!git config --global credential.helper store" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "sEawW_i0OvJV" + }, + "outputs": [], + "source": [ + "from sample_factory.enjoy import enjoy\n", + "\n", + "hf_username = \"ThomasSimonini\" # insert your HuggingFace username here\n", + "\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--max_num_frames=100000\", \"--push_to_hub\", f\"--hf_repository={hf_username}/rl_course_vizdoom_health_gathering_supreme\"], evaluation=True)\n", + "status = enjoy(cfg)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Let's load another model\n", + "\n", + "\n" + ], + "metadata": { + "id": "9PzeXx-qxVvw" + } + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mHZAWSgL5F7P" + }, + "source": [ + "This agent's performance was good, but can do better! Let's download and visualize an agent trained for 10B timesteps from the hub." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Ud6DwAUl5S-l" + }, + "outputs": [], + "source": [ + "#download the agent from the hub\n", + "!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_health_gathering_supreme_2222 -d ./train_dir\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qoUJhL6x6sY5" + }, + "outputs": [], + "source": [ + "!ls train_dir/doom_health_gathering_supreme_2222" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lZskc8LG8qr8" + }, + "outputs": [], + "source": [ + "env = \"doom_health_gathering_supreme\"\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--experiment=doom_health_gathering_supreme_2222\", \"--train_dir=train_dir\"], evaluation=True)\n", + "status = enjoy(cfg)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BtzXBoj65Wmq" + }, + "outputs": [], + "source": [ + "mp4 = open('/content/train_dir/doom_health_gathering_supreme_2222/replay.mp4','rb').read()\n", + "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", + "HTML(\"\"\"\n", + "\n", + "\"\"\" % data_url)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Some additional challenges 🏆: Doom Deathmatch\n", + "\n", + "Training an agent to play a Doom deathmatch **takes many hours on a more beefy machine than is available in Colab**. \n", + "\n", + "Fortunately, we have have **already trained an agent in this scenario and it is available in the 🤗 Hub!** Let’s download the model and visualize the agent’s performance." + ], + "metadata": { + "id": "ie5YWC3NyKO8" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "fq3WFeus81iI" + }, + "outputs": [], + "source": [ + "# Download the agent from the hub\n", + "!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_deathmatch_bots_2222 -d ./train_dir" + ] + }, + { + "cell_type": "markdown", + "source": [ + "Given the agent plays for a long time the video generation can take **10 minutes**." + ], + "metadata": { + "id": "7AX_LwxR2FQ0" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0hq6XL__85Bv" + }, + "outputs": [], + "source": [ + "\n", + "from sample_factory.enjoy import enjoy\n", + "register_vizdoom_components()\n", + "env = \"doom_deathmatch_bots\"\n", + "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=1\", \"--experiment=doom_deathmatch_bots_2222\", \"--train_dir=train_dir\"], evaluation=True)\n", + "status = enjoy(cfg)\n", + "mp4 = open('/content/train_dir/doom_deathmatch_bots_2222/replay.mp4','rb').read()\n", + "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", + "HTML(\"\"\"\n", + "\n", + "\"\"\" % data_url)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "You **can try to train your agent in this environment** using the code above, but not on colab.\n", + "**Good luck 🤞**" + ], + "metadata": { + "id": "N6mEC-4zyihx" + } + }, + { + "cell_type": "markdown", + "source": [ + "If you prefer an easier scenario, **why not try training in another ViZDoom scenario such as `doom_deadly_corridor` or `doom_defend_the_center`.**\n", + "\n", + "\n", + "\n", + "---\n", + "\n", + "\n", + "This concludes the last unit. But we are not finished yet! 🤗 The following **bonus section include some of the most interesting, advanced and cutting edge work in Deep Reinforcement Learning**.\n", + "\n", + "## Keep learning, stay awesome 🤗" + ], + "metadata": { + "id": "YnDAngN6zeeI" + } + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "provenance": [], + "collapsed_sections": [ + "PU4FVzaoM6fC", + "nB68Eb9UgC94", + "ez5UhUtYcWXF", + "sgRy6wnrgnij" + ], + "private_outputs": true, + "include_colab_link": true + }, + "gpuClass": "standard", + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file From f2b01924204c06c3e2ffb3e2fba3a07aa447842a Mon Sep 17 00:00:00 2001 From: simoninithomas Date: Tue, 21 Feb 2023 16:13:49 +0100 Subject: [PATCH 12/14] Add unit8 Part 2 --- notebooks/unit8/unit8_part1.mdx | 1068 --------------------------- units/en/unit8/hands-on-cleanrl.mdx | 3 +- units/en/unit8/hands-on-sf.mdx | 431 ++++++++++- 3 files changed, 432 insertions(+), 1070 deletions(-) delete mode 100644 notebooks/unit8/unit8_part1.mdx diff --git a/notebooks/unit8/unit8_part1.mdx b/notebooks/unit8/unit8_part1.mdx deleted file mode 100644 index 0606dac..0000000 --- a/notebooks/unit8/unit8_part1.mdx +++ /dev/null @@ -1,1068 +0,0 @@ -Open In Colab - -# Unit 8: Proximal Policy Gradient (PPO) with PyTorch 🤖 - -Unit 8 - - -In this notebook, you'll learn to **code your PPO agent from scratch with PyTorch using CleanRL implementation as model**. - -To test its robustness, we're going to train it in: - -- [LunarLander-v2 🚀](https://www.gymlibrary.dev/environments/box2d/lunar_lander/) - - -⬇️ Here is an example of what you will achieve. ⬇️ - -```python -%%html - -``` - -We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues). - -## Objectives of this notebook 🏆 - -At the end of the notebook, you will: - -- Be able to **code your PPO agent from scratch using PyTorch**. -- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥. - - - - -## This notebook is from the Deep Reinforcement Learning Course -Deep RL Course illustration - -In this free course, you will: - -- 📖 Study Deep Reinforcement Learning in **theory and practice**. -- 🧑‍💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0. -- 🤖 Train **agents in unique environments** - -Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).** - - -The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5 - -## Prerequisites 🏗️ -Before diving into the notebook, you need to: - -🔲 📚 Study [PPO by reading Unit 8](https://huggingface.co/deep-rl-course/unit8/introduction) 🤗 - -To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model, we don't ask for a minimal result but we **advise you to try different hyperparameters settings to get better results**. - -If you don't find your model, **go to the bottom of the page and click on the refresh button** - -For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process - -## Set the GPU 💪 -- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type` - -GPU Step 1 - -- `Hardware Accelerator > GPU` - -GPU Step 2 - -## Create a virtual display 🔽 - -During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). - -Hence the following cell will install the librairies and create and run a virtual screen 🖥 - -```python -%%capture -!apt install python-opengl -!apt install ffmpeg -!apt install xvfb -!pip install pyglet==1.5 -!pip3 install pyvirtualdisplay -``` - -```python -# Virtual display -from pyvirtualdisplay import Display - -virtual_display = Display(visible=0, size=(1400, 900)) -virtual_display.start() -``` - -## Install dependencies 🔽 -For this exercise, we use `gym==0.21` - - -```python -!pip install gym==0.21 -!pip install imageio-ffmpeg -!pip install huggingface_hub -!pip install box2d -``` - -## Let's code PPO from scratch with Costa Huang tutorial -- For the core implementation of PPO we're going to use the excellent [Costa Huang](https://costa.sh/) tutorial. -- In addition to the tutorial, to go deeper you can read the 37 core implementation details: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ - -👉 The video tutorial: https://youtu.be/MEt6rrxH8W4 - -```python -from IPython.display import HTML - -HTML( - '' -) -``` - -- The best is to code first on the cell below, this way, if you kill the machine **you don't loose the implementation**. - -```python -### Your code here: -``` - -## Add the Hugging Face Integration 🤗 -- In order to push our model to the Hub, we need to define a function `package_to_hub` - -- Add dependencies we need to push our model to the Hub - -```python -from huggingface_hub import HfApi, upload_folder -from huggingface_hub.repocard import metadata_eval_result, metadata_save - -from pathlib import Path -import datetime -import tempfile -import json -import shutil -import imageio - -from wasabi import Printer - -msg = Printer() -``` - -- Add new argument in `parse_args()` function to define the repo-id where we want to push the model. - -```python -# Adding HuggingFace argument -parser.add_argument( - "--repo-id", - type=str, - default="ThomasSimonini/ppo-CartPole-v1", - help="id of the model repository from the Hugging Face Hub {username/repo_name}", -) -``` - -- Next, we add the methods needed to push the model to the Hub - -- These methods will: - - `_evalutate_agent()`: evaluate the agent. - - `_generate_model_card()`: generate the model card of your agent. - - `_record_video()`: record a video of your agent. - -```python -def package_to_hub( - repo_id, - model, - hyperparameters, - eval_env, - video_fps=30, - commit_message="Push agent to the Hub", - token=None, - logs=None, -): - """ - Evaluate, Generate a video and Upload a model to Hugging Face Hub. - This method does the complete pipeline: - - It evaluates the model - - It generates the model card - - It generates a replay video of the agent - - It pushes everything to the hub - :param repo_id: id of the model repository from the Hugging Face Hub - :param model: trained model - :param eval_env: environment used to evaluate the agent - :param fps: number of fps for rendering the video - :param commit_message: commit message - :param logs: directory on local machine of tensorboard logs you'd like to upload - """ - msg.info( - "This function will save, evaluate, generate a video of your agent, " - "create a model card and push everything to the hub. " - "It might take up to 1min. \n " - "This is a work in progress: if you encounter a bug, please open an issue." - ) - # Step 1: Clone or create the repo - repo_url = HfApi().create_repo( - repo_id=repo_id, - token=token, - private=False, - exist_ok=True, - ) - - with tempfile.TemporaryDirectory() as tmpdirname: - tmpdirname = Path(tmpdirname) - - # Step 2: Save the model - torch.save(model.state_dict(), tmpdirname / "model.pt") - - # Step 3: Evaluate the model and build JSON - mean_reward, std_reward = _evaluate_agent(eval_env, 10, model) - - # First get datetime - eval_datetime = datetime.datetime.now() - eval_form_datetime = eval_datetime.isoformat() - - evaluate_data = { - "env_id": hyperparameters.env_id, - "mean_reward": mean_reward, - "std_reward": std_reward, - "n_evaluation_episodes": 10, - "eval_datetime": eval_form_datetime, - } - - # Write a JSON file - with open(tmpdirname / "results.json", "w") as outfile: - json.dump(evaluate_data, outfile) - - # Step 4: Generate a video - video_path = tmpdirname / "replay.mp4" - record_video(eval_env, model, video_path, video_fps) - - # Step 5: Generate the model card - generated_model_card, metadata = _generate_model_card( - "PPO", hyperparameters.env_id, mean_reward, std_reward, hyperparameters - ) - _save_model_card(tmpdirname, generated_model_card, metadata) - - # Step 6: Add logs if needed - if logs: - _add_logdir(tmpdirname, Path(logs)) - - msg.info(f"Pushing repo {repo_id} to the Hugging Face Hub") - - repo_url = upload_folder( - repo_id=repo_id, - folder_path=tmpdirname, - path_in_repo="", - commit_message=commit_message, - token=token, - ) - - msg.info(f"Your model is pushed to the Hub. You can view your model here: {repo_url}") - return repo_url - - -def _evaluate_agent(env, n_eval_episodes, policy): - """ - Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward. - :param env: The evaluation environment - :param n_eval_episodes: Number of episode to evaluate the agent - :param policy: The agent - """ - episode_rewards = [] - for episode in range(n_eval_episodes): - state = env.reset() - step = 0 - done = False - total_rewards_ep = 0 - - while done is False: - state = torch.Tensor(state).to(device) - action, _, _, _ = policy.get_action_and_value(state) - new_state, reward, done, info = env.step(action.cpu().numpy()) - total_rewards_ep += reward - if done: - break - state = new_state - episode_rewards.append(total_rewards_ep) - mean_reward = np.mean(episode_rewards) - std_reward = np.std(episode_rewards) - - return mean_reward, std_reward - - -def record_video(env, policy, out_directory, fps=30): - images = [] - done = False - state = env.reset() - img = env.render(mode="rgb_array") - images.append(img) - while not done: - state = torch.Tensor(state).to(device) - # Take the action (index) that have the maximum expected future reward given that state - action, _, _, _ = policy.get_action_and_value(state) - state, reward, done, info = env.step( - action.cpu().numpy() - ) # We directly put next_state = state for recording logic - img = env.render(mode="rgb_array") - images.append(img) - imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps) - - -def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters): - """ - Generate the model card for the Hub - :param model_name: name of the model - :env_id: name of the environment - :mean_reward: mean reward of the agent - :std_reward: standard deviation of the mean reward of the agent - :hyperparameters: training arguments - """ - # Step 1: Select the tags - metadata = generate_metadata(model_name, env_id, mean_reward, std_reward) - - # Transform the hyperparams namespace to string - converted_dict = vars(hyperparameters) - converted_str = str(converted_dict) - converted_str = converted_str.split(", ") - converted_str = "\n".join(converted_str) - - # Step 2: Generate the model card - model_card = f""" - # PPO Agent Playing {env_id} - - This is a trained model of a PPO agent playing {env_id}. - - # Hyperparameters - ```python - {converted_str} - ``` - """ - return model_card, metadata - - -def generate_metadata(model_name, env_id, mean_reward, std_reward): - """ - Define the tags for the model card - :param model_name: name of the model - :param env_id: name of the environment - :mean_reward: mean reward of the agent - :std_reward: standard deviation of the mean reward of the agent - """ - metadata = {} - metadata["tags"] = [ - env_id, - "ppo", - "deep-reinforcement-learning", - "reinforcement-learning", - "custom-implementation", - "deep-rl-course", - ] - - # Add metrics - eval = metadata_eval_result( - model_pretty_name=model_name, - task_pretty_name="reinforcement-learning", - task_id="reinforcement-learning", - metrics_pretty_name="mean_reward", - metrics_id="mean_reward", - metrics_value=f"{mean_reward:.2f} +/- {std_reward:.2f}", - dataset_pretty_name=env_id, - dataset_id=env_id, - ) - - # Merges both dictionaries - metadata = {**metadata, **eval} - - return metadata - - -def _save_model_card(local_path, generated_model_card, metadata): - """Saves a model card for the repository. - :param local_path: repository directory - :param generated_model_card: model card generated by _generate_model_card() - :param metadata: metadata - """ - readme_path = local_path / "README.md" - readme = "" - if readme_path.exists(): - with readme_path.open("r", encoding="utf8") as f: - readme = f.read() - else: - readme = generated_model_card - - with readme_path.open("w", encoding="utf-8") as f: - f.write(readme) - - # Save our metrics to Readme metadata - metadata_save(readme_path, metadata) - - -def _add_logdir(local_path: Path, logdir: Path): - """Adds a logdir to the repository. - :param local_path: repository directory - :param logdir: logdir directory - """ - if logdir.exists() and logdir.is_dir(): - # Add the logdir to the repository under new dir called logs - repo_logdir = local_path / "logs" - - # Delete current logs if they exist - if repo_logdir.exists(): - shutil.rmtree(repo_logdir) - - # Copy logdir into repo logdir - shutil.copytree(logdir, repo_logdir) -``` - -- Finally, we call this function at the end of the PPO training - -```python -# Create the evaluation environment -eval_env = gym.make(args.env_id) - -package_to_hub( - repo_id=args.repo_id, - model=agent, # The model we want to save - hyperparameters=args, - eval_env=gym.make(args.env_id), - logs=f"runs/{run_name}", -) -``` - -- Here's what look the ppo.py final file - -```python -# docs and experiment results can be found at https://docs.cleanrl.dev/rl-algorithms/ppo/#ppopy - -import argparse -import os -import random -import time -from distutils.util import strtobool - -import gym -import numpy as np -import torch -import torch.nn as nn -import torch.optim as optim -from torch.distributions.categorical import Categorical -from torch.utils.tensorboard import SummaryWriter - -from huggingface_hub import HfApi, upload_folder -from huggingface_hub.repocard import metadata_eval_result, metadata_save - -from pathlib import Path -import datetime -import tempfile -import json -import shutil -import imageio - -from wasabi import Printer - -msg = Printer() - - -def parse_args(): - # fmt: off - parser = argparse.ArgumentParser() - parser.add_argument("--exp-name", type=str, default=os.path.basename(__file__).rstrip(".py"), - help="the name of this experiment") - parser.add_argument("--seed", type=int, default=1, - help="seed of the experiment") - parser.add_argument("--torch-deterministic", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="if toggled, `torch.backends.cudnn.deterministic=False`") - parser.add_argument("--cuda", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="if toggled, cuda will be enabled by default") - parser.add_argument("--track", type=lambda x: bool(strtobool(x)), default=False, nargs="?", const=True, - help="if toggled, this experiment will be tracked with Weights and Biases") - parser.add_argument("--wandb-project-name", type=str, default="cleanRL", - help="the wandb's project name") - parser.add_argument("--wandb-entity", type=str, default=None, - help="the entity (team) of wandb's project") - parser.add_argument("--capture-video", type=lambda x: bool(strtobool(x)), default=False, nargs="?", const=True, - help="weather to capture videos of the agent performances (check out `videos` folder)") - - # Algorithm specific arguments - parser.add_argument("--env-id", type=str, default="CartPole-v1", - help="the id of the environment") - parser.add_argument("--total-timesteps", type=int, default=50000, - help="total timesteps of the experiments") - parser.add_argument("--learning-rate", type=float, default=2.5e-4, - help="the learning rate of the optimizer") - parser.add_argument("--num-envs", type=int, default=4, - help="the number of parallel game environments") - parser.add_argument("--num-steps", type=int, default=128, - help="the number of steps to run in each environment per policy rollout") - parser.add_argument("--anneal-lr", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="Toggle learning rate annealing for policy and value networks") - parser.add_argument("--gae", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="Use GAE for advantage computation") - parser.add_argument("--gamma", type=float, default=0.99, - help="the discount factor gamma") - parser.add_argument("--gae-lambda", type=float, default=0.95, - help="the lambda for the general advantage estimation") - parser.add_argument("--num-minibatches", type=int, default=4, - help="the number of mini-batches") - parser.add_argument("--update-epochs", type=int, default=4, - help="the K epochs to update the policy") - parser.add_argument("--norm-adv", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="Toggles advantages normalization") - parser.add_argument("--clip-coef", type=float, default=0.2, - help="the surrogate clipping coefficient") - parser.add_argument("--clip-vloss", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="Toggles whether or not to use a clipped loss for the value function, as per the paper.") - parser.add_argument("--ent-coef", type=float, default=0.01, - help="coefficient of the entropy") - parser.add_argument("--vf-coef", type=float, default=0.5, - help="coefficient of the value function") - parser.add_argument("--max-grad-norm", type=float, default=0.5, - help="the maximum norm for the gradient clipping") - parser.add_argument("--target-kl", type=float, default=None, - help="the target KL divergence threshold") - - # Adding HuggingFace argument - parser.add_argument("--repo-id", type=str, default="ThomasSimonini/ppo-CartPole-v1", help="id of the model repository from the Hugging Face Hub {username/repo_name}") - - args = parser.parse_args() - args.batch_size = int(args.num_envs * args.num_steps) - args.minibatch_size = int(args.batch_size // args.num_minibatches) - # fmt: on - return args - - -def package_to_hub( - repo_id, - model, - hyperparameters, - eval_env, - video_fps=30, - commit_message="Push agent to the Hub", - token=None, - logs=None, -): - """ - Evaluate, Generate a video and Upload a model to Hugging Face Hub. - This method does the complete pipeline: - - It evaluates the model - - It generates the model card - - It generates a replay video of the agent - - It pushes everything to the hub - :param repo_id: id of the model repository from the Hugging Face Hub - :param model: trained model - :param eval_env: environment used to evaluate the agent - :param fps: number of fps for rendering the video - :param commit_message: commit message - :param logs: directory on local machine of tensorboard logs you'd like to upload - """ - msg.info( - "This function will save, evaluate, generate a video of your agent, " - "create a model card and push everything to the hub. " - "It might take up to 1min. \n " - "This is a work in progress: if you encounter a bug, please open an issue." - ) - # Step 1: Clone or create the repo - repo_url = HfApi().create_repo( - repo_id=repo_id, - token=token, - private=False, - exist_ok=True, - ) - - with tempfile.TemporaryDirectory() as tmpdirname: - tmpdirname = Path(tmpdirname) - - # Step 2: Save the model - torch.save(model.state_dict(), tmpdirname / "model.pt") - - # Step 3: Evaluate the model and build JSON - mean_reward, std_reward = _evaluate_agent(eval_env, 10, model) - - # First get datetime - eval_datetime = datetime.datetime.now() - eval_form_datetime = eval_datetime.isoformat() - - evaluate_data = { - "env_id": hyperparameters.env_id, - "mean_reward": mean_reward, - "std_reward": std_reward, - "n_evaluation_episodes": 10, - "eval_datetime": eval_form_datetime, - } - - # Write a JSON file - with open(tmpdirname / "results.json", "w") as outfile: - json.dump(evaluate_data, outfile) - - # Step 4: Generate a video - video_path = tmpdirname / "replay.mp4" - record_video(eval_env, model, video_path, video_fps) - - # Step 5: Generate the model card - generated_model_card, metadata = _generate_model_card( - "PPO", hyperparameters.env_id, mean_reward, std_reward, hyperparameters - ) - _save_model_card(tmpdirname, generated_model_card, metadata) - - # Step 6: Add logs if needed - if logs: - _add_logdir(tmpdirname, Path(logs)) - - msg.info(f"Pushing repo {repo_id} to the Hugging Face Hub") - - repo_url = upload_folder( - repo_id=repo_id, - folder_path=tmpdirname, - path_in_repo="", - commit_message=commit_message, - token=token, - ) - - msg.info(f"Your model is pushed to the Hub. You can view your model here: {repo_url}") - return repo_url - - -def _evaluate_agent(env, n_eval_episodes, policy): - """ - Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward. - :param env: The evaluation environment - :param n_eval_episodes: Number of episode to evaluate the agent - :param policy: The agent - """ - episode_rewards = [] - for episode in range(n_eval_episodes): - state = env.reset() - step = 0 - done = False - total_rewards_ep = 0 - - while done is False: - state = torch.Tensor(state).to(device) - action, _, _, _ = policy.get_action_and_value(state) - new_state, reward, done, info = env.step(action.cpu().numpy()) - total_rewards_ep += reward - if done: - break - state = new_state - episode_rewards.append(total_rewards_ep) - mean_reward = np.mean(episode_rewards) - std_reward = np.std(episode_rewards) - - return mean_reward, std_reward - - -def record_video(env, policy, out_directory, fps=30): - images = [] - done = False - state = env.reset() - img = env.render(mode="rgb_array") - images.append(img) - while not done: - state = torch.Tensor(state).to(device) - # Take the action (index) that have the maximum expected future reward given that state - action, _, _, _ = policy.get_action_and_value(state) - state, reward, done, info = env.step( - action.cpu().numpy() - ) # We directly put next_state = state for recording logic - img = env.render(mode="rgb_array") - images.append(img) - imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps) - - -def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters): - """ - Generate the model card for the Hub - :param model_name: name of the model - :env_id: name of the environment - :mean_reward: mean reward of the agent - :std_reward: standard deviation of the mean reward of the agent - :hyperparameters: training arguments - """ - # Step 1: Select the tags - metadata = generate_metadata(model_name, env_id, mean_reward, std_reward) - - # Transform the hyperparams namespace to string - converted_dict = vars(hyperparameters) - converted_str = str(converted_dict) - converted_str = converted_str.split(", ") - converted_str = "\n".join(converted_str) - - # Step 2: Generate the model card - model_card = f""" - # PPO Agent Playing {env_id} - - This is a trained model of a PPO agent playing {env_id}. - - # Hyperparameters - ```python - {converted_str} - ``` - """ - return model_card, metadata - - -def generate_metadata(model_name, env_id, mean_reward, std_reward): - """ - Define the tags for the model card - :param model_name: name of the model - :param env_id: name of the environment - :mean_reward: mean reward of the agent - :std_reward: standard deviation of the mean reward of the agent - """ - metadata = {} - metadata["tags"] = [ - env_id, - "ppo", - "deep-reinforcement-learning", - "reinforcement-learning", - "custom-implementation", - "deep-rl-course", - ] - - # Add metrics - eval = metadata_eval_result( - model_pretty_name=model_name, - task_pretty_name="reinforcement-learning", - task_id="reinforcement-learning", - metrics_pretty_name="mean_reward", - metrics_id="mean_reward", - metrics_value=f"{mean_reward:.2f} +/- {std_reward:.2f}", - dataset_pretty_name=env_id, - dataset_id=env_id, - ) - - # Merges both dictionaries - metadata = {**metadata, **eval} - - return metadata - - -def _save_model_card(local_path, generated_model_card, metadata): - """Saves a model card for the repository. - :param local_path: repository directory - :param generated_model_card: model card generated by _generate_model_card() - :param metadata: metadata - """ - readme_path = local_path / "README.md" - readme = "" - if readme_path.exists(): - with readme_path.open("r", encoding="utf8") as f: - readme = f.read() - else: - readme = generated_model_card - - with readme_path.open("w", encoding="utf-8") as f: - f.write(readme) - - # Save our metrics to Readme metadata - metadata_save(readme_path, metadata) - - -def _add_logdir(local_path: Path, logdir: Path): - """Adds a logdir to the repository. - :param local_path: repository directory - :param logdir: logdir directory - """ - if logdir.exists() and logdir.is_dir(): - # Add the logdir to the repository under new dir called logs - repo_logdir = local_path / "logs" - - # Delete current logs if they exist - if repo_logdir.exists(): - shutil.rmtree(repo_logdir) - - # Copy logdir into repo logdir - shutil.copytree(logdir, repo_logdir) - - -def make_env(env_id, seed, idx, capture_video, run_name): - def thunk(): - env = gym.make(env_id) - env = gym.wrappers.RecordEpisodeStatistics(env) - if capture_video: - if idx == 0: - env = gym.wrappers.RecordVideo(env, f"videos/{run_name}") - env.seed(seed) - env.action_space.seed(seed) - env.observation_space.seed(seed) - return env - - return thunk - - -def layer_init(layer, std=np.sqrt(2), bias_const=0.0): - torch.nn.init.orthogonal_(layer.weight, std) - torch.nn.init.constant_(layer.bias, bias_const) - return layer - - -class Agent(nn.Module): - def __init__(self, envs): - super().__init__() - self.critic = nn.Sequential( - layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)), - nn.Tanh(), - layer_init(nn.Linear(64, 64)), - nn.Tanh(), - layer_init(nn.Linear(64, 1), std=1.0), - ) - self.actor = nn.Sequential( - layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)), - nn.Tanh(), - layer_init(nn.Linear(64, 64)), - nn.Tanh(), - layer_init(nn.Linear(64, envs.single_action_space.n), std=0.01), - ) - - def get_value(self, x): - return self.critic(x) - - def get_action_and_value(self, x, action=None): - logits = self.actor(x) - probs = Categorical(logits=logits) - if action is None: - action = probs.sample() - return action, probs.log_prob(action), probs.entropy(), self.critic(x) - - -if __name__ == "__main__": - args = parse_args() - run_name = f"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}" - if args.track: - import wandb - - wandb.init( - project=args.wandb_project_name, - entity=args.wandb_entity, - sync_tensorboard=True, - config=vars(args), - name=run_name, - monitor_gym=True, - save_code=True, - ) - writer = SummaryWriter(f"runs/{run_name}") - writer.add_text( - "hyperparameters", - "|param|value|\n|-|-|\n%s" % ("\n".join([f"|{key}|{value}|" for key, value in vars(args).items()])), - ) - - # TRY NOT TO MODIFY: seeding - random.seed(args.seed) - np.random.seed(args.seed) - torch.manual_seed(args.seed) - torch.backends.cudnn.deterministic = args.torch_deterministic - - device = torch.device("cuda" if torch.cuda.is_available() and args.cuda else "cpu") - - # env setup - envs = gym.vector.SyncVectorEnv( - [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)] - ) - assert isinstance(envs.single_action_space, gym.spaces.Discrete), "only discrete action space is supported" - - agent = Agent(envs).to(device) - optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5) - - # ALGO Logic: Storage setup - obs = torch.zeros((args.num_steps, args.num_envs) + envs.single_observation_space.shape).to(device) - actions = torch.zeros((args.num_steps, args.num_envs) + envs.single_action_space.shape).to(device) - logprobs = torch.zeros((args.num_steps, args.num_envs)).to(device) - rewards = torch.zeros((args.num_steps, args.num_envs)).to(device) - dones = torch.zeros((args.num_steps, args.num_envs)).to(device) - values = torch.zeros((args.num_steps, args.num_envs)).to(device) - - # TRY NOT TO MODIFY: start the game - global_step = 0 - start_time = time.time() - next_obs = torch.Tensor(envs.reset()).to(device) - next_done = torch.zeros(args.num_envs).to(device) - num_updates = args.total_timesteps // args.batch_size - - for update in range(1, num_updates + 1): - # Annealing the rate if instructed to do so. - if args.anneal_lr: - frac = 1.0 - (update - 1.0) / num_updates - lrnow = frac * args.learning_rate - optimizer.param_groups[0]["lr"] = lrnow - - for step in range(0, args.num_steps): - global_step += 1 * args.num_envs - obs[step] = next_obs - dones[step] = next_done - - # ALGO LOGIC: action logic - with torch.no_grad(): - action, logprob, _, value = agent.get_action_and_value(next_obs) - values[step] = value.flatten() - actions[step] = action - logprobs[step] = logprob - - # TRY NOT TO MODIFY: execute the game and log data. - next_obs, reward, done, info = envs.step(action.cpu().numpy()) - rewards[step] = torch.tensor(reward).to(device).view(-1) - next_obs, next_done = torch.Tensor(next_obs).to(device), torch.Tensor(done).to(device) - - for item in info: - if "episode" in item.keys(): - print(f"global_step={global_step}, episodic_return={item['episode']['r']}") - writer.add_scalar("charts/episodic_return", item["episode"]["r"], global_step) - writer.add_scalar("charts/episodic_length", item["episode"]["l"], global_step) - break - - # bootstrap value if not done - with torch.no_grad(): - next_value = agent.get_value(next_obs).reshape(1, -1) - if args.gae: - advantages = torch.zeros_like(rewards).to(device) - lastgaelam = 0 - for t in reversed(range(args.num_steps)): - if t == args.num_steps - 1: - nextnonterminal = 1.0 - next_done - nextvalues = next_value - else: - nextnonterminal = 1.0 - dones[t + 1] - nextvalues = values[t + 1] - delta = rewards[t] + args.gamma * nextvalues * nextnonterminal - values[t] - advantages[t] = lastgaelam = delta + args.gamma * args.gae_lambda * nextnonterminal * lastgaelam - returns = advantages + values - else: - returns = torch.zeros_like(rewards).to(device) - for t in reversed(range(args.num_steps)): - if t == args.num_steps - 1: - nextnonterminal = 1.0 - next_done - next_return = next_value - else: - nextnonterminal = 1.0 - dones[t + 1] - next_return = returns[t + 1] - returns[t] = rewards[t] + args.gamma * nextnonterminal * next_return - advantages = returns - values - - # flatten the batch - b_obs = obs.reshape((-1,) + envs.single_observation_space.shape) - b_logprobs = logprobs.reshape(-1) - b_actions = actions.reshape((-1,) + envs.single_action_space.shape) - b_advantages = advantages.reshape(-1) - b_returns = returns.reshape(-1) - b_values = values.reshape(-1) - - # Optimizing the policy and value network - b_inds = np.arange(args.batch_size) - clipfracs = [] - for epoch in range(args.update_epochs): - np.random.shuffle(b_inds) - for start in range(0, args.batch_size, args.minibatch_size): - end = start + args.minibatch_size - mb_inds = b_inds[start:end] - - _, newlogprob, entropy, newvalue = agent.get_action_and_value( - b_obs[mb_inds], b_actions.long()[mb_inds] - ) - logratio = newlogprob - b_logprobs[mb_inds] - ratio = logratio.exp() - - with torch.no_grad(): - # calculate approx_kl http://joschu.net/blog/kl-approx.html - old_approx_kl = (-logratio).mean() - approx_kl = ((ratio - 1) - logratio).mean() - clipfracs += [((ratio - 1.0).abs() > args.clip_coef).float().mean().item()] - - mb_advantages = b_advantages[mb_inds] - if args.norm_adv: - mb_advantages = (mb_advantages - mb_advantages.mean()) / (mb_advantages.std() + 1e-8) - - # Policy loss - pg_loss1 = -mb_advantages * ratio - pg_loss2 = -mb_advantages * torch.clamp(ratio, 1 - args.clip_coef, 1 + args.clip_coef) - pg_loss = torch.max(pg_loss1, pg_loss2).mean() - - # Value loss - newvalue = newvalue.view(-1) - if args.clip_vloss: - v_loss_unclipped = (newvalue - b_returns[mb_inds]) ** 2 - v_clipped = b_values[mb_inds] + torch.clamp( - newvalue - b_values[mb_inds], - -args.clip_coef, - args.clip_coef, - ) - v_loss_clipped = (v_clipped - b_returns[mb_inds]) ** 2 - v_loss_max = torch.max(v_loss_unclipped, v_loss_clipped) - v_loss = 0.5 * v_loss_max.mean() - else: - v_loss = 0.5 * ((newvalue - b_returns[mb_inds]) ** 2).mean() - - entropy_loss = entropy.mean() - loss = pg_loss - args.ent_coef * entropy_loss + v_loss * args.vf_coef - - optimizer.zero_grad() - loss.backward() - nn.utils.clip_grad_norm_(agent.parameters(), args.max_grad_norm) - optimizer.step() - - if args.target_kl is not None: - if approx_kl > args.target_kl: - break - - y_pred, y_true = b_values.cpu().numpy(), b_returns.cpu().numpy() - var_y = np.var(y_true) - explained_var = np.nan if var_y == 0 else 1 - np.var(y_true - y_pred) / var_y - - # TRY NOT TO MODIFY: record rewards for plotting purposes - writer.add_scalar("charts/learning_rate", optimizer.param_groups[0]["lr"], global_step) - writer.add_scalar("losses/value_loss", v_loss.item(), global_step) - writer.add_scalar("losses/policy_loss", pg_loss.item(), global_step) - writer.add_scalar("losses/entropy", entropy_loss.item(), global_step) - writer.add_scalar("losses/old_approx_kl", old_approx_kl.item(), global_step) - writer.add_scalar("losses/approx_kl", approx_kl.item(), global_step) - writer.add_scalar("losses/clipfrac", np.mean(clipfracs), global_step) - writer.add_scalar("losses/explained_variance", explained_var, global_step) - print("SPS:", int(global_step / (time.time() - start_time))) - writer.add_scalar("charts/SPS", int(global_step / (time.time() - start_time)), global_step) - - envs.close() - writer.close() - - # Create the evaluation environment - eval_env = gym.make(args.env_id) - - package_to_hub( - repo_id=args.repo_id, - model=agent, # The model we want to save - hyperparameters=args, - eval_env=gym.make(args.env_id), - logs=f"runs/{run_name}", - ) -``` - -To be able to share your model with the community there are three more steps to follow: - -1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join - -2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website. -- Create a new token (https://huggingface.co/settings/tokens) **with write role** - -Create HF Token - -- Copy the token -- Run the cell below and paste the token - -```python -from huggingface_hub import notebook_login -notebook_login() -!git config --global credential.helper store -``` - -If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login` - -## Let's start the training 🔥 -- Now that you've coded from scratch PPO and added the Hugging Face Integration, we're ready to start the training 🔥 - -- First, you need to copy all your code to a file you create called `ppo.py` - -PPO - -PPO - -- Now we just need to run this python script using `python .py` with the additional parameters we defined with `argparse` - -- You should modify more hyperparameters otherwise the training will not be super stable. - -```python -!python ppo.py --env-id="LunarLander-v2" --repo-id="YOUR_REPO_ID" --total-timesteps=50000 -``` - -## Some additional challenges 🏆 -The best way to learn **is to try things by your own**! Why not trying another environment? - - -See you on Unit 8, part 2 where we going to train agents to play Doom 🔥 -## Keep learning, stay awesome 🤗 \ No newline at end of file diff --git a/units/en/unit8/hands-on-cleanrl.mdx b/units/en/unit8/hands-on-cleanrl.mdx index 65a1270..88d1033 100644 --- a/units/en/unit8/hands-on-cleanrl.mdx +++ b/units/en/unit8/hands-on-cleanrl.mdx @@ -39,8 +39,9 @@ LunarLander-v2 is the first environment you used when you started this course. A Let's get started! 🚀 -Open In Colab +The colab notebook: +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/master/notebooks/unit8/unit8_part1.ipynb) # Unit 8: Proximal Policy Gradient (PPO) with PyTorch 🤖 diff --git a/units/en/unit8/hands-on-sf.mdx b/units/en/unit8/hands-on-sf.mdx index a93b860..6bde46e 100644 --- a/units/en/unit8/hands-on-sf.mdx +++ b/units/en/unit8/hands-on-sf.mdx @@ -1 +1,430 @@ -# Hands-on Doom +# Hands-on: advanced Deep Reinforcement Learning. Using Sample Factory to play Doom from pixels + + + +The colab notebook: +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/deep-rl-class/blob/master/notebooks/unit8/unit8_part2.ipynb) + +# Unit 8 Part 2: Advanced Deep Reinforcement Learning. Using Sample Factory to play Doom from pixels + +Thumbnail + +In this notebook, we will learn how to train a Deep Neural Network to collect objects in a 3D environment based on the game of Doom, a video of the resulting policy is shown below. We train this policy using [Sample Factory](https://www.samplefactory.dev/), an asynchronous implementation of the PPO algorithm. + +Please note the following points: + +* [Sample Factory](https://www.samplefactory.dev/) is an advanced RL framework and **only functions on Linux and Mac** (not Windows). + +* The framework performs best on a **GPU machine with many CPU cores**, where it can achieve speeds of 100k interactions per second. The resources available on a standard Colab notebook **limit the performance of this library**. So the speed in this setting **does not reflect the real-world performance**. +* Benchmarks for Sample Factory are available in a number of settings, check out the [examples](https://github.com/alex-petrenko/sample-factory/tree/master/sf_examples) if you want to find out more. + + +```python +from IPython.display import HTML + +HTML( + """""" +) +``` + +To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model: + +- `doom_health_gathering_supreme` get a result of >= 5. + +To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward** + +If you don't find your model, **go to the bottom of the page and click on the refresh button** + +For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process + +## Set the GPU 💪 + +- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type` + +GPU Step 1 + +- `Hardware Accelerator > GPU` + +GPU Step 2 + +Before starting to train our agent, let's **study the library and environments we're going to use**. + +## Sample Factory + +[Sample Factory](https://www.samplefactory.dev/) is one of the **fastest RL libraries focused on very efficient synchronous and asynchronous implementations of policy gradients (PPO)**. + +Sample Factory is thoroughly **tested, used by many researchers and practitioners**, and is actively maintained. Our implementation is known to **reach SOTA performance in a variety of domains while minimizing RL experiment training time and hardware requirements**. + +Sample factory + +### Key features + +- Highly optimized algorithm [architecture](https://www.samplefactory.dev/06-architecture/overview/) for maximum learning throughput +- [Synchronous and asynchronous](https://www.samplefactory.dev/07-advanced-topics/sync-async/) training regimes +- [Serial (single-process) mode](https://www.samplefactory.dev/07-advanced-topics/serial-mode/) for easy debugging +- Optimal performance in both CPU-based and [GPU-accelerated environments](https://www.samplefactory.dev/09-environment-integrations/isaacgym/) +- Single- & multi-agent training, self-play, supports [training multiple policies](https://www.samplefactory.dev/07-advanced-topics/multi-policy-training/) at once on one or many GPUs +- Population-Based Training ([PBT](https://www.samplefactory.dev/07-advanced-topics/pbt/)) +- Discrete, continuous, hybrid action spaces +- Vector-based, image-based, dictionary observation spaces +- Automatically creates a model architecture by parsing action/observation space specification. Supports [custom model architectures](https://www.samplefactory.dev/03-customization/custom-models/) +- Designed to be imported into other projects, [custom environments](https://www.samplefactory.dev/03-customization/custom-environments/) are first-class citizens +- Detailed [WandB and Tensorboard summaries](https://www.samplefactory.dev/05-monitoring/metrics-reference/), [custom metrics](https://www.samplefactory.dev/05-monitoring/custom-metrics/) +- [HuggingFace 🤗 integration](https://www.samplefactory.dev/10-huggingface/huggingface/) (upload trained models and metrics to the Hub) +- [Multiple](https://www.samplefactory.dev/09-environment-integrations/mujoco/) [example](https://www.samplefactory.dev/09-environment-integrations/atari/) [environment](https://www.samplefactory.dev/09-environment-integrations/vizdoom/) [integrations](https://www.samplefactory.dev/09-environment-integrations/dmlab/) with tuned parameters and trained models + +All of the above policies are available on the 🤗 hub. Search for the tag [sample-factory](https://huggingface.co/models?library=sample-factory&sort=downloads) + +### How sample-factory works + +Sample-factory is one of the **most highly optimized RL implementations available to the community**. + +It works by **spawning multiple processes that run rollout workers, inference workers and a learner worker**. + +The *workers* **communicate through shared memory, which lowers the communication cost between processes**. + +The *rollout workers* interact with the environment and send observations to the *inference workers*. + +The *inferences workers* query a fixed version of the policy and **send actions back to the rollout worker**. + +After *k* steps the rollout works send a trajectory of experience to the learner worker, **which it uses to update the agent’s policy network**. + +Sample factory + +### Actor Critic models in Sample-factory + +Actor Critic models in Sample Factory are composed of three components: + +- **Encoder** - Process input observations (images, vectors) and map them to a vector. This is the part of the model you will most likely want to customize. +- **Core** - Intergrate vectors from one or more encoders, can optionally include a single- or multi-layer LSTM/GRU in a memory-based agent. +- **Decoder** - Apply additional layers to the output of the model core before computing the policy and value outputs. + +The library has been designed to automatically support any observation and action spaces. Users can easily add their custom models. You can find out more in the [documentation](https://www.samplefactory.dev/03-customization/custom-models/#actor-critic-models-in-sample-factory). + +## ViZDoom + +[ViZDoom](https://vizdoom.cs.put.edu.pl/) is an **open-source python interface for the Doom Engine**. + +The library was created in 2016 by Marek Wydmuch, Michal Kempka at the Institute of Computing Science, Poznan University of Technology, Poland. + +The library enables the **training of agents directly from the screen pixels in a number of scenarios**, including team deathmatch, shown in the video below. Because the ViZDoom environment is based on a game the was created in the 90s, it can be run on modern hardware at accelerated speeds, **allowing us to learn complex AI behaviors fairly quickly**. + +The library includes feature such as: + +- Multi-platform (Linux, macOS, Windows), +- API for Python and C++, +- [OpenAI Gym](https://www.gymlibrary.dev/) environment wrappers +- Easy-to-create custom scenarios (visual editors, scripting language, and examples available), +- Async and sync single-player and multiplayer modes, +- Lightweight (few MBs) and fast (up to 7000 fps in sync mode, single-threaded), +- Customizable resolution and rendering parameters, +- Access to the depth buffer (3D vision), +- Automatic labeling of game objects visible in the frame, +- Access to the audio buffer +- Access to the list of actors/objects and map geometry, +- Off-screen rendering and episode recording, +- Time scaling in async mode. + +## We first need to install some dependencies that are required for the ViZDoom environment + +Now that our Colab runtime is set up, we can start by installing the dependencies required to run ViZDoom on linux. + +If you are following on your machine on Mac, you will want to follow the installation instructions on the [github page](https://github.com/Farama-Foundation/ViZDoom/blob/master/doc/Quickstart.md#-quickstart-for-macos-and-anaconda3-python-36). + +```python +# Install ViZDoom deps from +# https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md#-linux + +apt-get install build-essential zlib1g-dev libsdl2-dev libjpeg-dev \ +nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \ +libopenal-dev timidity libwildmidi-dev unzip ffmpeg + +# Boost libraries +apt-get install libboost-all-dev + +# Lua binding dependencies +apt-get install liblua5.1-dev +``` + +## Then we can install Sample Factory and ViZDoom + +- This can take 7min + +```bash +pip install sample-factory +pip install vizdoom +``` + +## Setting up the Doom Environment in sample-factory + +```python +import functools + +from sample_factory.algo.utils.context import global_model_factory +from sample_factory.cfg.arguments import parse_full_cfg, parse_sf_args +from sample_factory.envs.env_utils import register_env +from sample_factory.train import run_rl + +from sf_examples.vizdoom.doom.doom_model import make_vizdoom_encoder +from sf_examples.vizdoom.doom.doom_params import add_doom_env_args, doom_override_defaults +from sf_examples.vizdoom.doom.doom_utils import DOOM_ENVS, make_doom_env_from_spec + + +# Registers all the ViZDoom environments +def register_vizdoom_envs(): + for env_spec in DOOM_ENVS: + make_env_func = functools.partial(make_doom_env_from_spec, env_spec) + register_env(env_spec.name, make_env_func) + + +# Sample Factory allows the registration of a custom Neural Network architecture +# See https://github.com/alex-petrenko/sample-factory/blob/master/sf_examples/vizdoom/doom/doom_model.py for more details +def register_vizdoom_models(): + global_model_factory().register_encoder_factory(make_vizdoom_encoder) + + +def register_vizdoom_components(): + register_vizdoom_envs() + register_vizdoom_models() + + +# parse the command line args and create a config +def parse_vizdoom_cfg(argv=None, evaluation=False): + parser, _ = parse_sf_args(argv=argv, evaluation=evaluation) + # parameters specific to Doom envs + add_doom_env_args(parser) + # override Doom default values for algo parameters + doom_override_defaults(parser) + # second parsing pass yields the final configuration + final_cfg = parse_full_cfg(parser, argv) + return final_cfg +``` + +Now that the setup if complete, we can train the agent. We have chosen here to learn a ViZDoom task called `Health Gathering Supreme`. + +### The scenario: Health Gathering Supreme + +Health-Gathering-Supreme + + + +The objective of this scenario is to **teach the agent how to survive without knowing what makes him survive**. Agent know only that **life is precious** and death is bad so **it must learn what prolongs his existence and that his health is connected with it**. + +Map is a rectangle containing walls and with a green, acidic floor which **hurts the player periodically**. Initially there are some medkits spread uniformly over the map. A new medkit falls from the skies every now and then. **Medkits heal some portions of player's health** - to survive agent needs to pick them up. Episode finishes after player's death or on timeout. + +Further configuration: +- Living_reward = 1 +- 3 available buttons: turn left, turn right, move forward +- 1 available game variable: HEALTH +- death penalty = 100 + +You can find out more about the scenarios available in ViZDoom [here](https://github.com/Farama-Foundation/ViZDoom/tree/master/scenarios). + +There are also a number of more complex scenarios that have been create for ViZDoom, such as the ones detailed on [this github page](https://github.com/edbeeching/3d_control_deep_rl). + + + +## Training the agent + +- We're going to train the agent for 4000000 steps it will take approximately 20min + +```python +## Start the training, this should take around 15 minutes +register_vizdoom_components() + +# The scenario we train on today is health gathering +# other scenarios include "doom_basic", "doom_two_colors_easy", "doom_dm", "doom_dwango5", "doom_my_way_home", "doom_deadly_corridor", "doom_defend_the_center", "doom_defend_the_line" +env = "doom_health_gathering_supreme" +cfg = parse_vizdoom_cfg( + argv=[f"--env={env}", "--num_workers=8", "--num_envs_per_worker=4", "--train_for_env_steps=4000000"] +) + +status = run_rl(cfg) +``` + +## Let's take a look at the performance of the trained policy and output a video of the agent. + +```python +from sample_factory.enjoy import enjoy + +cfg = parse_vizdoom_cfg( + argv=[f"--env={env}", "--num_workers=1", "--save_video", "--no_render", "--max_num_episodes=10"], evaluation=True +) +status = enjoy(cfg) +``` + +## Now lets visualize the performance of the agent + +```python +from base64 import b64encode +from IPython.display import HTML + +mp4 = open("/content/train_dir/default_experiment/replay.mp4", "rb").read() +data_url = "data:video/mp4;base64," + b64encode(mp4).decode() +HTML( + """ + +""" + % data_url +) +``` + +The agent has learned something, but its performance could be better. We would clearly need to train for longer. But let's upload this model to the Hub. + +## Now lets upload your checkpoint and video to the Hugging Face Hub + + + + +To be able to share your model with the community there are three more steps to follow: + +1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join + +2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website. +- Create a new token (https://huggingface.co/settings/tokens) **with write role** + +Create HF Token + +- Copy the token +- Run the cell below and paste the token + +If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login` + +```python +from huggingface_hub import notebook_login +notebook_login() +!git config --global credential.helper store +``` + +```python +from sample_factory.enjoy import enjoy + +hf_username = "ThomasSimonini" # insert your HuggingFace username here + +cfg = parse_vizdoom_cfg( + argv=[ + f"--env={env}", + "--num_workers=1", + "--save_video", + "--no_render", + "--max_num_episodes=10", + "--max_num_frames=100000", + "--push_to_hub", + f"--hf_repository={hf_username}/rl_course_vizdoom_health_gathering_supreme", + ], + evaluation=True, +) +status = enjoy(cfg) +``` + +## Let's load another model + + + + +This agent's performance was good, but can do better! Let's download and visualize an agent trained for 10B timesteps from the hub. + +```bash +#download the agent from the hub +python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_health_gathering_supreme_2222 -d ./train_dir +``` + +```bash +ls train_dir/doom_health_gathering_supreme_2222 +``` + +```python +env = "doom_health_gathering_supreme" +cfg = parse_vizdoom_cfg( + argv=[ + f"--env={env}", + "--num_workers=1", + "--save_video", + "--no_render", + "--max_num_episodes=10", + "--experiment=doom_health_gathering_supreme_2222", + "--train_dir=train_dir", + ], + evaluation=True, +) +status = enjoy(cfg) +``` + +```python +mp4 = open("/content/train_dir/doom_health_gathering_supreme_2222/replay.mp4", "rb").read() +data_url = "data:video/mp4;base64," + b64encode(mp4).decode() +HTML( + """ + +""" + % data_url +) +``` + +## Some additional challenges 🏆: Doom Deathmatch + +Training an agent to play a Doom deathmatch **takes many hours on a more beefy machine than is available in Colab**. + +Fortunately, we have have **already trained an agent in this scenario and it is available in the 🤗 Hub!** Let’s download the model and visualize the agent’s performance. + +```python +# Download the agent from the hub +python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_deathmatch_bots_2222 -d ./train_dir +``` + +Given the agent plays for a long time the video generation can take **10 minutes**. + +```python +from sample_factory.enjoy import enjoy + +register_vizdoom_components() +env = "doom_deathmatch_bots" +cfg = parse_vizdoom_cfg( + argv=[ + f"--env={env}", + "--num_workers=1", + "--save_video", + "--no_render", + "--max_num_episodes=1", + "--experiment=doom_deathmatch_bots_2222", + "--train_dir=train_dir", + ], + evaluation=True, +) +status = enjoy(cfg) +mp4 = open("/content/train_dir/doom_deathmatch_bots_2222/replay.mp4", "rb").read() +data_url = "data:video/mp4;base64," + b64encode(mp4).decode() +HTML( + """ + +""" + % data_url +) +``` + + +You **can try to train your agent in this environment** using the code above, but not on colab. +**Good luck 🤞** + +If you prefer an easier scenario, **why not try training in another ViZDoom scenario such as `doom_deadly_corridor` or `doom_defend_the_center`.** + + + +--- + + +This concludes the last unit. But we are not finished yet! 🤗 The following **bonus section include some of the most interesting, advanced and cutting edge work in Deep Reinforcement Learning**. + +## Keep learning, stay awesome 🤗 From 516d082981d150626f7b5825da9ef8f06aa878d7 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Tue, 21 Feb 2023 22:07:59 +0100 Subject: [PATCH 13/14] Apply suggestions from code review Co-authored-by: Nathan Raw --- units/en/unit8/conclusion-sf.mdx | 6 +++--- units/en/unit8/introduction-sf.mdx | 2 +- units/en/unit8/introduction.mdx | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/units/en/unit8/conclusion-sf.mdx b/units/en/unit8/conclusion-sf.mdx index ac9893f..34c85df 100644 --- a/units/en/unit8/conclusion-sf.mdx +++ b/units/en/unit8/conclusion-sf.mdx @@ -1,12 +1,12 @@ # Conclusion -That's all for today. Congrats on finishing this Unit and the tutorial! +That's all for today. Congrats on finishing this Unit and the tutorial! ⭐️ -Now that you've successfully trained your Doom agent, why not try deathmatch? But remember, that's a much more complex level than the one you've just trained. **But it's a nice experiment, and I advise you to try it.** +Now that you've successfully trained your Doom agent, why not try deathmatch? Remember, that's a much more complex level than the one you've just trained, **but it's a nice experiment and I advise you to try it.** If you do it, don't hesitate to share your model in the `#rl-i-made-this` channel in our [discord server](https://www.hf.co/join/discord). -This concludes the last unit. But we are not finished yet! 🤗 The following **bonus unit include some of the most interesting, advanced and cutting edge work in Deep Reinforcement Learning**. +This concludes the last unit, but we are not finished yet! 🤗 The following **bonus unit includes some of the most interesting, advanced and cutting edge work in Deep Reinforcement Learning**. See you next time 🔥, diff --git a/units/en/unit8/introduction-sf.mdx b/units/en/unit8/introduction-sf.mdx index 486b416..9250cf4 100644 --- a/units/en/unit8/introduction-sf.mdx +++ b/units/en/unit8/introduction-sf.mdx @@ -4,7 +4,7 @@ In this second part of Unit 8, we'll get deeper into PPO optimization by using [Sample-Factory](https://samplefactory.dev/), an **asynchronous implementation of the PPO algorithm**, to train our agent playing [vizdoom](https://vizdoom.cs.put.edu.pl/) (an open source version of Doom). -During the notebook, **you'll train your agent to play Health Gathering level**, where our agent must collect health packs to avoid dying. And after that, you can **train your agent to play more complex versions of the levels, such as Deathmatch**. +In the notebook, **you'll train your agent to play the Health Gathering level**, where the agent must collect health packs to avoid dying. After that, you can **train your agent to play more complex levels, such as Deathmatch**. Environment diff --git a/units/en/unit8/introduction.mdx b/units/en/unit8/introduction.mdx index 7657ec1..6e8645d 100644 --- a/units/en/unit8/introduction.mdx +++ b/units/en/unit8/introduction.mdx @@ -17,7 +17,7 @@ This Unit is in two parts:
Environment -
This is the environments you're going to use to train your agents: VizDoom environments
+
These are the environments you're going to use to train your agents: VizDoom environments
Sounds exciting? Let's get started! 🚀 From 1a2374cbd09490752b14aee6fdecfe64db411550 Mon Sep 17 00:00:00 2001 From: Thomas Simonini Date: Thu, 23 Feb 2023 16:21:46 +0100 Subject: [PATCH 14/14] Update hands-on-sf.mdx --- units/en/unit8/hands-on-sf.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/units/en/unit8/hands-on-sf.mdx b/units/en/unit8/hands-on-sf.mdx index 6bde46e..1a71f2f 100644 --- a/units/en/unit8/hands-on-sf.mdx +++ b/units/en/unit8/hands-on-sf.mdx @@ -2,7 +2,7 @@