mirror of
https://github.com/huggingface/deep-rl-class.git
synced 2026-02-03 02:14:53 +08:00
691 lines
28 KiB
Plaintext
691 lines
28 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "view-in-github",
|
||
"colab_type": "text"
|
||
},
|
||
"source": [
|
||
"<a href=\"https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/notebooks/unit8/unit8_part2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "OVx1gdg9wt9t"
|
||
},
|
||
"source": [
|
||
"# Unit 8 Part 2: Advanced Deep Reinforcement Learning. Using Sample Factory to play Doom from pixels\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit9/thumbnail2.png\" alt=\"Thumbnail\"/>\n",
|
||
"\n",
|
||
"In this notebook, we will learn how to train a Deep Neural Network to collect objects in a 3D environment based on the game of Doom, a video of the resulting policy is shown below. We train this policy using [Sample Factory](https://www.samplefactory.dev/), an asynchronous implementation of the PPO algorithm.\n",
|
||
"\n",
|
||
"Please note the following points:\n",
|
||
"\n",
|
||
"* [Sample Factory](https://www.samplefactory.dev/) is an advanced RL framework and **only functions on Linux and Mac** (not Windows).\n",
|
||
"\n",
|
||
"* The framework performs best on a **GPU machine with many CPU cores**, where it can achieve speeds of 100k interactions per second. The resources available on a standard Colab notebook **limit the performance of this library**. So the speed in this setting **does not reflect the real-world performance**.\n",
|
||
"* Benchmarks for Sample Factory are available in a number of settings, check out the [examples](https://github.com/alex-petrenko/sample-factory/tree/master/sf_examples) if you want to find out more.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "I6_67HfI1CKg"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from IPython.display import HTML\n",
|
||
"\n",
|
||
"HTML('''<video width=\"640\" height=\"480\" controls>\n",
|
||
" <source src=\"https://huggingface.co/edbeeching/doom_health_gathering_supreme_3333/resolve/main/replay.mp4\"\n",
|
||
" type=\"video/mp4\">Your browser does not support the video tag.</video>'''\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "DgHRAsYEXdyw"
|
||
},
|
||
"source": [
|
||
"To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model:\n",
|
||
"\n",
|
||
"- `doom_health_gathering_supreme` get a result of >= 5.\n",
|
||
"\n",
|
||
"To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward**\n",
|
||
"\n",
|
||
"If you don't find your model, **go to the bottom of the page and click on the refresh button**\n",
|
||
"\n",
|
||
"For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "PU4FVzaoM6fC"
|
||
},
|
||
"source": [
|
||
"## Set the GPU 💪\n",
|
||
"- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\" alt=\"GPU Step 1\">"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "KV0NyFdQM9ZG"
|
||
},
|
||
"source": [
|
||
"- `Hardware Accelerator > GPU`\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\" alt=\"GPU Step 2\">"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "-fSy5HzUcMWB"
|
||
},
|
||
"source": [
|
||
"Before starting to train our agent, let's **study the library and environments we're going to use**.\n",
|
||
"\n",
|
||
"## Sample Factory\n",
|
||
"\n",
|
||
"[Sample Factory](https://www.samplefactory.dev/) is one of the **fastest RL libraries focused on very efficient synchronous and asynchronous implementations of policy gradients (PPO)**.\n",
|
||
"\n",
|
||
"Sample Factory is thoroughly **tested, used by many researchers and practitioners**, and is actively maintained. Our implementation is known to **reach SOTA performance in a variety of domains while minimizing RL experiment training time and hardware requirements**.\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit9/samplefactoryenvs.png\" alt=\"Sample factory\"/>\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"### Key features\n",
|
||
"\n",
|
||
"- Highly optimized algorithm [architecture](https://www.samplefactory.dev/06-architecture/overview/) for maximum learning throughput\n",
|
||
"- [Synchronous and asynchronous](https://www.samplefactory.dev/07-advanced-topics/sync-async/) training regimes\n",
|
||
"- [Serial (single-process) mode](https://www.samplefactory.dev/07-advanced-topics/serial-mode/) for easy debugging\n",
|
||
"- Optimal performance in both CPU-based and [GPU-accelerated environments](https://www.samplefactory.dev/09-environment-integrations/isaacgym/)\n",
|
||
"- Single- & multi-agent training, self-play, supports [training multiple policies](https://www.samplefactory.dev/07-advanced-topics/multi-policy-training/) at once on one or many GPUs\n",
|
||
"- Population-Based Training ([PBT](https://www.samplefactory.dev/07-advanced-topics/pbt/))\n",
|
||
"- Discrete, continuous, hybrid action spaces\n",
|
||
"- Vector-based, image-based, dictionary observation spaces\n",
|
||
"- Automatically creates a model architecture by parsing action/observation space specification. Supports [custom model architectures](https://www.samplefactory.dev/03-customization/custom-models/)\n",
|
||
"- Designed to be imported into other projects, [custom environments](https://www.samplefactory.dev/03-customization/custom-environments/) are first-class citizens\n",
|
||
"- Detailed [WandB and Tensorboard summaries](https://www.samplefactory.dev/05-monitoring/metrics-reference/), [custom metrics](https://www.samplefactory.dev/05-monitoring/custom-metrics/)\n",
|
||
"- [HuggingFace 🤗 integration](https://www.samplefactory.dev/10-huggingface/huggingface/) (upload trained models and metrics to the Hub)\n",
|
||
"- [Multiple](https://www.samplefactory.dev/09-environment-integrations/mujoco/) [example](https://www.samplefactory.dev/09-environment-integrations/atari/) [environment](https://www.samplefactory.dev/09-environment-integrations/vizdoom/) [integrations](https://www.samplefactory.dev/09-environment-integrations/dmlab/) with tuned parameters and trained models\n",
|
||
"\n",
|
||
"All of the above policies are available on the 🤗 hub. Search for the tag [sample-factory](https://huggingface.co/models?library=sample-factory&sort=downloads)\n",
|
||
"\n",
|
||
"### How sample-factory works\n",
|
||
"\n",
|
||
"Sample-factory is one of the **most highly optimized RL implementations available to the community**.\n",
|
||
"\n",
|
||
"It works by **spawning multiple processes that run rollout workers, inference workers and a learner worker**.\n",
|
||
"\n",
|
||
"The *workers* **communicate through shared memory, which lowers the communication cost between processes**.\n",
|
||
"\n",
|
||
"The *rollout workers* interact with the environment and send observations to the *inference workers*.\n",
|
||
"\n",
|
||
"The *inferences workers* query a fixed version of the policy and **send actions back to the rollout worker**.\n",
|
||
"\n",
|
||
"After *k* steps the rollout works send a trajectory of experience to the learner worker, **which it uses to update the agent’s policy network**.\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit9/samplefactory.png\" alt=\"Sample factory\"/>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "nB68Eb9UgC94"
|
||
},
|
||
"source": [
|
||
"### Actor Critic models in Sample-factory\n",
|
||
"\n",
|
||
"Actor Critic models in Sample Factory are composed of three components:\n",
|
||
"\n",
|
||
"- **Encoder** - Process input observations (images, vectors) and map them to a vector. This is the part of the model you will most likely want to customize.\n",
|
||
"- **Core** - Intergrate vectors from one or more encoders, can optionally include a single- or multi-layer LSTM/GRU in a memory-based agent.\n",
|
||
"- **Decoder** - Apply additional layers to the output of the model core before computing the policy and value outputs.\n",
|
||
"\n",
|
||
"The library has been designed to automatically support any observation and action spaces. Users can easily add their custom models. You can find out more in the [documentation](https://www.samplefactory.dev/03-customization/custom-models/#actor-critic-models-in-sample-factory)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "ez5UhUtYcWXF"
|
||
},
|
||
"source": [
|
||
"## ViZDoom\n",
|
||
"\n",
|
||
"[ViZDoom](https://vizdoom.cs.put.edu.pl/) is an **open-source python interface for the Doom Engine**.\n",
|
||
"\n",
|
||
"The library was created in 2016 by Marek Wydmuch, Michal Kempka at the Institute of Computing Science, Poznan University of Technology, Poland.\n",
|
||
"\n",
|
||
"The library enables the **training of agents directly from the screen pixels in a number of scenarios**, including team deathmatch, shown in the video below. Because the ViZDoom environment is based on a game the was created in the 90s, it can be run on modern hardware at accelerated speeds, **allowing us to learn complex AI behaviors fairly quickly**.\n",
|
||
"\n",
|
||
"The library includes feature such as:\n",
|
||
"\n",
|
||
"- Multi-platform (Linux, macOS, Windows),\n",
|
||
"- API for Python and C++,\n",
|
||
"- [OpenAI Gym](https://www.gymlibrary.dev/) environment wrappers\n",
|
||
"- Easy-to-create custom scenarios (visual editors, scripting language, and examples available),\n",
|
||
"- Async and sync single-player and multiplayer modes,\n",
|
||
"- Lightweight (few MBs) and fast (up to 7000 fps in sync mode, single-threaded),\n",
|
||
"- Customizable resolution and rendering parameters,\n",
|
||
"- Access to the depth buffer (3D vision),\n",
|
||
"- Automatic labeling of game objects visible in the frame,\n",
|
||
"- Access to the audio buffer\n",
|
||
"- Access to the list of actors/objects and map geometry,\n",
|
||
"- Off-screen rendering and episode recording,\n",
|
||
"- Time scaling in async mode."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "wAMwza0d5QVj"
|
||
},
|
||
"source": [
|
||
"## We first need to install some dependencies that are required for the ViZDoom environment\n",
|
||
"\n",
|
||
"Now that our Colab runtime is set up, we can start by installing the dependencies required to run ViZDoom on linux.\n",
|
||
"\n",
|
||
"If you are following on your machine on Mac, you will want to follow the installation instructions on the [github page](https://github.com/Farama-Foundation/ViZDoom/blob/master/doc/Quickstart.md#-quickstart-for-macos-and-anaconda3-python-36)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "RJMxkaldwIVx"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"%%capture\n",
|
||
"%%bash\n",
|
||
"# Install ViZDoom deps from\n",
|
||
"# https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md#-linux\n",
|
||
"\n",
|
||
"apt-get install build-essential zlib1g-dev libsdl2-dev libjpeg-dev \\\n",
|
||
"nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \\\n",
|
||
"libopenal-dev timidity libwildmidi-dev unzip ffmpeg\n",
|
||
"\n",
|
||
"# Boost libraries\n",
|
||
"apt-get install libboost-all-dev\n",
|
||
"\n",
|
||
"# Lua binding dependencies\n",
|
||
"apt-get install liblua5.1-dev"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "JT4att2c57MW"
|
||
},
|
||
"source": [
|
||
"## Then we can install Sample Factory and ViZDoom\n",
|
||
"- This can take 7min"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "bbqfPZnIsvA6"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# install python libraries\n",
|
||
"# thanks toinsson\n",
|
||
"!pip install faster-fifo==1.4.2\n",
|
||
"!pip install vizdoom"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"!pip install sample-factory==2.1.1"
|
||
],
|
||
"metadata": {
|
||
"id": "alxUt7Au-O8e"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "1jizouGpghUZ"
|
||
},
|
||
"source": [
|
||
"## Setting up the Doom Environment in sample-factory"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "bCgZbeiavcDU"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import functools\n",
|
||
"\n",
|
||
"from sample_factory.algo.utils.context import global_model_factory\n",
|
||
"from sample_factory.cfg.arguments import parse_full_cfg, parse_sf_args\n",
|
||
"from sample_factory.envs.env_utils import register_env\n",
|
||
"from sample_factory.train import run_rl\n",
|
||
"\n",
|
||
"from sf_examples.vizdoom.doom.doom_model import make_vizdoom_encoder\n",
|
||
"from sf_examples.vizdoom.doom.doom_params import add_doom_env_args, doom_override_defaults\n",
|
||
"from sf_examples.vizdoom.doom.doom_utils import DOOM_ENVS, make_doom_env_from_spec\n",
|
||
"\n",
|
||
"\n",
|
||
"# Registers all the ViZDoom environments\n",
|
||
"def register_vizdoom_envs():\n",
|
||
" for env_spec in DOOM_ENVS:\n",
|
||
" make_env_func = functools.partial(make_doom_env_from_spec, env_spec)\n",
|
||
" register_env(env_spec.name, make_env_func)\n",
|
||
"\n",
|
||
"# Sample Factory allows the registration of a custom Neural Network architecture\n",
|
||
"# See https://github.com/alex-petrenko/sample-factory/blob/master/sf_examples/vizdoom/doom/doom_model.py for more details\n",
|
||
"def register_vizdoom_models():\n",
|
||
" global_model_factory().register_encoder_factory(make_vizdoom_encoder)\n",
|
||
"\n",
|
||
"\n",
|
||
"def register_vizdoom_components():\n",
|
||
" register_vizdoom_envs()\n",
|
||
" register_vizdoom_models()\n",
|
||
"\n",
|
||
"# parse the command line args and create a config\n",
|
||
"def parse_vizdoom_cfg(argv=None, evaluation=False):\n",
|
||
" parser, _ = parse_sf_args(argv=argv, evaluation=evaluation)\n",
|
||
" # parameters specific to Doom envs\n",
|
||
" add_doom_env_args(parser)\n",
|
||
" # override Doom default values for algo parameters\n",
|
||
" doom_override_defaults(parser)\n",
|
||
" # second parsing pass yields the final configuration\n",
|
||
" final_cfg = parse_full_cfg(parser, argv)\n",
|
||
" return final_cfg"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "sgRy6wnrgnij"
|
||
},
|
||
"source": [
|
||
"Now that the setup if complete, we can train the agent. We have chosen here to learn a ViZDoom task called `Health Gathering Supreme`.\n",
|
||
"\n",
|
||
"### The scenario: Health Gathering Supreme\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit9/Health-Gathering-Supreme.png\" alt=\"Health-Gathering-Supreme\"/>\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"The objective of this scenario is to **teach the agent how to survive without knowing what makes him survive**. Agent know only that **life is precious** and death is bad so **it must learn what prolongs his existence and that his health is connected with it**.\n",
|
||
"\n",
|
||
"Map is a rectangle containing walls and with a green, acidic floor which **hurts the player periodically**. Initially there are some medkits spread uniformly over the map. A new medkit falls from the skies every now and then. **Medkits heal some portions of player's health** - to survive agent needs to pick them up. Episode finishes after player's death or on timeout.\n",
|
||
"\n",
|
||
"Further configuration:\n",
|
||
"- Living_reward = 1\n",
|
||
"- 3 available buttons: turn left, turn right, move forward\n",
|
||
"- 1 available game variable: HEALTH\n",
|
||
"- death penalty = 100\n",
|
||
"\n",
|
||
"You can find out more about the scenarios available in ViZDoom [here](https://github.com/Farama-Foundation/ViZDoom/tree/master/scenarios).\n",
|
||
"\n",
|
||
"There are also a number of more complex scenarios that have been create for ViZDoom, such as the ones detailed on [this github page](https://github.com/edbeeching/3d_control_deep_rl).\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "siHZZ34DiZEp"
|
||
},
|
||
"source": [
|
||
"## Training the agent\n",
|
||
"- We're going to train the agent for 4000000 steps it will take approximately 20min"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "y_TeicMvyKHP"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"## Start the training, this should take around 15 minutes\n",
|
||
"register_vizdoom_components()\n",
|
||
"\n",
|
||
"# The scenario we train on today is health gathering\n",
|
||
"# other scenarios include \"doom_basic\", \"doom_two_colors_easy\", \"doom_dm\", \"doom_dwango5\", \"doom_my_way_home\", \"doom_deadly_corridor\", \"doom_defend_the_center\", \"doom_defend_the_line\"\n",
|
||
"env = \"doom_health_gathering_supreme\"\n",
|
||
"cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=8\", \"--num_envs_per_worker=4\", \"--train_for_env_steps=4000000\"])\n",
|
||
"\n",
|
||
"status = run_rl(cfg)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "5L0nBS9e_jqC"
|
||
},
|
||
"source": [
|
||
"## Let's take a look at the performance of the trained policy and output a video of the agent."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "MGSA4Kg5_i0j"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sample_factory.enjoy import enjoy\n",
|
||
"cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\"], evaluation=True)\n",
|
||
"status = enjoy(cfg)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Lj5L1x0WLxwB"
|
||
},
|
||
"source": [
|
||
"## Now lets visualize the performance of the agent"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "WsXhBY7JNOdJ"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from base64 import b64encode\n",
|
||
"from IPython.display import HTML\n",
|
||
"\n",
|
||
"mp4 = open('/content/train_dir/default_experiment/replay.mp4','rb').read()\n",
|
||
"data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
|
||
"HTML(\"\"\"\n",
|
||
"<video width=640 controls>\n",
|
||
" <source src=\"%s\" type=\"video/mp4\">\n",
|
||
"</video>\n",
|
||
"\"\"\" % data_url)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"The agent has learned something, but its performance could be better. We would clearly need to train for longer. But let's upload this model to the Hub."
|
||
],
|
||
"metadata": {
|
||
"id": "2A4pf_1VwPqR"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "CSQVWF0kNuy9"
|
||
},
|
||
"source": [
|
||
"## Now lets upload your checkpoint and video to the Hugging Face Hub\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "JquRrWytA6eo"
|
||
},
|
||
"source": [
|
||
"To be able to share your model with the community there are three more steps to follow:\n",
|
||
"\n",
|
||
"1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
|
||
"\n",
|
||
"2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
|
||
"- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/create-token.jpg\" alt=\"Create HF Token\">\n",
|
||
"\n",
|
||
"- Copy the token\n",
|
||
"- Run the cell below and paste the token"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "_tsf2uv0g_4p"
|
||
},
|
||
"source": [
|
||
"If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "GoQm_jYSOts0"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from huggingface_hub import notebook_login\n",
|
||
"notebook_login()\n",
|
||
"!git config --global credential.helper store"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "sEawW_i0OvJV"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sample_factory.enjoy import enjoy\n",
|
||
"\n",
|
||
"hf_username = \"ThomasSimonini\" # insert your HuggingFace username here\n",
|
||
"\n",
|
||
"cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--max_num_frames=100000\", \"--push_to_hub\", f\"--hf_repository={hf_username}/rl_course_vizdoom_health_gathering_supreme\"], evaluation=True)\n",
|
||
"status = enjoy(cfg)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## Let's load another model\n",
|
||
"\n",
|
||
"\n"
|
||
],
|
||
"metadata": {
|
||
"id": "9PzeXx-qxVvw"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "mHZAWSgL5F7P"
|
||
},
|
||
"source": [
|
||
"This agent's performance was good, but can do better! Let's download and visualize an agent trained for 10B timesteps from the hub."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "Ud6DwAUl5S-l"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"#download the agent from the hub\n",
|
||
"!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_health_gathering_supreme_2222 -d ./train_dir\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "qoUJhL6x6sY5"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!ls train_dir/doom_health_gathering_supreme_2222"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "lZskc8LG8qr8"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"env = \"doom_health_gathering_supreme\"\n",
|
||
"cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--experiment=doom_health_gathering_supreme_2222\", \"--train_dir=train_dir\"], evaluation=True)\n",
|
||
"status = enjoy(cfg)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "BtzXBoj65Wmq"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"mp4 = open('/content/train_dir/doom_health_gathering_supreme_2222/replay.mp4','rb').read()\n",
|
||
"data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
|
||
"HTML(\"\"\"\n",
|
||
"<video width=640 controls>\n",
|
||
" <source src=\"%s\" type=\"video/mp4\">\n",
|
||
"</video>\n",
|
||
"\"\"\" % data_url)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## Some additional challenges 🏆: Doom Deathmatch\n",
|
||
"\n",
|
||
"Training an agent to play a Doom deathmatch **takes many hours on a more beefy machine than is available in Colab**.\n",
|
||
"\n",
|
||
"Fortunately, we have have **already trained an agent in this scenario and it is available in the 🤗 Hub!** Let’s download the model and visualize the agent’s performance."
|
||
],
|
||
"metadata": {
|
||
"id": "ie5YWC3NyKO8"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "fq3WFeus81iI"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Download the agent from the hub\n",
|
||
"!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_deathmatch_bots_2222 -d ./train_dir"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"Given the agent plays for a long time the video generation can take **10 minutes**."
|
||
],
|
||
"metadata": {
|
||
"id": "7AX_LwxR2FQ0"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "0hq6XL__85Bv"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\n",
|
||
"from sample_factory.enjoy import enjoy\n",
|
||
"register_vizdoom_components()\n",
|
||
"env = \"doom_deathmatch_bots\"\n",
|
||
"cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=1\", \"--experiment=doom_deathmatch_bots_2222\", \"--train_dir=train_dir\"], evaluation=True)\n",
|
||
"status = enjoy(cfg)\n",
|
||
"mp4 = open('/content/train_dir/doom_deathmatch_bots_2222/replay.mp4','rb').read()\n",
|
||
"data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
|
||
"HTML(\"\"\"\n",
|
||
"<video width=640 controls>\n",
|
||
" <source src=\"%s\" type=\"video/mp4\">\n",
|
||
"</video>\n",
|
||
"\"\"\" % data_url)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"\n",
|
||
"You **can try to train your agent in this environment** using the code above, but not on colab.\n",
|
||
"**Good luck 🤞**"
|
||
],
|
||
"metadata": {
|
||
"id": "N6mEC-4zyihx"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"If you prefer an easier scenario, **why not try training in another ViZDoom scenario such as `doom_deadly_corridor` or `doom_defend_the_center`.**\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"\n",
|
||
"This concludes the last unit. But we are not finished yet! 🤗 The following **bonus section include some of the most interesting, advanced and cutting edge work in Deep Reinforcement Learning**.\n",
|
||
"\n",
|
||
"## Keep learning, stay awesome 🤗"
|
||
],
|
||
"metadata": {
|
||
"id": "YnDAngN6zeeI"
|
||
}
|
||
}
|
||
],
|
||
"metadata": {
|
||
"accelerator": "GPU",
|
||
"colab": {
|
||
"provenance": [],
|
||
"collapsed_sections": [
|
||
"PU4FVzaoM6fC",
|
||
"nB68Eb9UgC94",
|
||
"ez5UhUtYcWXF",
|
||
"sgRy6wnrgnij"
|
||
],
|
||
"private_outputs": true,
|
||
"include_colab_link": true
|
||
},
|
||
"gpuClass": "standard",
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"name": "python"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 0
|
||
}
|