{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "k7xBVPzoXxOg"
},
"source": [
"# Unit 3: Deep Q-Learning with Atari Games ๐พ using RL Baselines3 Zoo\n",
"\n",
"\n",
"\n",
"In this notebook, **you'll train a Deep Q-Learning agent** playing Space Invaders using [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), a training framework based on [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.\n",
"\n",
"We're using the [RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) with no extensions such as Double-DQN, Dueling-DQN, and Prioritized Experience Replay.\n",
"\n",
"โฌ๏ธ Here is an example of what **you will achieve** โฌ๏ธ"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "J9S713biXntc"
},
"outputs": [],
"source": [
"%%html\n",
""
]
},
{
"cell_type": "markdown",
"source": [
"### ๐ฎ Environments: \n",
"\n",
"- SpacesInvadersNoFrameskip-v4 \n",
"\n",
"### ๐ RL-Library: \n",
"\n",
"- [RL-Baselines3-Zoo](https://github.com/DLR-RM/rl-baselines3-zoo)"
],
"metadata": {
"id": "ykJiGevCMVc5"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "wciHGjrFYz9m"
},
"source": [
"## Objectives of this notebook ๐\n",
"At the end of the notebook, you will:\n",
"- Be able to understand deeper **how RL Baselines3 Zoo works**.\n",
"- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score ๐ฅ.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"source": [
"## This notebook is from Deep Reinforcement Learning Course\n",
""
],
"metadata": {
"id": "TsnP0rjxMn1e"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "nw6fJHIAZd-J"
},
"source": [
"In this free course, you will:\n",
"\n",
"- ๐ Study Deep Reinforcement Learning in **theory and practice**.\n",
"- ๐งโ๐ป Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
"- ๐ค Train **agents in unique environments** \n",
"\n",
"And more check ๐ the syllabus ๐ https://simoninithomas.github.io/deep-rl-course\n",
"\n",
"Donโt forget to **sign up to the course** (we are collecting your email to be able toย **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
"\n",
"\n",
"The best way to keep in touch is to join our discord server to exchange with the community and with us ๐๐ป https://discord.gg/ydHrjt3WP5"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0vgANIBBZg1p"
},
"source": [
"## Prerequisites ๐๏ธ\n",
"Before diving into the notebook, you need to:\n",
"\n",
"๐ฒ ๐ **[Study Deep Q-Learning by reading Unit 3](https://huggingface.co/deep-rl-course/unit3/introduction)** ๐ค "
]
},
{
"cell_type": "markdown",
"source": [
"We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
],
"metadata": {
"id": "7kszpGFaRVhq"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "QR0jZtYreSI5"
},
"source": [
"# Let's train a Deep Q-Learning agent playing Atari' Space Invaders ๐พ and upload it to the Hub.\n",
"\n",
"To validate this hands-on for the certification process, you need to push your trained model to the Hub and **get a result of >= 200**.\n",
"\n",
"To find your result, go to the leaderboard and find your model, **the result = mean_reward - std of reward**\n",
"\n",
"For more information about the certification process, check this section ๐ https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process"
]
},
{
"cell_type": "markdown",
"source": [
"## An advice ๐ก\n",
"It's better to run this colab in a copy on your Google Drive, so that **if it timeouts** you still have the saved notebook on your Google Drive and do not need to fill everything from scratch.\n",
"\n",
"To do that you can either do `Ctrl + S` or `File > Save a copy in Google Drive.`\n",
"\n",
"Also, we're going to **train it for 90 minutes with 1M timesteps**. By typing `!nvidia-smi` will tell you what GPU you're using.\n",
"\n",
"And if you want to train more such 10 million steps, this will take about 9 hours, potentially resulting in Colab timing out. In that case, I recommend running this on your local computer (or somewhere else). Just click on: `File>Download`. "
],
"metadata": {
"id": "Nc8BnyVEc3Ys"
}
},
{
"cell_type": "markdown",
"source": [
"## Set the GPU ๐ช\n",
"- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
"\n",
""
],
"metadata": {
"id": "PU4FVzaoM6fC"
}
},
{
"cell_type": "markdown",
"source": [
"- `Hardware Accelerator > GPU`\n",
"\n",
""
],
"metadata": {
"id": "KV0NyFdQM9ZG"
}
},
{
"cell_type": "markdown",
"source": [
"## Create a virtual display ๐ฝ\n",
"\n",
"During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
"\n",
"Hence the following cell will install the librairies and create and run a virtual screen ๐ฅ"
],
"metadata": {
"id": "bTpYcVZVMzUI"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jV6wjQ7Be7p5"
},
"outputs": [],
"source": [
"%%capture\n",
"!apt install python-opengl\n",
"!apt install ffmpeg\n",
"!apt install xvfb\n",
"!pip3 install pyvirtualdisplay"
]
},
{
"cell_type": "code",
"source": [
"# Additional dependencies for RL Baselines3 Zoo\n",
"!apt-get install swig cmake freeglut3-dev "
],
"metadata": {
"id": "fWyKJCy_NJBX"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"!pip install pyglet==1.5.1"
],
"metadata": {
"id": "C5LwHrISW7Q5"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Virtual display\n",
"from pyvirtualdisplay import Display\n",
"\n",
"virtual_display = Display(visible=0, size=(1400, 900))\n",
"virtual_display.start()"
],
"metadata": {
"id": "ww5PQH1gNLI4"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "mYIMvl5X9NAu"
},
"source": [
"## Clone RL-Baselines3 Zoo Repo ๐\n",
"You can now directly install from python package `pip install rl_zoo3` but since we want **the full installation with extra environments and dependencies** we're going to clone `RL-Baselines3-Zoo` repository and install from source."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eu5ZDPZ09VNQ"
},
"outputs": [],
"source": [
"!git clone https://github.com/DLR-RM/rl-baselines3-zoo"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HCIoSbvbfAQh"
},
"source": [
"## Install dependencies ๐ฝ\n",
"We can now install the dependencies RL-Baselines3 Zoo needs (this can take 5min โฒ)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "s2QsFAk29h-D"
},
"outputs": [],
"source": [
"%cd /content/rl-baselines3-zoo/"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3QaOS7Xj9j1s"
},
"outputs": [],
"source": [
"!pip install setuptools==65.5.0\n",
"!pip install -r requirements.txt\n",
"# Since colab uses Python 3.9 we need to add this installation\n",
"!pip install gym[atari,accept-rom-license]==0.21.0"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5iPgzluo9z-u"
},
"source": [
"## Train our Deep Q-Learning Agent to Play Space Invaders ๐พ\n",
"\n",
"To train an agent with RL-Baselines3-Zoo, we just need to do two things:\n",
"1. We define the hyperparameters in `/content/rl-baselines3-zoo/hyperparams/dqn.yml`\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_VjblFSVDQOj"
},
"source": [
"Here we see that:\n",
"- We use the `Atari Wrapper` that preprocess the input (Frame reduction ,grayscale, stack 4 frames)\n",
"- We use `CnnPolicy`, since we use Convolutional layers to process the frames\n",
"- We train it for 10 million `n_timesteps` \n",
"- Memory (Experience Replay) size is 100000, aka the amount of experience steps you saved to train again your agent with.\n",
"\n",
"๐ก My advice is to **reduce the training timesteps to 1M,** which will take about 90 minutes on a P100. `!nvidia-smi` will tell you what GPU you're using. At 10 million steps, this will take about 9 hours, which could likely result in Colab timing out. I recommend running this on your local computer (or somewhere else). Just click on: `File>Download`. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5qTkbWrkECOJ"
},
"source": [
"In terms of hyperparameters optimization, my advice is to focus on these 3 hyperparameters:\n",
"- `learning_rate`\n",
"- `buffer_size (Experience Memory size)`\n",
"- `batch_size`\n",
"\n",
"As a good practice, you need to **check the documentation to understand what each hyperparameters does**: https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html#parameters\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Hn8bRTHvERRL"
},
"source": [
"2. We run `train.py` and save the models on `logs` folder ๐"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Xr1TVW4xfbz3"
},
"outputs": [],
"source": [
"!python train.py --algo ________ --env SpaceInvadersNoFrameskip-v4 -f _________"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SeChoX-3SZfP"
},
"source": [
"#### Solution"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PuocgdokSab9"
},
"outputs": [],
"source": [
"!python train.py --algo dqn --env SpaceInvadersNoFrameskip-v4 -f logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_dLomIiMKQaf"
},
"source": [
"## Let's evaluate our agent ๐\n",
"- RL-Baselines3-Zoo provides `enjoy.py`, a python script to evaluate our agent. In most RL libraries, we call the evaluation script `enjoy.py`.\n",
"- Let's evaluate it for 5000 timesteps ๐ฅ"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "co5um_KeKbBJ"
},
"outputs": [],
"source": [
"!python enjoy.py --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps _________ --folder logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q24K1tyWSj7t"
},
"source": [
"#### Solution"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "P_uSmwGRSk0z"
},
"outputs": [],
"source": [
"!python enjoy.py --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps 5000 --folder logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "liBeTltiHJtr"
},
"source": [
"## Publish our trained model on the Hub ๐\n",
"Now that we saw we got good results after the training, we can publish our trained model on the hub ๐ค with one line of code.\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ezbHS1q3HYVV"
},
"source": [
"By using `rl_zoo3.push_to_hub.py` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
"\n",
"This way:\n",
"- You can **showcase our work** ๐ฅ\n",
"- You can **visualize your agent playing** ๐\n",
"- You can **share with the community an agent that others can use** ๐พ\n",
"- You can **access a leaderboard ๐ to see how well your agent is performing compared to your classmates** ๐ https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XMSeZRBiHk6X"
},
"source": [
"To be able to share your model with the community there are three more steps to follow:\n",
"\n",
"1๏ธโฃ (If it's not already done) create an account to HF โก https://huggingface.co/join\n",
"\n",
"2๏ธโฃ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
"- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9O6FI0F8HnzE"
},
"source": [
"- Copy the token \n",
"- Run the cell below and past the token"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ppu9yePwHrZX"
},
"outputs": [],
"source": [
"from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
"notebook_login()\n",
"!git config --global credential.helper store"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2RVEdunPHs8B"
},
"source": [
"If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dSLwdmvhHvjw"
},
"source": [
"3๏ธโฃ We're now ready to push our trained agent to the ๐ค Hub ๐ฅ"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PW436XnhHw1H"
},
"source": [
"Let's run push_to_hub.py file to upload our trained agent to the Hub.\n",
"\n",
"`--repo-name `: The name of the repo\n",
"\n",
"`-orga`: Your Hugging Face username\n",
"\n",
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ygk2sEktTDEw"
},
"outputs": [],
"source": [
"!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name _____________________ -orga _____________________ -f logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "otgpa0rhS9wR"
},
"source": [
"#### Solution"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_HQNlAXuEhci"
},
"outputs": [],
"source": [
"!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name dqn-SpaceInvadersNoFrameskip-v4 -orga ThomasSimonini -f logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0D4F5zsTTJ-L"
},
"source": [
"###."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ff89kd2HL1_s"
},
"source": [
"Congrats ๐ฅณ you've just trained and uploaded your first Deep Q-Learning agent using RL-Baselines-3 Zoo. The script above should have displayed a link to a model repository such as https://huggingface.co/ThomasSimonini/dqn-SpaceInvadersNoFrameskip-v4. When you go to this link, you can:\n",
"\n",
"- See a **video preview of your agent** at the right. \n",
"- Click \"Files and versions\" to see all the files in the repository.\n",
"- Click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
"- A model card (`README.md` file) which gives a description of the model and the hyperparameters you used.\n",
"\n",
"Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
"\n",
"**Compare the results of your agents with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) ๐"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fyRKcCYY-dIo"
},
"source": [
"## Load a powerful trained model ๐ฅ\n",
"- The Stable-Baselines3 team uploaded **more than 150 trained Deep Reinforcement Learning agents on the Hub**.\n",
"\n",
"You can find them here: ๐ https://huggingface.co/sb3\n",
"\n",
"Some examples:\n",
"- Asteroids: https://huggingface.co/sb3/dqn-AsteroidsNoFrameskip-v4\n",
"- Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4\n",
"- Breakout: https://huggingface.co/sb3/dqn-BreakoutNoFrameskip-v4\n",
"- Road Runner: https://huggingface.co/sb3/dqn-RoadRunnerNoFrameskip-v4\n",
"\n",
"Let's load an agent playing Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "B-9QVFIROI5Y"
},
"outputs": [],
"source": [
"%%html\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7ZQNY_r6NJtC"
},
"source": [
"1. We download the model using `rl_zoo3.load_from_hub`, and place it in a new folder that we can call `rl_trained`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OdBNZHy0NGTR"
},
"outputs": [],
"source": [
"# Download model and save it into the logs/ folder\n",
"!python -m rl_zoo3.load_from_hub --algo dqn --env BeamRiderNoFrameskip-v4 -orga sb3 -f rl_trained/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LFt6hmWsNdBo"
},
"source": [
"2. Let's evaluate if for 5000 timesteps"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "aOxs0rNuN0uS"
},
"outputs": [],
"source": [
"!python enjoy.py --algo dqn --env BeamRiderNoFrameskip-v4 -n 5000 -f rl_trained/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kxMDuDfPON57"
},
"source": [
"Why not trying to train your own **Deep Q-Learning Agent playing BeamRiderNoFrameskip-v4? ๐.**\n",
"\n",
"If you want to try, check https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4#hyperparameters **in the model card, you have the hyperparameters of the trained agent.**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xL_ZtUgpOuY6"
},
"source": [
"But finding hyperparameters can be a daunting task. Fortunately, we'll see in the next Unit, how we can **use Optuna for optimizing the Hyperparameters ๐ฅ.**\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-pqaco8W-huW"
},
"source": [
"## Some additional challenges ๐\n",
"The best way to learn **is to try things by your own**!\n",
"\n",
"In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
"\n",
"Here's a list of environments you can try to train your agent with:\n",
"- BeamRiderNoFrameskip-v4\n",
"- BreakoutNoFrameskip-v4 \n",
"- EnduroNoFrameskip-v4\n",
"- PongNoFrameskip-v4\n",
"\n",
"Also, **if you want to learn to implement Deep Q-Learning by yourself**, you definitely should look at CleanRL implementation: https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "paS-XKo4-kmu"
},
"source": [
"________________________________________________________________________\n",
"Congrats on finishing this chapter!\n",
"\n",
"If youโre still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
"\n",
"Take time to really **grasp the material before continuing and try the additional challenges**. Itโs important to master these elements and having a solid foundations.\n",
"\n",
"In the next unit, **weโre going to learn about [Optuna](https://optuna.org/)**. One of the most critical task in Deep Reinforcement Learning is to find a good set of training hyperparameters. And Optuna is a library that helps you to automate the search.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5WRx7tO7-mvC"
},
"source": [
"\n",
"\n",
"### This is a course built with you ๐ท๐ฟโโ๏ธ\n",
"\n",
"Finally, we want to improve and update the course iteratively with your feedback. If you have some, please fill this form ๐ https://forms.gle/3HgA7bEHwAmmLfwh9\n",
"\n",
"We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
]
},
{
"cell_type": "markdown",
"source": [
"See you on Bonus unit 2! ๐ฅ "
],
"metadata": {
"id": "Kc3udPT-RcXc"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "fS3Xerx0fIMV"
},
"source": [
"### Keep Learning, Stay Awesome ๐ค"
]
}
],
"metadata": {
"colab": {
"private_outputs": true,
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
},
"accelerator": "GPU",
"gpuClass": "standard"
},
"nbformat": 4,
"nbformat_minor": 0
}
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
![\"Unit](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit4/thumbnail.jpg\")
\n",
"\n",
"In this notebook, **you'll train a Deep Q-Learning agent** playing Space Invaders using [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), a training framework based on [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.\n",
"\n",
"We're using the [RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) with no extensions such as Double-DQN, Dueling-DQN, and Prioritized Experience Replay.\n",
"\n",
"โฌ๏ธ Here is an example of what **you will achieve** โฌ๏ธ"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "J9S713biXntc"
},
"outputs": [],
"source": [
"%%html\n",
""
]
},
{
"cell_type": "markdown",
"source": [
"### ๐ฎ Environments: \n",
"\n",
"- SpacesInvadersNoFrameskip-v4 \n",
"\n",
"### ๐ RL-Library: \n",
"\n",
"- [RL-Baselines3-Zoo](https://github.com/DLR-RM/rl-baselines3-zoo)"
],
"metadata": {
"id": "ykJiGevCMVc5"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "wciHGjrFYz9m"
},
"source": [
"## Objectives of this notebook ๐\n",
"At the end of the notebook, you will:\n",
"- Be able to understand deeper **how RL Baselines3 Zoo works**.\n",
"- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score ๐ฅ.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"source": [
"## This notebook is from Deep Reinforcement Learning Course\n",
"![\"Deep](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/deep-rl-course-illustration.jpg\")
"
],
"metadata": {
"id": "TsnP0rjxMn1e"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "nw6fJHIAZd-J"
},
"source": [
"In this free course, you will:\n",
"\n",
"- ๐ Study Deep Reinforcement Learning in **theory and practice**.\n",
"- ๐งโ๐ป Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
"- ๐ค Train **agents in unique environments** \n",
"\n",
"And more check ๐ the syllabus ๐ https://simoninithomas.github.io/deep-rl-course\n",
"\n",
"Donโt forget to **sign up to the course** (we are collecting your email to be able toย **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
"\n",
"\n",
"The best way to keep in touch is to join our discord server to exchange with the community and with us ๐๐ป https://discord.gg/ydHrjt3WP5"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0vgANIBBZg1p"
},
"source": [
"## Prerequisites ๐๏ธ\n",
"Before diving into the notebook, you need to:\n",
"\n",
"๐ฒ ๐ **[Study Deep Q-Learning by reading Unit 3](https://huggingface.co/deep-rl-course/unit3/introduction)** ๐ค "
]
},
{
"cell_type": "markdown",
"source": [
"We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
],
"metadata": {
"id": "7kszpGFaRVhq"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "QR0jZtYreSI5"
},
"source": [
"# Let's train a Deep Q-Learning agent playing Atari' Space Invaders ๐พ and upload it to the Hub.\n",
"\n",
"To validate this hands-on for the certification process, you need to push your trained model to the Hub and **get a result of >= 200**.\n",
"\n",
"To find your result, go to the leaderboard and find your model, **the result = mean_reward - std of reward**\n",
"\n",
"For more information about the certification process, check this section ๐ https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process"
]
},
{
"cell_type": "markdown",
"source": [
"## An advice ๐ก\n",
"It's better to run this colab in a copy on your Google Drive, so that **if it timeouts** you still have the saved notebook on your Google Drive and do not need to fill everything from scratch.\n",
"\n",
"To do that you can either do `Ctrl + S` or `File > Save a copy in Google Drive.`\n",
"\n",
"Also, we're going to **train it for 90 minutes with 1M timesteps**. By typing `!nvidia-smi` will tell you what GPU you're using.\n",
"\n",
"And if you want to train more such 10 million steps, this will take about 9 hours, potentially resulting in Colab timing out. In that case, I recommend running this on your local computer (or somewhere else). Just click on: `File>Download`. "
],
"metadata": {
"id": "Nc8BnyVEc3Ys"
}
},
{
"cell_type": "markdown",
"source": [
"## Set the GPU ๐ช\n",
"- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
"\n",
"![\"GPU](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\")
"
],
"metadata": {
"id": "PU4FVzaoM6fC"
}
},
{
"cell_type": "markdown",
"source": [
"- `Hardware Accelerator > GPU`\n",
"\n",
"![\"GPU](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\")
"
],
"metadata": {
"id": "KV0NyFdQM9ZG"
}
},
{
"cell_type": "markdown",
"source": [
"## Create a virtual display ๐ฝ\n",
"\n",
"During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
"\n",
"Hence the following cell will install the librairies and create and run a virtual screen ๐ฅ"
],
"metadata": {
"id": "bTpYcVZVMzUI"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jV6wjQ7Be7p5"
},
"outputs": [],
"source": [
"%%capture\n",
"!apt install python-opengl\n",
"!apt install ffmpeg\n",
"!apt install xvfb\n",
"!pip3 install pyvirtualdisplay"
]
},
{
"cell_type": "code",
"source": [
"# Additional dependencies for RL Baselines3 Zoo\n",
"!apt-get install swig cmake freeglut3-dev "
],
"metadata": {
"id": "fWyKJCy_NJBX"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"!pip install pyglet==1.5.1"
],
"metadata": {
"id": "C5LwHrISW7Q5"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Virtual display\n",
"from pyvirtualdisplay import Display\n",
"\n",
"virtual_display = Display(visible=0, size=(1400, 900))\n",
"virtual_display.start()"
],
"metadata": {
"id": "ww5PQH1gNLI4"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "mYIMvl5X9NAu"
},
"source": [
"## Clone RL-Baselines3 Zoo Repo ๐\n",
"You can now directly install from python package `pip install rl_zoo3` but since we want **the full installation with extra environments and dependencies** we're going to clone `RL-Baselines3-Zoo` repository and install from source."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eu5ZDPZ09VNQ"
},
"outputs": [],
"source": [
"!git clone https://github.com/DLR-RM/rl-baselines3-zoo"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HCIoSbvbfAQh"
},
"source": [
"## Install dependencies ๐ฝ\n",
"We can now install the dependencies RL-Baselines3 Zoo needs (this can take 5min โฒ)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "s2QsFAk29h-D"
},
"outputs": [],
"source": [
"%cd /content/rl-baselines3-zoo/"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3QaOS7Xj9j1s"
},
"outputs": [],
"source": [
"!pip install setuptools==65.5.0\n",
"!pip install -r requirements.txt\n",
"# Since colab uses Python 3.9 we need to add this installation\n",
"!pip install gym[atari,accept-rom-license]==0.21.0"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5iPgzluo9z-u"
},
"source": [
"## Train our Deep Q-Learning Agent to Play Space Invaders ๐พ\n",
"\n",
"To train an agent with RL-Baselines3-Zoo, we just need to do two things:\n",
"1. We define the hyperparameters in `/content/rl-baselines3-zoo/hyperparams/dqn.yml`\n",
"\n",
"![\"DQN](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit3/hyperparameters.png\")
\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_VjblFSVDQOj"
},
"source": [
"Here we see that:\n",
"- We use the `Atari Wrapper` that preprocess the input (Frame reduction ,grayscale, stack 4 frames)\n",
"- We use `CnnPolicy`, since we use Convolutional layers to process the frames\n",
"- We train it for 10 million `n_timesteps` \n",
"- Memory (Experience Replay) size is 100000, aka the amount of experience steps you saved to train again your agent with.\n",
"\n",
"๐ก My advice is to **reduce the training timesteps to 1M,** which will take about 90 minutes on a P100. `!nvidia-smi` will tell you what GPU you're using. At 10 million steps, this will take about 9 hours, which could likely result in Colab timing out. I recommend running this on your local computer (or somewhere else). Just click on: `File>Download`. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5qTkbWrkECOJ"
},
"source": [
"In terms of hyperparameters optimization, my advice is to focus on these 3 hyperparameters:\n",
"- `learning_rate`\n",
"- `buffer_size (Experience Memory size)`\n",
"- `batch_size`\n",
"\n",
"As a good practice, you need to **check the documentation to understand what each hyperparameters does**: https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html#parameters\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Hn8bRTHvERRL"
},
"source": [
"2. We run `train.py` and save the models on `logs` folder ๐"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Xr1TVW4xfbz3"
},
"outputs": [],
"source": [
"!python train.py --algo ________ --env SpaceInvadersNoFrameskip-v4 -f _________"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SeChoX-3SZfP"
},
"source": [
"#### Solution"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PuocgdokSab9"
},
"outputs": [],
"source": [
"!python train.py --algo dqn --env SpaceInvadersNoFrameskip-v4 -f logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_dLomIiMKQaf"
},
"source": [
"## Let's evaluate our agent ๐\n",
"- RL-Baselines3-Zoo provides `enjoy.py`, a python script to evaluate our agent. In most RL libraries, we call the evaluation script `enjoy.py`.\n",
"- Let's evaluate it for 5000 timesteps ๐ฅ"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "co5um_KeKbBJ"
},
"outputs": [],
"source": [
"!python enjoy.py --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps _________ --folder logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q24K1tyWSj7t"
},
"source": [
"#### Solution"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "P_uSmwGRSk0z"
},
"outputs": [],
"source": [
"!python enjoy.py --algo dqn --env SpaceInvadersNoFrameskip-v4 --no-render --n-timesteps 5000 --folder logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "liBeTltiHJtr"
},
"source": [
"## Publish our trained model on the Hub ๐\n",
"Now that we saw we got good results after the training, we can publish our trained model on the hub ๐ค with one line of code.\n",
"\n",
"![\"Space](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit3/space-invaders-model.gif\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ezbHS1q3HYVV"
},
"source": [
"By using `rl_zoo3.push_to_hub.py` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
"\n",
"This way:\n",
"- You can **showcase our work** ๐ฅ\n",
"- You can **visualize your agent playing** ๐\n",
"- You can **share with the community an agent that others can use** ๐พ\n",
"- You can **access a leaderboard ๐ to see how well your agent is performing compared to your classmates** ๐ https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XMSeZRBiHk6X"
},
"source": [
"To be able to share your model with the community there are three more steps to follow:\n",
"\n",
"1๏ธโฃ (If it's not already done) create an account to HF โก https://huggingface.co/join\n",
"\n",
"2๏ธโฃ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
"- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
"\n",
"![\"Create](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/create-token.jpg\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9O6FI0F8HnzE"
},
"source": [
"- Copy the token \n",
"- Run the cell below and past the token"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ppu9yePwHrZX"
},
"outputs": [],
"source": [
"from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
"notebook_login()\n",
"!git config --global credential.helper store"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2RVEdunPHs8B"
},
"source": [
"If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dSLwdmvhHvjw"
},
"source": [
"3๏ธโฃ We're now ready to push our trained agent to the ๐ค Hub ๐ฅ"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PW436XnhHw1H"
},
"source": [
"Let's run push_to_hub.py file to upload our trained agent to the Hub.\n",
"\n",
"`--repo-name `: The name of the repo\n",
"\n",
"`-orga`: Your Hugging Face username\n",
"\n",
"![\"Select](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit3/select-id.png\")
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ygk2sEktTDEw"
},
"outputs": [],
"source": [
"!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name _____________________ -orga _____________________ -f logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "otgpa0rhS9wR"
},
"source": [
"#### Solution"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_HQNlAXuEhci"
},
"outputs": [],
"source": [
"!python -m rl_zoo3.push_to_hub --algo dqn --env SpaceInvadersNoFrameskip-v4 --repo-name dqn-SpaceInvadersNoFrameskip-v4 -orga ThomasSimonini -f logs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0D4F5zsTTJ-L"
},
"source": [
"###."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ff89kd2HL1_s"
},
"source": [
"Congrats ๐ฅณ you've just trained and uploaded your first Deep Q-Learning agent using RL-Baselines-3 Zoo. The script above should have displayed a link to a model repository such as https://huggingface.co/ThomasSimonini/dqn-SpaceInvadersNoFrameskip-v4. When you go to this link, you can:\n",
"\n",
"- See a **video preview of your agent** at the right. \n",
"- Click \"Files and versions\" to see all the files in the repository.\n",
"- Click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
"- A model card (`README.md` file) which gives a description of the model and the hyperparameters you used.\n",
"\n",
"Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
"\n",
"**Compare the results of your agents with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) ๐"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fyRKcCYY-dIo"
},
"source": [
"## Load a powerful trained model ๐ฅ\n",
"- The Stable-Baselines3 team uploaded **more than 150 trained Deep Reinforcement Learning agents on the Hub**.\n",
"\n",
"You can find them here: ๐ https://huggingface.co/sb3\n",
"\n",
"Some examples:\n",
"- Asteroids: https://huggingface.co/sb3/dqn-AsteroidsNoFrameskip-v4\n",
"- Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4\n",
"- Breakout: https://huggingface.co/sb3/dqn-BreakoutNoFrameskip-v4\n",
"- Road Runner: https://huggingface.co/sb3/dqn-RoadRunnerNoFrameskip-v4\n",
"\n",
"Let's load an agent playing Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "B-9QVFIROI5Y"
},
"outputs": [],
"source": [
"%%html\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7ZQNY_r6NJtC"
},
"source": [
"1. We download the model using `rl_zoo3.load_from_hub`, and place it in a new folder that we can call `rl_trained`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OdBNZHy0NGTR"
},
"outputs": [],
"source": [
"# Download model and save it into the logs/ folder\n",
"!python -m rl_zoo3.load_from_hub --algo dqn --env BeamRiderNoFrameskip-v4 -orga sb3 -f rl_trained/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LFt6hmWsNdBo"
},
"source": [
"2. Let's evaluate if for 5000 timesteps"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "aOxs0rNuN0uS"
},
"outputs": [],
"source": [
"!python enjoy.py --algo dqn --env BeamRiderNoFrameskip-v4 -n 5000 -f rl_trained/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kxMDuDfPON57"
},
"source": [
"Why not trying to train your own **Deep Q-Learning Agent playing BeamRiderNoFrameskip-v4? ๐.**\n",
"\n",
"If you want to try, check https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4#hyperparameters **in the model card, you have the hyperparameters of the trained agent.**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xL_ZtUgpOuY6"
},
"source": [
"But finding hyperparameters can be a daunting task. Fortunately, we'll see in the next Unit, how we can **use Optuna for optimizing the Hyperparameters ๐ฅ.**\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-pqaco8W-huW"
},
"source": [
"## Some additional challenges ๐\n",
"The best way to learn **is to try things by your own**!\n",
"\n",
"In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
"\n",
"Here's a list of environments you can try to train your agent with:\n",
"- BeamRiderNoFrameskip-v4\n",
"- BreakoutNoFrameskip-v4 \n",
"- EnduroNoFrameskip-v4\n",
"- PongNoFrameskip-v4\n",
"\n",
"Also, **if you want to learn to implement Deep Q-Learning by yourself**, you definitely should look at CleanRL implementation: https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py\n",
"\n",
"![\"Environments\"/](\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit4/atari-envs.gif\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "paS-XKo4-kmu"
},
"source": [
"________________________________________________________________________\n",
"Congrats on finishing this chapter!\n",
"\n",
"If youโre still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
"\n",
"Take time to really **grasp the material before continuing and try the additional challenges**. Itโs important to master these elements and having a solid foundations.\n",
"\n",
"In the next unit, **weโre going to learn about [Optuna](https://optuna.org/)**. One of the most critical task in Deep Reinforcement Learning is to find a good set of training hyperparameters. And Optuna is a library that helps you to automate the search.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5WRx7tO7-mvC"
},
"source": [
"\n",
"\n",
"### This is a course built with you ๐ท๐ฟโโ๏ธ\n",
"\n",
"Finally, we want to improve and update the course iteratively with your feedback. If you have some, please fill this form ๐ https://forms.gle/3HgA7bEHwAmmLfwh9\n",
"\n",
"We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
]
},
{
"cell_type": "markdown",
"source": [
"See you on Bonus unit 2! ๐ฅ "
],
"metadata": {
"id": "Kc3udPT-RcXc"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "fS3Xerx0fIMV"
},
"source": [
"### Keep Learning, Stay Awesome ๐ค"
]
}
],
"metadata": {
"colab": {
"private_outputs": true,
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
},
"accelerator": "GPU",
"gpuClass": "standard"
},
"nbformat": 4,
"nbformat_minor": 0
}