CS_ALearn/deep-rl-class
1
0
Fork 0
mirror of https://github.com/huggingface/deep-rl-class.git synced 2026-02-02 18:09:24 +08:00
Code Issues Projects Releases Wiki Activity

Add new python venv for ML-Agents & Colab compatibility (Bonus-Unit-1)

Browse Source
This commit is contained in:
unknown
2025-01-18 15:02:00 +05:30
parent 4991ad06b9
commit 8de22a7619
1 changed files with 188 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files

311
notebooks/bonus-unit1/bonus_unit1.ipynb
View File

@@ -3,8 +3,8 @@
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
"colab_type": "text",
"id": "view-in-github"
},
"source": [
"<a href=\"https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/notebooks/bonus-unit1/bonus_unit1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
@@ -21,31 +21,34 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "FMYrDriDujzX"
},
"source": [
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit2/thumbnail.png\" alt=\"Bonus Unit 1Thumbnail\">\n",
"\n",
"In this notebook, we'll reinforce what we learned in the first Unit by **teaching Huggy the Dog to fetch the stick and then play with it directly in your browser**\n",
"\n",
"⬇️ Here is an example of what **you will achieve at the end of the unit.** ⬇️ (launch ▶ to see)"
],
"metadata": {
"id": "FMYrDriDujzX"
}
]
},
{
"cell_type": "code",
"source": [
"%%html\n",
"<video controls autoplay><source src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/huggy.mp4\" type=\"video/mp4\"></video>"
],
"execution_count": null,
"metadata": {
"id": "PnVhs1yYNyUF"
},
"execution_count": null,
"outputs": []
"outputs": [],
"source": [
"%%html\n",
"<video controls autoplay><source src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/huggy.mp4\" type=\"video/mp4\"></video>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "x7oR6R-ZIbeS"
},
"source": [
"### The environment 🎮\n",
"\n",
@@ -54,22 +57,22 @@
"### The library used 📚\n",
"\n",
"- [MLAgents](https://github.com/Unity-Technologies/ml-agents)"
],
"metadata": {
"id": "x7oR6R-ZIbeS"
}
]
},
{
"cell_type": "markdown",
"source": [
"We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
],
"metadata": {
"id": "60yACvZwO0Cy"
}
},
"source": [
"We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Oks-ETYdO2Dc"
},
"source": [
"## Objectives of this notebook 🏆\n",
"\n",
@@ -80,23 +83,23 @@
"- Be able to play **with your trained Huggy directly in your browser**.\n",
"\n",
"\n"
],
"metadata": {
"id": "Oks-ETYdO2Dc"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mUlVrqnBv2o1"
},
"source": [
"## This notebook is from Deep Reinforcement Learning Course\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/deep-rl-course-illustration.jpg\" alt=\"Deep RL Course illustration\"/>"
],
"metadata": {
"id": "mUlVrqnBv2o1"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pAMjaQpHwB_s"
},
"source": [
"In this free course, you will:\n",
"\n",
@@ -110,13 +113,13 @@
"\n",
"\n",
"The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
],
"metadata": {
"id": "pAMjaQpHwB_s"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6r7Hl0uywFSO"
},
"source": [
"## Prerequisites 🏗️\n",
"\n",
@@ -125,41 +128,36 @@
"🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by doing Unit 1\n",
"\n",
"🔲 📚 **Read the introduction to Huggy** by doing Bonus Unit 1"
],
"metadata": {
"id": "6r7Hl0uywFSO"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DssdIjk_8vZE"
},
"source": [
"## Set the GPU 💪\n",
"- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
"\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\" alt=\"GPU Step 1\">"
],
"metadata": {
"id": "DssdIjk_8vZE"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sTfCXHy68xBv"
},
"source": [
"- `Hardware Accelerator > GPU`\n",
"\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\" alt=\"GPU Step 2\">"
],
"metadata": {
"id": "sTfCXHy68xBv"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "an3ByrXYQ4iK"
},
"metadata": {},
"source": [
"## Clone the repository and install the dependencies 🔽\n",
"## Clone the repository 🔽\n",
"\n",
"- We need to clone the repository, that contains **ML-Agents.**"
]
@@ -167,9 +165,7 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6WNoL04M7rTa"
},
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
@@ -177,12 +173,81 @@
"!git clone --depth 1 https://github.com/Unity-Technologies/ml-agents"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup the Virtual Environment 🔽\n",
"- In order for the **ML-Agents** to run successfully in Colab, Colab's Python version must meet the library's Python requirements.\n",
"\n",
"- We can check for the supported Python version under the `python_requires` parameter in the `setup.py` files. These files are required to set up the **ML-Agents** library for use and can be found in the following locations:\n",
" - `/content/ml-agents/ml-agents/setup.py`\n",
" - `/content/ml-agents/ml-agents-envs/setup.py`\n",
"\n",
"- Colab's Current Python version(can be checked using `!python --version`) doesn't match the library's `python_requires` parameter, as a result installation may silently fail and lead to errors like these, when executing the same commands later:\n",
" - `/bin/bash: line 1: mlagents-learn: command not found`\n",
" - `/bin/bash: line 1: mlagents-push-to-hf: command not found`\n",
"\n",
"- To resolve this, we'll create a virtual environment with a Python version compatible with the **ML-Agents** library.\n",
"\n",
"`Note:` *For future compatibility, always check the `python_requires` parameter in the installation files and set your virtual environment to the maximum supported Python version in the given below script if the Colab's Python version is not compatible*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d8wmVcMk7xKo"
},
"metadata": {},
"outputs": [],
"source": [
"# Colab's Current Python Version (Incompatible with ML-Agents)\n",
"!python --version"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Install virtualenv and create a virtual environment\n",
"!pip install virtualenv\n",
"!virtualenv myenv\n",
"\n",
"# Download and install Miniconda\n",
"!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n",
"!chmod +x Miniconda3-latest-Linux-x86_64.sh\n",
"!./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local\n",
"\n",
"# Activate Miniconda and install Python ver 3.10.12\n",
"!source /usr/local/bin/activate\n",
"!conda install -q -y --prefix /usr/local python=3.10.12 ujson # Specify the version here\n",
"\n",
"# Set environment variables for Python and conda paths\n",
"!export PYTHONPATH=/usr/local/lib/python3.10/site-packages/\n",
"!export CONDA_PREFIX=/usr/local/envs/myenv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Python Version in New Virtual Environment (Compatible with ML-Agents)\n",
"!python --version"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing the dependencies 🔽"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
@@ -218,23 +283,23 @@
},
{
"cell_type": "markdown",
"source": [
"We downloaded the file Huggy.zip from https://github.com/huggingface/Huggy using `wget`"
],
"metadata": {
"id": "IHh_LXsRrrbM"
}
},
"source": [
"We downloaded the file Huggy.zip from https://github.com/huggingface/Huggy using `wget`"
]
},
{
"cell_type": "code",
"source": [
"!wget \"https://github.com/huggingface/Huggy/raw/main/Huggy.zip\" -O ./trained-envs-executables/linux/Huggy.zip"
],
"execution_count": null,
"metadata": {
"id": "8xNAD1tRpy0_"
},
"execution_count": null,
"outputs": []
"outputs": [],
"source": [
"!wget \"https://github.com/huggingface/Huggy/raw/main/Huggy.zip\" -O ./trained-envs-executables/linux/Huggy.zip"
]
},
{
"cell_type": "code",
@@ -270,6 +335,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "dYKVj8yUvj55"
},
"source": [
"## Let's recap how this environment works\n",
"\n",
@@ -307,13 +375,13 @@
"- *Time penalty*: a fixed-time penalty given at every action to **force him to get to the stick as fast as possible**.\n",
"- *Rotation penalty*: we penalize Huggy if **he spins too much and turns too quickly**.\n",
"- *Getting to the target reward*: we reward Huggy for **reaching the target**."
],
"metadata": {
"id": "dYKVj8yUvj55"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NAuEq32Mwvtz"
},
"source": [
"## Create the Huggy config file\n",
"\n",
@@ -333,13 +401,15 @@
" <img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit1/create-huggy.png\" alt=\"Create huggy.yaml\" width=\"20%\">\n",
"\n",
"- Copy and paste the content below 🔽"
],
"metadata": {
"id": "NAuEq32Mwvtz"
}
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "loQ0N5jhXW71"
},
"outputs": [],
"source": [
"behaviors:\n",
" Huggy:\n",
@@ -367,24 +437,22 @@
" max_steps: 2e6\n",
" time_horizon: 1000\n",
" summary_freq: 50000"
],
"metadata": {
"id": "loQ0N5jhXW71"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"- Don't forget to save the file!"
],
"metadata": {
"id": "oakN7UHwXdCX"
}
},
"source": [
"- Don't forget to save the file!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r9wv5NYGw-05"
},
"source": [
"- **In the case you want to modify the hyperparameters**, in Google Colab notebook, you can click here to open the config.yaml: `/content/ml-agents/config/ppo/Huggy.yaml`\n",
"\n",
@@ -394,10 +462,7 @@
"\n",
"=> Just keep in mind that **decreasing the `checkpoint_interval` means more models to upload to the Hub and so a longer uploading time**\n",
"We’re now ready to train our agent 🔥."
],
"metadata": {
"id": "r9wv5NYGw-05"
}
]
},
{
"cell_type": "markdown",
@@ -426,12 +491,12 @@
},
{
"cell_type": "markdown",
"source": [
"The training will take 30 to 45min depending on your machine (don't forget to **set up a GPU**), go take a ☕️you deserve it 🤗."
],
"metadata": {
"id": "lN32oWF8zPjs"
}
},
"source": [
"The training will take 30 to 45min depending on your machine (don't forget to **set up a GPU**), go take a ☕️you deserve it 🤗."
]
},
{
"cell_type": "code",
@@ -457,6 +522,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "izT6FpgNzZ6R"
},
"source": [
"To be able to share your model with the community there are three more steps to follow:\n",
"\n",
@@ -469,10 +537,7 @@
"\n",
"- Copy the token\n",
"- Run the cell below and paste the token"
],
"metadata": {
"id": "izT6FpgNzZ6R"
}
]
},
{
"cell_type": "code",
@@ -488,12 +553,12 @@
},
{
"cell_type": "markdown",
"source": [
"If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
],
"metadata": {
"id": "ew59mK19zjtN"
}
},
"source": [
"If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
]
},
{
"cell_type": "markdown",
@@ -508,6 +573,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "KK4fPfnczunT"
},
"source": [
"And we define 4 parameters:\n",
"\n",
@@ -516,10 +584,7 @@
"3. `--repo-id`: the name of the Hugging Face repo you want to create or update. It’s always <your huggingface username>/<the repo name>\n",
"If the repo does not exist **it will be created automatically**\n",
"4. `--commit-message`: since HF repos are git repository you need to define a commit message."
],
"metadata": {
"id": "KK4fPfnczunT"
}
]
},
{
"cell_type": "code",
@@ -534,6 +599,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "yborB0850FTM"
},
"source": [
"Else, if everything worked you should have this at the end of the process(but with a different url 😆) :\n",
"\n",
@@ -546,22 +614,22 @@
"It’s the link to your model repository. The repository contains a model card that explains how to use the model, your Tensorboard logs and your config file. **What’s awesome is that it’s a git repository, which means you can have different commits, update your repository with a new push, open Pull Requests, etc.**\n",
"\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/modelcard.png\" alt=\"ml learn function\" width=\"100%\">"
],
"metadata": {
"id": "yborB0850FTM"
}
]
},
{
"cell_type": "markdown",
"source": [
"But now comes the best: **being able to play with Huggy online 👀.**"
],
"metadata": {
"id": "5Uaon2cg0NrL"
}
},
"source": [
"But now comes the best: **being able to play with Huggy online 👀.**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VMc4oOsE0QiZ"
},
"source": [
"## Play with your Huggy 🐕\n",
"\n",
@@ -572,13 +640,13 @@
"- Click on Play with my Huggy model\n",
"\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit-bonus1/load-huggy.jpg\" alt=\"load-huggy\" width=\"100%\">"
],
"metadata": {
"id": "VMc4oOsE0QiZ"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Djs8c5rR0Z8a"
},
"source": [
"1. In step 1, choose your model repository which is the model id (in my case ThomasSimonini/ppo-Huggy).\n",
"\n",
@@ -587,13 +655,13 @@
" - But since I want the more recent, I choose `Huggy.onnx`\n",
"\n",
"👉 What’s nice **is to try with different models steps to see the improvement of the agent.**"
],
"metadata": {
"id": "Djs8c5rR0Z8a"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PI6dPWmh064H"
},
"source": [
"Congrats on finishing this bonus unit!\n",
"\n",
@@ -603,18 +671,15 @@
"\n",
"\n",
"## Keep Learning, Stay awesome 🤗"
],
"metadata": {
"id": "PI6dPWmh064H"
}
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"provenance": [],
"include_colab_link": true,
"private_outputs": true,
"include_colab_link": true
"provenance": []
},
"gpuClass": "standard",
"kernelspec": {