CS_ALearn/deep-rl-class
1
0
Fork 0
mirror of https://github.com/huggingface/deep-rl-class.git synced 2026-04-14 18:31:36 +08:00
Code Issues Projects Releases Wiki Activity

Add new python venv for ML-Agents & Colab compatibility (Unit-5)

Browse Source
This commit is contained in:
unknown
2025-01-18 16:13:32 +05:30
parent 8de22a7619
commit 07b977930e
1 changed files with 247 additions and 180 deletions
Download Patch File Download Diff File Expand all files Collapse all files

427
notebooks/unit5/unit5.ipynb
View File

@@ -3,8 +3,8 @@
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
"colab_type": "text",
"id": "view-in-github"
},
"source": [
"<a href=\"https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main/notebooks/unit5/unit5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
@@ -22,6 +22,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "97ZiytXEgqIz"
},
"source": [
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit7/thumbnail.png\" alt=\"Thumbnail\"/>\n",
"\n",
@@ -33,33 +36,33 @@
"After that, you'll be able **to watch your agents playing directly on your browser**.\n",
"\n",
"For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process"
],
"metadata": {
"id": "97ZiytXEgqIz"
}
]
},
{
"cell_type": "markdown",
"source": [
"⬇️ Here is an example of what **you will achieve at the end of this unit.** ⬇️\n"
],
"metadata": {
"id": "FMYrDriDujzX"
}
},
"source": [
"⬇️ Here is an example of what **you will achieve at the end of this unit.** ⬇️\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cBmFlh8suma-"
},
"source": [
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit7/pyramids.gif\" alt=\"Pyramids\"/>\n",
"\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit7/snowballtarget.gif\" alt=\"SnowballTarget\"/>"
],
"metadata": {
"id": "cBmFlh8suma-"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A-cYE0K5iL-w"
},
"source": [
"### 🎮 Environments:\n",
"\n",
@@ -69,22 +72,22 @@
"### 📚 RL-Library:\n",
"\n",
"- [ML-Agents](https://github.com/Unity-Technologies/ml-agents)\n"
],
"metadata": {
"id": "A-cYE0K5iL-w"
}
]
},
{
"cell_type": "markdown",
"source": [
"We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)."
],
"metadata": {
"id": "qEhtaFh9i31S"
}
},
"source": [
"We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "j7f63r3Yi5vE"
},
"source": [
"## Objectives of this notebook 🏆\n",
"\n",
@@ -92,20 +95,17 @@
"\n",
"- Understand how works **ML-Agents**, the environment library.\n",
"- Be able to **train agents in Unity Environments**.\n"
],
"metadata": {
"id": "j7f63r3Yi5vE"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "viNzVbVaYvY3"
},
"source": [
"## This notebook is from the Deep Reinforcement Learning Course\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/deep-rl-course-illustration.jpg\" alt=\"Deep RL Course illustration\"/>"
],
"metadata": {
"id": "viNzVbVaYvY3"
}
]
},
{
"cell_type": "markdown",
@@ -141,6 +141,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "xYO1uD5Ujgdh"
},
"source": [
"# Let's train our agents 🚀\n",
"\n",
@@ -148,66 +151,130 @@
"\n",
"- For `Pyramids` : Mean Reward = 1.75\n",
"- For `SnowballTarget` : Mean Reward = 15 or 30 targets hit in an episode.\n"
],
"metadata": {
"id": "xYO1uD5Ujgdh"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DssdIjk_8vZE"
},
"source": [
"## Set the GPU 💪\n",
"- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
"\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\" alt=\"GPU Step 1\">"
],
"metadata": {
"id": "DssdIjk_8vZE"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sTfCXHy68xBv"
},
"source": [
"- `Hardware Accelerator > GPU`\n",
"\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\" alt=\"GPU Step 2\">"
],
"metadata": {
"id": "sTfCXHy68xBv"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "an3ByrXYQ4iK"
},
"metadata": {},
"source": [
"## Clone the repository and install the dependencies 🔽\n"
"## Clone the repository 🔽\n",
"\n",
"- We need to clone the repository, that contains **ML-Agents.**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6WNoL04M7rTa"
},
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"# Clone the repository\n",
"# Clone the repository (can take 3min)\n",
"!git clone --depth 1 https://github.com/Unity-Technologies/ml-agents"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup the Virtual Environment 🔽\n",
"- In order for the **ML-Agents** to run successfully in Colab, Colab's Python version must meet the library's Python requirements.\n",
"\n",
"- We can check for the supported Python version under the `python_requires` parameter in the `setup.py` files. These files are required to set up the **ML-Agents** library for use and can be found in the following locations:\n",
" - `/content/ml-agents/ml-agents/setup.py`\n",
" - `/content/ml-agents/ml-agents-envs/setup.py`\n",
"\n",
"- Colab's Current Python version(can be checked using `!python --version`) doesn't match the library's `python_requires` parameter, as a result installation may silently fail and lead to errors like these, when executing the same commands later:\n",
" - `/bin/bash: line 1: mlagents-learn: command not found`\n",
" - `/bin/bash: line 1: mlagents-push-to-hf: command not found`\n",
"\n",
"- To resolve this, we'll create a virtual environment with a Python version compatible with the **ML-Agents** library.\n",
"\n",
"`Note:` *For future compatibility, always check the `python_requires` parameter in the installation files and set your virtual environment to the maximum supported Python version in the given below script if the Colab's Python version is not compatible*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d8wmVcMk7xKo"
},
"metadata": {},
"outputs": [],
"source": [
"# Colab's Current Python Version (Incompatible with ML-Agents)\n",
"!python --version"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Install virtualenv and create a virtual environment\n",
"!pip install virtualenv\n",
"!virtualenv myenv\n",
"\n",
"# Go inside the repository and install the package\n",
"# Download and install Miniconda\n",
"!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n",
"!chmod +x Miniconda3-latest-Linux-x86_64.sh\n",
"!./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local\n",
"\n",
"# Activate Miniconda and install Python ver 3.10.12\n",
"!source /usr/local/bin/activate\n",
"!conda install -q -y --prefix /usr/local python=3.10.12 ujson # Specify the version here\n",
"\n",
"# Set environment variables for Python and conda paths\n",
"!export PYTHONPATH=/usr/local/lib/python3.10/site-packages/\n",
"!export CONDA_PREFIX=/usr/local/envs/myenv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Python Version in New Virtual Environment (Compatible with ML-Agents)\n",
"!python --version"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing the dependencies 🔽"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"# Go inside the repository and install the package (can take 3min)\n",
"%cd ml-agents\n",
"!pip3 install -e ./ml-agents-envs\n",
"!pip3 install -e ./ml-agents"
@@ -215,15 +282,15 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "R5_7Ptd_kEcG"
},
"source": [
"## SnowballTarget ⛄\n",
"\n",
"If you need a refresher on how this environments work check this section 👉\n",
"https://huggingface.co/deep-rl-course/unit5/snowball-target"
],
"metadata": {
"id": "R5_7Ptd_kEcG"
}
]
},
{
"cell_type": "markdown",
@@ -252,32 +319,32 @@
},
{
"cell_type": "markdown",
"source": [
"We downloaded the file SnowballTarget.zip from https://github.com/huggingface/Snowball-Target using `wget`"
],
"metadata": {
"id": "ekSh8LWawkB5"
}
},
"source": [
"We downloaded the file SnowballTarget.zip from https://github.com/huggingface/Snowball-Target using `wget`"
]
},
{
"cell_type": "code",
"source": [
"!wget \"https://github.com/huggingface/Snowball-Target/raw/main/SnowballTarget.zip\" -O ./training-envs-executables/linux/SnowballTarget.zip"
],
"execution_count": null,
"metadata": {
"id": "6LosWO50wa77"
},
"execution_count": null,
"outputs": []
"outputs": [],
"source": [
"!wget \"https://github.com/huggingface/Snowball-Target/raw/main/SnowballTarget.zip\" -O ./training-envs-executables/linux/SnowballTarget.zip"
]
},
{
"cell_type": "markdown",
"source": [
"We unzip the executable.zip file"
],
"metadata": {
"id": "_LLVaEEK3ayi"
}
},
"source": [
"We unzip the executable.zip file"
]
},
{
"cell_type": "code",
@@ -313,6 +380,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "NAuEq32Mwvtz"
},
"source": [
"### Define the SnowballTarget config file\n",
"- In ML-Agents, you define the **training hyperparameters into config.yaml files.**\n",
@@ -353,31 +423,28 @@
" gamma: 0.99\n",
" strength: 1.0\n",
"```"
],
"metadata": {
"id": "NAuEq32Mwvtz"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4U3sRH4N4h_l"
},
"source": [
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit7/snowballfight_config1.png\" alt=\"Config SnowballTarget\"/>\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit7/snowballfight_config2.png\" alt=\"Config SnowballTarget\"/>"
],
"metadata": {
"id": "4U3sRH4N4h_l"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JJJdo_5AyoGo"
},
"source": [
"As an experimentation, you should also try to modify some other hyperparameters. Unity provides very [good documentation explaining each of them here](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Training-Configuration-File.md).\n",
"\n",
"Now that you've created the config file and understand what most hyperparameters do, we're ready to train our agent 🔥."
],
"metadata": {
"id": "JJJdo_5AyoGo"
}
]
},
{
"cell_type": "markdown",
@@ -406,12 +473,12 @@
},
{
"cell_type": "markdown",
"source": [
"The training will take 10 to 35min depending on your config, go take a ☕️you deserve it 🤗."
],
"metadata": {
"id": "lN32oWF8zPjs"
}
},
"source": [
"The training will take 10 to 35min depending on your config, go take a ☕️you deserve it 🤗."
]
},
{
"cell_type": "code",
@@ -437,6 +504,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "izT6FpgNzZ6R"
},
"source": [
"To be able to share your model with the community there are three more steps to follow:\n",
"\n",
@@ -449,10 +519,7 @@
"\n",
"- Copy the token\n",
"- Run the cell below and paste the token"
],
"metadata": {
"id": "izT6FpgNzZ6R"
}
]
},
{
"cell_type": "code",
@@ -468,15 +535,18 @@
},
{
"cell_type": "markdown",
"source": [
"If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
],
"metadata": {
"id": "aSU9qD9_6dem"
}
},
"source": [
"If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KK4fPfnczunT"
},
"source": [
"Then, we simply need to run `mlagents-push-to-hf`.\n",
"\n",
@@ -493,21 +563,18 @@
"For instance:\n",
"\n",
"`!mlagents-push-to-hf --run-id=\"SnowballTarget1\" --local-dir=\"./results/SnowballTarget1\" --repo-id=\"ThomasSimonini/ppo-SnowballTarget\" --commit-message=\"First Push\"`"
],
"metadata": {
"id": "KK4fPfnczunT"
}
]
},
{
"cell_type": "code",
"source": [
"!mlagents-push-to-hf --run-id=\"SnowballTarget1\" --local-dir=\"./results/SnowballTarget1\" --repo-id=\"ThomasSimonini/ppo-SnowballTarget\" --commit-message=\"First Push\""
],
"execution_count": null,
"metadata": {
"id": "kAFzVB7OYj_H"
},
"execution_count": null,
"outputs": []
"outputs": [],
"source": [
"!mlagents-push-to-hf --run-id=\"SnowballTarget1\" --local-dir=\"./results/SnowballTarget1\" --repo-id=\"ThomasSimonini/ppo-SnowballTarget\" --commit-message=\"First Push\""
]
},
{
"cell_type": "code",
@@ -522,6 +589,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "yborB0850FTM"
},
"source": [
"Else, if everything worked you should have this at the end of the process(but with a different url 😆) :\n",
"\n",
@@ -532,22 +602,22 @@
"```\n",
"\n",
"It’s the link to your model, it contains a model card that explains how to use it, your Tensorboard and your config file. **What’s awesome is that it’s a git repository, that means you can have different commits, update your repository with a new push etc.**"
],
"metadata": {
"id": "yborB0850FTM"
}
]
},
{
"cell_type": "markdown",
"source": [
"But now comes the best: **being able to visualize your agent online 👀.**"
],
"metadata": {
"id": "5Uaon2cg0NrL"
}
},
"source": [
"But now comes the best: **being able to visualize your agent online 👀.**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VMc4oOsE0QiZ"
},
"source": [
"### Watch your agent playing 👀\n",
"\n",
@@ -558,19 +628,19 @@
"2. Launch the game and put it in full screen by clicking on the bottom right button\n",
"\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit7/snowballtarget_load.png\" alt=\"Snowballtarget load\"/>"
],
"metadata": {
"id": "VMc4oOsE0QiZ"
}
]
},
{
{
"cell_type": "markdown",
"metadata": {
"id": "Djs8c5rR0Z8a"
},
"source": [
"1. In step 1, type your username (your username is case sensitive: for instance, my username is ThomasSimonini not thomassimonini or ThOmasImoNInI) and click on the search button.",
"1. In step 1, type your username (your username is case sensitive: for instance, my username is ThomasSimonini not thomassimonini or ThOmasImoNInI) and click on the search button.\n",
"\n",
"2. In step 2, select your model repository.",
"2. In step 2, select your model repository.\n",
"\n",
"3. In step 3, **choose which model you want to replay**:",
"3. In step 3, **choose which model you want to replay**:\n",
" - I have multiple ones, since we saved a model every 500000 timesteps.\n",
" - But since I want the more recent, I choose `SnowballTarget.onnx`\n",
"\n",
@@ -579,13 +649,13 @@
"And don't hesitate to share the best score your agent gets on discord in #rl-i-made-this channel 🔥\n",
"\n",
"Let's now try a harder environment called Pyramids..."
],
"metadata": {
"id": "Djs8c5rR0Z8a"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rVMwRi4y_tmx"
},
"source": [
"## Pyramids 🏆\n",
"\n",
@@ -593,39 +663,36 @@
"- Our environment executable is in a zip file.\n",
"- We need to download it and place it to `./training-envs-executables/linux/`\n",
"- We use a linux executable because we use colab, and colab machines OS is Ubuntu (linux)"
],
"metadata": {
"id": "rVMwRi4y_tmx"
}
]
},
{
"cell_type": "markdown",
"source": [
"We downloaded the file Pyramids.zip from from https://huggingface.co/spaces/unity/ML-Agents-Pyramids/resolve/main/Pyramids.zip using `wget`"
],
"metadata": {
"id": "x2C48SGZjZYw"
}
},
"source": [
"We downloaded the file Pyramids.zip from from https://huggingface.co/spaces/unity/ML-Agents-Pyramids/resolve/main/Pyramids.zip using `wget`"
]
},
{
"cell_type": "code",
"source": [
"!wget \"https://huggingface.co/spaces/unity/ML-Agents-Pyramids/resolve/main/Pyramids.zip\" -O ./training-envs-executables/linux/Pyramids.zip"
],
"execution_count": null,
"metadata": {
"id": "eWh8Pl3sjZY2"
},
"execution_count": null,
"outputs": []
"outputs": [],
"source": [
"!wget \"https://huggingface.co/spaces/unity/ML-Agents-Pyramids/resolve/main/Pyramids.zip\" -O ./training-envs-executables/linux/Pyramids.zip"
]
},
{
"cell_type": "markdown",
"source": [
"We unzip the executable.zip file"
],
"metadata": {
"id": "V5LXPOPujZY3"
}
},
"source": [
"We unzip the executable.zip file"
]
},
{
"cell_type": "code",
@@ -661,6 +728,9 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "fqceIATXAgih"
},
"source": [
"### Modify the PyramidsRND config file\n",
"- Contrary to the first environment which was a custom one, **Pyramids was made by the Unity team**.\n",
@@ -672,32 +742,29 @@
"👉 To do that, we go to config/ppo/PyramidsRND.yaml,**and modify these to max_steps to 1000000.**\n",
"\n",
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit7/pyramids-config.png\" alt=\"Pyramids config\"/>"
],
"metadata": {
"id": "fqceIATXAgih"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RI-5aPL7BWVk"
},
"source": [
"As an experimentation, you should also try to modify some other hyperparameters, Unity provides a very [good documentation explaining each of them here](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Training-Configuration-File.md).\n",
"\n",
"We’re now ready to train our agent 🔥."
],
"metadata": {
"id": "RI-5aPL7BWVk"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s5hr1rvIBdZH"
},
"source": [
"### Train the agent\n",
"\n",
"The training will take 30 to 45min depending on your machine, go take a ☕️you deserve it 🤗."
],
"metadata": {
"id": "s5hr1rvIBdZH"
}
]
},
{
"cell_type": "code",
@@ -723,50 +790,53 @@
},
{
"cell_type": "code",
"source": [
"!mlagents-push-to-hf --run-id= # Add your run id --local-dir= # Your local dir --repo-id= # Your repo id --commit-message= # Your commit message"
],
"execution_count": null,
"metadata": {
"id": "yiEQbv7rB4mU"
},
"execution_count": null,
"outputs": []
"outputs": [],
"source": [
"!mlagents-push-to-hf --run-id= # Add your run id --local-dir= # Your local dir --repo-id= # Your repo id --commit-message= # Your commit message"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7aZfgxo-CDeQ"
},
"source": [
"### Watch your agent playing 👀\n",
"\n",
"👉 https://huggingface.co/spaces/unity/ML-Agents-Pyramids"
],
"metadata": {
"id": "7aZfgxo-CDeQ"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hGG_oq2n0wjB"
},
"source": [
"### 🎁 Bonus: Why not train on another environment?\n",
"Now that you know how to train an agent using MLAgents, **why not try another environment?**\n",
"\n",
"MLAgents provides 17 different and we’re building some custom ones. The best way to learn is to try things of your own, have fun.\n",
"\n"
],
"metadata": {
"id": "hGG_oq2n0wjB"
}
]
},
{
"cell_type": "markdown",
"source": [
"![cover](https://miro.medium.com/max/1400/0*xERdThTRRM2k_U9f.png)"
],
"metadata": {
"id": "KSAkJxSr0z6-"
}
},
"source": [
"![cover](https://miro.medium.com/max/1400/0*xERdThTRRM2k_U9f.png)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YiyF4FX-04JB"
},
"source": [
"You have the full list of the Unity official environments here 👉 https://github.com/Unity-Technologies/ml-agents/blob/develop/docs/Learning-Environment-Examples.md\n",
"\n",
@@ -775,13 +845,13 @@
"For now we have integrated:\n",
"- [Worm](https://huggingface.co/spaces/unity/ML-Agents-Worm) demo where you teach a **worm to crawl**.\n",
"- [Walker](https://huggingface.co/spaces/unity/ML-Agents-Walker) demo where you teach an agent **to walk towards a goal**."
],
"metadata": {
"id": "YiyF4FX-04JB"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PI6dPWmh064H"
},
"source": [
"That’s all for today. Congrats on finishing this tutorial!\n",
"\n",
@@ -790,18 +860,15 @@
"See you on Unit 6 🔥,\n",
"\n",
"## Keep Learning, Stay awesome 🤗"
],
"metadata": {
"id": "PI6dPWmh064H"
}
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"provenance": [],
"include_colab_link": true,
"private_outputs": true,
"include_colab_link": true
"provenance": []
},
"gpuClass": "standard",
"kernelspec": {