From 07b977930e4d56e24551c6d5573d766553bbb792 Mon Sep 17 00:00:00 2001 From: unknown Date: Sat, 18 Jan 2025 16:13:32 +0530 Subject: [PATCH] Add new python venv for ML-Agents & Colab compatibility (Unit-5) --- notebooks/unit5/unit5.ipynb | 427 +++++++++++++++++++++--------------- 1 file changed, 247 insertions(+), 180 deletions(-) diff --git a/notebooks/unit5/unit5.ipynb b/notebooks/unit5/unit5.ipynb index e0220c3..e9f817a 100644 --- a/notebooks/unit5/unit5.ipynb +++ b/notebooks/unit5/unit5.ipynb @@ -3,8 +3,8 @@ { "cell_type": "markdown", "metadata": { - "id": "view-in-github", - "colab_type": "text" + "colab_type": "text", + "id": "view-in-github" }, "source": [ "\"Open" @@ -22,6 +22,9 @@ }, { "cell_type": "markdown", + "metadata": { + "id": "97ZiytXEgqIz" + }, "source": [ "\"Thumbnail\"/\n", "\n", @@ -33,33 +36,33 @@ "After that, you'll be able **to watch your agents playing directly on your browser**.\n", "\n", "For more information about the certification process, check this section ๐Ÿ‘‰ https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" - ], - "metadata": { - "id": "97ZiytXEgqIz" - } + ] }, { "cell_type": "markdown", - "source": [ - "โฌ‡๏ธ Here is an example of what **you will achieve at the end of this unit.** โฌ‡๏ธ\n" - ], "metadata": { "id": "FMYrDriDujzX" - } + }, + "source": [ + "โฌ‡๏ธ Here is an example of what **you will achieve at the end of this unit.** โฌ‡๏ธ\n" + ] }, { "cell_type": "markdown", + "metadata": { + "id": "cBmFlh8suma-" + }, "source": [ "\"Pyramids\"/\n", "\n", "\"SnowballTarget\"/" - ], - "metadata": { - "id": "cBmFlh8suma-" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "A-cYE0K5iL-w" + }, "source": [ "### ๐ŸŽฎ Environments:\n", "\n", @@ -69,22 +72,22 @@ "### ๐Ÿ“š RL-Library:\n", "\n", "- [ML-Agents](https://github.com/Unity-Technologies/ml-agents)\n" - ], - "metadata": { - "id": "A-cYE0K5iL-w" - } + ] }, { "cell_type": "markdown", - "source": [ - "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)." - ], "metadata": { "id": "qEhtaFh9i31S" - } + }, + "source": [ + "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)." + ] }, { "cell_type": "markdown", + "metadata": { + "id": "j7f63r3Yi5vE" + }, "source": [ "## Objectives of this notebook ๐Ÿ†\n", "\n", @@ -92,20 +95,17 @@ "\n", "- Understand how works **ML-Agents**, the environment library.\n", "- Be able to **train agents in Unity Environments**.\n" - ], - "metadata": { - "id": "j7f63r3Yi5vE" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "viNzVbVaYvY3" + }, "source": [ "## This notebook is from the Deep Reinforcement Learning Course\n", "\"Deep" - ], - "metadata": { - "id": "viNzVbVaYvY3" - } + ] }, { "cell_type": "markdown", @@ -141,6 +141,9 @@ }, { "cell_type": "markdown", + "metadata": { + "id": "xYO1uD5Ujgdh" + }, "source": [ "# Let's train our agents ๐Ÿš€\n", "\n", @@ -148,66 +151,130 @@ "\n", "- For `Pyramids` : Mean Reward = 1.75\n", "- For `SnowballTarget` : Mean Reward = 15 or 30 targets hit in an episode.\n" - ], - "metadata": { - "id": "xYO1uD5Ujgdh" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "DssdIjk_8vZE" + }, "source": [ "## Set the GPU ๐Ÿ’ช\n", "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n", "\n", "\"GPU" - ], - "metadata": { - "id": "DssdIjk_8vZE" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "sTfCXHy68xBv" + }, "source": [ "- `Hardware Accelerator > GPU`\n", "\n", "\"GPU" - ], - "metadata": { - "id": "sTfCXHy68xBv" - } + ] }, { "cell_type": "markdown", - "metadata": { - "id": "an3ByrXYQ4iK" - }, + "metadata": {}, "source": [ - "## Clone the repository and install the dependencies ๐Ÿ”ฝ\n" + "## Clone the repository ๐Ÿ”ฝ\n", + "\n", + "- We need to clone the repository, that contains **ML-Agents.**" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "6WNoL04M7rTa" - }, + "metadata": {}, "outputs": [], "source": [ "%%capture\n", - "# Clone the repository\n", + "# Clone the repository (can take 3min)\n", "!git clone --depth 1 https://github.com/Unity-Technologies/ml-agents" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup the Virtual Environment ๐Ÿ”ฝ\n", + "- In order for the **ML-Agents** to run successfully in Colab, Colab's Python version must meet the library's Python requirements.\n", + "\n", + "- We can check for the supported Python version under the `python_requires` parameter in the `setup.py` files. These files are required to set up the **ML-Agents** library for use and can be found in the following locations:\n", + " - `/content/ml-agents/ml-agents/setup.py`\n", + " - `/content/ml-agents/ml-agents-envs/setup.py`\n", + "\n", + "- Colab's Current Python version(can be checked using `!python --version`) doesn't match the library's `python_requires` parameter, as a result installation may silently fail and lead to errors like these, when executing the same commands later:\n", + " - `/bin/bash: line 1: mlagents-learn: command not found`\n", + " - `/bin/bash: line 1: mlagents-push-to-hf: command not found`\n", + "\n", + "- To resolve this, we'll create a virtual environment with a Python version compatible with the **ML-Agents** library.\n", + "\n", + "`Note:` *For future compatibility, always check the `python_requires` parameter in the installation files and set your virtual environment to the maximum supported Python version in the given below script if the Colab's Python version is not compatible*" + ] + }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "d8wmVcMk7xKo" - }, + "metadata": {}, "outputs": [], "source": [ + "# Colab's Current Python Version (Incompatible with ML-Agents)\n", + "!python --version" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Install virtualenv and create a virtual environment\n", + "!pip install virtualenv\n", + "!virtualenv myenv\n", "\n", - "# Go inside the repository and install the package\n", + "# Download and install Miniconda\n", + "!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n", + "!chmod +x Miniconda3-latest-Linux-x86_64.sh\n", + "!./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local\n", + "\n", + "# Activate Miniconda and install Python ver 3.10.12\n", + "!source /usr/local/bin/activate\n", + "!conda install -q -y --prefix /usr/local python=3.10.12 ujson # Specify the version here\n", + "\n", + "# Set environment variables for Python and conda paths\n", + "!export PYTHONPATH=/usr/local/lib/python3.10/site-packages/\n", + "!export CONDA_PREFIX=/usr/local/envs/myenv" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Python Version in New Virtual Environment (Compatible with ML-Agents)\n", + "!python --version" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Installing the dependencies ๐Ÿ”ฝ" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%capture\n", + "# Go inside the repository and install the package (can take 3min)\n", "%cd ml-agents\n", "!pip3 install -e ./ml-agents-envs\n", "!pip3 install -e ./ml-agents" @@ -215,15 +282,15 @@ }, { "cell_type": "markdown", + "metadata": { + "id": "R5_7Ptd_kEcG" + }, "source": [ "## SnowballTarget โ›„\n", "\n", "If you need a refresher on how this environments work check this section ๐Ÿ‘‰\n", "https://huggingface.co/deep-rl-course/unit5/snowball-target" - ], - "metadata": { - "id": "R5_7Ptd_kEcG" - } + ] }, { "cell_type": "markdown", @@ -252,32 +319,32 @@ }, { "cell_type": "markdown", - "source": [ - "We downloaded the file SnowballTarget.zip from https://github.com/huggingface/Snowball-Target using `wget`" - ], "metadata": { "id": "ekSh8LWawkB5" - } + }, + "source": [ + "We downloaded the file SnowballTarget.zip from https://github.com/huggingface/Snowball-Target using `wget`" + ] }, { "cell_type": "code", - "source": [ - "!wget \"https://github.com/huggingface/Snowball-Target/raw/main/SnowballTarget.zip\" -O ./training-envs-executables/linux/SnowballTarget.zip" - ], + "execution_count": null, "metadata": { "id": "6LosWO50wa77" }, - "execution_count": null, - "outputs": [] + "outputs": [], + "source": [ + "!wget \"https://github.com/huggingface/Snowball-Target/raw/main/SnowballTarget.zip\" -O ./training-envs-executables/linux/SnowballTarget.zip" + ] }, { "cell_type": "markdown", - "source": [ - "We unzip the executable.zip file" - ], "metadata": { "id": "_LLVaEEK3ayi" - } + }, + "source": [ + "We unzip the executable.zip file" + ] }, { "cell_type": "code", @@ -313,6 +380,9 @@ }, { "cell_type": "markdown", + "metadata": { + "id": "NAuEq32Mwvtz" + }, "source": [ "### Define the SnowballTarget config file\n", "- In ML-Agents, you define the **training hyperparameters into config.yaml files.**\n", @@ -353,31 +423,28 @@ " gamma: 0.99\n", " strength: 1.0\n", "```" - ], - "metadata": { - "id": "NAuEq32Mwvtz" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "4U3sRH4N4h_l" + }, "source": [ "\"Config\n", "\"Config" - ], - "metadata": { - "id": "4U3sRH4N4h_l" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "JJJdo_5AyoGo" + }, "source": [ "As an experimentation, you should also try to modify some other hyperparameters. Unity provides very [good documentation explaining each of them here](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Training-Configuration-File.md).\n", "\n", "Now that you've created the config file and understand what most hyperparameters do, we're ready to train our agent ๐Ÿ”ฅ." - ], - "metadata": { - "id": "JJJdo_5AyoGo" - } + ] }, { "cell_type": "markdown", @@ -406,12 +473,12 @@ }, { "cell_type": "markdown", - "source": [ - "The training will take 10 to 35min depending on your config, go take a โ˜•๏ธyou deserve it ๐Ÿค—." - ], "metadata": { "id": "lN32oWF8zPjs" - } + }, + "source": [ + "The training will take 10 to 35min depending on your config, go take a โ˜•๏ธyou deserve it ๐Ÿค—." + ] }, { "cell_type": "code", @@ -437,6 +504,9 @@ }, { "cell_type": "markdown", + "metadata": { + "id": "izT6FpgNzZ6R" + }, "source": [ "To be able to share your model with the community there are three more steps to follow:\n", "\n", @@ -449,10 +519,7 @@ "\n", "- Copy the token\n", "- Run the cell below and paste the token" - ], - "metadata": { - "id": "izT6FpgNzZ6R" - } + ] }, { "cell_type": "code", @@ -468,15 +535,18 @@ }, { "cell_type": "markdown", - "source": [ - "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`" - ], "metadata": { "id": "aSU9qD9_6dem" - } + }, + "source": [ + "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`" + ] }, { "cell_type": "markdown", + "metadata": { + "id": "KK4fPfnczunT" + }, "source": [ "Then, we simply need to run `mlagents-push-to-hf`.\n", "\n", @@ -493,21 +563,18 @@ "For instance:\n", "\n", "`!mlagents-push-to-hf --run-id=\"SnowballTarget1\" --local-dir=\"./results/SnowballTarget1\" --repo-id=\"ThomasSimonini/ppo-SnowballTarget\" --commit-message=\"First Push\"`" - ], - "metadata": { - "id": "KK4fPfnczunT" - } + ] }, { "cell_type": "code", - "source": [ - "!mlagents-push-to-hf --run-id=\"SnowballTarget1\" --local-dir=\"./results/SnowballTarget1\" --repo-id=\"ThomasSimonini/ppo-SnowballTarget\" --commit-message=\"First Push\"" - ], + "execution_count": null, "metadata": { "id": "kAFzVB7OYj_H" }, - "execution_count": null, - "outputs": [] + "outputs": [], + "source": [ + "!mlagents-push-to-hf --run-id=\"SnowballTarget1\" --local-dir=\"./results/SnowballTarget1\" --repo-id=\"ThomasSimonini/ppo-SnowballTarget\" --commit-message=\"First Push\"" + ] }, { "cell_type": "code", @@ -522,6 +589,9 @@ }, { "cell_type": "markdown", + "metadata": { + "id": "yborB0850FTM" + }, "source": [ "Else, if everything worked you should have this at the end of the process(but with a different url ๐Ÿ˜†) :\n", "\n", @@ -532,22 +602,22 @@ "```\n", "\n", "Itโ€™s the link to your model, it contains a model card that explains how to use it, your Tensorboard and your config file. **Whatโ€™s awesome is that itโ€™s a git repository, that means you can have different commits, update your repository with a new push etc.**" - ], - "metadata": { - "id": "yborB0850FTM" - } + ] }, { "cell_type": "markdown", - "source": [ - "But now comes the best: **being able to visualize your agent online ๐Ÿ‘€.**" - ], "metadata": { "id": "5Uaon2cg0NrL" - } + }, + "source": [ + "But now comes the best: **being able to visualize your agent online ๐Ÿ‘€.**" + ] }, { "cell_type": "markdown", + "metadata": { + "id": "VMc4oOsE0QiZ" + }, "source": [ "### Watch your agent playing ๐Ÿ‘€\n", "\n", @@ -558,19 +628,19 @@ "2. Launch the game and put it in full screen by clicking on the bottom right button\n", "\n", "\"Snowballtarget" - ], - "metadata": { - "id": "VMc4oOsE0QiZ" - } + ] }, - { + { "cell_type": "markdown", + "metadata": { + "id": "Djs8c5rR0Z8a" + }, "source": [ - "1. In step 1, type your username (your username is case sensitive: for instance, my username is ThomasSimonini not thomassimonini or ThOmasImoNInI) and click on the search button.", + "1. In step 1, type your username (your username is case sensitive: for instance, my username is ThomasSimonini not thomassimonini or ThOmasImoNInI) and click on the search button.\n", "\n", - "2. In step 2, select your model repository.", + "2. In step 2, select your model repository.\n", "\n", - "3. In step 3, **choose which model you want to replay**:", + "3. In step 3, **choose which model you want to replay**:\n", " - I have multiple ones, since we saved a model every 500000 timesteps.\n", " - But since I want the more recent, I choose `SnowballTarget.onnx`\n", "\n", @@ -579,13 +649,13 @@ "And don't hesitate to share the best score your agent gets on discord in #rl-i-made-this channel ๐Ÿ”ฅ\n", "\n", "Let's now try a harder environment called Pyramids..." - ], - "metadata": { - "id": "Djs8c5rR0Z8a" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "rVMwRi4y_tmx" + }, "source": [ "## Pyramids ๐Ÿ†\n", "\n", @@ -593,39 +663,36 @@ "- Our environment executable is in a zip file.\n", "- We need to download it and place it to `./training-envs-executables/linux/`\n", "- We use a linux executable because we use colab, and colab machines OS is Ubuntu (linux)" - ], - "metadata": { - "id": "rVMwRi4y_tmx" - } + ] }, { "cell_type": "markdown", - "source": [ - "We downloaded the file Pyramids.zip from from https://huggingface.co/spaces/unity/ML-Agents-Pyramids/resolve/main/Pyramids.zip using `wget`" - ], "metadata": { "id": "x2C48SGZjZYw" - } + }, + "source": [ + "We downloaded the file Pyramids.zip from from https://huggingface.co/spaces/unity/ML-Agents-Pyramids/resolve/main/Pyramids.zip using `wget`" + ] }, { "cell_type": "code", - "source": [ - "!wget \"https://huggingface.co/spaces/unity/ML-Agents-Pyramids/resolve/main/Pyramids.zip\" -O ./training-envs-executables/linux/Pyramids.zip" - ], + "execution_count": null, "metadata": { "id": "eWh8Pl3sjZY2" }, - "execution_count": null, - "outputs": [] + "outputs": [], + "source": [ + "!wget \"https://huggingface.co/spaces/unity/ML-Agents-Pyramids/resolve/main/Pyramids.zip\" -O ./training-envs-executables/linux/Pyramids.zip" + ] }, { "cell_type": "markdown", - "source": [ - "We unzip the executable.zip file" - ], "metadata": { "id": "V5LXPOPujZY3" - } + }, + "source": [ + "We unzip the executable.zip file" + ] }, { "cell_type": "code", @@ -661,6 +728,9 @@ }, { "cell_type": "markdown", + "metadata": { + "id": "fqceIATXAgih" + }, "source": [ "### Modify the PyramidsRND config file\n", "- Contrary to the first environment which was a custom one, **Pyramids was made by the Unity team**.\n", @@ -672,32 +742,29 @@ "๐Ÿ‘‰ To do that, we go to config/ppo/PyramidsRND.yaml,**and modify these to max_steps to 1000000.**\n", "\n", "\"Pyramids" - ], - "metadata": { - "id": "fqceIATXAgih" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "RI-5aPL7BWVk" + }, "source": [ "As an experimentation, you should also try to modify some other hyperparameters, Unity provides a very [good documentation explaining each of them here](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Training-Configuration-File.md).\n", "\n", "Weโ€™re now ready to train our agent ๐Ÿ”ฅ." - ], - "metadata": { - "id": "RI-5aPL7BWVk" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "s5hr1rvIBdZH" + }, "source": [ "### Train the agent\n", "\n", "The training will take 30 to 45min depending on your machine, go take a โ˜•๏ธyou deserve it ๐Ÿค—." - ], - "metadata": { - "id": "s5hr1rvIBdZH" - } + ] }, { "cell_type": "code", @@ -723,50 +790,53 @@ }, { "cell_type": "code", - "source": [ - "!mlagents-push-to-hf --run-id= # Add your run id --local-dir= # Your local dir --repo-id= # Your repo id --commit-message= # Your commit message" - ], + "execution_count": null, "metadata": { "id": "yiEQbv7rB4mU" }, - "execution_count": null, - "outputs": [] + "outputs": [], + "source": [ + "!mlagents-push-to-hf --run-id= # Add your run id --local-dir= # Your local dir --repo-id= # Your repo id --commit-message= # Your commit message" + ] }, { "cell_type": "markdown", + "metadata": { + "id": "7aZfgxo-CDeQ" + }, "source": [ "### Watch your agent playing ๐Ÿ‘€\n", "\n", "๐Ÿ‘‰ https://huggingface.co/spaces/unity/ML-Agents-Pyramids" - ], - "metadata": { - "id": "7aZfgxo-CDeQ" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "hGG_oq2n0wjB" + }, "source": [ "### ๐ŸŽ Bonus: Why not train on another environment?\n", "Now that you know how to train an agent using MLAgents, **why not try another environment?**\n", "\n", "MLAgents provides 17 different and weโ€™re building some custom ones. The best way to learn is to try things of your own, have fun.\n", "\n" - ], - "metadata": { - "id": "hGG_oq2n0wjB" - } + ] }, { "cell_type": "markdown", - "source": [ - "![cover](https://miro.medium.com/max/1400/0*xERdThTRRM2k_U9f.png)" - ], "metadata": { "id": "KSAkJxSr0z6-" - } + }, + "source": [ + "![cover](https://miro.medium.com/max/1400/0*xERdThTRRM2k_U9f.png)" + ] }, { "cell_type": "markdown", + "metadata": { + "id": "YiyF4FX-04JB" + }, "source": [ "You have the full list of the Unity official environments here ๐Ÿ‘‰ https://github.com/Unity-Technologies/ml-agents/blob/develop/docs/Learning-Environment-Examples.md\n", "\n", @@ -775,13 +845,13 @@ "For now we have integrated:\n", "- [Worm](https://huggingface.co/spaces/unity/ML-Agents-Worm) demo where you teach a **worm to crawl**.\n", "- [Walker](https://huggingface.co/spaces/unity/ML-Agents-Walker) demo where you teach an agent **to walk towards a goal**." - ], - "metadata": { - "id": "YiyF4FX-04JB" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "PI6dPWmh064H" + }, "source": [ "Thatโ€™s all for today. Congrats on finishing this tutorial!\n", "\n", @@ -790,18 +860,15 @@ "See you on Unit 6 ๐Ÿ”ฅ,\n", "\n", "## Keep Learning, Stay awesome ๐Ÿค—" - ], - "metadata": { - "id": "PI6dPWmh064H" - } + ] } ], "metadata": { "accelerator": "GPU", "colab": { - "provenance": [], + "include_colab_link": true, "private_outputs": true, - "include_colab_link": true + "provenance": [] }, "gpuClass": "standard", "kernelspec": {