mirror of
https://github.com/huggingface/deep-rl-class.git
synced 2026-04-24 10:51:12 +08:00
1145 lines
652 KiB
Plaintext
1145 lines
652 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "njb_ProuHiOe"
|
||
},
|
||
"source": [
|
||
"# Unit 1: Train your first Deep Reinforcement Learning Agent 🤖\n",
|
||
"\n",
|
||
"\n",
|
||
"In this notebook, you'll train your **first Deep Reinforcement Learning agent** a Lunar Lander agent that will learn to **land correctly on the Moon 🌕**. Using [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) a Deep Reinforcement Learning library, share them with the community, and experiment with different configurations\n",
|
||
"\n",
|
||
"⬇️ Here is an example of what **you will achieve in just a couple of minutes.** ⬇️\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "PF46MwbZD00b"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"%%html\n",
|
||
"<video controls autoplay><source src=\"https://huggingface.co/ThomasSimonini/ppo-LunarLander-v2/resolve/main/replay.mp4\" type=\"video/mp4\"></video>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"TODO: ADD TEXT LIVE INFO"
|
||
],
|
||
"metadata": {
|
||
"id": "QQnzAjfcIO2z"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"TODO: ADD IF YOU HAVE QUESTIONS\n",
|
||
"\n",
|
||
"\n",
|
||
"### The environment 🎮\n",
|
||
"- [LunarLander-v2](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n",
|
||
"\n",
|
||
"### The library used 📚\n",
|
||
"- [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/)"
|
||
],
|
||
"metadata": {
|
||
"id": "x7oR6R-ZIbeS"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "4i6tjI2tHQ8j"
|
||
},
|
||
"source": [
|
||
"## Objectives of this notebook 🏆\n",
|
||
"At the end of the notebook, you will:\n",
|
||
"- Be able to use **Gym**, the environment library.\n",
|
||
"- Be able to use **Stable-Baselines3**, the deep reinforcement learning library.\n",
|
||
"- Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score 🔥.\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## This notebook is from Deep Reinforcement Learning Course\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/deep-rl-course-illustration.jpg\" alt=\"Deep RL Course illustration\"/>"
|
||
],
|
||
"metadata": {
|
||
"id": "Ff-nyJdzJPND"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "6p5HnEefISCB"
|
||
},
|
||
"source": [
|
||
"In this free course, you will:\n",
|
||
"\n",
|
||
"- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
|
||
"- 🧑💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, and RLlib.\n",
|
||
"- 🤖 Train **agents in unique environments** \n",
|
||
"\n",
|
||
"And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course\n",
|
||
"\n",
|
||
"Don’t forget to **<a href=\"http://eepurl.com/ic5ZUD\">sign up to the course</a>** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
|
||
"\n",
|
||
"\n",
|
||
"The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Y-mo_6rXIjRi"
|
||
},
|
||
"source": [
|
||
"## Prerequisites 🏗️\n",
|
||
"Before diving into the notebook, you need to:\n",
|
||
"\n",
|
||
"🔲 📝 **Done Unit 0** that gives you all the **information about the course and help you to onboard** 🤗 ADD LINK \n",
|
||
"\n",
|
||
"🔲 📚 **Develop an understanding of the foundations of Reinforcement learning** (MC, TD, Rewards hypothesis...) by doing Unit 1 👉 ADD LINK"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## A small recap of what is Deep Reinforcement Learning 📚\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit1/RL_process_game.jpg\" alt=\"The RL process\" width=\"100%\">"
|
||
],
|
||
"metadata": {
|
||
"id": "HoeqMnr5LuYE"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "xcQYx9ynaFMD"
|
||
},
|
||
"source": [
|
||
"Let's do a small recap on what we learned in the first Unit:\n",
|
||
"- Reinforcement Learning is a **computational approach to learning from action**. We build an agent that learns from the environment by **interacting with it through trial and error** and receiving rewards (negative or positive) as feedback.\n",
|
||
"\n",
|
||
"- The goal of any RL agent is to **maximize its expected cumulative reward** (also called expected return) because RL is based on the _reward hypothesis_, which is that all goals can be described as the maximization of the expected cumulative reward.\n",
|
||
"\n",
|
||
"- The RL process is a **loop that outputs a sequence of state, action, reward, and next state**.\n",
|
||
"\n",
|
||
"- To calculate the expected cumulative reward (expected return), **we discount the rewards**: the rewards that come sooner (at the beginning of the game) are more probable to happen since they are more predictable than the long-term future reward.\n",
|
||
"\n",
|
||
"- To solve an RL problem, you want to **find an optimal policy**; the policy is the \"brain\" of your AI that will tell us what action to take given a state. The optimal one is the one that gives you the actions that max the expected return.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "8X2WN2X99BEz"
|
||
},
|
||
"source": [
|
||
"There are **two** ways to find your optimal policy:\n",
|
||
"- By **training your policy directly**: policy-based methods.\n",
|
||
"- By **training a value function** that tells us the expected return the agent will get at each state and use this function to define our policy: value-based methods."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BPy6dfcm8_WT"
|
||
},
|
||
"source": [
|
||
"- Finally, we spoke about Deep RL because **we introduce deep neural networks to estimate the action to take (policy-based) or to estimate the value of a state (value-based) hence the name \"deep.\"**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "LzGDDOTSKaF8"
|
||
},
|
||
"source": [
|
||
"# Let's train our first Deep Reinforcement Learning agents and upload them to the Hub 🚀\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## Step 0: Set the GPU 💪 and install the virtual screen\n",
|
||
"- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\" alt=\"GPU Step 1\">"
|
||
],
|
||
"metadata": {
|
||
"id": "HqzznTzhNfAC"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "38HBd3t1SHJ8"
|
||
},
|
||
"source": [
|
||
"- `Hardware Accelerator > GPU`\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\" alt=\"GPU Step 2\">"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "jeDAH0h0EBiG"
|
||
},
|
||
"source": [
|
||
"## Install dependencies and create a virtual screen 🔽\n",
|
||
"The first step is to install the dependencies, we’ll install multiple ones.\n",
|
||
"\n",
|
||
"- `gym[box2D]`: Contains the LunarLander-v2 environment 🌛 (we use `gym==0.21`)\n",
|
||
"- `stable-baselines3[extra]`: The deep reinforcement learning library.\n",
|
||
"- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face 🤗 Hub.\n",
|
||
"\n",
|
||
"To make things easier, we created a script to install all these dependencies."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"!apt install swig cmake"
|
||
],
|
||
"metadata": {
|
||
"id": "yQIGLPDkGhgG"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"TODO CHANGE LINK OF THE REQUIREMENTS"
|
||
],
|
||
"metadata": {
|
||
"id": "32e3NPYgH5ET"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "9XaULfDZDvrC"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip install -r https://huggingface.co/spaces/ThomasSimonini/temp-space-requirements/raw/main/requirements/requirements-unit1.txt"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
|
||
"\n",
|
||
"Hence the following cell will install virtual screen libraries and create and run a virtual screen 🖥"
|
||
],
|
||
"metadata": {
|
||
"id": "BEKeXQJsQCYm"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"!sudo apt-get update\n",
|
||
"!apt install python-opengl\n",
|
||
"!apt install ffmpeg\n",
|
||
"!apt install xvfb\n",
|
||
"!pip3 install pyvirtualdisplay"
|
||
],
|
||
"metadata": {
|
||
"id": "j5f2cGkdP-mb"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"# Virtual display\n",
|
||
"from pyvirtualdisplay import Display\n",
|
||
"\n",
|
||
"virtual_display = Display(visible=0, size=(1400, 900))\n",
|
||
"virtual_display.start()"
|
||
],
|
||
"metadata": {
|
||
"id": "BE5JWP5rQIKf"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "wrgpVFqyENVf"
|
||
},
|
||
"source": [
|
||
"## Step 2: Import the packages 📦\n",
|
||
"\n",
|
||
"One additional library we import is huggingface_hub **to be able to upload and download trained models from the hub**.\n",
|
||
"\n",
|
||
"\n",
|
||
"The Hugging Face Hub 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations and other features that will allow you to easily collaborate with others.\n",
|
||
"\n",
|
||
"You can see here all the Deep reinforcement Learning models available 👉 https://huggingface.co/models?pipeline_tag=reinforcement-learning&sort=downloads\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "cygWLPGsEQ0m"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import gym\n",
|
||
"\n",
|
||
"from huggingface_sb3 import load_from_hub, package_to_hub, push_to_hub\n",
|
||
"from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
|
||
"\n",
|
||
"from stable_baselines3 import PPO\n",
|
||
"from stable_baselines3.common.evaluation import evaluate_policy\n",
|
||
"from stable_baselines3.common.env_util import make_vec_env"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "MRqRuRUl8CsB"
|
||
},
|
||
"source": [
|
||
"## Step 3: Understand what is Gym and how it works? 🤖\n",
|
||
"\n",
|
||
"🏋 The library containing our environment is called Gym.\n",
|
||
"**You'll use Gym a lot in Deep Reinforcement Learning.**\n",
|
||
"\n",
|
||
"The Gym library provides two things:\n",
|
||
"- An interface that allows you to **create RL environments**.\n",
|
||
"- A **collection of environments** (gym-control, atari, box2D...).\n",
|
||
"\n",
|
||
"Let's look at an example, but first let's remember what's the RL Loop.\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit1/RL_process_game.jpg\" alt=\"The RL process\" width=\"100%\">"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "-TzNN0bQ_j-3"
|
||
},
|
||
"source": [
|
||
"At each step:\n",
|
||
"- Our Agent receives **state S0** from the **Environment** — we receive the first frame of our game (Environment).\n",
|
||
"- Based on that **state S0,** the Agent takes **action A0** — our Agent will move to the right.\n",
|
||
"- Environment to a **new** **state S1** — new frame.\n",
|
||
"- The environment gives some **reward R1** to the Agent — we’re not dead *(Positive Reward +1)*.\n",
|
||
"\n",
|
||
"\n",
|
||
"With Gym:\n",
|
||
"\n",
|
||
"1️⃣ We create our environment using `gym.make()`\n",
|
||
"\n",
|
||
"2️⃣ We reset the environment to its initial state with `observation = env.reset()`\n",
|
||
"\n",
|
||
"At each step:\n",
|
||
"\n",
|
||
"3️⃣ Get an action using our model (in our example we take a random action)\n",
|
||
"\n",
|
||
"4️⃣ Using `env.step(action)`, we perform this action in the environment and get\n",
|
||
"- `observation`: The new state (st+1)\n",
|
||
"- `reward`: The reward we get after executing the action\n",
|
||
"- `done`: Indicates if the episode terminated\n",
|
||
"- `info`: A dictionary that provides additional information (depends on the environment).\n",
|
||
"\n",
|
||
"If the episode is done:\n",
|
||
"- We reset the environment to its initial state with `observation = env.reset()`\n",
|
||
"\n",
|
||
"**Let's look at an example!** Make sure to read the code\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "w7vOFlpA_ONz"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import gym\n",
|
||
"\n",
|
||
"# First, we create our environment called LunarLander-v2\n",
|
||
"env = gym.make(\"LunarLander-v2\")\n",
|
||
"\n",
|
||
"# Then we reset this environment\n",
|
||
"observation = env.reset()\n",
|
||
"\n",
|
||
"for _ in range(20):\n",
|
||
" # Take a random action\n",
|
||
" action = env.action_space.sample()\n",
|
||
" print(\"Action taken:\", action)\n",
|
||
"\n",
|
||
" # Do this action in the environment and get\n",
|
||
" # next_state, reward, done and info\n",
|
||
" observation, reward, done, info = env.step(action)\n",
|
||
" \n",
|
||
" # If the game is done (in our case we land, crashed or timeout)\n",
|
||
" if done:\n",
|
||
" # Reset the environment\n",
|
||
" print(\"Environment is reset\")\n",
|
||
" observation = env.reset()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "XIrKGGSlENZB"
|
||
},
|
||
"source": [
|
||
"## Create the LunarLander environment 🌛 and understand how it works\n",
|
||
"### [The environment 🎮](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n",
|
||
"In this first tutorial, we’re going to train our agent, a [Lunar Lander](https://www.gymlibrary.dev/environments/box2d/lunar_lander/), **to land correctly on the moon**. To do that, the agent needs to learn **to adapt its speed and position(horizontal, vertical, and angular) to land correctly.**\n",
|
||
"\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"\n",
|
||
"💡 A good habit when you start to use an environment is to check its documentation \n",
|
||
"\n",
|
||
"👉 https://www.gymlibrary.dev/environments/box2d/lunar_lander/\n",
|
||
"\n",
|
||
"---\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "poLBgRocF9aT"
|
||
},
|
||
"source": [
|
||
"Let's see what the Environment looks like:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "ZNPG0g_UGCfh"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# We create our environment with gym.make(\"<name_of_the_environment>\")\n",
|
||
"env = gym.make(\"LunarLander-v2\")\n",
|
||
"env.reset()\n",
|
||
"print(\"_____OBSERVATION SPACE_____ \\n\")\n",
|
||
"print(\"Observation Space Shape\", env.observation_space.shape)\n",
|
||
"print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "2MXc15qFE0M9"
|
||
},
|
||
"source": [
|
||
"We see with `Observation Space Shape (8,)` that the observation is a vector of size 8, where each value contains different information about the lander:\n",
|
||
"- Horizontal pad coordinate (x)\n",
|
||
"- Vertical pad coordinate (y)\n",
|
||
"- Horizontal speed (x)\n",
|
||
"- Vertical speed (y)\n",
|
||
"- Angle\n",
|
||
"- Angular speed\n",
|
||
"- If the left leg has contact point touched the land\n",
|
||
"- If the right leg has contact point touched the land\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "We5WqOBGLoSm"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(\"\\n _____ACTION SPACE_____ \\n\")\n",
|
||
"print(\"Action Space Shape\", env.action_space.n)\n",
|
||
"print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "MyxXwkI2Magx"
|
||
},
|
||
"source": [
|
||
"The action space (the set of possible actions the agent can take) is discrete with 4 actions available 🎮: \n",
|
||
"\n",
|
||
"- Do nothing,\n",
|
||
"- Fire left orientation engine,\n",
|
||
"- Fire the main engine,\n",
|
||
"- Fire right orientation engine.\n",
|
||
"\n",
|
||
"Reward function (the function that will gives a reward at each timestep) 💰:\n",
|
||
"\n",
|
||
"- Moving from the top of the screen to the landing pad and zero speed is about 100~140 points.\n",
|
||
"- Firing main engine is -0.3 each frame\n",
|
||
"- Each leg ground contact is +10 points\n",
|
||
"- Episode finishes if the lander crashes (additional - 100 points) or come to rest (+100 points)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "dFD9RAFjG8aq"
|
||
},
|
||
"source": [
|
||
"#### Vectorized Environment\n",
|
||
"- We create a vectorized environment (method for stacking multiple independent environments into a single environment) of 16 environments, this way, **we'll have more diverse experiences during the training.**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "99hqQ_etEy1N"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create the environment\n",
|
||
"env = make_vec_env('LunarLander-v2', n_envs=16)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "VgrE86r5E5IK"
|
||
},
|
||
"source": [
|
||
"## Create the Model 🤖\n",
|
||
"- Now that we studied our environment and we understood the problem: **being able to land correctly the Lunar Lander to the Landing Pad by controlling left, right and main orientation engine**. Let's build the algorithm we're going to use to solve this Problem 🚀.\n",
|
||
"\n",
|
||
"- To do so, we're going to use our first Deep RL library, [Stable Baselines3 (SB3)](https://stable-baselines3.readthedocs.io/en/master/).\n",
|
||
"\n",
|
||
"- SB3 is a set of **reliable implementations of reinforcement learning algorithms in PyTorch**.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"💡 A good habit when using a new library is to dive first on the documentation: https://stable-baselines3.readthedocs.io/en/master/ and then try some tutorials.\n",
|
||
"\n",
|
||
"----"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "3wgg-y0-92Wj"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "HV4yiUM_9_Ka"
|
||
},
|
||
"source": [
|
||
"To solve this problem, we're going to use SB3 **PPO**. [PPO (aka Proximal Policy Optimization) is one of the of the SOTA (state of the art) Deep Reinforcement Learning algorithms that you'll study during this course](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#example%5D).\n",
|
||
"\n",
|
||
"PPO is a combination of:\n",
|
||
"- *Value-based reinforcement learning method*: learning an action-value function that will tell us what's the **most valuable action to take given a state and action**.\n",
|
||
"- *Policy-based reinforcement learning method*: learning a policy that will **gives us a probability distribution over actions**.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "5qL_4HeIOrEJ"
|
||
},
|
||
"source": [
|
||
"Stable-Baselines3 is easy to set up:\n",
|
||
"\n",
|
||
"1️⃣ You **create your environment** (in our case it was done above)\n",
|
||
"\n",
|
||
"2️⃣ You define the **model you want to use and instantiate this model** `model = PPO(\"MlpPolicy\")`\n",
|
||
"\n",
|
||
"3️⃣ You **train the agent** with `model.learn` and define the number of training timesteps\n",
|
||
"\n",
|
||
"```\n",
|
||
"# Create environment\n",
|
||
"env = gym.make('LunarLander-v2')\n",
|
||
"\n",
|
||
"# Instantiate the agent\n",
|
||
"model = PPO('MlpPolicy', env, verbose=1)\n",
|
||
"# Train the agent\n",
|
||
"model.learn(total_timesteps=int(2e5))\n",
|
||
"```\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "nxI6hT1GE4-A"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# TODO: Define a PPO MlpPolicy architecture\n",
|
||
"# We use MultiLayerPerceptron (MLPPolicy) because the input is a vector,\n",
|
||
"# if we had frames as input we would use CnnPolicy\n",
|
||
"model = "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "QAN7B0_HCVZC"
|
||
},
|
||
"source": [
|
||
"#### Solution"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "543OHYDfcjK4"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# SOLUTION\n",
|
||
"# We added some parameters to accelerate the training\n",
|
||
"model = PPO(\n",
|
||
" policy = 'MlpPolicy',\n",
|
||
" env = env,\n",
|
||
" n_steps = 1024,\n",
|
||
" batch_size = 64,\n",
|
||
" n_epochs = 4,\n",
|
||
" gamma = 0.999,\n",
|
||
" gae_lambda = 0.98,\n",
|
||
" ent_coef = 0.01,\n",
|
||
" verbose=1)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "ClJJk88yoBUi"
|
||
},
|
||
"source": [
|
||
"### Train the PPO agent 🏃\n",
|
||
"- Let's train our agent for 500,000 timesteps, don't forget to use GPU on Colab. It will take approximately ~10min, but you can use less timesteps if you just want to try it out.\n",
|
||
"- During the training, take a ☕ break you deserved it 🤗"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "qKnYkNiVp89p"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# TODO: Train it for 500,000 timesteps\n",
|
||
"\n",
|
||
"# TODO: Specify file name for model and save the model to file\n",
|
||
"model_name = \"\"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "1bQzQ-QcE3zo"
|
||
},
|
||
"source": [
|
||
"#### Solution"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "poBCy9u_csyR"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# SOLUTION\n",
|
||
"# Train it for 500,000 timesteps\n",
|
||
"model.learn(total_timesteps=500000)\n",
|
||
"# Save the model\n",
|
||
"model_name = \"ppo-LunarLander-v2\"\n",
|
||
"model.save(model_name)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BY_HuedOoISR"
|
||
},
|
||
"source": [
|
||
"### Evaluate the agent 📈\n",
|
||
"- Now that our Lunar Lander agent is trained 🚀, we need to **check its performance**.\n",
|
||
"- Stable-Baselines3 provides a method to do that: `evaluate_policy`.\n",
|
||
"- To fill that part you need to [check the documentation](https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#basic-usage-training-saving-loading)\n",
|
||
"- In the next step, we'll see **how to automatically evaluate and share your agent to compete in a leaderboard, but for now let's do it ourselves**\n",
|
||
"\n",
|
||
"\n",
|
||
"💡 When you evaluate your agent, you should not use your training environment but create an evaluation environment."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "yRpno0glsADy"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# TODO: Evaluate the agent\n",
|
||
"# Create a new environment for evaluation\n",
|
||
"eval_env =\n",
|
||
"\n",
|
||
"# Evaluate the model with 10 evaluation episodes and deterministic=True\n",
|
||
"mean_reward, std_reward = \n",
|
||
"\n",
|
||
"# Print the results\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BqPKw3jt_pG5"
|
||
},
|
||
"source": [
|
||
"#### Solution"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "zpz8kHlt_a_m"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"#@title\n",
|
||
"eval_env = gym.make(\"LunarLander-v2\")\n",
|
||
"mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)\n",
|
||
"print(f\"mean_reward={mean_reward:.2f} +/- {std_reward}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "reBhoODwcXfr"
|
||
},
|
||
"source": [
|
||
"- In my case, I got a mean reward is `200.20 +/- 20.80` after training for 1 million steps, which means that our lunar lander agent is ready to land on the moon 🌛🥳."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "IK_kR78NoNb2"
|
||
},
|
||
"source": [
|
||
"### Publish our trained model on the Hub 🔥\n",
|
||
"Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
|
||
"\n",
|
||
"📚 The libraries documentation 👉 https://github.com/huggingface/huggingface_sb3/tree/main#hugging-face--x-stable-baselines3-v20\n",
|
||
"\n",
|
||
"Here's an example of a Model Card (with Space Invaders):"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Gs-Ew7e1gXN3"
|
||
},
|
||
"source": [
|
||
"By using `package_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
|
||
"\n",
|
||
"This way:\n",
|
||
"- You can **showcase our work** 🔥\n",
|
||
"- You can **visualize your agent playing** 👀\n",
|
||
"- You can **share with the community an agent that others can use** 💾\n",
|
||
"- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "JquRrWytA6eo"
|
||
},
|
||
"source": [
|
||
"To be able to share your model with the community there are three more steps to follow:\n",
|
||
"\n",
|
||
"1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
|
||
"\n",
|
||
"2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
|
||
"- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
|
||
"\n",
|
||
"<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/create-token.jpg\" alt=\"Create HF Token\">\n",
|
||
"\n",
|
||
"- Copy the token \n",
|
||
"- Run the cell below and paste the token"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "GZiFBBlzxzxY"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"notebook_login()\n",
|
||
"!git config --global credential.helper store"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "_tsf2uv0g_4p"
|
||
},
|
||
"source": [
|
||
"If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "FGNh9VsZok0i"
|
||
},
|
||
"source": [
|
||
"3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using `package_to_hub()` function"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Ay24l6bqFF18"
|
||
},
|
||
"source": [
|
||
"Let's fill the `package_to_hub` function:\n",
|
||
"- `model`: our trained model.\n",
|
||
"- `model_name`: the name of the trained model that we defined in `model_save`\n",
|
||
"- `model_architecture`: the model architecture we used: in our case PPO\n",
|
||
"- `env_id`: the name of the environment, in our case `LunarLander-v2`\n",
|
||
"- `eval_env`: the evaluation environment defined in eval_env\n",
|
||
"- `repo_id`: the name of the Hugging Face Hub Repository that will be created/updated `(repo_id = {username}/{repo_name})`\n",
|
||
"\n",
|
||
"💡 **A good name is {username}/{model_architecture}-{env_id}**\n",
|
||
"\n",
|
||
"- `commit_message`: message of the commit"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "JPG7ofdGIHN8"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import gym\n",
|
||
"from stable_baselines3.common.vec_env import DummyVecEnv\n",
|
||
"from stable_baselines3.common.env_util import make_vec_env\n",
|
||
"\n",
|
||
"from huggingface_sb3 import package_to_hub\n",
|
||
"\n",
|
||
"## TODO: Define a repo_id\n",
|
||
"## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
|
||
"repo_id = \n",
|
||
"\n",
|
||
"# TODO: Define the name of the environment\n",
|
||
"env_id = \n",
|
||
"\n",
|
||
"# Create the evaluation env\n",
|
||
"eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
|
||
"\n",
|
||
"\n",
|
||
"# TODO: Define the model architecture we used\n",
|
||
"model_architecture = \"\"\n",
|
||
"\n",
|
||
"## TODO: Define the commit message\n",
|
||
"commit_message = \"\"\n",
|
||
"\n",
|
||
"# method save, evaluate, generate a model card and record a replay video of your agent before pushing the repo to the hub\n",
|
||
"package_to_hub(model=model, # Our trained model\n",
|
||
" model_name=model_name, # The name of our trained model \n",
|
||
" model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
|
||
" env_id=env_id, # Name of the environment\n",
|
||
" eval_env=eval_env, # Evaluation Environment\n",
|
||
" repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
|
||
" commit_message=commit_message)\n",
|
||
"\n",
|
||
"# Note: if after running the package_to_hub function and it gives an issue of rebasing, please run the following code\n",
|
||
"# cd <path_to_repo> && git add . && git commit -m \"Add message\" && git pull \n",
|
||
"# And don't forget to do a \"git push\" at the end to push the change to the hub."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Avf6gufJBGMw"
|
||
},
|
||
"source": [
|
||
"#### Solution\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "xMkkkukIBQJM"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import gym\n",
|
||
"\n",
|
||
"from stable_baselines3 import PPO\n",
|
||
"from stable_baselines3.common.vec_env import DummyVecEnv\n",
|
||
"from stable_baselines3.common.env_util import make_vec_env\n",
|
||
"\n",
|
||
"from huggingface_sb3 import package_to_hub\n",
|
||
"\n",
|
||
"# PLACE the variables you've just defined two cells above\n",
|
||
"# Define the name of the environment\n",
|
||
"env_id = \"LunarLander-v2\"\n",
|
||
"\n",
|
||
"# TODO: Define the model architecture we used\n",
|
||
"model_architecture = \"PPO\"\n",
|
||
"\n",
|
||
"## Define a repo_id\n",
|
||
"## repo_id is the id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
|
||
"## CHANGE WITH YOUR REPO ID\n",
|
||
"repo_id = \"ThomasSimonini/ppo-LunarLander-v2\" # Change with your repo id, you can't push with mine 😄\n",
|
||
"\n",
|
||
"## Define the commit message\n",
|
||
"commit_message = \"Upload PPO LunarLander-v2 trained agent\"\n",
|
||
"\n",
|
||
"# Create the evaluation env\n",
|
||
"eval_env = DummyVecEnv([lambda: gym.make(env_id)])\n",
|
||
"\n",
|
||
"# PLACE the package_to_hub function you've just filled here\n",
|
||
"package_to_hub(model=model, # Our trained model\n",
|
||
" model_name=model_name, # The name of our trained model \n",
|
||
" model_architecture=model_architecture, # The model architecture we used: in our case PPO\n",
|
||
" env_id=env_id, # Name of the environment\n",
|
||
" eval_env=eval_env, # Evaluation Environment\n",
|
||
" repo_id=repo_id, # id of the model repository from the Hugging Face Hub (repo_id = {organization}/{repo_name} for instance ThomasSimonini/ppo-LunarLander-v2\n",
|
||
" commit_message=commit_message)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "pN0-W6kkKn44"
|
||
},
|
||
"source": [
|
||
"Congrats 🥳 you've just trained and uploaded your first Deep Reinforcement Learning agent. The script above should have displayed a link to a model repository such as https://huggingface.co/osanseviero/test_sb3. When you go to this link, you can:\n",
|
||
"* see a video preview of your agent at the right. \n",
|
||
"* click \"Files and versions\" to see all the files in the repository.\n",
|
||
"* click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
|
||
"* a model card (`README.md` file) which gives a description of the model\n",
|
||
"\n",
|
||
"Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
|
||
"\n",
|
||
"Compare the results of your LunarLander-v2 with your classmates using the leaderboard 🏆 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "9nWnuQHRfFRa"
|
||
},
|
||
"source": [
|
||
"## Load a saved LunarLander model from the Hub 🤗\n",
|
||
"Thanks to [ironbar](https://github.com/ironbar) for the contribution.\n",
|
||
"\n",
|
||
"Loading a saved model from the Hub is really easy. \n",
|
||
"\n",
|
||
"You go https://huggingface.co/models?library=stable-baselines3 to see the list of all the Stable-baselines3 saved models.\n",
|
||
"1. You select one and copy its repo_id"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "qDwKbhJ0ffuG"
|
||
},
|
||
"source": [
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "hNPLJF2bfiUw"
|
||
},
|
||
"source": [
|
||
"2. Then we just need to use load_from_hub with:\n",
|
||
"- The repo_id\n",
|
||
"- The filename: the saved model inside the repo and its extension (*.zip)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "oj8PSGHJfwz3"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from huggingface_sb3 import load_from_hub\n",
|
||
"repo_id = \"\" # The repo_id\n",
|
||
"filename = \"\" # The model filename.zip\n",
|
||
"\n",
|
||
"# When the model was trained on Python 3.8 the pickle protocol is 5\n",
|
||
"# But Python 3.6, 3.7 use protocol 4\n",
|
||
"# In order to get compatibility we need to:\n",
|
||
"# 1. Install pickle5 (we done it at the beginning of the colab)\n",
|
||
"# 2. Create a custom empty object we pass as parameter to PPO.load()\n",
|
||
"custom_objects = {\n",
|
||
" \"learning_rate\": 0.0,\n",
|
||
" \"lr_schedule\": lambda _: 0.0,\n",
|
||
" \"clip_range\": lambda _: 0.0,\n",
|
||
"}\n",
|
||
"\n",
|
||
"checkpoint = load_from_hub(repo_id, filename)\n",
|
||
"model = PPO.load(checkpoint, custom_objects=custom_objects, print_system_info=True)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "Fs0Y-qgPgLUf"
|
||
},
|
||
"source": [
|
||
"Let's evaluate this agent:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"#@title\n",
|
||
"eval_env = gym.make(\"LunarLander-v2\")\n",
|
||
"mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True)\n",
|
||
"print(f\"mean_reward={mean_reward:.2f} +/- {std_reward}\")"
|
||
],
|
||
"metadata": {
|
||
"id": "PAEVwK-aahfx"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BQAwLnYFPk-s"
|
||
},
|
||
"source": [
|
||
"## Some additional challenges 🏆\n",
|
||
"The best way to learn **is to try things by your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. With 1,000,000 steps, we saw some great results! \n",
|
||
"\n",
|
||
"In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
|
||
"\n",
|
||
"Here are some ideas to achieve so:\n",
|
||
"* Train more steps\n",
|
||
"* Try different hyperparameters of `PPO`. You can see them at https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html#parameters. \n",
|
||
"* Check the [Stable-Baselines3 documentation](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) and try another models such as DQN.\n",
|
||
"* **Push your new trained model** on the Hub 🔥\n",
|
||
"\n",
|
||
"**Compare the results of your LunarLander-v2 with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) 🏆\n",
|
||
"\n",
|
||
"Is moon landing too boring to you? Try to **change the environment**, why not using MountainCar-v0, CartPole-v1 or CarRacing-v0? Check how they works [using the gym documentation](https://www.gymlibrary.dev/) and have fun 🎉."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "9lM95-dvmif8"
|
||
},
|
||
"source": [
|
||
"________________________________________________________________________\n",
|
||
"Congrats on finishing this chapter! That was the biggest one, **and there was a lot of information.**\n",
|
||
"\n",
|
||
"If you’re still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
|
||
"\n",
|
||
"Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations.\n",
|
||
"\n",
|
||
"Naturally, during the course, we’re going to use and deeper explain again these terms but **it’s better to have a good understanding of them now before diving into the next chapters.**\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "feR90OUSEXq9"
|
||
},
|
||
"source": [
|
||
"\n",
|
||
"\n",
|
||
"## This is a course built with you 👷🏿♀️\n",
|
||
"\n",
|
||
"We want to improve and update the course iteratively with your feedback. If you have some, please fill this form 👉 https://forms.gle/3HgA7bEHwAmmLfwh9\n",
|
||
"\n",
|
||
"If you found some issues in this notebook, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues).\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "BjLhT70TEZIn"
|
||
},
|
||
"source": [
|
||
"See you on [Unit 2](https://github.com/huggingface/deep-rl-class/tree/main/unit2#unit-2-introduction-to-q-learning)! 🔥\n",
|
||
"TODO CHANGE LINK\n",
|
||
"## Keep learning, stay awesome 🤗"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"accelerator": "GPU",
|
||
"colab": {
|
||
"private_outputs": true,
|
||
"provenance": [],
|
||
"collapsed_sections": [
|
||
"feR90OUSEXq9"
|
||
]
|
||
},
|
||
"gpuClass": "standard",
|
||
"kernelspec": {
|
||
"display_name": "Python 3.9.7",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"name": "python",
|
||
"version": "3.9.7"
|
||
},
|
||
"vscode": {
|
||
"interpreter": {
|
||
"hash": "ed7f8024e43d3b8f5ca3c5e1a8151ab4d136b3ecee1e3fd59e0766ccc55e1b10"
|
||
}
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 0
|
||
} |