merge upstream

2026-04-15 10:59:56 +08:00 · 2022-03-14 15:27:44 -04:00
parent c2ef729df0 9b19d5d762
commit 254fecdda8
76 changed files with 38004 additions and 136 deletions
--- a/STYLE_GUIDE.md
+++ b/STYLE_GUIDE.md
@@ -10,7 +10,7 @@
 ## 文本

 * LaTeX文件转换Markdown文件
-    * Linux下安装Pandoc命令：agt-get install pandoc
+    * Linux下安装Pandoc命令：apt-get install pandoc
    * 使用Pandoc命令：pandoc -s example.tex -o example.md
 * 使用Pandoc转换后需要注意修改的地方
    * 表格需要手动改
--- a/chapter_accelerator/accelerator_architecture.md
+++ b/chapter_accelerator/accelerator_architecture.md
@@ -5,7 +5,7 @@

 ### 硬件加速器的架构

-现代GPU在十分有限的面积上实现了极强的计算能力和极高的储存器以及IO带宽。一块高端的GPU中，晶体管数量已经达到主流CPU的两倍，而且显存已经达到了16GB以上,工作频率也达到了1GHz。GPU的体系架构由两部分组成，分别是流处理阵列和存储器系统，两部分通过一个片上互联网络连接。流处理器阵列和存储器系统都可以单独扩展，规格可以根据产品的市场定位单独裁剪。如GV100的组成如 :numref:`gv100`所示：
+现代GPU在十分有限的面积上实现了极强的计算能力和极高的储存器以及IO带宽。一块高端的GPU中，晶体管数量已经达到主流CPU的两倍，而且显存已经达到了16GB以上,工作频率也达到了1GHz。GPU的体系架构由两部分组成，分别是流处理阵列和存储器系统，两部分通过一个片上互联网络连接。流处理器阵列和存储器系统都可以单独扩展，规格可以根据产品的市场定位单独裁剪。如GV100的组成 :cite:`2017NVIDIA`如 :numref:`gv100`所示：

 ![Volta GV100](../img/ch06/V100.svg)
 :width:`800px`
--- a/chapter_accelerator/accelerator_programming.md
+++ b/chapter_accelerator/accelerator_programming.md
@@ -2,12 +2,16 @@
 ------------------
 :label:`accelerator-program-title`

-本章前两节主要介绍了这些硬件加速器设计的意义、思路以及基本组成原理。软硬件协同优化作为构建高效AI系统的一个重要指导思想，需要软件算法/软件栈和硬件架构在神经网络应用中互相影响、紧密耦合。为了最大限度地发挥加速器的优势，要求能够基于硬件系统架构设计出一套较为匹配的指令或编程（操纵）方法。因此，本节将以 :numref:`compute-unit-title`中介绍的Tensor Core为例，着重介绍加速器的可编程性，以及如何通过编程使能加速器，提升神经网络算子的计算效率。
+本章前两节主要介绍了硬件加速器设计的意义、思路以及基本组成原理。软硬件协同优化作为构建高效AI系统的一个重要指导思想，需要软件算法/软件栈和硬件架构在神经网络应用中互相影响、紧密耦合。为了最大限度地发挥加速器的优势，要求能够基于硬件系统架构提供易用、高效的编程方法。因此，在本节中将着重介绍加速器的可编程性，包括编程接口直接调用方式及算子编译器优化方式。最后，通过示例介绍如何通过编程使能加速器，提升神经网络算子的计算效率。

 ### 硬件加速器的可编程性
 :label:`accelerator-programable-title`

- :numref:`accelerator-design-title`节中列出的硬件加速器均具有一定的可编程性，程序员可以通过软件编程，有效的使能上述加速器进行计算加速。但出于计算效率和易用性等方面考虑，将编程使能方式分为不同等级，一般包括：算子库层级，编程原语层级，以及指令层级。为了更具象的解释上述层级的区别，仍以Volta架构的Tensor Core加速器为例，由高层至底层对比介绍这三种不同编程方式：
+ :numref:`accelerator-design-title`节中列出的硬件加速器均具有一定的可编程性，程序员可以通过软件编程，有效的使能上述加速器进行计算加速。现有硬件加速器常见的两类编程方式主要有编程接口调用以及算子编译器优化。
+
+ #### 编程接口使能加速器
+
+硬件加速器出于计算效率和易用性等方面考虑，将编程接口使能方式分为不同等级，一般包括：算子库层级，编程原语层级，以及指令层级。为了更具象的解释上述层级的区别，我们以Volta架构的Tensor Core加速器为例，由高层至底层对比介绍这三种不同编程方式：

 -   **算子库层级**：如cuBLAS基本矩阵与向量运算库，cuDNN深度学习加速库，均通过Host端调用算子库提供的核函数使能TensorCore；

@@ -15,23 +19,55 @@

 -   **指令层级**：如PTX ISA MMA指令集，提供更细粒度的mma指令，便于用户组成更多种形状的接口，通过CUDA Device端内联编程使能TensorCore。

-矩阵乘法运算作为深度学习网络中占比最大的计算，对其进行优化是十分必要的。因此本节将统一以矩阵乘法$D[M, N] = C[M, N] + A[M, K] * B[K, N]$为实例，对比介绍如何通过不同编程方式使能加速器。
+ #### 算子编译器使能加速器

-```
-$A, B$矩阵 $D$矩阵  $C[i][j] \gets 0$ 
-$C[i][j] \gets C[i][j] + A[i, k] \times B[k, j]$ 
-$D[i][j] \gets C[i][j]$ 
-```
+ DSA架构的多维度AI加速器通常提供了更多的指令选择（3D-Matrix/2D-Vector/1D-Scalar），以及更加复杂的数据流处理，通过提供接口调用的方式对程序开发人员带来较大的挑战。此外，由于调度、切分的复杂度增加，直接提供算子库的方式由于缺少根据目标shape调优的能力，往往无法在所有shape下均得到最优的性能。因此，对于DSA加速器，业界通常采用算子编译器的解决方案。
+
+ 随着深度学习模型的迭代更新及各类AI芯片的层出不穷，基于人工优化算子的方式给算子开发团队带来沉重的负担。因此，开发一种能够将High-level的算子表示编译成目标硬件可执行代码的算子编译器，逐渐成为学术界及工业界的共识。算子编译器前端通常提供了特定领域描述语言（DSL），用于定义算子的计算范式；类似于传统编译器，算子编译器也会将算子计算表示转换为中间表示，如HalideIR :cite:`ragan2013halide`、TVM :cite:`chen2018tvm`的TIR、Schedule Tree :cite:`verdoolaege2010isl`等，基于模板（手动）、搜索算法或优化求解算法（自动）等方式完成循环变换、循环切分等调度相关优化，以及硬件指令映射、内存分配、指令流水等后端pass优化，最后通过codegen模块将IR转换为DSA加速器可执行的kernel。
+
+ 当前业界的算子编译器/编译框架主要有TVM/Ansor :cite:`zheng2020ansor`、MLIR :cite:`lattner2020mlir`、以及华为Ascend芯片上的TBE/AKG :cite:`zhao2021akg`等。
+
+ -   **TVM/Ansor**
+ 
+ ![TVM](../img/ch06/TVM.svg)
+:width:`800px`
+:label:`tvm`
+
+ TVM是陈天奇博士等人开发的开源深度学习编译框架，提供了端到端的编译优化（图优化/算子优化）能力，在工业界应用较广。在架构上，主要包括relay和tir两层。通过relay导入推理模型，进行算子融合等图层优化，通过tir生成融合算子。在算子编译方面，TVM采用了计算和调度分离的技术，为不同的算子提供了不同的模板，同时支持自定义模板，优化特定算子类型调度。为了更进一步优化算子性能，TVM支持对算子进行自动tuning，来生成较优的切分参数。此外，为了简化用户开发模板的工作，TVM在0.8版本后提供了自动调度能力Ansor，通过搜索的方式，为目标算子生成调度及切分参数。
+
+
+ -   **MLIR**
+ 前面的章节介绍过，Google开发的MLIR并不是一个单一的算子编译器，而是一套编译器基础设施，提供了工具链的组合与复用能力。基于MLIR，DSA加速器厂商可以快速的搭建其定制化算子编译器。如Google论文 :cite:`vasilache2022composable`中所述，当前的算子编译器大多提供了一整套自顶向下的编译优化pass，包括调度优化、切分优化、窥孔优化、后端优化、指令生成等，彼此之间大多无法复用，导致新的场景中通常又得从头开发。而在MLIR中，将功能相近的IR优化pass封装为方言（Dialect），并且提供了多个代码生成相关的基础方言，如vector、memref、tensor、scf、affine、linalg等。硬件厂商可以基于这些Dialect，快速构建一整套lower优化及codegen流程。如下图所示，利用scf、affine、linalg等方言，对结构化的计算IR完成循环并行优化、切分、向量化等，最后基于LLVM完成指令映射。
+
+ -   **华为TBE/AKG**
+ TBE（Tensor Boost Engine）是华为的Ascend芯片及其CANN软件栈基于TVM 开发的一套算子编译优化工具，用于对Ascend芯片进行调度优化、指令映射、及后端pass优化等。不仅提供了一个优化过的神经网络标准算子库，同时还提供了算子开发能力及融合能力。通过TBE提供的API和自定义算子编程开发界面可以完成相应神经网络算子的开发，帮助用户较容易的去使能硬件加速器上的AI_CORE 相关指令，以实现高性能的神经网络计算。为了简化算子开发流程，TBE还实现了一个Auto Schedule工具，开放了自定义算子编程DSL，用于自动完成复杂算子的调度生成。此外，TBE还实现了端到端的动态shape算子编译能力。
+![TBE](../img/ch06/tbe.png)
+:width:`800px`
+:label:`tbe`
+
+ AKG则是MindSpore社区的开源算子编译工具。与上述介绍的算子编译器不同，AKG基于Polyhedral多面体编译技术 :cite:`bastoul2004code`，支持在CPU/GPU/Ascend多硬件上自动生成满足并行性与数据局部性的调度。Polyhedral编译技术的核心思想是将程序中循环的迭代空间映射为高维空间多面体，通过分析语句读写依赖关系，将循环调度优化问题转换为整数规划求解问题。
+ AKG的编译流程如下图所示，主要包含程序规范化、自动调度优化、指令生成、后端优化几个模块。AKG同样基于TVM实现，支持TVM compute/Hybrid DSL编写的算子表示，以及MindSpore图算融合模块优化后的融合子图。通过IR规范化，将DSL/子图IR转换为polyhedral编译的调度树。在polyhedral模块中，利用其提供的调度算法，实现循环的自动融合、自动重排等变换，为融合算子自动生成满足并行性、数据局部性的初始调度。为了能够快速适配不同的硬件后端，我们在poly模块内将优化pass识别为硬件无关的通用优化与硬件相关的特定优化，编译时按照硬件特征拼接组合，实现异构硬件后端的快速适配。
+
+![AKG](../img/ch06/akg.png)
+:width:`800px`
+:label:`akg`
+
+ 在polyhedral模块中，实现了算子的自动调度生成、自动切分以及自动数据搬移。为了进一步提升算子的性能，我们针对不同硬件后端开发了相应的优化pass，如Ascend后端中实现数据对齐、指令映射，GPU后端中实现向量化存取，插入同步指令等，最终生成相应平台代码。

 ### 硬件加速器的多样化编程方法
 :label:`diversified-programming-title`

-#### 算子库使能加速器
+矩阵乘法运算作为深度学习网络中占比最大的计算，对其进行优化是十分必要的。因此本节将统一以矩阵乘法$D[M, N] = C[M, N] + A[M, K] \times B[K, N]$为实例，对比介绍如何通过不同编程方式使能加速器。

-在上述三种层级的编程方式中，直接调用算子加速库使能加速器无疑是最快捷高效的方式。NVIDIA提供了cuBLAS/cuDNN两类算子计算库，cuBLAS提供了使能Tensor Core单元的接口，用以加速矩阵乘法(GEMM)运算，cuDNN提供了对应接口加速卷积(CONV)运算等。
+![矩阵乘法GEMM运算](../img/ch06/gemm.svg)
+:width:`800px`
+:label:`gemm-algorith`

+#### 编程接口使能加速器
+
+-   **算子库层级**
+在上述不同层级的编程方式中，直接调用算子加速库使能加速器无疑是最快捷高效的方式。NVIDIA提供了cuBLAS/cuDNN两类算子计算库，cuBLAS提供了使能Tensor Core单元的接口，用以加速矩阵乘法(GEMM)运算，cuDNN提供了对应接口加速卷积(CONV)运算等。
 以 :numref:`accelerator-programable-title`小节的GEMM运算为例，与常规CUDA调用cuBLAS算子库相似，通过cuBLAS加速库使能Tensor Core步骤包括：
-
 1.  创建cuBLAS对象句柄且设置对应数学计算模式

 ```cpp
@@ -74,10 +110,8 @@ cudaDestroy(handle);

 该方式由于能够隐藏体系结构细节，易用性较好，且一般官方提供的算子库吞吐量较高。但与此同时，这种算子颗粒度的库也存在一些问题，如不足以应对复杂多变的网络模型导致的算子长尾问题（虽然常规形式算子占据绝大多数样本，但仍有源源不断的新增算子，因其出现机会较少，算子库未对其进行有效优化。），以及错失了较多神经网络框架优化（如算子融合）的机会。

-#### 编程原语使能加速器
-
+-   **编程原语层级**
 第二种加速器编程方式为编程原语使能加速器，如通过在Device端调用CUDA WMMA (Warp Matrix Multiply Accumulate) API接口。以线程束（即Warp，是调度的基本单位）为操纵对象，使能多个Tensor Core单元。该方式在CUDA 9.0中被公开，程序员可通过添加API头文件的引用和命名空间定义来使用上述API接口。基于软硬件协同设计的基本思想，该层级编程API的设计多与架构绑定，如WMMA操纵的总是$16\times16$大小的矩阵块，并且操作一次跨两个TensorCore进行处理，本质是与TensorCore如何集成进SM中强相关的。针对Float16输入数据类型，NVIDIA官方提供了三种不同矩阵规模的WMMA乘累加计算接口，分别为$16\times16\times16$，$32\times8\times16$，$8\times32\times16$。
-
 该API接口操纵的基本单位为Fragment，是一种指明了矩阵含义（乘法器/累加器）、矩阵形状（$WMMA\_M, WMMA\_N, WMMA\_K$）、数据类型（Half/ Float）、排布方式（$row\_major/ col\_major$）等信息的模板类型，包括如下：

 ```cpp
@@ -86,12 +120,11 @@ wmma::fragment<wmma::matrix_b, WMMA_M, WMMA_N, WMMA_K, half, wmma::col_major> b_
 wmma::fragment<wmma::accumulator, WMMA_M, WMMA_N, WMMA_K, float> acc_frag;
 wmma::fragment<wmma::accumulator, WMMA_M, WMMA_N, WMMA_K, float> c_frag;
 ```
+ 使用时，我们需要将待执行乘法操作矩阵块的数据，作为Fragment，由寄存器加载至TensorCore，在将累加Fragment初始化/清零操作后，通过TensorCore单元执行乘累加运算，最后将运算结果的Fragment存回寄存器或其他内存区域。与上述操作对应的，NVIDIA提供了$wmma.load\_matrix\_sync(), wmma.store\_matrix\_sync()$接口用于将参与计算的子矩阵块写入/载出Fragment片段；$wmma.fill\_fragment()$接口用于初始化对应Fragment的数据；$wmma.mma\_sync()$接口用于对Fragment进行乘累加运算。

-使用时，我们需要将待执行乘法操作矩阵块的数据，作为Fragment，由寄存器加载至TensorCore，在将累加Fragment初始化/清零操作后，通过TensorCore单元执行乘累加运算，最后将运算结果的Fragment存回寄存器或其他内存区域。与上述操作对应的，NVIDIA提供了$wmma.load\_matrix\_sync(), wmma.store\_matrix\_sync()$接口用于将参与计算的子矩阵块写入/载出Fragment片段；$wmma.fill\_fragment()$接口用于初始化对应Fragment的数据；$wmma.mma\_sync()$接口用于对Fragment进行乘累加运算。
+-   **指令层级**

-#### 指令集编程使能加速器
-
-在NVIDIA PTX ISA (Instruction Set Architecture)中提供了另一个编程接口，如Volta架构中的$mma.sync.m8n8k4$指令，它使用$M=8, N=8, K=4$的形状配置执行乘累加操作。具体地，它由线程组（黑色椭圆表示）或octet执行，如 :numref:`PTX`显示了线程和数据的映射关系。每个线程组由四个连续的线程组成，使用不同颜色的圆圈表示。图中还指出了一个octet里面的线程在线程束内的分布，Float16乘法器A或B的四个连续元素（使用具有相同颜色的块表示），以及Float32累加器C或D的八个分散元素（同样使用相同颜色的块表示）。彩色块上的数字代表对应的线程ID。
+在NVIDIA PTX ISA (Instruction Set Architecture)中提供了另一个编程接口，如Volta架构中的$mma.sync.m8n8k4$指令，它使用$M=8, N=8, K=4$的形状配置执行乘累加操作。具体地，它由线程组（黑色椭圆表示）或octet执行 :cite:`2018Modeling`，如 :numref:`PTX`显示了线程和数据的映射关系。每个线程组由四个连续的线程组成，使用不同颜色的圆圈表示。图中还指出了一个octet里面的线程在线程束内的分布，Float16乘法器A或B的四个连续元素（使用具有相同颜色的块表示），以及Float32累加器C或D的八个分散元素（同样使用相同颜色的块表示）。彩色块上的数字代表对应的线程ID。

 ![mma指令之线程与矩阵元素映射关系](../img/ch06/ptx.svg)
 :width:`800px`
@@ -101,11 +134,7 @@ wmma::fragment<wmma::accumulator, WMMA_M, WMMA_N, WMMA_K, float> c_frag;

 #### 算子编译器编程使能加速器

-除了上述三种层级的编程方式，算子编译器也逐渐成为DSA加速器的关注热点。随着深度学习模型的迭代更新以及各类DSA加速器的层出不穷，手写算子或高性能算子库（如cuDNN/cuBLAS）等基于人工优化算子的方式给算子开发团队带来沉重的负担。因此，开发一种能够将High-level的算子表示编译成目标硬件可执行代码的算子编译器，成为了学术界、业界的共识。
-
-近年来涌现出许多算子编译器/编译框架，如TVM，为不同的硬件后端提供了编译优化支持。在昇腾芯片上，基于TVM开发了TBE(Tensor Boost Engine)，不仅提供了一个优化过的神经网络标准算子库，同时还提供了算子开发能力及融合能力。通过TBE提供的API和自定义算子编程开发界面可以完成相应神经网络算子的开发，帮助用户较容易的去使能硬件加速器上的AI\_CORE相关指令，以实现高性能的神经网络计算。此外，为了更好的支持复杂算子融合场景，还提供了基于polyhedral多面体编译技术的AKG(Auto kernel generator)，提供算子的自动生成能力。
-
-基于算子编译器使能加速器实现矩阵乘的流程则对用户更加友好，用户只需基于python定义矩阵乘的tensor信息（数据类型及形状等），调用对应TBE接口即可。如下所示：
+基于算子编译器使能加速器实现矩阵乘的流程则对用户更加友好。以在Ascend中使用TBE为例，用户只需基于python定义矩阵乘的tensor信息（数据类型及形状等），调用对应TBE接口即可。如下所示：

 ```python
 a_shape = (1024, 256)
@@ -123,21 +152,11 @@ res = te.lang.cce.matmul(tensor_a, tensor_b, False, False, False, dst_dtype=dst_

 本节 :numref:`accelerator-program-title`前几个小节主要介绍了硬件加速器的不同层级的多样化编程方法。调用计算库的方式留给程序员的优化空间较少，合理利用硬件加速器不同层级的编程，可以实现更好的性能优化。 为了更好的让读者理解硬件加速器的使用，本节会继续 :numref:`accelerator-programable-title`节中的GEMM运算，仍以WMMA API使能Tensor Core加速单元为例，介绍如何通过矩阵分块、资源映射等方式更高效的利用硬件加速器。

-[\[alg:TensorCore\]]{#alg:TensorCore label="alg:TensorCore"}
+![TensorCore矩阵乘法GEMM运算](../img/ch06/gemm_tensor_core.svg)
+:width:`800px`
+:label:`gemm-tensor-core-algorith`

-```
-$A, B$矩阵 $D$矩阵   Mapping to Block.Idx  Mapping to Block.Idy  Mapping
-to Block.Idz
-$A_{Shared}[i_o][k_o] \gets A[i_o][k_o]$ $B_{Shared}[k_o][j_o] \gets B[k_o][j_o]$ $Syncthreads()$ 
- Mapping to Warp.Idx  Mapping to Warp.Idy
-$A_{Register}[i_i][k_i] \gets A_{Shared}[i_i][k_i]$ $B_{Register}[k_i][j_i] \gets B_{Shared}[k_i][j_i]$ $pragma\ unroll$ 
-$wmma.load\_matrix\_sync(A_{Fragment}, A_{Register})$ $wmma.load\_matrix\_sync(B_{Fragment}, B_{Register})$ 
-$wmma.fill\_fragment(C_{Fragment}, 0)$ 
-$wmma.mma\_sync(D_{Fragment}, C_{Fragment}, A_{Fragment}, B_{Fragment})$ 
-$Syncthreads()$  $wmma.store\_matrix\_sync(D, D_{Fragment})$ 
-```
-
-若要得到高性能CUDA程序，提高并行性、增大吞吐量、优化指令执行是至关重要的三个优化目标。针对该实例，具体地实现和优化方案列出如下，对应到具体实例伪代码如算法2所示：
+若要得到高性能CUDA程序，提高并行性、增大吞吐量、优化指令执行是至关重要的三个优化目标。针对该实例，具体地实现和优化方案列出如下，对应到具体实例伪代码如 :numref:`gemm-tensor-core-algorith`所示：

 1.  **优化内存结构------增大吞吐量**：将原始大规模矩阵根据不同阈值切分成不同层级的子矩阵块，使得子矩阵块能被如共享内存、寄存器等高性能体系结构存储下来，以此提高吞吐量。设置切分参数为$BlockTile[Ms, Ns, Ks]$和$WarpTile[Mw, Nw, Kw]$，对应的将BlockTile下的矩阵由全局内存搬移至共享内存，以提高全局内存合并访问和数据局部性，如 :numref:`GEMM-BlockTile`所示；再将内层WarpTile下的矩阵由共享内存搬移至寄存器中，如 :numref:`GEMM-WarpTile`所示，以备Tensor Core加速器数据存取。

@@ -149,7 +168,7 @@ $Syncthreads()$  $wmma.store\_matrix\_sync(D, D_{Fragment})$
 :width:`800px`
 :label:`GEMM-WarpTile`

-2.  **并行资源映射------提高并行性**：将多层级的并行资源（Block、Warp、Thread）与对应需要计算/搬移的数据建立映射关系，提高程序并行性。将可并行的计算/数据搬移操作映射到并行资源上，对于GEMM实例，M/N轴即为可并行轴，将数据搬移操作中的循环指令映射分配到Block层级（即算法3中的2-4行$For$循环），将内层循环指令映射分配到Warp层级（即算法3中的8-10行$For$循环）。（前文介绍，线程束Warp作为调度的基本单位，且是WMMA API操纵的基本层级，因此对Warp层级进行数据映射比Thread层级映射更为合适）
+2.  **并行资源映射------提高并行性**：将多层级的并行资源（Block、Warp、Thread）与对应需要计算/搬移的数据建立映射关系，提高程序并行性。将可并行的计算/数据搬移操作映射到并行资源上，对于GEMM实例，M/N轴即为可并行轴，将数据搬移操作中的循环指令映射分配到Block层级（即 :numref:`gemm-tensor-core-algorith`中的2-4行$For$循环），将内层循环指令映射分配到Warp层级（即 :numref:`gemm-tensor-core-algorith`中的8-9行$For$循环）。（前文介绍，线程束Warp作为调度的基本单位，且是WMMA API操纵的基本层级，因此对Warp层级进行数据映射比Thread层级映射更为合适）

 3.  **Warp统一的Tensor Core数据交互------增大吞吐量**：根据 :numref:`diversified-programming-title`节中介绍的编程方法，除调用算子库外，均需要使用或将指令封装成WMMA接口形式统一进行Warp层级的数据存取和计算。如 :numref:`GEMM-TensorCore`所示，Tensor Core加速器需要从局部内存/寄存器中读取数据，存于虚拟Fragment数据结构中，对应使用$wmma.load\_matrix\_sync()$接口，将累加Fragment $C$ 通过$wmma.fill\_fragment()$接口进行初始化后，使用$wmma.mma\_sync()$使能加速器进行乘累加运算，后将结果Fragment $D$通过调用$wmma.store\_matrix\_sync()$接口拷贝至目标内存地址。

@@ -160,6 +179,3 @@ $Syncthreads()$  $wmma.store\_matrix\_sync(D, D_{Fragment})$
 4.  **优化数据访存------提高并行性**：在进行内存结构变化（矩阵数据搬移）时，需要注意全局内存的合并访问、共享内存的存储体冲突等常见性能瓶颈点。

 5.  **资源负载均衡------增大吞吐量**：调整平衡每个线程处理的数据量、共享内存使用量、寄存器使用量，以获得更高的SM占用率。一般在实际程序中BlockTile和WarpTile的选取至关重要。
-
-6.  **优化指令执行**：使用\#unroll功能进行循环展开以避免分支冲突，如算法3中13行；使用向量化加载指令减少PTX指令执行次数以提高带宽等，对于GPU Volta架构，最大向量化加载指令为ldg128，即128比特带宽，对于算法3中5-6行数据由全局内存加载至共享内存时，即可采用Float4\*类型指针进行内存读取。
-
--- a/chapter_computational_graph/generation_of_computational_graph.md
+++ b/chapter_computational_graph/generation_of_computational_graph.md
@@ -1,5 +1,5 @@
 ## 计算图的生成
-计算框架执行深度学习模型训练时，会根据模型结构生成计算图，通过调度计算图完成模型计算。在计算框架中可以生成静态图和动态图两种计算图。静态图对应声明式编程范式，动态图对应命令式编程范式。静态生成可以根据前端语言描述的神经网络拓扑结构以及参数变量等信息构建一份固定的计算图，因此静态图在执行期间可以不依赖前端语言描述常用于神经网络模型的部署，比如移动端人脸识别场景中的应用等。动态图则需要在每一次执行神经网络模型依据前端语言描述动态生成一份临时的计算图，这意味着计算图的动态生成过程灵活可变，该特性有助于我们在神经网络结构调整阶段提高效率。主流计算框架TensorFlow、MindSpore、PyTorch均支持动态图和静态图模式。了解两种计算图生成方式的优缺点及构建执行特点，可以针对待解决的任务需求，选择合适的生成方式调用执行神经网络模型。
+计算框架执行深度学习模型训练时，会根据模型结构生成计算图，通过调度计算图完成模型计算。在计算框架中可以生成静态图和动态图两种计算图。静态图对应声明式编程范式，动态图对应命令式编程范式。静态生成可以根据前端语言描述的神经网络拓扑结构以及参数变量等信息构建一份固定的计算图，因此静态图在执行期间可以不依赖前端语言描述常用于神经网络模型的部署，比如移动端人脸识别场景中的应用等。动态图则需要在每一次执行神经网络模型依据前端语言描述动态生成一份临时的计算图，这意味着计算图的动态生成过程灵活可变，该特性有助于我们在神经网络结构调整阶段提高效率。主流计算框架TensorFlow、MindSpore均支持动态图和静态图模式；PyTorch则可以通过工具将构建的动态图神经网络模型转化为静态结构，以获得高效的计算执行效率。了解两种计算图生成方式的优缺点及构建执行特点，可以针对待解决的任务需求，选择合适的生成方式调用执行神经网络模型。

 ### 静态生成

--- a/chapter_distributed_training/collective.md
+++ b/chapter_distributed_training/collective.md
@@ -6,11 +6,11 @@ Centers），以及如何在数据中心中高效实现集合通讯，从而让

 ### 在数据中心的梯度计算

-![数据中心](../img/ch09/ch10-datacentre.pdf)
+![数据中心](../img/ch09/ch10-datacentre.svg)
 :width:`800px`
 :label:`ch10-datacentre`

-:numref:`ch10-datacentre` 描述了一个典型的用于深度学习模型训练的数据中心。数据中心中的训练服务器一般会有多个设备。如需增加服务器，我们会将多个训练服务器放置在一个机柜（Rack）上，同时接入一个架顶交换机（Top
+ :numref:`ch10-datacentre` 描述了一个典型的用于深度学习模型训练的数据中心。数据中心中的训练服务器一般会有多个设备。如需增加服务器，我们会将多个训练服务器放置在一个机柜（Rack）上，同时接入一个架顶交换机（Top
 of Rack
 Switch）来连接多个服务器。当一个机柜满的时候，我们可以通过在架顶交换机之间增加骨干交换机（Spine
 Switch），接入新的机柜。通过这种方式，我们可以在数据中心内不断增加服务器，从而为神经网络的训练提供海量的算力和内存。目前的商用数据中心可能拥有超过一百万台服务器。
@@ -31,7 +31,7 @@ GPU），而在一个服务器内的多个设备可以通过高速机内网络
 为了在数据中心中高效完成梯度平均的操作，我们往往会实现
 Allreduce算法。这个算法诞生的背景是：传统计算平均梯度的方法往往是在集群中找出一个设备来收集本地梯度，计算平均梯度，然后再将平均梯度广播到全部的设备。这种做法易于实现，但是其引入了两个问题。首先，多设备共同给这个聚合设备发送数据的时候，在聚合设备上往往会产生严重的带宽不足和网络拥塞。其次，单设备需要负担大量的梯度平均的计算，而受限于单设备上的有限算力，这种平均计算会受限于算力瓶颈。

-![Allreduce初始状态和终止状态](../img/ch09/ch10-allreduce-state.pdf)
+![Allreduce初始状态和终止状态](../img/ch09/ch10-allreduce-state.svg)
 :width:`800px`
 :label:`ch10-allreduce-state`

@@ -41,12 +41,12 @@ Allreduce算法。这个算法诞生的背景是：传统计算平均梯度的
 = 1 + 2 +
 4）。为了计算平均梯度，每个设备只需要在最后将梯度之和除以设备数量即可（分区1的最终结果为7除以3）。

-![Allreduce算法的过程](../img/ch09/ch10-allreduce-process.pdf)
+![Allreduce算法的过程](../img/ch09/ch10-allreduce-process.svg)
 :width:`800px`
 :label:`ch10-allreduce-process`

 Allreduce算法会把梯度的加和计算拆分成$M-1$个Reduce步骤和$M-1$个Broadcast步骤（其中$M$是节点的数量）。Reduce步骤是为了计算出梯度的和（Summation），Broadcast步骤是为了把梯度之和广播给全部的节点。为了说明这些步骤的执行过程，我们利用
-:numref:`ch10-allreduce-process` 。Allreduce算法由Reduce步骤开始，在第一个Reduce步骤中，Allreduce算法会对全部节点进行配对（Pairing），让他们共同完成梯度相加的操作。在 :numref:`ch10-allreduce-process` 的第一个Reduce步骤中，设备1和设备2进行了配对共同对分区1的数据相加。其中，设备2把本地的梯度数据1发送给设备1，设备将接收到1和本地的分区1内的梯度数据：2进行相加，计算出中间（intermediate）梯度相加的结果：3。于此同时，设备1和设备3进行配对，共同完成对分区3的数据相加。而设备3和设备2进行配对，共同完成对于分区2的数据相加。
+ :numref:`ch10-allreduce-process` 。Allreduce算法由Reduce步骤开始，在第一个Reduce步骤中，Allreduce算法会对全部节点进行配对（Pairing），让他们共同完成梯度相加的操作。在 :numref:`ch10-allreduce-process` 的第一个Reduce步骤中，设备1和设备2进行了配对共同对分区1的数据相加。其中，设备2把本地的梯度数据1发送给设备1，设备将接收到1和本地的分区1内的梯度数据：2进行相加，计算出中间（intermediate）梯度相加的结果：3。于此同时，设备1和设备3进行配对，共同完成对分区3的数据相加。而设备3和设备2进行配对，共同完成对于分区2的数据相加。

 在上述Reduce的步骤中，梯度的计算实现了以下几个特性:

--- a/chapter_distributed_training/methods.md
+++ b/chapter_distributed_training/methods.md
@@ -4,11 +4,11 @@

 ### 概述

-![单节点训练系统](../img/ch09/ch10-single-node.pdf)
+![单节点训练系统](../img/ch09/ch10-single-node.svg)
 :width:`800px`
 :label:`ch10-single-node`

-分布式训练系统的设计目标是：将单节点训练系统转化成**等价的**并行训练系统，从而在不影响模型精度的条件下完成训练过程的加速。一个单节点训练系统往往如:numref:`ch10-single-node`所示。一个训练过程会由多个数据小批次（mini-batch）完成。在图中，一个数据小批次被标示为**数据**。训练系统会利用数据小批次来生成梯度，提升模型精度。这个过程由一个训练**程序**实现。在实际中，这个程序往往实现了一个多层神经网络的执行过程。
+分布式训练系统的设计目标是：将单节点训练系统转化成**等价的**并行训练系统，从而在不影响模型精度的条件下完成训练过程的加速。一个单节点训练系统往往如 :numref:`ch10-single-node`所示。一个训练过程会由多个数据小批次（mini-batch）完成。在图中，一个数据小批次被标示为**数据**。训练系统会利用数据小批次来生成梯度，提升模型精度。这个过程由一个训练**程序**实现。在实际中，这个程序往往实现了一个多层神经网络的执行过程。
 该神经网络的执行由一个计算图（Computational
 Graph）表达。这个图有多个相互连接的算子（Operator），每个算子会拥有计算参数。每个算子往往会实现一个神经网络层（Neural
 Network Layer），而参数则代表了这个层在训练中所更新的的权重（Weights）。
@@ -27,7 +27,7 @@ Network Layer），而参数则代表了这个层在训练中所更新的的权
 | 多程序 | 多程序单数据：模型并行 | 多程序多数据：混合并行 |
 :label:`ch10-parallel-methods`

-给定一个单节点训练系统，人们会对**数据**和**程序**分区（Partition），从而完成并行加速。:numref:`ch10-parallel-methods`总结了不同的切分方法。单节点训练系统可以被归类于
+给定一个单节点训练系统，人们会对**数据**和**程序**分区（Partition），从而完成并行加速。 :numref:`ch10-parallel-methods`总结了不同的切分方法。单节点训练系统可以被归类于
 单程序单数据模式。而假如用户希望使用更多的设备来实现并行计算，他们首先可以选择对数据进行分区，并将同一个程序复制到多个设备上并行执行。这种方式是单程序多数据模式，常被称为**数据并行**（Data
 Parallelism）。另一种并行方式是对程序进行分区：程序的算子会被分发给多个设备按照依次完成。这种模式是
 多程序单数据模式，常被称为**模型并行**（Model
@@ -38,7 +38,7 @@ Parallelism）。

 ### 数据并行

-![数据并行训练系统](../img/ch09/ch10-data-parallel.pdf)
+![数据并行训练系统](../img/ch09/ch10-data-parallel.svg)
 :width:`800px`
 :label:`ch10-data-parallel`

@@ -51,7 +51,7 @@ Communication）的Allreduce操作来完成。

 ### 模型并行

-![模型并行系统：算子内并行](../img/ch09/ch10-model-parallel-intra-op.pdf)
+![模型并行系统：算子内并行](../img/ch09/ch10-model-parallel-intra-op.svg)
 :width:`800px`
 :label:`ch10-model-parallel-intra-op`

@@ -65,7 +65,7 @@ Parallelism）。
 另一种内存不足的场景是：模型的总内存需求超过了单设备的内存容量。在这种场景下，假如我们总共有$N$个算子和$M$个设备，我们可以将算子平摊给这$M$个设备，让每个设备仅需负责$N/M$个算子的前向和反向计算，降低设备的内存开销。这种并行方式是模型并行的另一种应用，被称为**算子间并行**（Inter-operator
 Parallelism）。

-![模型并行系统：算子间并行](../img/ch09/ch10-model-parallel-inter-op.pdf)
+![模型并行系统：算子间并行](../img/ch09/ch10-model-parallel-inter-op.svg)
 :width:`800px`
 :label:`ch10-model-parallel-inter-op`

@@ -73,8 +73,8 @@ Parallelism）。

 ### 混合并行

-![混合并行系统](../img/ch09/ch10-hybrid-parallel.pdf)
+![混合并行系统](../img/ch09/ch10-hybrid-parallel.svg)
 :width:`800px`
 :label:`ch10-hybrid-parallel`

-在训练大型人工智能模型中，我们往往会同时面对算力不足和内存不足。因此，我们需要混合使用数据并行和模型并行，这种方法被称为混合并行。:numref:`ch10-hybrid-parallel`提供了一个由4个设备实现的混合并行的例子。在这个例子中，我们首先实现算子间并行来解决训练程序内存开销过大的问题：该训练程序的算子1和算子2被分摊到了设备1和设备2上。进一步，我们通过数据并行来添加3和设备4，提升系统算力。为了达到这一点，我们对训练数据进行分区（数据分区1和数据分区2），并将模型（算子1和算子2）分配复制到设备3和设备4上生成可以并行执行的程序副本。在前向计算的过程中，设备1和设备3上的算子1副本同时开始，计算结果分别发送（Send）给设备2和设备4完成算子2副本的计算。在反向计算中，设备2和设备4同时开始计算梯度，本地梯度通过Allreduce进行平均。反向计算传递到设备1和设备3上的算子1副本结束。
+在训练大型人工智能模型中，我们往往会同时面对算力不足和内存不足。因此，我们需要混合使用数据并行和模型并行，这种方法被称为混合并行。 :numref:`ch10-hybrid-parallel`提供了一个由4个设备实现的混合并行的例子。在这个例子中，我们首先实现算子间并行来解决训练程序内存开销过大的问题：该训练程序的算子1和算子2被分摊到了设备1和设备2上。进一步，我们通过数据并行来添加3和设备4，提升系统算力。为了达到这一点，我们对训练数据进行分区（数据分区1和数据分区2），并将模型（算子1和算子2）分配复制到设备3和设备4上生成可以并行执行的程序副本。在前向计算的过程中，设备1和设备3上的算子1副本同时开始，计算结果分别发送（Send）给设备2和设备4完成算子2副本的计算。在反向计算中，设备2和设备4同时开始计算梯度，本地梯度通过Allreduce进行平均。反向计算传递到设备1和设备3上的算子1副本结束。
--- a/chapter_distributed_training/overview.md
+++ b/chapter_distributed_training/overview.md
@@ -4,7 +4,7 @@

 接下来，我们详细讨论分布式训练系统的设计动机

-![对比机器学习模型参数量增长和计算硬件的算力增长](../img/ch09/ch10-computation-increase.pdf)
+![对比机器学习模型参数量增长和计算硬件的算力增长](../img/ch09/ch10-computation-increase.svg)
 :width:`800px`
 :label:`ch10-computation-increase`

@@ -12,7 +12,7 @@

 单处理器的算力不足是促使人们设计分布式训练系统的一个主要原因。一个处理器的算力可以用**每秒钟浮点数操作**（Floating
 Point Operations Per Second，FLOPS）来衡量。
-如:numref:`ch10-computation-increase`所示，根据摩尔定律（Moore's
+如 :numref:`ch10-computation-increase`所示，根据摩尔定律（Moore's
 Law），中央处理器的算力每18个月增长2倍。虽然计算加速卡，如GPU和Tensor
 Processing
 Unit（TPU），针对机器学习计算（如矩阵相乘）提供了大量的算力。这些加速卡的发展最终也受限于摩尔定律，增长速度也停留在每18个月2倍。而与此同时，机器学习模型正在快速发展。短短数年，我们从仅能识别有限物体的AlexNet模型，一路发展到在复杂任务中打败人类的AlphaStar。这期间，模型对于算力需求每18个月增长了35倍。解决处理器性能和算力需求之间的鸿沟
@@ -29,12 +29,12 @@ A100）仅能提供最高80GB的内存。单卡内存空间的增长受到硬件

 受限于单节点的有限算力，内存和存储资源，人们把关注投向了日益成熟的云计算数据中心。一个数据中心管理着数十万个计算服务器。随着数据中心的全球部署，人们可以很方便地获得数百个服务器。这些服务器可以通过分布式训练系统来协调和管理，解决训练大型机器学习模型过程遇到的算力，内存和存储不足，从而完成训练过程的加速。

-![单节点计算和多节点分布式计算](../img/ch09/ch10-single-vs-multi.pdf)
+![单节点计算和多节点分布式计算](../img/ch09/ch10-single-vs-multi.svg)
 :width:`800px`
 :label:`ch10-single-vs-multi`

 在设计分布式训练系统的过程中，我们需要找出有资源瓶颈的计算任务，根据计算任务的特点，将其拆分成多个子任务，然后将子任务分发给多个节点（可以是服务器，机器，或者是加速卡）并行完成。
-:numref:`ch10-single-vs-multi`描述了如何将单节点执行转换为分布式执行的一般过程。在机器学习系统中，一个计算任务往往会有一组数据（例如训练样本）或者任务（例如算子）作为输入，利用一个计算节点（例如GPU）生成一组输出（例如梯度）。假如单节点成为瓶颈，我们可以利用分布式计算进行加速。分布式执行一般具有三个步骤：第一步，我们需要将输入进行**切分**。第二步，每个输入部分会分发给不同的计算节点，实现**并行**计算。第三步，每个计算节点的输出，进一步**合并**，最终得到和单节点等价的计算结果。这种切分-并行-合并的模式，本质上实现了分而治之算法（Divide-and-Conquer
+ :numref:`ch10-single-vs-multi`描述了如何将单节点执行转换为分布式执行的一般过程。在机器学习系统中，一个计算任务往往会有一组数据（例如训练样本）或者任务（例如算子）作为输入，利用一个计算节点（例如GPU）生成一组输出（例如梯度）。假如单节点成为瓶颈，我们可以利用分布式计算进行加速。分布式执行一般具有三个步骤：第一步，我们需要将输入进行**切分**。第二步，每个输入部分会分发给不同的计算节点，实现**并行**计算。第三步，每个计算节点的输出，进一步**合并**，最终得到和单节点等价的计算结果。这种切分-并行-合并的模式，本质上实现了分而治之算法（Divide-and-Conquer
 Algorithm）的设计思想：由于每个计算节点只需要负责更小的子任务，因此其可以更快速的完成计算，最终形成对整个计算过程的加速。

 ### 用户益处
--- a/chapter_distributed_training/parameter_servers.md
+++ b/chapter_distributed_training/parameter_servers.md
@@ -13,11 +13,11 @@ Perception），训练这种神经网络只需要几个GPU即可。而另一方
 feature）和产品特征（Item
 feature）构成。这些特征往往是大型向量（Vector）。现代推荐系统需要服务数亿的用户，推荐数以千万的商品。假设用户的特征是1MB，而系统需要服务10亿的用户，那么用户的嵌入表就会有1PB的大小。而这个大小远远超过了一个深度学习服务器所具有的内存。假如我们部署大量的昂贵的深度学习服务器来存储海量嵌入表，那么这些服务器上的加速卡的使用率将会极低，无法实现对于硬件的高效利用。

-![参数服务器](../img/ch09/ch10-parameter-servers.pdf)
+![参数服务器](../img/ch09/ch10-parameter-servers.svg)
 :width:`800px`
 :label:`ch10-parameter-servers`

-为了解决上述问题，人们往往会在稀疏模型集群中混合部署：训练服务器和参数服务器，从而实现对于计算需求和内存需求分别满足。:numref:`ch10-parameter-servers` 描述了带有参数服务器的机器学习集群。这个集群中含有2个训练服务器和2个参数服务器，训练服务器一般是拥有加速卡的计算优化服务器（Compute-optimised
+为了解决上述问题，人们往往会在稀疏模型集群中混合部署：训练服务器和参数服务器，从而实现对于计算需求和内存需求分别满足。 :numref:`ch10-parameter-servers` 描述了带有参数服务器的机器学习集群。这个集群中含有2个训练服务器和2个参数服务器，训练服务器一般是拥有加速卡的计算优化服务器（Compute-optimised
 server）。而参数服务器一般是内存优化服务器（Memory-optimised
 server），其的内存大小一般远远大于计算优化服务器。在一个稀疏模型中往往拥有神经网络参数和嵌入表参数。神经网络较小，其可以存储在训练服务器内存中。而嵌入表很大，因此需要存储在额外的参数服务器中。参数服务器一般会按照键-值对（Key-value
 pairs）的方式来存储参数。常用的键包括用户名（User ID），产品名（Item
--- a/chapter_distributed_training/pipeline.md
+++ b/chapter_distributed_training/pipeline.md
@@ -5,7 +5,7 @@
 然而，当这类系统的运行中，计算图中的下游设备需要长期持续处于空闲状态，等待上游设备的计算完成，才可以开始计算，这极大降低了设备的平均使用率。这种现象被称为模型并行空洞（Model
 Parallelism Bubble）。

-![流水线并行系统。注意！图的F和B任务的编号需要更新！](../img/ch09/ch10-pipeline-parallel.pdf)
+![流水线并行系统。注意！图的F和B任务的编号需要更新！](../img/ch09/ch10-pipeline-parallel.svg)
 :width:`800px`
 :label:`ch10-pipeline-parallel`

@@ -14,7 +14,7 @@ Mini-batch）划分为多个微批量（Micro-batch）。假设一个数据小
 propagation）和反向传播（Backwards
 propagation），计算出梯度。每个微批量对应的梯度将会缓存，等到全部微批量完成，缓存的梯度会被加和，算出平均梯度，更新模型参数。

-:numref:`ch10-pipeline-parallel` 进一步给出了一个流水线并行的执行例子。在本例中，模型参数需要切分给4个设备存储。为了充分利用起来这4个设备，我们将小批量切分为2个微批量。当设备1完成第一个微批量的前向传播后（表示为$F_{0,0}$）后，他会将中间结果发送给设备2，触发响应的前向传播任务（表示为$F_{1,0}$）。与此同时，设备1也可以开始第二个微批量的前向传播任务（表示为$F_{0,1}$）。前向传播会在流水线的最后一个设备--设备3--完成。系统于是开始反向传播。设备4开始第1个微批量的反向传播任务（表示为$B_{3,0}$）。该任务完成后的中间结果会被发送给设备3，触发响应的反向传播任务（表示为$B_{2,0}$）。与此同时，设备4会缓存好对应第1个微批量的梯度，接下来开始第2个微批量计算（表示为$B_{3,1}$）。当设备4完成了全部的反向传播计算后，他会将本地缓存的梯度进行相加，并且除以微批量数量，计算出平均梯度，该梯度用于更新模型参数。
+ :numref:`ch10-pipeline-parallel` 进一步给出了一个流水线并行的执行例子。在本例中，模型参数需要切分给4个设备存储。为了充分利用起来这4个设备，我们将小批量切分为2个微批量。当设备1完成第一个微批量的前向传播后（表示为$F_{0,0}$）后，他会将中间结果发送给设备2，触发响应的前向传播任务（表示为$F_{1,0}$）。与此同时，设备1也可以开始第二个微批量的前向传播任务（表示为$F_{0,1}$）。前向传播会在流水线的最后一个设备--设备3--完成。系统于是开始反向传播。设备4开始第1个微批量的反向传播任务（表示为$B_{3,0}$）。该任务完成后的中间结果会被发送给设备3，触发响应的反向传播任务（表示为$B_{2,0}$）。与此同时，设备4会缓存好对应第1个微批量的梯度，接下来开始第2个微批量计算（表示为$B_{3,1}$）。当设备4完成了全部的反向传播计算后，他会将本地缓存的梯度进行相加，并且除以微批量数量，计算出平均梯度，该梯度用于更新模型参数。

 流水线并行的关键因素是流水线泡沫（Bubble）。当设备完成前向传播后，必须等到全部反向传播开发，在此期间设备会处于空闲状态。在 :numref:`ch10-pipeline-parallel` 中，我们可以看到设备1在完成2个前向传播任务后，要等很多时间才能开始2个传向传播任务。这其中的等待时间即被称为泡沫。为了减少设备的等待时间，一种常见的做法是尽可能的增加微批量的数量，从而让反向传播尽可能早的开始。然而，使用非常小的微批量大小，可能会造成加速器无法被充分利用。因此最优的微批量大小是多种因素的折中。其中最核心的因素是流水线泡沫的大小和加速器的计算能力。

--- a/chapter_explainable_AI/explainable_ai.md
+++ b/chapter_explainable_AI/explainable_ai.md
@@ -1,8 +1,5 @@
-可解释机器学习系统
-============================

-背景
----
+## 背景

 在人类历史上，技术进步、生产关系逻辑、伦理法规的发展是动态演进的。当一种新的技术在实验室获得突破后，其引发的价值产生方式的变化会依次对商品形态、生产关系等带来冲击，而同时当新技术带来的价值提升得到认可后，商业逻辑的组织形态在自发的调整过程中，也会对技术发展的路径、内容甚至速度提出诉求，并当诉求得到满足时适配以新型的伦理法规。在这样的相互作用中，技术系统与社会体系会共振完成演进，是谓技术革命。

@@ -26,8 +23,7 @@

 因此，从商业推广层面以及从法规层面，我们都需要打开黑盒模型，对模型进行解释，可解释AI正是解决该类问题的技术。

-可解释AI定义
------------
+## 可解释AI定义

 按DARPA（美国国防部先进研究项目局）的描述，如 :numref:`xai_concept`所示，
 可解释AI的概念在于区别于现有的AI系统，可解释AI系统可以解决用户面对模型黑盒遇到的问题，是的用户知其然并知其所以然。
@@ -50,8 +46,7 @@ AI(XAI))，不论是学术界还是工业界都没有一个统一的定义。这

 我们根据自身的实践经验和理解，将可解释AI定义为：一套面向机器学习（主要是深度神经网络）的技术合集，包括可视化、数据挖掘、逻辑推理、知识图谱等，目的是通过此技术合集，使深度神经网络呈现一定的可理解性，以满足相关使用者对模型及应用服务产生的信息诉求（如因果或背景信息），从而为使用者对人工智能服务建立认知层面的信任。

-可解释AI算法现状介绍
--------------------
+## 可解释AI算法现状介绍

 随着可解释AI概念的提出，可解释AI越来越受到学术界及工业界的关注，下图展示了人工智能领域顶级学术会议中可解释AI关键字的趋势。为了供读者更好的对现有可解释AI算法有一个整体认知，我们这里参考[@2020tkde_li]总结归纳了可解释AI的算法类型，如图\[fig:ch011/xai\_methods\]
 所示。
@@ -166,8 +161,7 @@ MOBA.）[]{data-label="fig:ch011/xai_steam"}](../img/ch11/xai_kg_recommendataion
 Fortress 2”推荐给用户的解释可以翻译成固定话术：“Team Fortress
 2”是游戏公司“Valve”开发的一款动作类、多人在线、射击类“action”电子游戏。这与用户历史玩过的游戏“Half-Life”有高度关联。

-未来可解释AI
------------
+## 未来可解释AI

 为了进一步推动这个方向，我们在此总结了一些值得注意的前景：

--- a/chapter_federated_learning/challenge.md
+++ b/chapter_federated_learning/challenge.md
@@ -0,0 +1,30 @@
+## 实际部署时的挑战
+
+### 部署流程
+
+随着端上算力的提升和算法的快速发展，跨设备联邦学习在智能手机或智能设备中的应用越来越广泛。其主要流程如 :numfef:`ch10-federated-learning-flow`所示，可分为如下几步：
+
+1. FL-Client选择：FL-Client主动向FL-Server发起参与联邦学习的请求。FL-Server根据配置筛选出满足条件的FL-Client。选择成功后FL-Server下发模型以及相关联邦学习配置。
+2. FL-Client训练：在终端设备上进行本地模型训练。
+3. 联邦聚合：FL-Client上传模型权重到FL-Server，并由FL-Server选择聚合算法进行计算，并对某些数据进行持久化存储。
+4. FL-Client模型更新：终端向FL-Server查询联邦聚合后的模型等数据。
+5. 重复1-4步，完成联邦学习任务。
+
+![跨设备联邦学习流程图](../img/ch10/ch10-federated-learning-flow.png)
+
+:label:`ch10-federated-learning-flow`
+
+### 部署挑战
+
+然而，由于跨设备联邦学习的特殊性，其挑战主要包括：
+
+1. 跨设备联邦学习的FL-Client的网络连接常常是不稳定的，在任何时候都只有部分FL-Client可用。
+
+2. 跨设备联邦学习往往是大规模并行的场景，通信时间会成为瓶颈。当千万级FL-Client同时进行联邦学习请求时，需要大量网络带宽的支持，并且单台FL-Server必然承接不住庞大的数据和参数。
+
+### 解决方案
+
+为了解决跨设备联邦学习带来的挑战，MindSpore Federated Learning给出两个解决方案：
+
+1. 限时通信：在FL-Server和FL-Client建立连接后，启动全局的计时器和计数器。当预先设定的时间窗口内的FL-Server接收到FL-Client训练后的模型参数满足初始接入的所有FL-Client的一定比例后，就可以进行聚合。若时间窗内没有达到比例阈值，则进入下一轮迭代。保证即使有海量FL-Client接入的情况下，也不会由于个别FL-Client训练时间过长或掉线导致的整个联邦学习过程卡死。
+2. 松耦合组网：使用FL-Server集群。每个FL-Server接收和下发权重给部分FL-Client，减少单个FL-Server的带宽压力。此外，支持FL-Client以松散的方式接入。任意FL-Client的中途退出都不会影响全局任务，并且FL-Client在任意时刻访问任意FL-Server都能获得训练所需的全量数据。
--- a/chapter_federated_learning/fedavg.md
+++ b/chapter_federated_learning/fedavg.md
@@ -0,0 +1,8 @@
+## 联邦平均算法
+
+和传统分布式学习相比，联邦学习存在训练结点不稳定和通信代价大的难点。这些难点导致了联邦学习无法和传统分布式学习一样：在每次单步训练之后，同步不同训练结点上的权重。为了提高计算通信比并降低频繁通信带来的高能耗，谷歌公司提出了联邦平均算法（Federated Averaging，FedAvg）。 :numfef:`ch10-federated-learning-fedavg`展示了FedAvg的整体流程。在每轮联邦训练过程中，端侧进行多次单步训练。然后云侧聚合多个端侧权重，并取加权平均。
+
+![联邦平均算法](../img/ch10/ch10-federated-learning-fedavg.png)
+
+:label:`ch10-federated-learning-fedavg`
+
--- a/chapter_federated_learning/index.md
+++ b/chapter_federated_learning/index.md
@@ -1 +1,19 @@
-# 联邦学习系统
+# 联邦学习系统
+
+在本章中，我们介绍深度学习的一个重要分支——联邦学习及其在系统方面的知识。本章的学习目标包括：
+
+- 掌握联邦学习基本知识。
+- 掌握联邦系统架构组成部分。
+- 掌握联邦平均和隐私加密算法。
+- 了解联邦学习实际部署时的挑战。
+
+```toc
+:maxdepth: 2
+
+overview
+system_architecture
+fedavg
+privacy_encryption_algorithm
+challenge
+summary
+```
--- a/chapter_federated_learning/overview.md
+++ b/chapter_federated_learning/overview.md
@@ -0,0 +1,11 @@
+## 概述
+
+为了解决“隐私保护”与“数据孤岛”这两大难题，联邦学习（Federated Learning，FL）应运而生。联邦学习的概念最早在2016年被提了出来，能有效帮助多个机构在满足用户隐私保护、数据安全和政府法规的要求下，进行数据使用和机器学习建模。在联邦学习场景中，每个用户，被定义为客户端这一角色，使用各自的本地数据进行训练、模型更新、权重上传，并在中央服务器的协调下，多个客户端协作建立机器学习模型。
+
+根据数据分布的不同，联邦学习可以分为跨设备（cross-device）与跨组织（cross-silo）联邦学习。一般而言，跨组织联邦学习的用户一般是企业、机构单位级别的，而跨设备联邦学习针对的则是便携式电子设备、移动端设备等。 :numref:`ch10-federated-learning-different-connection`展示了两者的区别和联系：
+
+![跨设备和跨组织联邦学习的区别和联系](../img/ch10/ch10-federated-learning-different-connection.png)
+
+:label:`ch10-federated-learning-different-connection`
+
+下面就联邦学习中的要点：系统架构、联邦平均算法、隐私加密算法以及实际部署时的挑战进行详细描述。
--- a/chapter_federated_learning/privacy_encryption_algorithm.md
+++ b/chapter_federated_learning/privacy_encryption_algorithm.md
@@ -0,0 +1,52 @@
+## 隐私加密算法
+
+联邦学习过程中，用户数据仅用于本地设备训练，不需要上传至中央FL-Server。这样可以避免用户个人数据的直接泄露。然而联邦学习框架中，模型的权重以明文形式上云仍然存在间接泄露用户隐私的风险。敌手获取到用户上传的明文权重后，可以通过重构、模型逆向等攻击恢复用户的个人训练数据，导致用户隐私泄露。
+
+MindSpore Federated Learning框架，提供了基于本地差分隐私（LDP）和基于多方安全计算（MPC）的安全聚合算法，在本地模型的权重上云前对其进行加噪或加扰。在保证模型可用性的前提下，解决联邦学习中的隐私泄露问题。
+
+### 基于LDP的安全聚合
+
+差分隐私（differential privacy）是一种保护用户数据隐私的机制。差分隐私定义为：
+
+$$
+Pr[\mathcal{K}(D)\in S] \le e^{\epsilon} Pr[\mathcal{K}(D’) \in S]+\delta
+$$
+
+对于两个差别只有一条记录的数据集$D, D’$，通过随机算法$\mathcal{K}$，输出结果为集合$S$子集的概率满足上面公式。$\epsilon$为差分隐私预算，$\delta$扰动，$\epsilon$和$\delta$越小，说明$\mathcal{K}$在$D$和$D’$上输出的数据分布越接近。
+
+在联邦学习中，假设FL-Client本地训练之后的模型权重矩阵是$W$，由于模型在训练过程中会“记住”训练集的特征，所以敌手可以借助$W$还原出用户的训练数据集。
+
+MindSpore Federated Learning提供基于本地差分隐私的安全聚合算法，防止本地模型的权重上云时泄露隐私数据。
+
+FL-Client会生成一个与本地模型权重$W$相同维度的差分噪声矩阵$G$，然后将二者相加，得到一个满足差分隐私定义的权重$W_p$:
+
+$$
+W_p=W+G
+$$
+
+FL-Client将加噪后的模型权重$W_p$上传至云侧FL-Server进行联邦聚合。噪声矩阵$G$相当于给原模型加上了一层掩码，在降低模型泄露敏感数据风险的同时，也会影响模型训练的收敛性。如何在模型隐私性和可用性之间取得更好的平衡，仍然是一个值得研究的问题。实验表明，当参与方的数量$n$足够大时（一般指1000以上），大部分噪声能够相互抵消，本地差分机制对聚合模型的精度和收敛性没有明显影响。
+
+### 基于MPC的安全聚合
+
+尽管差分隐私技术可以适当保护用户数据隐私，但是当参与FL-Client数量比较少或者高斯噪声幅值较大时，模型精度会受较大影响。为了同时满足模型保护和模型收敛这两个要求，MindSpore Federated Learning提供了基于MPC的安全聚合方案。
+
+在这种训练模式下，假设参与的FL-Client集合为$U$，对于任意FL-Client $u$和$v$，
+它们会两两协商出一对随机扰动$p_{uv}$、$p_{vu}$，满足
+
+$$
+\label{puv}
+    p_{uv}=
+    \begin{cases}
+    -p_{vu}, &u{\neq}v\\
+    0, &u=v
+    \end{cases}
+$$
+于是每个FL-Client $u$ 在上传模型权重至FL-Server前，会在原模型权重$x_u$加上它与其它用户协商的扰动：
+
+$$
+x_{encrypt}=x_u+\sum\limits_{v{\in}U}p_{uv}
+$$
+
+从而FL-Server聚合结果$\overline{x}$为：
+
+上面的过程只是介绍了聚合算法的主要思想，基于MPC的聚合方案是精度无损的，代价是通讯轮次的增加。
--- a/chapter_federated_learning/summary.md
+++ b/chapter_federated_learning/summary.md
@@ -0,0 +1,3 @@
+## 小结
+
+在这一章，我们简单介绍了联邦学习的背景、系统架构、联邦平均算法、隐私加密算法以及实际部署时的挑战。联邦学习是一个新起步的人工智能算法，可以在“数据保护”与“数据孤岛”这两大约束条件下，建立有效的机器学习模型。此外，由于联邦学习场景的特殊性（端侧数据不上传、安全隐私要求高和数据非独立同分布等特点），使得系统和算法的开发难度更高：如何平衡计算和通讯的开销？如何保证模型不会泄露隐私？算法如何在非独立同分布场景下收敛？等等，都需要开发人员对实际的联邦学习场景有更深刻的认识。
--- a/chapter_federated_learning/system_architecture.md
+++ b/chapter_federated_learning/system_architecture.md
@@ -0,0 +1,20 @@
+## 系统架构
+
+联邦学习系统由调度器模块、服务器模块和客户端模块三个部分组成，其系统架构如 :numref:`ch10-federated-learning-architecture`所示。其中：
+
+- 联邦学习调度器：
+
+  联邦学习调度器（FL-Scheduler）协助集群组网，并负责管理面任务的下发。
+
+-  联邦学习服务器：
+
+  联邦学习服务器（FL-Server）提供客户端选择、限时通信、分布式联邦聚合功能。FL-Server需要具备支持端云千万台设备的能力以及边缘服务器的接入和安全处理的逻辑。
+
+- 联邦学习客户端：
+
+  联邦学习客户端（FL-Client）负责本地数据训练，并在和FL-Server进行通信时，对上传权重进行安全加密。
+
+![联邦学习系统架构图](../img/ch10/ch10-federated-learning-architecture.png)
+
+:label:`ch10-federated-learning-architecture`
+
--- a/chapter_frontend_and_ir/ad.md
+++ b/chapter_frontend_and_ir/ad.md
@@ -6,16 +6,16 @@
 ### 自动微分的基本概念

 自动微分（Automatic
-Differentiation，AD）是一种对计算机程序进行高效且准确求导的技术，在上个世纪六七十年代就已经被广泛应用于流体力学、天文学、数学金融等领域（[@10.5555/1455489]）。时至今日，自动微分的实现及其理论仍然是一个活跃的研究领域。随着近些年深度学习在越来越多的机器学习任务上取得领先成果（[@lecun2015deep]），自动微分被广泛的应用于机器学习领域。许多机器学习模型使用的优化算法都需要获取模型的导数，因此自动微分技术成为了一些热门的机器学习框架（例如TensorFlow和PyTorch）的核心特性。
+Differentiation，AD）是一种对计算机程序进行高效且准确求导的技术，在上个世纪六七十年代就已经被广泛应用于流体力学、天文学、数学金融等领域 :cite:`10.5555/1455489`。时至今日，自动微分的实现及其理论仍然是一个活跃的研究领域。随着近些年深度学习在越来越多的机器学习任务上取得领先成果，自动微分被广泛的应用于机器学习领域。许多机器学习模型使用的优化算法都需要获取模型的导数，因此自动微分技术成为了一些热门的机器学习框架（例如TensorFlow和PyTorch）的核心特性。

-常见的计算机程序求导的方法可以归纳为以下四种（[@2015Automatic]）：手工微分（Manual
+常见的计算机程序求导的方法可以归纳为以下四种 :cite:`2015Automatic`：手工微分（Manual
 Differentiation）、数值微分（Numerical
 Differentiation）、符号微分（Symbolic
 Differentiation）和自动微分（Automatic Differentiation）。

 （1）手工微分：需手工求解函数导数的表达式，并在程序运行时根据输入的数值直接计算结果。手工微分需根据函数的变化重新推导表达式，工作量大且容易出错。

-（2）数值微分（[@2015Numerical]）：数值微分通过差分近似方法完成，其本质是根据导数的定义推导而来。
+（2）数值微分 :cite:`2015Numerical`：数值微分通过差分近似方法完成，其本质是根据导数的定义推导而来。

 $$f^{'}(x)=\lim_{h \to 0}\frac{f(x+h)-f(x)}{h}$$

@@ -24,12 +24,12 @@ error）。理论上，数值微分中的截断误差与步长$h$有关，$h$越
 Error）。舍入误差会随着$h$变小而逐渐增大。当h较大时，截断误差占主导。而当h较小时，舍入误差占主导。
 在截断误差和舍入误差的共同作用下，数值微分的精度将会在某一个$h$值处达到最小值，并不会无限的减小。因此，虽然数值微分容易实现，但是存在精度误差问题。

-（3）符号微分（[@2003Computer]）：利用计算机程序自动地通过如下的数学规则对函数表达式进行递归变换来完成求导。
+（3）符号微分 :cite:`2003Computer`：利用计算机程序自动地通过如下的数学规则对函数表达式进行递归变换来完成求导。
 $$\frac{d}{dx}(f(x)+g(x))\rightsquigarrow\frac{d}{dx}f(x)+\frac{d}{dx}g(x)$$

 $$\frac{d}{dx}(f(x)g(x))\rightsquigarrow(\frac{d}{dx}f(x))g(x)+f(x)(\frac{d}{dx}g(x))$$
 符号微分常被应用于现代代数系统工具中，例如Mathematica、Maxima和Maple，以及机器学习框架，如Theano。符号微分虽然消除了手工微分硬编码的缺陷。但因为对表达式进行严格的递归变换和展开，不复用产生的变换结果，很容易产生表达式膨胀（expression
-swell（[@10.5555/60181.60188]））问题。如图:numref:`symbolic_differentiation`所示，用符号微分计算递归表达式$l_{n+1}=4l_n(1-l_n)$，$l_1=x$的导数表达式，其结果随着迭代次数增加快速膨胀。
+swell  :cite:`10.5555/60181.60188` ）问题。如 :numref:`symbolic_differentiation` 所示，用符号微分计算递归表达式$l_{n+1}=4l_n(1-l_n)$，$l_1=x$的导数表达式，其结果随着迭代次数增加快速膨胀。

 ![符号微分的表达式膨胀问题](../img/ch04/符号微分的表达式膨胀问题.png)
 :width:`800px`
@@ -37,7 +37,7 @@ swell（[@10.5555/60181.60188]））问题。如图:numref:`symbolic_differentia

 并且符号微分需要表达式被定义成闭合式的（closed-form），不能带有或者严格限制控制流的语句表达，使用符号微分会很大程度上地限制了机器学习框架网络的设计与表达。

-（4）自动微分（[@2000An]）：自动微分的思想是将计算机程序中的运算操作分解为一个有限的基本操作集合，且集合中基本操作的求导规则均为已知，在完成每一个基本操作的求导后，使用链式法则将结果组合得到整体程序的求导结果。自动微分是一种介于数值微分和符号微分之间的求导方法，结合了数值微分和符号微分的思想。相比于数值微分，自动微分可以精确地计算函数的导数；相比符号微分，自动微分将程序分解为基本表达式的组合，仅对基本表达式应用符号微分规则，并复用每一个基本表达式的求导结果，从而避免了符号微分中的表达式膨胀问题。而且自动微分可以处理分支、循环和递归等控制流语句。目前的深度学习框架基本都采用自动微分机制进行求导运算，下面我们将重点介绍自动微分机制以及自动微分的实现。
+（4）自动微分 :cite:`2000An`：自动微分的思想是将计算机程序中的运算操作分解为一个有限的基本操作集合，且集合中基本操作的求导规则均为已知，在完成每一个基本操作的求导后，使用链式法则将结果组合得到整体程序的求导结果。自动微分是一种介于数值微分和符号微分之间的求导方法，结合了数值微分和符号微分的思想。相比于数值微分，自动微分可以精确地计算函数的导数；相比符号微分，自动微分将程序分解为基本表达式的组合，仅对基本表达式应用符号微分规则，并复用每一个基本表达式的求导结果，从而避免了符号微分中的表达式膨胀问题。而且自动微分可以处理分支、循环和递归等控制流语句。目前的深度学习框架基本都采用自动微分机制进行求导运算，下面我们将重点介绍自动微分机制以及自动微分的实现。

 ### 前向与反向自动微分

@@ -52,7 +52,7 @@ $$\frac{dy}{dx}=(((\frac{dy}{da}\frac{da}{db})\frac{db}{dc})\frac{dc}{dx})$$
 我们以下面的函数为例介绍两种模式的计算方式，我们希望计算函数在$(x_1, x_2)=(2,5)$处的导数$\frac{\partial y}{\partial x_1}$：
 $$y=f(x_1,x_2)=ln(x_1)+{x_1}{x_2}-sin(x_2)$$

-该函数对应的计算图如:numref:`example_compute_graph`：
+该函数对应的计算图如 :numref:`example_compute_graph`：

 ![示例计算图](../img/ch04/自动微分-示例计算图.svg)
 :width:`800px`
@@ -64,7 +64,7 @@ $$y=f(x_1,x_2)=ln(x_1)+{x_1}{x_2}-sin(x_2)$$
 :width:`800px`
 :label:`forward_AD`

-前向模式的计算过程如图:numref:`forward_AD`所示，左侧是源程序分解后得到的基本操作集合，右侧展示了运用链式法则和已知的求导规则，从上至下计算每一个中间变量${\dot{v}_i}=\frac{\partial v_i}{\partial x_1}$，从而计算出最后的变量${\dot{v}_5}=\frac{\partial y}{\partial x_1}$。
+前向模式的计算过程如 :numref:`forward_AD`所示，左侧是源程序分解后得到的基本操作集合，右侧展示了运用链式法则和已知的求导规则，从上至下计算每一个中间变量${\dot{v}_i}=\frac{\partial v_i}{\partial x_1}$，从而计算出最后的变量${\dot{v}_5}=\frac{\partial y}{\partial x_1}$。

 当我们想要对一个函数求导时，我们想要得到的是该函数的任意一个输出对任意一个输入的偏微分的集合。对于一个带有$n$个独立输入$x_i$和$m$个独立输出$y_i$的函数$f:{\mathbf{R}^n}\to \mathbf{R}^m$，该函数的求导结果可以构成如下的雅克比矩阵（Jacobian
 Matrix）： $$\mathbf{J}_{f}=
@@ -101,7 +101,7 @@ $$\mathbf{J}_{f}\mathbf{r}=
 :width:`800px`
 :label:`backward_AD`

-反向模式的计算过程如上图:numref:`backward_AD`所示，左侧是源程序分解后得到的基本操作集合，右侧展示了运用链式法则和已知的求导规则，从$\bar{v}_5=\bar{y}=\frac{\partial y}{\partial y}=1$开始，
+反向模式的计算过程如上 :numref:`backward_AD`所示，左侧是源程序分解后得到的基本操作集合，右侧展示了运用链式法则和已知的求导规则，从$\bar{v}_5=\bar{y}=\frac{\partial y}{\partial y}=1$开始，
 由下至上地计算每一个中间变量${\bar{v}_i}=\frac{\partial y_j}{\partial v_i}$，从而计算出最后的变量${\bar{x}_1}=\frac{\partial y}{\partial x_1}$和${\bar{x}_2}=\frac{\partial y}{\partial x_2}$。

 反向模式每次计算的是函数$f$的某一个输出对任一输入的偏微分，也就是雅克比矩阵的某一行，如下面的向量所示。因此通过运行m次反向模式自动微分，我们就可以得到整个雅克比矩阵。
@@ -125,11 +125,11 @@ $$\mathbf{r}^{T}\mathbf{J}_{f}=

 但是反向模式也存在一定的缺陷。在源程序分解为一系列基本操作后，前向模式由于求导顺序与基本操作的执行顺序一致，输入值可以在执行基本操作的过程中同步获得。而在反向模式中，由于求导顺序与源程序的执行顺序是相反的，计算过程需要分为两个阶段，第一个阶段先执行源程序，且将源程序的中间结果保存起来，在第二阶段才把中间结果取出来去计算导数。因此反向模式会有额外的内存消耗。业界也一直在研究反向模式的内存占用优化方法，例如检查点策略（checkpointing
 strategies）和数据流分析（data-flow
-analysis）（[@2006The];[@2017Divide]）。
+analysis） :cite:`2006The,2017Divide` 。

 ### 自动微分的实现

-上一节我们介绍了自动微分的基本概念，可以总结为将程序分解为一系列微分规则已知的基本操作，然后运用链式法则将它们的微分结果组合起来得到程序的微分结果。而在机器学习的应用中，因为输入的数量远远大于输出的数量，所以反向模式的自动微分更受青睐。虽然自动微分的基本思想是明确的，但是具体的实现方法也分为几类（[@2015Automatic]），大体可以划分为基本表达式法（Elemental
+上一节我们介绍了自动微分的基本概念，可以总结为将程序分解为一系列微分规则已知的基本操作，然后运用链式法则将它们的微分结果组合起来得到程序的微分结果。而在机器学习的应用中，因为输入的数量远远大于输出的数量，所以反向模式的自动微分更受青睐。虽然自动微分的基本思想是明确的，但是具体的实现方法也分为几类 :cite:`2015Automatic` ，大体可以划分为基本表达式法（Elemental
 Libraries）、操作符重载法（Operator
 Overloading，OO）和代码变换法（Source Code Transformation，ST）。

@@ -201,7 +201,7 @@ OO对程序的运行跟踪经过了函数调用和控制流，因此实现起来
 （3）代码变换法（Source
 Transformation，ST）：提供对编程语言的扩展，分析程序的源码或抽象语法树（AST），将程序自动地分解为一系列可微分的基本操作，而这些基本操作的微分规则已预定义好，最后使用链式法则对基本操作的微分表达式进行组合生成新的程序表达来完成微分。TensorFlow，MindSpore等机器学习框架都采用了该方式。

-不同于OO在编程语言内部操作，ST需要语法分析器（parser）和操作中间表示的工具。除此以外，ST需要定义对函数调用和控制流语句（如循环和条件等）的转换规则。其优势在于对每一个程序，自动微分的转换只做一次，因此不会造成运行时的额外性能损耗。而且，因为整个微分程序在编译时就能获得，编译器可以对微分程序进行进一步的编译优化。但ST实现起来更加复杂，需要扩展语言的预处理器、编译器或解释器，且需要支持更多的数据类型和操作，需要更强的类型检查系统。另外，虽然ST不需要在运行时做自动微分的转换，但是对于反向模式，在反向部分执行时，仍然需要确保前向执行的一部分中间变量可以被获取到，有两种方式可以解决该问题（[@van2018Automatic]）：
+不同于OO在编程语言内部操作，ST需要语法分析器（parser）和操作中间表示的工具。除此以外，ST需要定义对函数调用和控制流语句（如循环和条件等）的转换规则。其优势在于对每一个程序，自动微分的转换只做一次，因此不会造成运行时的额外性能损耗。而且，因为整个微分程序在编译时就能获得，编译器可以对微分程序进行进一步的编译优化。但ST实现起来更加复杂，需要扩展语言的预处理器、编译器或解释器，且需要支持更多的数据类型和操作，需要更强的类型检查系统。另外，虽然ST不需要在运行时做自动微分的转换，但是对于反向模式，在反向部分执行时，仍然需要确保前向执行的一部分中间变量可以被获取到，有两种方式可以解决该问题  :cite:`van2018Automatic` ：

 （1）基于Tape的方式。该方式使用一个全局的"tape"去确保中间变量可以被获取到。原始函数被扩展为在前向部分执行时把中间变量写入到tape中的函数，在程序执行反向部分时会从tape中读取这些中间变量。除了存储中间变量外，OO中的tape还会存储执行的操作类型。然而因为tape是一个在运行时构造的数据结构，所以需要添加一些定制化的编译器优化方法。且为了支持高阶微分，对于tape的读写都需要是可微分的。而大多数基于tape的工具都没有实现对tape的读写操作的微分，因此它们都不支持多次嵌套执行反向模式的自动微分（reverse-over-reverse）。机器学习框架Tangent采用了该方式。

--- a/chapter_frontend_and_ir/common_frontend_optimization_pass.md
+++ b/chapter_frontend_and_ir/common_frontend_optimization_pass.md
@@ -19,7 +19,7 @@

 1\. 无用与不可达代码消除

-如图 :numref:`pass_useless_code_elimination`所示。无用代码是指输出结果没有被任何其他代码所使用的代码。不可达代码是指没有有效的控制流路径包含该代码。删除无用或不可达的代码可以使得中间表示更小，提高程序的编译与执行速度。无用与不可达代码一方面有可能来自于程序编写者的编写失误，也有可能是其他编译优化所产生的结果。
+如 :numref:`pass_useless_code_elimination`所示。无用代码是指输出结果没有被任何其他代码所使用的代码。不可达代码是指没有有效的控制流路径包含该代码。删除无用或不可达的代码可以使得中间表示更小，提高程序的编译与执行速度。无用与不可达代码一方面有可能来自于程序编写者的编写失误，也有可能是其他编译优化所产生的结果。

 ![无用代码消除](../img/ch04/编译优化-无用代码消除.svg)
 :width:`600px`
@@ -27,9 +27,9 @@

 2\. 常量传播、常量折叠

-常量传播：如图 :numref:`pass_constant_broadcast`所示，如果某些量为已知值的常量，那么可以在编译时刻将使用这些量的地方进行替换。
+常量传播：如 :numref:`pass_constant_broadcast`所示，如果某些量为已知值的常量，那么可以在编译时刻将使用这些量的地方进行替换。

-常量折叠：如图 :numref:`pass_constant_broadcast`所示，多个量进行计算时，如果能够在编译时刻直接计算出其结果，那么变量将由常量替换。
+常量折叠：如 :numref:`pass_constant_broadcast`所示，多个量进行计算时，如果能够在编译时刻直接计算出其结果，那么变量将由常量替换。

 ![常量传播与常量折叠](../img/ch04/编译优化-常量传播与常量折叠.svg)
 :width:`600px`
@@ -37,7 +37,7 @@

 3\. 公共子表达式消除

-如图 :numref:`pass_CSE`所示，如果一个表达式E已经计算过了，并且从先前的计算到现在E中所有变量的值都没有发生变化，那么E就成为了公共子表达式。对于这种表达式，没有必要花时间再对它进行计算，只需要直接用前面计算过的表达式结果代替E就可以了。
+如 :numref:`pass_CSE`所示，如果一个表达式E已经计算过了，并且从先前的计算到现在E中所有变量的值都没有发生变化，那么E就成为了公共子表达式。对于这种表达式，没有必要花时间再对它进行计算，只需要直接用前面计算过的表达式结果代替E就可以了。

 ![公共子表达式消除](../img/ch04/编译优化-公共子表达式消除.svg)
 :width:`600px`
--- a/chapter_frontend_and_ir/index.md
+++ b/chapter_frontend_and_ir/index.md
@@ -2,7 +2,7 @@
 在上一章节中，我们详细讨论了计算图的生成和调度，在进阶部分的介绍中简单介绍了深度学习编译器的作用。定义深度学习模型、计算图使用系统为用户提供的高级编程API，我们将用户使用高级编程API编写的程序称为源程序，将与硬件相关的程序称为目标程序，深度学习编译器需要理解输入的源程序并将其映射到目标机。为了实现这两项任务，编译器的设计被分解为两个主要部分：前端和后端。传统编译器的前端专注于理解源程序，后端则专注于将功能映射到目标机。为了将前后端相连接，我们需要一种结构来表示转换后的源代码，这就是中间表示（Intermediate
 Representation, IR）。

-图 :numref:`compiler_frontend_structure`展示了机器学习编译器的前端的流程。其中，对源程序的解析过程与传统编译器是大致相同的，本章节不对这部分进行更细致的讨论。机器学习框架的编译器前端的独特之处主要在于自动微分功能的支持。为了满足自动微分功能带来的新需求，机器学习框架需要在传统中间表示的基础上设计新的中间表示结构。因此，本章节的介绍重点会放在中间表示以及自动微分这两个部分。最后，我们会简要探讨类型系统，静态分析和前端优化等编译器基础概念。
+:numref:`compiler_frontend_structure`展示了机器学习编译器的前端的流程。其中，对源程序的解析过程与传统编译器是大致相同的，本章节不对这部分进行更细致的讨论。机器学习框架的编译器前端的独特之处主要在于自动微分功能的支持。为了满足自动微分功能带来的新需求，机器学习框架需要在传统中间表示的基础上设计新的中间表示结构。因此，本章节的介绍重点会放在中间表示以及自动微分这两个部分。最后，我们会简要探讨类型系统，静态分析和前端优化等编译器基础概念。

 ![编译器前端基础结构](../img/ch04/编译器前端基础架构.svg)
 :width:`1000px`
--- a/chapter_frontend_and_ir/intermediate_representation.md
+++ b/chapter_frontend_and_ir/intermediate_representation.md
@@ -8,7 +8,7 @@

 中间表示(IR)，是编译器用于表示源代码的数据结构或代码，是程序编译过程中介于源语言和目标语言之间的程序表示。几乎所有的编译器都需要某种形式的中间表示，来对被分析、转换和优化的代码进行建模。在编译过程中，中间表示必须具备足够的表达力，在不丢失信息的情况下准确表达源代码，并且充分考虑从源代码到目标代码编译的完备性、编译优化的易用性和性能。

-引入中间表示后，中间表示既能面向多个前端，表达多种源程序语言，又能对接多个后端，连接不同目标机器，如图 :numref:`intermediate_representation`所示。在此基础上，编译流程就可以在前后端直接增加更多的优化流程，这些优化流程以现有IR为输入，又以新生成的IR为输出，被称为优化器。优化器负责分析并改进中间表示，极大程度的提高了编译流程的可拓展性，也降低了优化流程对前端和后端的破坏。
+引入中间表示后，中间表示既能面向多个前端，表达多种源程序语言，又能对接多个后端，连接不同目标机器，如 :numref:`intermediate_representation`所示。在此基础上，编译流程就可以在前后端直接增加更多的优化流程，这些优化流程以现有IR为输入，又以新生成的IR为输出，被称为优化器。优化器负责分析并改进中间表示，极大程度的提高了编译流程的可拓展性，也降低了优化流程对前端和后端的破坏。

 ![中间表示](../img/ch04/中间表示-中间表示结构.svg)
 :width:`800px`
@@ -18,7 +18,7 @@

 ### 中间表示的种类

-上一节介绍了中间表示的基本概念，初步阐述了中间表示的重要作用和发展历程。接下来从组织结构的角度出发，介绍通用编译器的中间表示的类型以及各自特点（[@2007Engineering]），如下表所示。中间表示组织结构的设计，对编译阶段的分析优化、代码生成等有着重要影响。编译器的设计需求不同，采用的中间表示组织结构也有所不同。
+上一节介绍了中间表示的基本概念，初步阐述了中间表示的重要作用和发展历程。接下来从组织结构的角度出发，介绍通用编译器的中间表示的类型以及各自特点 :cite:`2020MLIR`，如下表所示。中间表示组织结构的设计，对编译阶段的分析优化、代码生成等有着重要影响。编译器的设计需求不同，采用的中间表示组织结构也有所不同。

 ::: {#tab:ch04/ch04-categorize}
  组织结构       特点                   举例
@@ -35,8 +35,7 @@
 线性中间表示类似抽象机的汇编代码，将被编译代码表示为操作的有序序列，对操作序列规定了一种清晰且实用的顺序。由于大多数处理器采用线性的汇编语言，线性中间表示广泛应用于编译器设计。

 常用线性中间表示有堆栈机代码(Stack-Machine Code)和三地址代码(Three
-Address Code) （[@2007Compilers]）
-。堆栈机代码是一种单地址代码，提供了简单紧凑的表示。堆栈机代码的指令通常只有一个操作码，其操作数存在一个栈中。大多数操作指令从栈获得操作数，并将其结果推入栈中。三地址代码，简称为3AC，模拟了现代RISC机器的指令格式。它通过一组四元组实现，每个四元组包括一个运算符和三个地址(两个操作数、一个目标)。对于表达式a-b\*5，堆栈机代码和三地址代码如图 :numref:`linear_ir`所示。
+Address Code) :cite:`2007Compilers` 。堆栈机代码是一种单地址代码，提供了简单紧凑的表示。堆栈机代码的指令通常只有一个操作码，其操作数存在一个栈中。大多数操作指令从栈获得操作数，并将其结果推入栈中。三地址代码，简称为3AC，模拟了现代RISC机器的指令格式。它通过一组四元组实现，每个四元组包括一个运算符和三个地址(两个操作数、一个目标)。对于表达式a-b\*5，堆栈机代码和三地址代码如 :numref:`linear_ir`所示。

 ![堆栈机代码和三地址代码](../img/ch04/中间表示-线性中间表示.svg)
 :width:`800px`
@@ -48,7 +47,7 @@ Address Code) （[@2007Compilers]）
 Syntax Tree，AST)、有向无环图(Directed Acyclic
 Graph，DAG)、控制流图(Control-Flow Graph，CFG)等。

-AST抽象语法树采用树型中间表示的形式，是一种接近源代码层次的表示。对于表达式$a*5+a*5*b$，其AST表示如图 :numref:`AST_DAG`所示。可以看到，AST形式包含$a*5$的两个不同副本，存在冗余。在AST的基础上，DAG提供了简化的表达形式，一个节点可以有多个父节点，相同子树可以重用。如果编译器能够证明$a$的值没有改变，则DAG可以重用子树，降低求值过程的代价。
+AST抽象语法树采用树型中间表示的形式，是一种接近源代码层次的表示。对于表达式$a*5+a*5*b$，其AST表示如 :numref:`AST_DAG`所示。可以看到，AST形式包含$a*5$的两个不同副本，存在冗余。在AST的基础上，DAG提供了简化的表达形式，一个节点可以有多个父节点，相同子树可以重用。如果编译器能够证明$a$的值没有改变，则DAG可以重用子树，降低求值过程的代价。

 ![AST图和DAG图](../img/ch04/中间表示-ASTDAG.svg)
 :width:`600px`
@@ -56,11 +55,10 @@ AST抽象语法树采用树型中间表示的形式，是一种接近源代码

 3、混合中间表示

-混合中间表示是线性中间表示和图中间表示的结合，这里以LLVM IR
-（[@2004LLVM]） 为例进行说明。LLVM(Low Level Virtual
+混合中间表示是线性中间表示和图中间表示的结合，这里以LLVM IR :cite:`2004LLVM` 为例进行说明。LLVM(Low Level Virtual
 Machine)是2000年提出的开源编译器框架项目，旨在为不同的前端后端提供统一的中间表示。LLVM
-IR使用线性中间表示表示基本块，使用图中间表示表示这些块之间的控制流，如图 :numref:`LLVM_IR`所示。基本块中，每条指令以静态单赋值(Static
-Single Assignment， SSA) （[@Richard1995A]）
+IR使用线性中间表示表示基本块，使用图中间表示表示这些块之间的控制流，如 :numref:`LLVM_IR`所示。基本块中，每条指令以静态单赋值(Static
+Single Assignment， SSA) :cite:`Richard1995A`
 形式呈现，这些指令构成一个指令线性列表。SSA形式要求每个变量只赋值一次，并且每个变量在使用之前定义。控制流图中，每个节点为一个基本块，基本块之间通过边实现控制转移。

 ![LLVM IR](../img/ch04/中间表示-LLVMIR.svg)
@@ -84,7 +82,7 @@ IR，能够很好地满足通用编译器的基本功能需求，包括类型系
 计算图模式。主流机器学习框架如TensorFlow、PyTorch、MindSpore等都提供了静态图和动态图两种计算图模式，静态计算图模式先创建定义计算图，再显式执行，有利于对计算图进行优化，高效但不灵活。动态计算图模式则是每使用一个算子后，该算子会在计算图中立即执行得到结果，使用灵活、便于调试，但运行速度较低。机器学习框架的中间表示设计同时支持静态图和动态图，可以针对待解决的任务需求，选择合适的模式构建算法模型。

 4\)
-支持高阶函数和闭包（[@2010C]）。高阶函数和闭包是函数式编程的重要特性，高阶函数是指使用其它函数作为参数、或者返回一个函数作为结果的函数，闭包是指代码块和作用域环境的结合，可以在另一个作用域中调用一个函数的内部函数，并访问到该函数作用域中的成员。支持高阶函数和闭包，可以抽象通用问题、减少重复代码、提升框架表达的灵活性和简洁性。
+支持高阶函数和闭包 :cite:`2010C`。高阶函数和闭包是函数式编程的重要特性，高阶函数是指使用其它函数作为参数、或者返回一个函数作为结果的函数，闭包是指代码块和作用域环境的结合，可以在另一个作用域中调用一个函数的内部函数，并访问到该函数作用域中的成员。支持高阶函数和闭包，可以抽象通用问题、减少重复代码、提升框架表达的灵活性和简洁性。

 5\)
 编译优化。机器学习框架的编译优化主要包括硬件无关的优化、硬件相关的优化、部署推理相关的优化等，这些优化都依赖于中间表示的实现。
@@ -101,7 +99,7 @@ IR作为PyTorch模型的中间表示，通过JIT即时编译的形式，将Pytho

 PyTorch框架采用命令式编程方式，其TorchScript
 IR以基于SSA的线性IR为基本组成形式，并通过JIT即时编译的Tracing和Scripting两种方法将Python代码转换成TorchScript
-IR。图 :numref:`TorchScript_IR`给出了Python示例代码及其TorchScript
+IR。如 :numref:`TorchScript_IR`给出了Python示例代码及其TorchScript
 IR。

 ![Python代码及输出的TorchScript IR](../img/ch04/中间表示-torchscript.png)
@@ -112,21 +110,7 @@ IR。
 TorchScript是PyTorch的JIT实现，支持使用Python训练模型，然后通过JIT转换为语言无关的模块，从而提升模型部署能力，提高编译性能。同时，TorchScript
 IR显著改善了Pytorch框架的模型可视化效果。

-2、TensorFlow
-
-与PyTorch框架的动态图机制不同，TensorFlow机器学习框架因为其静态图机制而被人熟知。关于静态图和动态图的介绍，请参考第3.3章节。
-
-TensorFlow框架同时支持静态图和动态图，是一个基于数据流编程的机器学习框架，使用数据流图作为数据结构进行各种数值计算。为了适配不同的硬件平台，基于静态计算图，TensorFlow采用了多种IR设计，其编译生态系统如图 :numref:`MLIR`所示。蓝色部分是基于图的中间表示，绿色部分是基于SSA的中间表示，各层级在结构和抽象层级上存在较大的差距，转换开销大，而且同一层级的中间表示优化是相互独立的，不利于协同优化。
-
-![TensorFlow MLIR](../img/ch04/中间表示-MLIR.svg)
-:width:`600px`
-:label:`MLIR`
-
-针对这个问题，TensorFlow团队提出了MLIR(Multi-Level Intermediate
-Represent，多级中间表示)
-([@2020MLIR]），允许使用TensorFlow和其它机器学习库的项目编译更有效的代码，从而最大程度地利用基础硬件。MLIR是用于现代优化编译器的灵活基础架构，旨在定义一个通用的中间表示，在统一的基础架构中支持多种不同的需求。MLIR采用混合中间表示，允许在同一编译单元中结合多个层级的抽象来表示、分析和转换计算图，利用其模块化、可扩展的特点，解决了各种中间表示之间转换效率和可迁移性不高的问题，从而适配多种硬件平台。
-
-3、Jax
+2、Jax

 Jax机器学习框架同时支持静态图和动态图，其中间表示采用Jaxpr(JAX Program
 Representation) IR。Jaxpr
@@ -137,15 +121,43 @@ IR是一种强类型、纯函数的中间表示，其输入、输出都带有类
 :label:`Jaxpr`

 Jaxpr IR的表达采用ANF(A-norm
-Form)函数式表达形式，如图 :numref:`Jaxpr`所示。ANF形式将表达式划分为两类：原子表达式(aexp)和复合表达式(cexp)。原子表达式用于表示常数、变量、原语、匿名函数，复合表达式由多个原子表达式组成，可看作一个匿名函数或原语函数调用，组合的第一个输入是调用的函数，其余输入是调用的参数。
+Form)函数式表达形式，如 :numref:`Jaxpr`所示。ANF形式将表达式划分为两类：原子表达式(aexp)和复合表达式(cexp)。原子表达式用于表示常数、变量、原语、匿名函数，复合表达式由多个原子表达式组成，可看作一个匿名函数或原语函数调用，组合的第一个输入是调用的函数，其余输入是调用的参数。

 Jax框架结合了Autograd 和 JIT，基于Jaxpr
 IR，支持循环、分支、递归、闭包函数求导以及三阶求导，并且支持自动微分的反向传播和前向传播。

-4、MindSpore
+3、TensorFlow

-与PyTorch、TensorFlow、Jax框架相同，MindSpore机器学习框架同时支持静态图和动态图。MindSpore框架采用的是一种基于图表示的函数式中间表示，即MindIR，全称MindSpore
-IR。MindIR通过统一的中间表示，定义了网络的逻辑结构和算子的属性，能够消除不同后端的模型差异，连接不同的目标机器。
+TensorFlow框架同时支持静态图和动态图，是一个基于数据流编程的机器学习框架，使用数据流图作为数据结构进行各种数值计算。TensorFlow机器学习框架的静态图机制更为人所熟知。在静态图机制中，运行TensorFlow的程序会经历一系列的抽象以及分析，程序会逐步从高层的中间表示向底层的中间表示进行转换，我们把这种变换成为lowering。
+
+为了适配不同的硬件平台，基于静态计算图，TensorFlow采用了多种IR设计，其编译生态系统如:numref:`TFIR`所示。蓝色部分是基于图的中间表示，绿色部分是基于SSA的中间表示。在中间表示的转换过程中，各个层级的中间表示各自为政，无法互相有效地沟通信息，也不清楚其他层级的中间表示做了哪些优化，因此每个中间表示只能尽力将当前的优化做到最好，造成了很多优化在每个层级的中间表示中重复进行, 从而导致优化效率的低下。尤其是从图中间表示到SSA中间表示的变化过大，转换开销极大。此外，各个层级的相同优化的代码无法复用，也降低了开发效率。
+
+![TensorFlow](../img/ch04/中间表示-MLIR.svg)
+:width:`600px`
+:label:`TFIR`
+
+4、MLIR
+
+针对这个问题，TensorFlow团队提出了MLIR(Multi-Level Intermediate
+Represent，多级中间表示) :cite:`2020MLIR`。MLIR不是一种具体的中间表示定义，而是为中间表示提供一个统一的抽象表达和概念。 开发者可以使用MLIR开发的一系列基础设施，来定义符合自己需求的中间表示， 因此我们可以把MLIR理解为“编译器的编译器”。MLIR不局限于TensorFlow框架， 还可以用于构建连接其他语言与后端（如LLVM）的中间表示。
+MLIR深受LLVM设计理念的影响，但与LLVM不同的是， MLIR是一个更开放的生态系统。 在MLIR中， 没有预设的操作与抽象类型， 这使得开发者可以更自由地定义中间表示，并更有针对性地解决其领域的问题。MLIR通过Dialect的概念来支持这种可拓展性， Dialect在特定的命名空间下为抽象提供了分组机制，分别为每种中间表示定义对应的产生式并绑定相应的Operation， 从而生成一个MLIR类型的中间表示。Operation是MLIR中抽象和计算的核心单元，其具有特定的语意，可以用于表示LLVM中所有核心的IR结构， 例如指令， 函数以及模块等。 如下就是一个MLIR定义下的Operation：
+
+```
+%tensor = "toy.transpose"(%tensor) {inplace = true} : (tensor<2x3xf64>) -> tensor<3x2xf64> loc("example/file/path":12:1)
+```
+- \% tensor: Operation定义的结果的名字， $\%$是为了避免冲突统一加入的。一个Operation可以定义0或者多个结果，它们是SSA值。
+- "toy.transpose": Operation的名字。它是一个唯一的字符串，其中Dialect为Toy。因此它可以理解为Toy Dialect 中的transpose Operation。
+- (\%tensor)：输入操作数（或参数）的列表，它们是由其它操作定义或引用块参数的 SSA 值。
+- {inplace = true}：零个或多个属性的字典，这些属性是始终为常量的特殊操作数。在这里，我们定义了一个名为“inplace”的布尔属性，它的常量值为 true。
+- (tensor<2x3xf64>)->tensor<3x2xf64>：函数形式表示的操作类型，前者是输入，后者是输出。尖括号内代表输入与输出的数据类型以及形状， 例如$<2x3xf64>$代表一个形状位2X3， 数据类型为float64的张量。
+- loc("example/file/path":12:1)：此操作的源代码中的位置。
+
+由于各层中间表示都遵循如上的样式进行定义，所以各个层级的中间表示之间可以更加方便的进行转换， 提高了中间表示转换的效率。各个不同层级的中间表示还可以协同进行优化。 此外，由于中间表示之间不再相互独立， 各层级的优化不必做到极致，而是可以将优化放到最适合的层级。 其他的中间表示只需要先转换为该层级的中间表示，就可以进行相关的优化，提高了优化的效率与开发效率。TensorFlow从图中间表示到SSA中间表示的转换也可以通过使用MLIR来进行多层转换， 使转换更加平滑， 降低了转化的难度。 针对MLIR的更多内容将会在第六章进行介绍。 
+
+5、MindSpore
+
+与PyTorch、Jax、TensorFlow框架相同，MindSpore机器学习框架同时支持静态图和动态图。MindSpore框架采用的是一种基于图表示的函数式中间表示，即MindIR，全称MindSpore
+IR。MindIR没有采用多层中间表示的结构，而是通过统一的中间表示，定义了网络的逻辑结构和算子的属性，能够消除不同后端的模型差异，连接不同的目标机器。

 MindIR最核心的目的是服务于自动微分变换，而自动微分采用的是基于函数式编程框架的变换方法，因此MindIR采用了接近于ANF函数式的语义。MindIR具有以下特点：

@@ -154,7 +166,7 @@ based）。与TensorFlow类似，程序使用图来表示，使其容易去做

 （2）纯函数的（Purely functional）。

-纯函数是指函数的结果只依赖函数的参数。若函数依赖或影响外部的状态，比如，函数会修改外部全局变量，或者函数的结果依赖全局变量的值，则称函数具有副作用（[@spuler1994compiler]）。若使用了带有副作用的函数，代码的执行顺序必须得到严格的保证，否则可能会得到错误的结果，比如对全局变量的先写后读变成了先读后写。同时，副作用的存在也会影响自动微分，因为反向部分需要从前向部分获取中间变量，需要确保该中间变量的正确。因此需要保证自动微分的函数是纯函数。
+纯函数是指函数的结果只依赖函数的参数。若函数依赖或影响外部的状态，比如，函数会修改外部全局变量，或者函数的结果依赖全局变量的值，则称函数具有副作用 :cite:`spuler1994compiler`。若使用了带有副作用的函数，代码的执行顺序必须得到严格的保证，否则可能会得到错误的结果，比如对全局变量的先写后读变成了先读后写。同时，副作用的存在也会影响自动微分，因为反向部分需要从前向部分获取中间变量，需要确保该中间变量的正确。因此需要保证自动微分的函数是纯函数。

 由于Python语言具有高度动态性的特点，纯函数式编程对用户使用上有一些编程限制。有些机器学习框架的自动微分功能只支持对纯函数求导，且要求用户自行保证这一点。如果用户代码中写了带有副作用的函数，那么求导的结果可能会不符合预期。MindIR支持副作用的表达，能够将副作用的表达转换为纯函数的表达，从而在保持ANF函数式语义不变的同时，确保执行顺序的正确性，从而实现自由度更高的自动微分。

@@ -164,19 +176,19 @@ representation）。反向模式的自动微分，需要存储基本操作的中
 （4）强类型的（Strongly
 typed）。每个节点需要有一个具体的类型，这个对于性能最大化很重要。在机器学习应用中，因为算子可能很耗费时间，所以越早捕获错误越好。因为需要支持函数调用和高阶函数，相比于TensorFlow的数据流图，MindIR的类型和形状推导更加复杂且强大。

-在结合MindSpore框架的自身特点后，MindIR的定义如图 :numref:`MindIR`所示。
+在结合MindSpore框架的自身特点后，MindIR的定义如 :numref:`MindIR`所示。

 ![MindIR文法。MindIR中的ANode对应于ANF的原子表达式，ValueNode用于表示常数值，ParameterNode用于表示函数的形参，CNode则对应于ANF的复合表达式，表示函数调用](../img/ch04/中间表示-MindIR.svg)
 :width:`800px`
 :label:`MindIR`

-接下来我们通过图 :numref:`MindIR_example`中的一段程序作为示例，来进一步分析MindIR。
+接下来我们通过 :numref:`MindIR_example`中的一段程序作为示例，来进一步分析MindIR。

 ![MindIR的ANF表达](../img/ch04/中间表示-MindIR示例.png)
 :width:`600px`
 :label:`MindIR_example`

-在ANF中，每个表达式都用let表达式绑定为一个变量，通过对变量的引用来表示对表达式输出的依赖，而在MindIR中，每个表达式都绑定为一个节点，通过节点与节点之间的有向边表示依赖关系。其函数图表示如图 :numref:`MindIR_graph`所示。
+在ANF中，每个表达式都用let表达式绑定为一个变量，通过对变量的引用来表示对表达式输出的依赖，而在MindIR中，每个表达式都绑定为一个节点，通过节点与节点之间的有向边表示依赖关系。其函数图表示如 :numref:`MindIR_graph`所示。

 ![MindIR的函数图表示](../img/ch04/中间表示-MindIR图.png)
 :width:`800px`
--- a/chapter_frontend_and_ir/type_system_and_static_analysis.md
+++ b/chapter_frontend_and_ir/type_system_and_static_analysis.md
@@ -16,9 +16,9 @@
 4）可读性。阅读代码时，明确的类型声明有助于理解程序代码。

 机器学习框架一般使用Python语言作为描述网络模型结构的前端语言。Python语言是一门动态弱类型的语言，入门简单易学习，开发代码简洁高效，但由于其解释执行的方式，运行速度往往较慢。Python前端语言给用户带来了动态灵活的语义和高效的开发效率，但是若想要生成运行高效的后端代码，后端框架需要优化友好的静态强类型中间表示。因此，需要一种高效可靠的静态分析方法作为桥梁，将Python前端表示转换成等价的静态强类型中间表示，以此给用户同时带来高效的开发效率和运行效率，例如Hindley--Milner（HM）类型系统。这是一种具有参数多态性的简单类型lambda演算的类型系统。它最初由J.
-Roger Hindley 提出（[@1969The]），并由Robin Milner
-进行扩展和验证（[@1978A]）。后来，路易斯·达马斯（Luis
-Damas）对HM类型推导方法进行了详尽的分析和证明（[@1982Principal]），并将其扩展到支持具有多态引用的系统。Hindley--Milner类型系统的目标是在没有给定类型注解的情况下，自动推导出任意表达式的类型。其算法具有抽象性和通用性，采用简洁的符号表示，能够根据表达式形式推导出明确直观的定义，常用于类型推导和类型检查。因此，Hindley--Milner类型系统广泛应用于编程语言设计中，比如Haskell和Ocaml。
+Roger Hindley 提出 :cite:`1969The`，并由Robin Milner
+进行扩展和验证 :cite:`1978A` 。后来，路易斯·达马斯（Luis
+Damas）对HM类型推导方法进行了详尽的分析和证明 :cite:`1982Principal`，并将其扩展到支持具有多态引用的系统。Hindley--Milner类型系统的目标是在没有给定类型注解的情况下，自动推导出任意表达式的类型。其算法具有抽象性和通用性，采用简洁的符号表示，能够根据表达式形式推导出明确直观的定义，常用于类型推导和类型检查。因此，Hindley--Milner类型系统广泛应用于编程语言设计中，比如Haskell和Ocaml。

 ### 静态分析概述

--- a/chapter_online_machine_learning/index.md
+++ b/chapter_online_machine_learning/index.md
@@ -1 +0,0 @@
-# 在线机器学习
--- a/chapter_recommender_system/future.md
+++ b/chapter_recommender_system/future.md
@@ -0,0 +1,10 @@
+## 未来可以探索的方向
+
+为了解决在线深度学习推荐系统的以上几点问题，研究人员也探索了几个潜在的方向。
+
+-   云--边--端协同推荐系统。随着边缘设备的增加以及用户端设备性能逐渐增强，服务提供者可以通过将部分计算服务从云服务器下放至边缘服务器乃至用户的设备上来提高模型的反应速度。例如，有研究 :cite:`gong2020edgerec`探索了将模型的前几层下放至客户端上，并且利用用户的本地数据进行个性化训练以给出更加准确的推荐结果。当用户的兴趣发生改变时，客户端上的小模型可以实时地更新以响应用户的请求。除此之外，还可以借鉴联邦学习中的概念，例如有研究 :cite:`NEURIPS2020_a1d4c20b`探索了利用知识迁移的方法在云-端之间传递信息。在在线推荐系统中使用这种方法可以彻底解耦云上的大模型与客户端的小模型。
+
+-   异构硬件多级存储。前文提到GPU显存无法装下完整的模型参数，一些现有的系统 :cite:`MLSYS2020_f7e6c855`为了充分利用GPU的计算优势，采用多级缓存的思想，将部分参数分级存储于显存、主存和固态硬盘上。在他们提出的这个分级系统中，主要解决了缓存策略和异构硬件的适配问题。然而在设计类似的存储系统时，还应该考虑到机器学习模型内在的一些访存特征以进一步优化。Kraken :cite:`9355295`这篇工作讨论了利用机器学习模型的特征对嵌入项的哈希表的存储进行优化的方法。此外，新型硬件的发展为解决大规模推荐模型的高效存储提供了新的可能。比如非易失存储可以作为主存的扩展，进一步提升系统可以支持的模型尺寸。然而目前还没有见到专门为在线机器学习优化的非易失存储系统。另外也有工作 :cite:`MLSYS2021_ec895663`讨论了利用FPGA加速嵌入表的访存并且相比于CPU服务器取得了非常显著的效果。
+
+
+-   内存高效的嵌入项存储与计算。除了系统上的设计，研究人员也在探索其他算法优化手段来压缩嵌入表的内存需求。直接使用低精度浮点数可以有效降低内存开销，但是还是会对模型的精度产生一定的影响。因此在在线推荐服务这种精度敏感的场景中并不适用。除此之外， :cite:`MLSYS2021_979d472a`利用低秩分解可以将一个大矩阵分解为两个小矩阵（向量）。这种方法可以在保留原矩阵大量信息的前提下显著减小内存开销。除了低秩分解外，还有其他 :cite:`10.1145/3394486.3403059`分解嵌入表的手段。还有研究 :cite:`ginart2021mixed`表明，没有必要为所有的项目都使用一样长的嵌入项，可以根据嵌入项的重要性动态决定其长度以节省内存开销。作为系统设计者，如何将层出不穷的算法优化手段高效地实现是需要考虑的问题。
--- a/chapter_recommender_system/index.md
+++ b/chapter_recommender_system/index.md
@@ -0,0 +1,13 @@
+# 深度学习推荐系统
+
+推荐系统通过对用户特征、历史行为等数据的分析，为用户推荐可能感兴趣的内容、商品或者广告。在信息爆炸的时代，高效且准确的推荐结果极大地提升了在线服务的质量。近年来，基于深度学习的推荐模型由于可以高效地利用在线服务中产生的海量数据，被谷歌、脸书、阿里巴巴等各大公司广泛应用于生产环境中。本节主要介绍深度学习推荐系统在工业界的主流系统架构、问题以及可能的解决方案。
+
+```toc
+:maxdepth: 2
+
+overview
+system_architecture
+system_problem
+future
+summary
+```
--- a/chapter_recommender_system/overview.md
+++ b/chapter_recommender_system/overview.md
@@ -0,0 +1,23 @@
+## 背景
+
+为了方便本节的讨论，我们首先简单介绍一些推荐系统的基本概念，包括三部分：推荐模型的结构，推荐系统的架构，和评估推荐系统的关键指标。
+
+基于深度学习的推荐模型在过去几年受到了学术界和工业界的高度关注，得到了快速发展。目前主流的推荐模型（1，2，3，4）的基本结构可以总结如图 :numref:`ch10-recommendation-models`。
+
+![推荐模型的基本结构](../img/ch10/ch10-recommendation-models.svg)
+:width:`800px`
+:label:`ch10-recommendation-models`
+
+
+推荐模型以用户和内容的交互历史、用户属性、内容属性等特征作为输入，对输入特征进行充分相互作用，再将交互结果交由稠密深度神经网络来预测用户点击候选内容的可能性。由于推荐模型的输入中包含大量无法直接进行矩阵运算的类别数据，例如用户和商品标识符，所以必须使用嵌入表将类别数据转换为数字向量形式。由于每种类别数据包含的每种情况都需要一个单独的嵌入项来表示，而稠密深度神经网络的参数可以共享，在大规模推荐模型中，嵌入表占据了绝大部分内存 :cite:`MLSYS2021_979d472a,MLSYS2020_f7e6c855`。举例说明，假设一个推荐模型需要处理1亿条短视频内容，而每条短视频对应的嵌入项为一个64维的32位浮点数向量，那么仅该内容嵌入表就需要就需要占据大约24GB内存。如果考虑到用户标识符等其他嵌入表，那么单个模型可以轻易占据近100GB内存。而在工业界生产环境中，TB级的推荐模型 :cite:`MLSYS2020_f7e6c855`也是非常常见的。
+
+
+在实际的生产环境中，除了推荐模型本身，推荐系统通常包括：数据收集、数据处理、数据存储、模型训练、模型存储、模型评估、推理服务等多个子系统。如图 :numref:`ch10-abstract-recommendation-systems`所示，这些子系统之间分工协作、紧密配合，构成一个从用户反馈、到模型更新、再到新推荐结果生成的闭环。下一小节中将重点介绍模型训练、推理子系统的结构。
+
+![推荐系统的抽象架构](../img/ch10/ch10-abstract-recommendation-systems.svg)
+:width:`800px`
+:label:`ch10-abstract-recommendation-systems`
+
+深度学习模型给出的推荐结果的准确性是推荐系统需要关注的一个基本指标。然而不同于一般学术论文中使用的基准数据集，生产环境中推荐模型面对的是动态变化的数据分布。例如，每天的热点内容不同，每个用户的兴趣也会不断变化。为了保持模型的性能，推荐系统需要不断根据用户的反馈对已有模型进行更新。而这就需要系统能够将用户的行为转换为模型可以处理的数据。系统首先在提供服务的同时收集用户的行为，例如用户对内容的浏览、点击动作。收集到的数据还需要进一步加工处理，从而得到模型可以接受的格式化数据。
+
+除了推荐准确性，对于在线服务的提供者而言，可用性是一个非常关键的指标。当用户需要一个推荐结果时，相比于给用户一个不完全准确的推荐，"无响应"的结果对于用户的体验伤害更大。因此，在某种程度上可以说系统可用性是比推荐结果的准确性更加关键的一个指标。然而这并不意味着准确性不重要，在一定的资源限制下，在线推荐系统的设计者必须谨慎地在准确性和可用性之间进行平衡。例如，使用更宽、更深、更复杂的神经网络模型可能会给出更加准确的推荐结果，但如果其推断延迟高于给定的阈值，那么这样的模型不能直接运用于生产环境中。
--- a/chapter_recommender_system/summary.md
+++ b/chapter_recommender_system/summary.md
@@ -0,0 +1,3 @@
+## 小结
+
+推荐系统作为深度学习在工业界最成功的落地成果之一，极大地提升了用户的在线服务体验，并且为各大公司创造了可观的利润，然而也带来了许多系统层面的挑战亟待解决。本节简单介绍了典型的工业界推荐系统架构及其面临的挑战，并给出了潜在的解决方案的方向。在实际生产环境中，具体的系统设计方案根据不同推荐场景的需求而变化，不存在一种万能的解决方案。
--- a/chapter_recommender_system/system_architecture.md
+++ b/chapter_recommender_system/system_architecture.md
@@ -0,0 +1,12 @@
+## 主流系统架构
+
+正如上文提到的，嵌入表占据了推荐模型绝大部分存储而其更新具有显著的稀疏性，因此推荐系统通常采用上一章介绍的参数服务器架构来存储模型。具体来讲，所有参数被分布存储在一组参数服务器上，而训练服务器一方面从数据存储模块拉取训练数据，另一方面根据训练数据从参数服务器上拉取对应的嵌入项和所有稠密神经网络参数。训练服务器本地更新之后将本地梯度或新的参数发送回参数服务器以更新全局参数。全局参数更新可以选择全同步，半同步，或异步更新。类似的，推理服务器在接到一批用户的推荐请求后，从参数服务器拉去相应的嵌入项和稠密神经网络参数来响应用户的请求。为了提升训练（推理）的吞吐，可以在训练（推理）服务器上缓存一部分参数。
+
+为了避免训练服务器和参数服务器之间的通信限制训练吞吐率，一些公司也在探索单机多GPU训练超大规模推荐系统。然而正如前文提到的，即使是单个推荐模型的参数量（1̃00GB）也超出了目前最新的GPU显存。有鉴于此，脸书公司的定制训练平台
+-- ZionEX :cite:`zionex`利用计算设备之间的高速链接将多台设备的存储共享起来可以单机训练TB级推荐模型。然而对于更大规模的模型或中小型企业、实验室，参数服务器架构依然是性价比最高的解决方案。
+
+为了提升在发生故障的情况下的可用性，在线服务中的深度学习推荐模型通常都采用多副本分布式部署。同一个模型的多个副本通常会被部署在至少两个不同的地理区域内的多个数据中心中，如图 :numref:`ch10-recommendation-systems`，以应对大面积停电或者网络中断而导致整个地区的所有副本都不可用。除了容错方面的考虑，部署多个副本还有其他几点优势。首先，将模型部署在靠近用户的云服务器上可以提升响应速度。其次，部署多份副本也可以拓展模型推理服务的吞吐率。
+
+![推荐系统的分布式架构](../img/ch10/ch10-recommendation-systems.svg)
+:width:`800px`
+:label:`ch10-recommendation-systems`
--- a/chapter_recommender_system/system_problem.md
+++ b/chapter_recommender_system/system_problem.md
@@ -0,0 +1,10 @@
+## 现有解决方案及其存在的问题
+
+在线服务系统的两个主要诉求：
+
+-   大模型的高效存储。为了提升训练和推理的性能，通常推荐模型全部存储在内存中，然而纯内存存储对于内存的需求极高。正如前文分析的，单个模型就要占据至少100GB的内存，而一个在线推荐系统中需要同时运行多个模型负责不同的服务。如果考虑到除了在线服务模型，算法研究人员还需要上线测试不同的模型结构或者训练策略，系统中通常会同时存在上百个超大模型。因此在线推荐系统亟需既能拓展存储容量，又不会影响训练和推理性能的存储解决方案。
+
+-   大模型的快速更新。
+    在线服务系统所面对的环境是复杂多变的，因此其中的机器学习模型必须不断更新以应对新的数据分布。以一个短视频推荐系统为例，其面对的变化主要来自三点。首先，每时每刻都有大量的新视频上传，这些新视频的特征分布和模型训练时所见到的数据不同；其次，对于不断加入的新用户，模型难以直接给出最优的推荐结果；最后，全部用户和内容之间的交互在不断改变，表现为热点视频在持续变化。因此，为了应对以上变化，在线服务中不可能奢望仅仅训练一次模型就能够一劳永逸地解决问题。目前业界主流的做法是利用新产生的数据不断地增量式更新所部属的模型。在学术界和工业界大量的研究和实践 :cite:`10.1145/2020408.2020444,10.1145/2648584.2648589,10.1145/3267809.3267817,9355295`中都发现模型更新可以有效缓解概念漂移带来的危害，而且更新的频率越高，模型的性能越好。
+
+在线推荐系统对跨地域地部署的大模型进行快速更新的需求在现有的系统中很难得到满足。一种最直观的解决方案是周期性地将训练服务器上的模型参数发给所有副本。然而这种方式面临着非常大的资源瓶颈。我们以网络开销为例进行分析。假设负责训练的参数服务器存储有100GB的参数，每10分钟将所有参数（在训练集群内部，模型更新的速度极快，10分钟足够将所有参数更新多次）发给其余2个副本。这就需要至少2.6Gbps的网络带宽。然而我们的分析只是最基本的情况，没有考虑网络传输的额外开销以及可能出现的失败重传，也没有考虑需要水平扩展至更多副本、更大模型、更高的更新频率的情况。为了缓解网络瓶颈，人们不得不选择以更慢的速度更新更大的模型，或者限制模型大小以追求更快的更新速度。简单的广播模型参数除了会有很大的资源瓶颈，还无法保证多副本之间的一致性。然而如果采用先用的数据库系统来保证一致性，只能使得资源开销更加严重，进一步限制系统的规模和效率。
--- a/chapter_reinforcement_learning/index.md
+++ b/chapter_reinforcement_learning/index.md
@@ -17,4 +17,4 @@ distributed_node_rl
 marl
 marl_sys
 summary
-```
+```
--- a/chapter_reinforcement_learning/marl.md
+++ b/chapter_reinforcement_learning/marl.md
@@ -12,13 +12,15 @@

 由上述介绍和定义可以发现，多智能体强化学习是一个比单智能体强化学习更加复杂的问题。而实际上，多个智能体的存在，对于每个智能体的决策而言，绝对不是简单的把每个单智能体决策累加的难度，实际情况要比单智能体决策问题复杂很多。多智能体系统的研究实际上是门古老的学科，它与博弈论（Game Theory）密切相关，在深度强化学习盛行以前早已有大量研究和许多理论上未解的难题。其中一个典型的问题是纳什均衡在双人非零和博弈下没有多项式时间内可解的方法（实际上，这是一个PPAD（Polynomial Parity Argument, Directed version）类的问题。见Settling the Complexity of Computing Two-Player Nash Equilibria. Xi Chen, et al.）。由于篇幅限制，我们这里无法对多智能体问题做深入探讨，我们可以用一个简单例子来介绍为什么多智能体强化学习问题无法简单地用单智能体强化学习算法来解。

+:剪刀石头布奖励值
+
 |     | 剪刀    | 石头    | 布     |
 | --- | ------- | ------- | ------- |
 | 剪刀  | (0,0)   | (-1,+1) | (+1,-1) |
 | 石头  | (+1,-1) | (0,0)   | (-1,+1) |
 | 布   | (-1,+1) | (+1,-1) | (0,0)   |
-:label:`tab:ch12/ch12-marl`
+:label:`tab_ch12_ch12_marl`

-我们考虑一个大家都熟悉的游戏就“剪刀、石头、布”，考虑两个玩家玩这个游戏的输赢情况，“剪刀<石头<布<剪刀...”，这里的“<”即前一个纯策略被后一个纯策略完全压制，我们给予奖励值-1、+1到这两个玩家，当他们选择相同的纯策略时，奖励值均为0。于是我们得到一个奖励值表如 :numref:`tab:ch12/ch12-marl`所示，横轴为玩家1，纵轴为玩家2，表内的数组为玩家1和玩家2各自在相应动作下得到的奖励值。由于这个矩阵的反对称性，这个问题的纳什均衡策略对两个玩家相同，均为$(\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$的策略分布，即有各$\frac{1}{3}$的概率出剪刀、石头或布。如果我们把得到这个纳什均衡策略作为多智能体学习的目标，那么我们可以简单分析得到这个均衡策略无法通过简单的单智能体算法得到。考虑我们随机初始化两个玩家为任意两个纯策略，比如玩家1出剪刀，玩家2出石头。这时假设玩家2策略固定，可以把玩家2看做固定环境的一部分，于是可以使用任意单智能体强化学习算法对玩家1进行训练，使其最大化自己的奖励值。于是，玩家1会收敛到布的纯策略。这时再把玩家1固定，训练玩家2，玩家2又收敛到剪刀的纯策略。于是循环往复，整个训练过程始终无法收敛，玩家1和2各自在3个策略中循环却无法得到正确的纳什均衡策略。
+我们考虑一个大家都熟悉的游戏就“剪刀、石头、布”，考虑两个玩家玩这个游戏的输赢情况，“剪刀<石头<布<剪刀...”，这里的“<”即前一个纯策略被后一个纯策略完全压制，我们给予奖励值-1、+1到这两个玩家，当他们选择相同的纯策略时，奖励值均为0。于是我们得到一个奖励值表如 :numref:`tab_ch12_ch12_marl`所示，横轴为玩家1，纵轴为玩家2，表内的数组为玩家1和玩家2各自在相应动作下得到的奖励值。由于这个矩阵的反对称性，这个问题的纳什均衡策略对两个玩家相同，均为$(\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$的策略分布，即有各$\frac{1}{3}$的概率出剪刀、石头或布。如果我们把得到这个纳什均衡策略作为多智能体学习的目标，那么我们可以简单分析得到这个均衡策略无法通过简单的单智能体算法得到。考虑我们随机初始化两个玩家为任意两个纯策略，比如玩家1出剪刀，玩家2出石头。这时假设玩家2策略固定，可以把玩家2看做固定环境的一部分，于是可以使用任意单智能体强化学习算法对玩家1进行训练，使其最大化自己的奖励值。于是，玩家1会收敛到布的纯策略。这时再把玩家1固定，训练玩家2，玩家2又收敛到剪刀的纯策略。于是循环往复，整个训练过程始终无法收敛，玩家1和2各自在3个策略中循环却无法得到正确的纳什均衡策略。

 我们在上面这个例子中采用的学习方法其实是多智能体强化学习中最基础的一种，叫自学习（Selfplay）。我们可以看到自学习在特定的任务设置下可能无法收敛到我们想要的最终目标。正是由于多智能体学习过程中有类似循环结构的出现，我们需要更复杂的训练方法，和专门针对多智能体的学习方式来达到我们想要的目标。一般来讲，多智能体强化学习是比单智能体强化学习更复杂的一类，对于自学习的方法而言，单智能体强化学习的过程可以看做一个多智能体强化学习的子任务。从前面这一小游戏的角度来理解，当玩家1策略固定时，玩家1加游戏环境构成玩家2的实际学习环境，由于这个环境是固定的，玩家2可以通过单智能体强化学习来达到自身奖励值最大化；这时再固定玩家2的策略，玩家1又可以进行单智能体强化学习......这样，单智能体强化学习是多智能体任务的子任务。其他算法如虚构自学习（Fictitious Self-play），需要在每个单智能体强化学习的步骤中，对对手历史策略的平均策略求得最优应对策略，而对手的训练也是如此，进行循环，能够在上面剪刀-石头-布一类的游戏中保证收敛到纳什均衡策略。
--- a/chapter_reinforcement_learning/single_node_rl.md
+++ b/chapter_reinforcement_learning/single_node_rl.md
@@ -3,7 +3,7 @@
 前面介绍了强化学习的基本知识和在系统层面的一般需求，这里我们介绍常见的单智能体强化学习系统中较为简单的一类，即单节点强化学习系统。这里，我们按照是否对模型训练和更新进行并行处理，将强化学习系统分为单节点和分布式强化学习系统。其中，单节点强化学习系统可以理解为只实例化一个类对象作为智能体，与环境交互进行采样和利用所采得的样本进行更新的过程分别视为这个类内的不同函数。除此之外的更为复杂的强化学习框架都可视为分布式强化学习系统。分布式强化学习系统的具体形式有很多，这也往往依赖于所实现的算法。从最简单的情况考虑，假设我们仍在同一个计算单元上实现算法，但是将强化学习的采样过程和更新过程实现为两个并行的进程，甚至各自实现为多个进程，以满足不同计算资源间的平衡。这时就需要进程间通信来协调采样和更新过程，这是一个最基础的分布式强化学习框架。更为复杂的情况是，整个算法的运行在多个计算设备上进行（如一个多机的计算集群），智能体的函数可能需要跨机跨进程间的通信来实现。对于多智能体系统，还需要同时对多个智能体的模型进行更新，则需要更为复杂的计算系统设计。我们将逐步介绍这些不同的系统内的实现机制。

 我们先对单节点强化学习系统进行介绍。
-在这里，我们以RLzoo :cite:`ding2020efficient`为例，讲解一个单节点强化学习系统构建所需要的基本模块。如 :numref:`ch12/ch12-rlzoo`所示，是RLzoo中采用的一个典型的单节点强化学习系统，它包括几个基本的组成部分：神经网络、适配器、策略网络和价值网络、环境实例、模型学习器、经验回放缓存（Experience Replay Buffer）等。我们先对前三个，神经网络、适配器、策略网络和价值网络进行介绍。神经网络即一般深度学习中的神经网络，用于实现基于数据的函数拟合，我们在图中简单列出常见的三类神经网络：全连接网络，卷积网络和循环网络。策略网络和价值网络是一般深度强化学习的常见组成部分，策略网络即一个由深度神经网络参数化的策略表示，而价值网络为神经网络表示的状态价值（State-Value）或状态-动作价值（State-Action Value）函数。这里我们不妨称前三类神经网络为一般神经网络，策略网络和价值网络为强化学习特定网络，前者往往是后者的重要组成部分。在RLzoo中，适配器则是为实现强化学习特定网络而选配一般神经网络的功能模块。首先，根据不同的观察量类型，强化学习智能体所用的神经网络头部会有不同的结构，这一选择可以由一个基于观察量的适配器来实现；其次，根据所采用的强化学习算法类型，相应的策略网络尾部需要有不同的输出类型，包括确定性策略和随机性策略，RLzoo中使用一个策略适配器来进行选择；最后，根据不同的动作输出，如离散型、连续型、类别型等，需要使用一个动作适配器来选择。 :numref:`fig:ch12/ch12-rlzoo`中我们统称这三个不类型的适配器为适配器。介绍完这些，我们已经有了可用的策略网络和价值网络，这构成了强化学习智能体核心学习模块。除此之外，还需要一个学习器（Learner）来更新这些学习模块，更新的规则就是强化学习算法给出的损失函数。而要想实现学习模块的更新，最重要的是输入的学习数据，即智能体跟环境交互过程中所采集的样本。对于**离线**（Off-Policy）强化学习，这些样本通常被存储于一个称为经验回放缓存的地方，学习器在需要更新模型时从该缓存中采得一些样本来进行更新。这里说到的离线强化学习是强化学习算法中的一类，强化学习算法可以分为**在线**（On-Policy）强化学习和离线强化学习两类，按照某个特定判据。这个判据是，用于更新的模型和用于采样的模型是否为同一个，如果是，则称在线强化学习算法，否则为离线强化学习算法。因而，离线强化学习通常允许与环境交互的策略采集的样本被存储于一个较大的缓存内，从而允许在许久之后再从这个缓存中抽取样本对模型进行更新。而对于在线强化学习，这个“缓存”有时其实也是存在的，只不过它所存储的是非常近期内采集的数据，从而被更新模型和用于采样的模型可以近似认为是同一个。从而，这里我们简单表示RLzoo的强化学习系统统一包括这个经验回放缓存模块。有了以上策略和价值网络、经验回放缓存、适配器、学习器，我们就得到了RLzoo中一个单节点的强化学习智能体，将这个智能体与环境实例交互，并采集数据进行模型更新，我们就得到了一个完整的单节点强化学习系统。这里的环境实例化我们允许多个环境并行采样。
+在这里，我们以RLzoo :cite:`ding2020efficient`为例，讲解一个单节点强化学习系统构建所需要的基本模块。如 :numref:`ch12/ch12-rlzoo`所示，是RLzoo中采用的一个典型的单节点强化学习系统，它包括几个基本的组成部分：神经网络、适配器、策略网络和价值网络、环境实例、模型学习器、经验回放缓存（Experience Replay Buffer）等。我们先对前三个，神经网络、适配器、策略网络和价值网络进行介绍。神经网络即一般深度学习中的神经网络，用于实现基于数据的函数拟合，我们在图中简单列出常见的三类神经网络：全连接网络，卷积网络和循环网络。策略网络和价值网络是一般深度强化学习的常见组成部分，策略网络即一个由深度神经网络参数化的策略表示，而价值网络为神经网络表示的状态价值（State-Value）或状态-动作价值（State-Action Value）函数。这里我们不妨称前三类神经网络为一般神经网络，策略网络和价值网络为强化学习特定网络，前者往往是后者的重要组成部分。在RLzoo中，适配器则是为实现强化学习特定网络而选配一般神经网络的功能模块。首先，根据不同的观察量类型，强化学习智能体所用的神经网络头部会有不同的结构，这一选择可以由一个基于观察量的适配器来实现；其次，根据所采用的强化学习算法类型，相应的策略网络尾部需要有不同的输出类型，包括确定性策略和随机性策略，RLzoo中使用一个策略适配器来进行选择；最后，根据不同的动作输出，如离散型、连续型、类别型等，需要使用一个动作适配器来选择。 :numref:`ch12/ch12-rlzoo`中我们统称这三个不类型的适配器为适配器。介绍完这些，我们已经有了可用的策略网络和价值网络，这构成了强化学习智能体核心学习模块。除此之外，还需要一个学习器（Learner）来更新这些学习模块，更新的规则就是强化学习算法给出的损失函数。而要想实现学习模块的更新，最重要的是输入的学习数据，即智能体跟环境交互过程中所采集的样本。对于**离线**（Off-Policy）强化学习，这些样本通常被存储于一个称为经验回放缓存的地方，学习器在需要更新模型时从该缓存中采得一些样本来进行更新。这里说到的离线强化学习是强化学习算法中的一类，强化学习算法可以分为**在线**（On-Policy）强化学习和离线强化学习两类，按照某个特定判据。这个判据是，用于更新的模型和用于采样的模型是否为同一个，如果是，则称在线强化学习算法，否则为离线强化学习算法。因而，离线强化学习通常允许与环境交互的策略采集的样本被存储于一个较大的缓存内，从而允许在许久之后再从这个缓存中抽取样本对模型进行更新。而对于在线强化学习，这个“缓存”有时其实也是存在的，只不过它所存储的是非常近期内采集的数据，从而被更新模型和用于采样的模型可以近似认为是同一个。从而，这里我们简单表示RLzoo的强化学习系统统一包括这个经验回放缓存模块。有了以上策略和价值网络、经验回放缓存、适配器、学习器，我们就得到了RLzoo中一个单节点的强化学习智能体，将这个智能体与环境实例交互，并采集数据进行模型更新，我们就得到了一个完整的单节点强化学习系统。这里的环境实例化我们允许多个环境并行采样。

 ![RLzoo算法库中使用的强化学习系统](../img/ch12/ch12-rlzoo.png)

--- a/img/ch06/TVM.svg
+++ b/img/ch06/TVM.svg
--- a/img/ch06/akg.png
+++ b/img/ch06/akg.png
--- a/img/ch06/gemm.svg
+++ b/img/ch06/gemm.svg
--- a/img/ch06/gemm_tensor_core.svg
+++ b/img/ch06/gemm_tensor_core.svg
--- a/img/ch06/tbe.png
+++ b/img/ch06/tbe.png
--- a/img/ch09/ch10-allreduce-process.pdf
+++ b/img/ch09/ch10-allreduce-process.pdf
--- a/img/ch09/ch10-allreduce-process.svg
+++ b/img/ch09/ch10-allreduce-process.svg
--- a/img/ch09/ch10-allreduce-state.pdf
+++ b/img/ch09/ch10-allreduce-state.pdf
--- a/img/ch09/ch10-allreduce-state.svg
+++ b/img/ch09/ch10-allreduce-state.svg
@@ -0,0 +1,732 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="1176"
+   height="457.09634"
+   viewBox="0 0 1176 457.09634"
+   sodipodi:docname="ch10-allreduce-state.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 883 V 364 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     inkscape:zoom="0.78378348"
+     inkscape:cx="290.25873"
+     inkscape:cy="247.51734"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-allreduce-state"
+     transform="matrix(1.3333333,0,0,-1.3333333,-1.3333325,475.63801)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="m 103.5,300.1664 c 0,8.4685 6.8651,15.3336 15.3336,15.3336 h 155.3328 c 8.4685,0 15.3336,-6.8651 15.3336,-15.3336 V 238.8336 C 289.5,230.3651 282.6349,223.5 274.1664,223.5 H 118.8336 c -8.4685,0 -15.3336,6.8651 -15.3336,15.3336 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,103.5,315.5)"><path
+           d="M 0,194737.2 C 0,87186.82 87186.65,0 194736.8,0 H 2167463 c 107550,0 194737,87186.82 194737,194737.2 v 778925.6 c 0,107550.2 -87187,194737.2 -194737,194737.2 H 194736.8 C 87186.65,1168400 0,1081213 0,973662.8 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path28" /></g><path
+         d="m 111.5,278.5 h 56 v -39 h -56 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path30" /><g
+         id="g32"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,111.5,278.5)"><path
+           d="M 0,0 H 711200 V 495300 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path34" /></g><g
+         id="g36"
+         transform="translate(134.5547,-110)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text40"><tspan
+             x="0"
+             y="0"
+             id="tspan38">2</tspan></text></g><g
+         id="g42"
+         transform="translate(113.4134,-72)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text46"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan44">设备</tspan></text></g><g
+         id="g48"
+         transform="translate(154.4134,-72)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text52"><tspan
+             x="0"
+             y="0"
+             id="tspan50">1</tspan></text></g><path
+         d="m 168.5,278.5 h 56 v -39 h -56 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path54" /><g
+         id="g56"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,168.5,278.5)"><path
+           d="M 0,0 H 711200 V 495300 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path58" /></g><g
+         id="g60"
+         transform="translate(191.3145,-110)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0"
+             y="0"
+             id="tspan62">4</tspan></text></g><path
+         d="m 225.5,278.5 h 55 v -39 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path66" /><g
+         id="g68"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,225.5,278.5)"><path
+           d="M 0,0 H 698500 V 495300 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path70" /></g><g
+         id="g72"
+         transform="translate(248.0741,-110)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text76"><tspan
+             x="0"
+             y="0"
+             id="tspan74">6</tspan></text></g><path
+         d="m 1.5,182.333 c 0,8.3765 6.7905,15.167 15.16701,15.167 H 171.333 c 8.3765,0 15.167,-6.7905 15.167,-15.167 v -60.666 c 0,-8.3765 -6.7905,-15.167 -15.167,-15.167 H 16.66701 C 8.2905,106.5 1.5,113.2905 1.5,121.667 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path78" /><g
+         id="g80"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,1.5,197.5)"><path
+           d="M 0,192621.4 C 0,86239.53 86239.35,0 192621,0 h 1964258 c 106382,0 192621,86239.53 192621,192621.4 v 770457.2 c 0,106382.4 -86239,192621.4 -192621,192621.4 H 192621 C 86239.35,1155700 0,1069461 0,963078.6 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path82" /></g><path
+         d="m 9.5,160.5 h 56 v -38 h -56 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path84" /><g
+         id="g86"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,9.5,160.5)"><path
+           d="M 0,0 H 711200 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path88" /></g><g
+         id="g90"
+         transform="translate(32.40973,-227)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text94"><tspan
+             x="0"
+             y="0"
+             id="tspan92">1</tspan></text></g><g
+         id="g96"
+         transform="translate(11.26842,-190)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text100"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan98">设备</tspan></text></g><g
+         id="g102"
+         transform="translate(52.26842,-190)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text106"><tspan
+             x="0"
+             y="0"
+             id="tspan104">2</tspan></text></g><path
+         d="m 66.5,160.5 h 55 v -38 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path108" /><g
+         id="g110"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,66.5,160.5)"><path
+           d="M 0,0 H 698500 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path112" /></g><g
+         id="g114"
+         transform="translate(89.16941,-227)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text118"><tspan
+             x="0"
+             y="0"
+             id="tspan116">2</tspan></text></g><path
+         d="m 123.5,160.5 h 55 v -38 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path120" /><g
+         id="g122"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,123.5,160.5)"><path
+           d="M 0,0 H 698500 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path124" /></g><g
+         id="g126"
+         transform="translate(145.9292,-227)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text130"><tspan
+             x="0"
+             y="0"
+             id="tspan128">3</tspan></text></g><path
+         d="m 201.5,182.333 c 0,8.3765 6.7905,15.167 15.167,15.167 h 154.666 c 8.3765,0 15.167,-6.7905 15.167,-15.167 v -60.666 c 0,-8.3765 -6.7905,-15.167 -15.167,-15.167 H 216.667 c -8.3765,0 -15.167,6.7905 -15.167,15.167 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path132" /><g
+         id="g134"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,201.5,197.5)"><path
+           d="M 0,192621.4 C 0,86239.53 86239.35,0 192621,0 h 1964258 c 106382,0 192621,86239.53 192621,192621.4 v 770457.2 c 0,106382.4 -86239,192621.4 -192621,192621.4 H 192621 C 86239.35,1155700 0,1069461 0,963078.6 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path136" /></g><path
+         d="m 209.5,160.5 h 56 v -38 h -56 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path138" /><g
+         id="g140"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,209.5,160.5)"><path
+           d="M 0,0 H 711200 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path142" /></g><g
+         id="g144"
+         transform="translate(232.5075,-227)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text148"><tspan
+             x="0"
+             y="0"
+             id="tspan146">4</tspan></text></g><g
+         id="g150"
+         transform="translate(211.3663,-190)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text154"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan152">设备</tspan></text></g><g
+         id="g156"
+         transform="translate(252.3663,-190)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text160"><tspan
+             x="0"
+             y="0"
+             id="tspan158">3</tspan></text></g><path
+         d="m 266.5,160.5 h 56 v -38 h -56 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path162" /><g
+         id="g164"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,266.5,160.5)"><path
+           d="M 0,0 H 711200 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path166" /></g><g
+         id="g168"
+         transform="translate(289.2673,-227)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text172"><tspan
+             x="0"
+             y="0"
+             id="tspan170">8</tspan></text></g><path
+         d="m 323.5,160.5 h 55 v -38 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path174" /><g
+         id="g176"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,323.5,160.5)"><path
+           d="M 0,0 H 698500 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path178" /></g><g
+         id="g180"
+         transform="translate(342.027,-227)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text184"><tspan
+             x="0 7.9991999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan182">12</tspan></text></g><path
+         d="m 599.5,297.333 c 0,8.3765 6.7905,15.167 15.167,15.167 h 154.666 c 8.3765,0 15.167,-6.7905 15.167,-15.167 v -60.666 c 0,-8.3765 -6.7905,-15.167 -15.167,-15.167 H 614.667 c -8.3765,0 -15.167,6.7905 -15.167,15.167 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path186" /><g
+         id="g188"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,599.5,312.5)"><path
+           d="M 0,192621.4 C 0,86239.53 86239.35,0 192621,0 h 1964258 c 106382,0 192621,86239.53 192621,192621.4 v 770457.2 c 0,106382.4 -86239,192621.4 -192621,192621.4 H 192621 C 86239.35,1155700 0,1069461 0,963078.6 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path190" /></g><path
+         d="m 607.5,275.5 h 55 v -38 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path192" /><g
+         id="g194"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,607.5,275.5)"><path
+           d="M 0,0 H 698500 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path196" /></g><g
+         id="g198"
+         transform="translate(630.1545,-112)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text202"><tspan
+             x="0"
+             y="0"
+             id="tspan200">7</tspan></text></g><g
+         id="g204"
+         transform="translate(609.0132,-75)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text208"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan206">设备</tspan></text></g><g
+         id="g210"
+         transform="translate(650.0132,-75)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text214"><tspan
+             x="0"
+             y="0"
+             id="tspan212">1</tspan></text></g><path
+         d="m 664.5,275.5 h 55 v -38 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path216" /><g
+         id="g218"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,664.5,275.5)"><path
+           d="M 0,0 H 698500 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path220" /></g><g
+         id="g222"
+         transform="translate(682.9143,-112)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text226"><tspan
+             x="0 7.9991999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan224">14</tspan></text></g><path
+         d="m 720.5,275.5 h 56 v -38 h -56 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path228" /><g
+         id="g230"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,720.5,275.5)"><path
+           d="M 0,0 H 711200 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path232" /></g><g
+         id="g234"
+         transform="translate(739.674,-112)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text238"><tspan
+             x="0 7.9991999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan236">21</tspan></text></g><path
+         d="m 496.5,179.333 c 0,8.3765 6.7905,15.167 15.167,15.167 h 155.666 c 8.3765,0 15.167,-6.7905 15.167,-15.167 v -60.666 c 0,-8.3765 -6.7905,-15.167 -15.167,-15.167 H 511.667 c -8.3765,0 -15.167,6.7905 -15.167,15.167 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path240" /><g
+         id="g242"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,496.5,194.5)"><path
+           d="M 0,192621.4 C 0,86239.54 86239.39,0 192621.1,0 H 2169579 c 106382,0 192621,86239.54 192621,192621.4 v 770457.2 c 0,106381.4 -86239,192621.4 -192621,192621.4 H 192621.1 C 86239.39,1155700 0,1069460 0,963078.6 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path244" /></g><path
+         d="m 505.5,157.5 h 55 v -38 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path246" /><g
+         id="g248"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,505.5,157.5)"><path
+           d="M 0,0 H 698500 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path250" /></g><g
+         id="g252"
+         transform="translate(528.0096,-230)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text256"><tspan
+             x="0"
+             y="0"
+             id="tspan254">7</tspan></text></g><g
+         id="g258"
+         transform="translate(506.8683,-192)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text262"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan260">设备</tspan></text></g><g
+         id="g264"
+         transform="translate(547.8683,-192)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text268"><tspan
+             x="0"
+             y="0"
+             id="tspan266">2</tspan></text></g><path
+         d="m 562.5,157.5 h 55 v -38 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path270" /><g
+         id="g272"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,562.5,157.5)"><path
+           d="M 0,0 H 698500 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path274" /></g><g
+         id="g276"
+         transform="translate(580.7692,-230)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text280"><tspan
+             x="0 7.9991999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan278">14</tspan></text></g><path
+         d="m 618.5,157.5 h 56 v -38 h -56 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path282" /><g
+         id="g284"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,618.5,157.5)"><path
+           d="M 0,0 H 711200 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path286" /></g><g
+         id="g288"
+         transform="translate(637.529,-230)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text292"><tspan
+             x="0 7.9991999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan290">21</tspan></text></g><path
+         d="m 697.5,179.333 c 0,8.3765 6.7905,15.167 15.167,15.167 h 154.666 c 8.3765,0 15.167,-6.7905 15.167,-15.167 v -60.666 c 0,-8.3765 -6.7905,-15.167 -15.167,-15.167 H 712.667 c -8.3765,0 -15.167,6.7905 -15.167,15.167 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path294" /><g
+         id="g296"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,697.5,194.5)"><path
+           d="M 0,192621.4 C 0,86239.53 86239.35,0 192621,0 h 1964258 c 106382,0 192621,86239.53 192621,192621.4 v 770457.2 c 0,106382.4 -86239,192621.4 -192621,192621.4 H 192621 C 86239.35,1155700 0,1069461 0,963078.6 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path298" /></g><path
+         d="m 705.5,157.5 h 55 v -38 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path300" /><g
+         id="g302"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,705.5,157.5)"><path
+           d="M 0,0 H 698500 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path304" /></g><g
+         id="g306"
+         transform="translate(728.1074,-230)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text310"><tspan
+             x="0"
+             y="0"
+             id="tspan308">7</tspan></text></g><g
+         id="g312"
+         transform="translate(706.9661,-192)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text316"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan314">设备</tspan></text></g><g
+         id="g318"
+         transform="translate(747.9661,-192)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text322"><tspan
+             x="0"
+             y="0"
+             id="tspan320">3</tspan></text></g><path
+         d="m 762.5,157.5 h 55 v -38 h -55 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path324" /><g
+         id="g326"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,762.5,157.5)"><path
+           d="M 0,0 H 698500 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path328" /></g><g
+         id="g330"
+         transform="translate(780.8671,-230)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text334"><tspan
+             x="0 7.9991999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan332">14</tspan></text></g><path
+         d="m 818.5,157.5 h 56 v -38 h -56 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path336" /><g
+         id="g338"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,818.5,157.5)"><path
+           d="M 0,0 H 711200 V 482600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path340" /></g><g
+         id="g342"
+         transform="translate(837.6268,-230)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text346"><tspan
+             x="0 7.9991999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan344">21</tspan></text></g><path
+         d="M 412.5,237.75 H 445 V 250.5 L 470.5,225 445,199.5 v 12.75 h -32.5 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path348" /><g
+         id="g350"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,412.5,250.5)"><path
+           d="M 0,161924.8 H 412750.6 V 0 L 736600,323850.4 412750.6,647700 V 485775.2 H 0 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path352" /></g><g
+         id="g354"
+         transform="translate(155.3719,-25)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:bold;font-size:20px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Bold;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text358"><tspan
+             x="0 20 40 60"
+             y="0"
+             sodipodi:role="line"
+             id="tspan356">初始状态</tspan></text></g><g
+         id="g360"
+         transform="translate(649.0545,-24)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:bold;font-size:20px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Bold;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text364"><tspan
+             x="0 20 40 60"
+             y="0"
+             sodipodi:role="line"
+             id="tspan362">最终状态</tspan></text></g><g
+         id="g366"
+         transform="translate(507.0483,-290)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text370"><tspan
+             x="0 7.9991999 12.9924 24.9876 29.980801 37.98 42.973202 54.968399 58.9716 66.970802 71.963997 83.959198 88.9524 96.951599"
+             y="0"
+             sodipodi:role="line"
+             id="tspan368">7 = 1 + 2 + 4 </tspan></text></g><g
+         id="g372"
+         transform="translate(609.0483,-290)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text376"><tspan
+             x="0"
+             y="0"
+             id="tspan374">(</tspan></text></g><g
+         id="g378"
+         transform="translate(614.0483,-290)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text382"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan380">分区</tspan></text></g><g
+         id="g384"
+         transform="translate(650.0483,-290)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text388"><tspan
+             x="0"
+             y="0"
+             id="tspan386">1</tspan></text></g><g
+         id="g390"
+         transform="translate(658.0483,-290)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text394"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan392">计算过程</tspan></text></g><g
+         id="g396"
+         transform="translate(730.0483,-290)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text400"><tspan
+             x="0"
+             y="0"
+             id="tspan398">)</tspan></text></g><g
+         id="g402"
+         transform="translate(507.0483,-319)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text406"><tspan
+             x="0 7.9991999 15.9984 21.999599 33.994801 37.998001 45.9972 50.990398 62.9856 67.978798 75.977997 80.971199 92.9664 96.969597 104.9688"
+             y="0"
+             sodipodi:role="line"
+             id="tspan404">14 = 2 + 4 + 8 </tspan></text></g><g
+         id="g408"
+         transform="translate(617.0483,-319)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text412"><tspan
+             x="0"
+             y="0"
+             id="tspan410">(</tspan></text></g><g
+         id="g414"
+         transform="translate(622.0483,-319)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text418"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan416">分区</tspan></text></g><g
+         id="g420"
+         transform="translate(658.0483,-319)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text424"><tspan
+             x="0"
+             y="0"
+             id="tspan422">2</tspan></text></g><g
+         id="g426"
+         transform="translate(666.0483,-319)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text430"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan428">计算过程</tspan></text></g><g
+         id="g432"
+         transform="translate(738.0483,-319)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text436"><tspan
+             x="0"
+             y="0"
+             id="tspan434">)</tspan></text></g><g
+         id="g438"
+         transform="translate(506.679,-347)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text442"><tspan
+             x="0 7.9991999 15.9984 21.999599 33.994801 37.998001 45.9972 50.990398 62.9856 67.978798 75.977997 80.971199 92.9664 96.969597 104.9688 112.968"
+             y="0"
+             sodipodi:role="line"
+             id="tspan440">21 = 3 + 6 + 12 </tspan></text></g><g
+         id="g444"
+         transform="translate(625.679,-347)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text448"><tspan
+             x="0"
+             y="0"
+             id="tspan446">(</tspan></text></g><g
+         id="g450"
+         transform="translate(630.679,-347)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text454"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan452">分区</tspan></text></g><g
+         id="g456"
+         transform="translate(666.679,-347)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text460"><tspan
+             x="0"
+             y="0"
+             id="tspan458">3</tspan></text></g><g
+         id="g462"
+         transform="translate(674.679,-347)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text466"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan464">计算过程</tspan></text></g><g
+         id="g468"
+         transform="translate(746.679,-347)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text472"><tspan
+             x="0"
+             y="0"
+             id="tspan470">)</tspan></text></g><g
+         id="g474"
+         transform="translate(405.2823,-100)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text478"><tspan
+             x="0 12.0006 16.999201 21.997801 28.9944 36.999001 45.993599 54.988201 61.984798"
+             y="0"
+             sodipodi:role="line"
+             id="tspan476">Allreduce</tspan></text></g><g
+         id="g480"
+         transform="translate(12.3878,-284)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text484"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan482">分区</tspan></text></g><g
+         id="g486"
+         transform="translate(48.38779,-284)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text490"><tspan
+             x="0"
+             y="0"
+             id="tspan488">1</tspan></text></g><g
+         id="g492"
+         transform="translate(67.5878,-284)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text496"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan494">分区</tspan></text></g><g
+         id="g498"
+         transform="translate(103.5878,-284)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text502"><tspan
+             x="0"
+             y="0"
+             id="tspan500">2</tspan></text></g><g
+         id="g504"
+         transform="translate(127.5878,-284)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text508"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan506">分区</tspan></text></g><g
+         id="g510"
+         transform="translate(163.5878,-284)"><text
+           transform="matrix(1,0,0,-1,0,364)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text514"><tspan
+             x="0"
+             y="0"
+             id="tspan512">3</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-averaged-gradient.pdf
+++ b/img/ch09/ch10-averaged-gradient.pdf
--- a/img/ch09/ch10-averaged-gradient.svg
+++ b/img/ch09/ch10-averaged-gradient.svg
@@ -0,0 +1,435 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="850.44928"
+   height="374.93619"
+   viewBox="0 0 850.44928 374.93618"
+   sodipodi:docname="ch10-averaged-gradient.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:xlink="http://www.w3.org/1999/xlink"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 650 V 298 H 0 Z"
+         id="path16" /></clipPath><mask
+       maskUnits="userSpaceOnUse"
+       x="0"
+       y="0"
+       width="1"
+       height="1"
+       id="mask54"><image
+         width="1"
+         height="1"
+         preserveAspectRatio="none"
+         xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD4AAAAqCAAAAADO7bcVAAAAAXNCSVQI5gpbmQAAAllJREFUSIntk91PE0EUxc98dNquGygtRRCKlUIBwWAlgpJoYuKDD/7HxhejIiCJoPIVIQKl9LtYaGl3Z2d8AIzdFWt87nmavZPfPWfuzAIddXSd6L9ucAaAUoBw8qsYjAeuwZmLvt9Tt1k80KRmqta4Ks6kzGNNDSgPzls/1cl84pUVTmaVEdUr8iJgZFzl+uCfriw32uGHQ4PhPNPL1mA4c2nmnyHfqekb6ya9Gbe/C4e1+qOkxbklHRCulQOIe+Za31Zj0jxcqnnSt+IMxLfZFF1VDYmpWPBoy+Gj/R+K9qNKIr1U8xy9dXRsJNL/OJDvG9ssK1v4BZp53Br+WFTW+MD5Yl23wWkocONmQc6m9zTDadennawkYveMReeNnDlsSodS2tqDtDajQ0+++e0IThXpDu29lwAlYjSZ2zgzJuIBp3ZW2an+3qD17I5WKnOsggvrVb5g7imAiViSrxZs1NfTzwUzKs0We/fkoRuWRmnoK+89yivQxHCdhSbGOLcdPzt/i1wTf8UByn2FVPo225YAygeY52/lSOp1WbzMZ6Xr6rw4i0xs55pz4aOiAlRJc02YZoRzBku1ezaE3QmtF8n+Q73hAIAGEJxVITHVpH5vUhdOe3lsMUMHxhWd+5yVF271ZZlIrVV80bY4j1sr2UBy9ORddPppPZc+cEA4M6VQhu3j3t/ehdtfRPVu7HQpa5UKD6IDFQJAiJhDd3tCjFLixt0FMWkU83VHAURErbIEfM/wRgKAeLG/Jr35XXE4u2p5YUZ7LkdGQn+45fYinkVHHXX0n/oJ/lnwjtc5nYUAAAAASUVORK5CYII="
+         id="image56" /></mask><mask
+       maskUnits="userSpaceOnUse"
+       x="0"
+       y="0"
+       width="1"
+       height="1"
+       id="mask68"><image
+         width="1"
+         height="1"
+         preserveAspectRatio="none"
+         xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACIAAAAqCAAAAADzf9ZzAAAAAXNCSVQI5gpbmQAAAHVJREFUOI1jYBgFtAOM6Hx2hp//UYWY0bhCNjwv/6GKMaGqkDYTY0G3CEUJI6/YmS8YbkHR8//jBeb/GEpQLfr/B1MFmhKsYMCUMDKihze6EiYuFm5WNCWoEcAkqcPGyv3qL6rBaKYwMzL8+0PYfaNgFNAWAAAFIhHYGrcC+QAAAABJRU5ErkJggg=="
+         id="image70" /></mask><mask
+       maskUnits="userSpaceOnUse"
+       x="0"
+       y="0"
+       width="1"
+       height="1"
+       id="mask112"><image
+         width="1"
+         height="1"
+         preserveAspectRatio="none"
+         xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD0AAAAqCAAAAAAl2gwWAAAAAXNCSVQI5gpbmQAAAlZJREFUSIntk1tPU0EUhddcOr14AqWlCEJrpVBAMFiJoCSamPjgg//Y+GJQBCQRVG4RIlBK78VCS885c2Z8AAw9JZQf0PU0s5Nvr32ZATrqyC16xzhnACgFCCf/g/6472aaueCnPXWbxX0mNVK1xlVwOmUcaxqActO8+apO5hIfrFAyqwIRvSIvyguPqVwfvFOV5UYb+nBoMJRnetkaDGUurbzT5A81PKPdpDejbqVhrf4taXFuSQeEa+UA4omx1rfVmDAOl2ru0ptpBuLZNEVXVUNiMuo/2nL4SP/Xov2ikkgv1dxtN0+NDYf7X/ryfaObZWULr4CZx4PYt6KyxgbOF+v6dpoGfffuF+RMek8znHZ938lKInbPWGQukDNihnQopU0pSHMuOvTqt9cO41SR7uDeFwlQIkaSuY2zwHjc59TOKjvVa3xz345WKnOs/PPrVT5v7CmAiWiSrxZs1NfTbwULVMzr5u6ZQzcsjdLQL957lFegiVidBcdHObcdLztfQM7EbTRAuaeQSj9k2xJA+QBzfEEOpz6Wxft8VjbvrJVm4fHtnDkbOioqQJU014RpRjhnsFSb10LYo+B6kew/1xsOAGgA/hkVFJMm9bY4uWjay6OLGTowpujsj6y88Kovy0RqreKJtKN53FrJ+pIjJ58jU6/rufSBA8KZIYUK2B7e8s1dtP1TVB9HT5eyVqnwLDJQIQCEiDp0tyfIKCUu2n0XE4Fivu4ogIiIVZaA5w0+SQAQ7/bXZEvxrmI4u8p4YUV7LqdFgjest61Iy6Gjjjq6s/4BUyfwjnJesYgAAAAASUVORK5CYII="
+         id="image114" /></mask><mask
+       maskUnits="userSpaceOnUse"
+       x="0"
+       y="0"
+       width="1"
+       height="1"
+       id="mask126"><image
+         width="1"
+         height="1"
+         preserveAspectRatio="none"
+         xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACEAAAAqCAAAAAAYSG1wAAAAAXNCSVQI5gpbmQAAALlJREFUOI3tkT0LwjAURW/atNShCB2UWhBaiovgor/AyR8vDm4OOih+IFSHgA4JjYNL7tBOuvUuIS+H8x4vQJdfx+erF0Q+LJUEA6NZ3942ymXY0Zufd7qMT7VTk6TIjgdTpWmomwhUSsO8Y1fBRP20QJhcDYnJYQFZ6m0LAXhpvn5YNEckq1y2vEPEy0LCoxVwl2hxv9hgULgIKeV0qCaQ2R5NRJjVY0CIlzsq/YsIvqehlXXp8ud8ABu/La4jm0WAAAAAAElFTkSuQmCC"
+         id="image128" /></mask><mask
+       maskUnits="userSpaceOnUse"
+       x="0"
+       y="0"
+       width="1"
+       height="1"
+       id="mask212"><image
+         width="1"
+         height="1"
+         preserveAspectRatio="none"
+         xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD0AAAAqCAAAAAAl2gwWAAAAAXNCSVQI5gpbmQAAAlZJREFUSIntk1tPU0EUhddcOr14AqWlCEJrpVBAMFiJoCSamPjgg//Y+GJQBCQRVG4RIlBK78VCS885c2Z8AAw9JZQf0PU0s5Nvr32ZATrqyC16xzhnACgFCCf/g/6472aaueCnPXWbxX0mNVK1xlVwOmUcaxqActO8+apO5hIfrFAyqwIRvSIvyguPqVwfvFOV5UYb+nBoMJRnetkaDGUurbzT5A81PKPdpDejbqVhrf4taXFuSQeEa+UA4omx1rfVmDAOl2ru0ptpBuLZNEVXVUNiMuo/2nL4SP/Xov2ikkgv1dxtN0+NDYf7X/ryfaObZWULr4CZx4PYt6KyxgbOF+v6dpoGfffuF+RMek8znHZ938lKInbPWGQukDNihnQopU0pSHMuOvTqt9cO41SR7uDeFwlQIkaSuY2zwHjc59TOKjvVa3xz345WKnOs/PPrVT5v7CmAiWiSrxZs1NfTbwULVMzr5u6ZQzcsjdLQL957lFegiVidBcdHObcdLztfQM7EbTRAuaeQSj9k2xJA+QBzfEEOpz6Wxft8VjbvrJVm4fHtnDkbOioqQJU014RpRjhnsFSb10LYo+B6kew/1xsOAGgA/hkVFJMm9bY4uWjay6OLGTowpujsj6y88Kovy0RqreKJtKN53FrJ+pIjJ58jU6/rufSBA8KZIYUK2B7e8s1dtP1TVB9HT5eyVqnwLDJQIQCEiDp0tyfIKCUu2n0XE4Fivu4ogIiIVZaA5w0+SQAQ7/bXZEvxrmI4u8p4YUV7LqdFgjest61Iy6Gjjjq6s/4BUyfwjnJesYgAAAAASUVORK5CYII="
+         id="image214" /></mask><mask
+       maskUnits="userSpaceOnUse"
+       x="0"
+       y="0"
+       width="1"
+       height="1"
+       id="mask226"><image
+         width="1"
+         height="1"
+         preserveAspectRatio="none"
+         xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACEAAAAqCAAAAAAYSG1wAAAAAXNCSVQI5gpbmQAAAMBJREFUOI3tkcsKwjAQRScm0YLFjbgoIhYUxAf4/7/gB7ioC+sDRASLpRYSp43bmUILgsveXe6c3JlMAFr9W6JylErgp4EQg02gb7ucepIReptGLnw/HfEUi9D3K+Zhxm7xDHyhnpsDUq/DCFd2V2ur2XA8A+QiyKajc1FPuDS+eONHRkblXQCMNTH4tE2VAIDC5a6WkBJADk1SEo/tQ87UEf1wz3bKCOgtJ0k/OtGnVP5FeV1rsYQmicZqq1a/6wsvajoAU7Q9BQAAAABJRU5ErkJggg=="
+         id="image228" /></mask></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:snap-text-baseline="false"
+     inkscape:zoom="0.76211537"
+     inkscape:cx="667.22182"
+     inkscape:cy="115.46808"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g22" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-averaged-gradient"
+     transform="matrix(1.3333333,0,0,-1.3333333,0,396.99999)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="M 93,274.3332 C 93,277.4629 95.53709,280 98.66675,280 H 167.3333 C 170.4629,280 173,277.4629 173,274.3332 V 251.6668 C 173,248.5371 170.4629,246 167.3333,246 H 98.66675 C 95.53709,246 93,248.5371 93,251.6668 Z"
+         style="fill:#def2eb;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="translate(114.6859,-40)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text30"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan28">模型</tspan></text></g><path
+         d="m 222,274.3332 c 0,3.1297 2.5371,5.6668 5.6668,5.6668 h 80.6664 C 311.4629,280 314,277.4629 314,274.3332 V 251.6668 C 314,248.5371 311.4629,246 308.3332,246 H 227.6668 C 224.5371,246 222,248.5371 222,251.6668 Z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path32" /><g
+         id="g34"
+         transform="translate(232.3546,-40)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text38"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan36">本地梯度</tspan></text></g><path
+         d="m 380,274.3333 c 0,3.1296 2.5371,5.6667 5.6667,5.6667 H 558.3333 C 561.4629,280 564,277.4629 564,274.3333 V 251.6667 C 564,248.5371 561.4629,246 558.3333,246 H 385.6667 C 382.5371,246 380,248.5371 380,251.6667 Z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path40" /><g
+         id="g42"
+         transform="translate(435.5474,-40)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text46"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan44">平均梯度</tspan></text></g><path
+         d="m 173.5,263 40.8545,-10e-5 v -1 L 173.5,262 Z m 39.8545,2.4999 10,-3 -10,-3 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path48" /><path
+         d="m 314.5,263 56.2069,-10e-5 v -1 L 314.5,262 Z m 55.2069,2.4999 10,-3 -10,-3 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path50" /><g
+         id="g52"
+         transform="matrix(62,0,0,42,0,240)"><image
+           width="1"
+           height="1"
+           preserveAspectRatio="none"
+           transform="matrix(1,0,0,-1,0,1)"
+           xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD4AAAAqCAYAAADrhujJAAAABHNCSVQICAgIfAhkiAAAAEtJREFUaIHtzzEBACAMwLCBf88gY0caBe2ZmTegux2wpXFN45rGNY1rGtc0rmlc07imcU3jmsY1jWsa1zSuaVzTuKZxTeOaxjWNaz6MiQFTg6TksQAAAABJRU5ErkJggg=="
+           mask="url(#mask54)"
+           id="image58" /></g><g
+         id="g60"
+         transform="translate(12.42319,-39)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan62">设备</tspan></text></g><g
+         id="g66"
+         transform="matrix(34,0,0,42,36,240)"><image
+           width="1"
+           height="1"
+           preserveAspectRatio="none"
+           transform="matrix(1,0,0,-1,0,1)"
+           xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACIAAAAqCAYAAADWFImvAAAABHNCSVQICAgIfAhkiAAAAEFJREFUWIXtzjEBwCAQALEH/57bieUMwJAoyJqZbx6wbwcOkRIpkRIpkRIpkRIpkRIpkRIpkRIpkRIpkRIpkRKpHyeOAVPaBYvcAAAAAElFTkSuQmCC"
+           mask="url(#mask68)"
+           id="image72" /></g><g
+         id="g74"
+         transform="translate(48.42319,-39)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text78"><tspan
+             x="0"
+             y="0"
+             id="tspan76">1</tspan></text></g><path
+         d="m 564.662,262.6276 h 16.285 l -0.25,-0.25 V 297.5 l 0.25,-0.25 H 133.5 l 0.25,0.25 v -9 h -0.5 v 9.25 H 581.197 V 262.1276 H 564.662 Z M 136.5,289.5 l -3,-10 -3,10 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path80" /><path
+         d="m 96,206.1666 c 0,3.2217 2.61171,5.8334 5.8334,5.8334 h 68.3332 C 173.3883,212 176,209.3883 176,206.1666 V 182.8334 C 176,179.6117 173.3883,177 170.1666,177 H 101.8334 C 98.61171,177 96,179.6117 96,182.8334 Z"
+         style="fill:#def2eb;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path82" /><g
+         id="g84"
+         transform="translate(117.7066,-109)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text88"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan86">模型</tspan></text></g><path
+         d="m 225,206.1666 c 0,3.2217 2.6117,5.8334 5.8334,5.8334 h 80.3332 C 314.3883,212 317,209.3883 317,206.1666 V 182.8334 C 317,179.6117 314.3883,177 311.1666,177 H 230.8334 C 227.6117,177 225,179.6117 225,182.8334 Z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path90" /><g
+         id="g92"
+         transform="translate(235.3753,-109)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text96"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan94">本地梯度</tspan></text></g><path
+         d="m 383,206.1665 c 0,3.2217 2.6118,5.8335 5.8335,5.8335 h 172.333 c 3.2218,0 5.8335,-2.6118 5.8335,-5.8335 v -23.333 C 567,179.6118 564.3883,177 561.1665,177 H 388.8335 C 385.6118,177 383,179.6118 383,182.8335 Z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path98" /><g
+         id="g100"
+         transform="translate(438.568,-109)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text104"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan102">平均梯度</tspan></text></g><path
+         d="m 176.5,195 40.8545,-1e-4 v -1 L 176.5,194 Z m 39.8545,2.4999 10,-3 -10,-3 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path106" /><path
+         d="m 317.5,195 56.2069,-1e-4 v -1 L 317.5,194 Z m 55.2069,2.4999 10,-3 -10,-3 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path108" /><g
+         id="g110"
+         transform="matrix(61,0,0,42,1,172)"><image
+           width="1"
+           height="1"
+           preserveAspectRatio="none"
+           transform="matrix(1,0,0,-1,0,1)"
+           xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD0AAAAqCAYAAAAAsVPKAAAABHNCSVQICAgIfAhkiAAAAEtJREFUaIHtzzEBACAMwLCBf88gY0caBe2ZmTeYux2woWlF04qmFU0rmlY0rWha0bSiaUXTiqYVTSuaVjStaFrRtKJpRdOKphVNKz7I6gFTPvQz4AAAAABJRU5ErkJggg=="
+           mask="url(#mask112)"
+           id="image116" /></g><g
+         id="g118"
+         transform="translate(12.96098,-108)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text122"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan120">设备</tspan></text></g><g
+         id="g124"
+         transform="matrix(33,0,0,42,37,172)"><image
+           width="1"
+           height="1"
+           preserveAspectRatio="none"
+           transform="matrix(1,0,0,-1,0,1)"
+           xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACEAAAAqCAYAAAA9IzKsAAAABHNCSVQICAgIfAhkiAAAAD5JREFUWIXtzkERgDAABLED/57LawcJ/SQK8mw7u+y9HdgkfhKRiEQkIhGJSEQiEpGIRCQiEYlIRCISkYhEPmPvAVNUz9k9AAAAAElFTkSuQmCC"
+           mask="url(#mask126)"
+           id="image130" /></g><g
+         id="g132"
+         transform="translate(48.96098,-108)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text136"><tspan
+             x="0"
+             y="0"
+             id="tspan134">2</tspan></text></g><path
+         d="m 567.662,194.6276 h 16.285 l -0.25,-0.25 V 229.5 l 0.25,-0.25 H 136.5 l 0.25,0.25 v -9 h -0.5 v 9.25 H 584.197 V 194.1276 H 567.662 Z M 139.5,221.5 l -3,-10 -3,10 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path138" /><g
+         id="g140"
+         transform="translate(359.2237,-235)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text144"><tspan
+             x="0 18 36 54 72 90 108 126"
+             y="0"
+             sodipodi:role="line"
+             id="tspan142">计算平均梯度步骤</tspan></text></g><g
+         id="g146"
+         transform="translate(503.2237,-235)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text150"><tspan
+             x="0"
+             y="0"
+             id="tspan148">：</tspan></text></g><g
+         id="g152"
+         transform="translate(359.2237,-257)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text156"><tspan
+             x="0 7.9991999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan154">1.</tspan></text></g><g
+         id="g158"
+         transform="translate(386.2237,-257)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text162"><tspan
+             x="0 18 36 54 72 90"
+             y="0"
+             sodipodi:role="line"
+             id="tspan160">收集本地梯度</tspan></text></g><g
+         id="g164"
+         transform="translate(494.2237,-257)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text168"><tspan
+             x="0 18 36 54 72 90 108 126"
+             y="0"
+             sodipodi:role="line"
+             id="tspan166">，并计算平均梯度</tspan></text></g><g
+         id="g170"
+         transform="translate(359.2237,-279)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text174"><tspan
+             x="0 7.9991999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan172">2.</tspan></text></g><g
+         id="g176"
+         transform="translate(386.2237,-279)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text180"><tspan
+             x="0 18 36 54 72 90 108 126 144 162 180"
+             sodipodi:role="line"
+             id="tspan178"
+             y="0">⼴播平局梯度到全部设备</tspan></text></g><path
+         d="m 96,131.1666 c 0,3.2217 2.61171,5.8334 5.8334,5.8334 h 68.3332 C 173.3883,137 176,134.3883 176,131.1666 V 107.8334 C 176,104.6117 173.3883,102 170.1666,102 H 101.8334 C 98.61171,102 96,104.6117 96,107.8334 Z"
+         style="fill:#def2eb;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path182" /><g
+         id="g184"
+         transform="translate(117.7066,-184)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text188"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan186">模型</tspan></text></g><path
+         d="m 225,131.1666 c 0,3.2217 2.6117,5.8334 5.8334,5.8334 h 80.3332 C 314.3883,137 317,134.3883 317,131.1666 V 107.8334 C 317,104.6117 314.3883,102 311.1666,102 H 230.8334 C 227.6117,102 225,104.6117 225,107.8334 Z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path190" /><g
+         id="g192"
+         transform="translate(235.3754,-184)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text196"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan194">本地梯度</tspan></text></g><path
+         d="m 383,131.1665 c 0,3.2217 2.6118,5.8335 5.8335,5.8335 h 172.333 c 3.2218,0 5.8335,-2.6118 5.8335,-5.8335 v -23.333 C 567,104.6118 564.3883,102 561.1665,102 H 388.8335 C 385.6118,102 383,104.6118 383,107.8335 Z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path198" /><g
+         id="g200"
+         transform="translate(438.5681,-184)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text204"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan202">平均梯度</tspan></text></g><path
+         d="m 176.5,120 40.8545,-1e-4 v -1 L 176.5,119 Z m 39.8545,2.4999 10,-3 -10,-3 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path206" /><path
+         d="m 317.5,120 56.2069,-1e-4 v -1 L 317.5,119 Z m 55.2069,2.4999 10,-3 -10,-3 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path208" /><g
+         id="g210"
+         transform="matrix(61,0,0,42,1,97)"><image
+           width="1"
+           height="1"
+           preserveAspectRatio="none"
+           transform="matrix(1,0,0,-1,0,1)"
+           xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD0AAAAqCAYAAAAAsVPKAAAABHNCSVQICAgIfAhkiAAAAEtJREFUaIHtzzEBACAMwLCBf88gY0caBe2ZmTeYux2woWlF04qmFU0rmlY0rWha0bSiaUXTiqYVTSuaVjStaFrRtKJpRdOKphVNKz7I6gFTPvQz4AAAAABJRU5ErkJggg=="
+           mask="url(#mask212)"
+           id="image216" /></g><g
+         id="g218"
+         transform="translate(12.96106,-183)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text222"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan220">设备</tspan></text></g><g
+         id="g224"
+         transform="matrix(33,0,0,42,37,97)"><image
+           width="1"
+           height="1"
+           preserveAspectRatio="none"
+           transform="matrix(1,0,0,-1,0,1)"
+           xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACEAAAAqCAYAAAA9IzKsAAAABHNCSVQICAgIfAhkiAAAAD5JREFUWIXtzkERgDAABLED/57LawcJ/SQK8mw7u+y9HdgkfhKRiEQkIhGJSEQiEpGIRCQiEYlIRCISkYhEPmPvAVNUz9k9AAAAAElFTkSuQmCC"
+           mask="url(#mask226)"
+           id="image230" /></g><g
+         id="g232"
+         transform="translate(48.96106,-183)"><text
+           transform="matrix(1,0,0,-1,0,298)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text236"><tspan
+             x="0"
+             y="0"
+             id="tspan234">3</tspan></text></g><path
+         d="m 567.662,119.6276 h 16.285 l -0.25,-0.25 V 154.5 l 0.25,-0.25 H 136.5 l 0.25,0.25 v -9 h -0.5 v 9.25 H 584.197 V 119.1276 H 567.662 Z M 139.5,146.5 l -3,-10 -3,10 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path238" /><g
+         id="g240"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,348,263)"><path
+           d="M 0,0 1,1821273"
+           style="fill:none;stroke:#2e75b6;stroke-width:25400;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path242" /></g><path
+         d="m 384.6971,82.14824 -29.7662,46.50216 0.8423,0.5391 29.7661,-46.50214 z M 353.3645,126.4603 350.5,136.5 l 7.9179,-6.805 z"
+         style="fill:#ff0000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path244" /><path
+         d="m 344,262.5 c 0,1.933 1.567,3.5 3.5,3.5 1.933,0 3.5,-1.567 3.5,-3.5 0,-1.933 -1.567,-3.5 -3.5,-3.5 -1.933,0 -3.5,1.567 -3.5,3.5 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path246" /><path
+         d="m 345,194.5 c 0,1.933 1.567,3.5 3.5,3.5 1.933,0 3.5,-1.567 3.5,-3.5 0,-1.933 -1.567,-3.5 -3.5,-3.5 -1.933,0 -3.5,1.567 -3.5,3.5 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path248" /><path
+         d="m 344,119.5 c 0,1.933 1.567,3.5 3.5,3.5 1.933,0 3.5,-1.567 3.5,-3.5 0,-1.933 -1.567,-3.5 -3.5,-3.5 -1.933,0 -3.5,1.567 -3.5,3.5 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path250" /></g></g></svg>
--- a/img/ch09/ch10-computation-increase.pdf
+++ b/img/ch09/ch10-computation-increase.pdf
--- a/img/ch09/ch10-computation-increase.svg
+++ b/img/ch09/ch10-computation-increase.svg
--- a/img/ch09/ch10-data-parallel.pdf
+++ b/img/ch09/ch10-data-parallel.pdf
--- a/img/ch09/ch10-data-parallel.svg
+++ b/img/ch09/ch10-data-parallel.svg
@@ -0,0 +1,539 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="929.85919"
+   height="522.151"
+   viewBox="0 0 929.85919 522.151"
+   sodipodi:docname="ch10-data-parallel.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 709 V 414 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.9210322"
+     inkscape:cx="413.12345"
+     inkscape:cy="275.77755"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-data-parallel"
+     transform="matrix(1.3333333,0,0,-1.3333333,-15.477411,541.54816)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="m 84.5,352.3325 c 0,13.8996 11.26785,25.1675 25.1675,25.1675 h 573.6651 c 13.8996,0 25.1674,-11.2679 25.1674,-25.1675 V 251.6675 C 708.5,237.7678 697.2322,226.5 683.3326,226.5 H 109.6675 C 95.76785,226.5 84.5,237.7678 84.5,251.6675 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,84.5,377.5)"><path
+           d="M 0,319626.8 C 0,143101.8 143101.7,0 319626.7,0 H 7605174 c 176524,0 319626,143101.8 319626,319626.8 V 1598073 c 0,176525 -143102,319627 -319626,319627 H 319626.7 C 143101.7,1917700 0,1774598 0,1598073 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path28" /></g><path
+         d="m 472.5,368.5 h 72 v -90 h -72 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path30" /><g
+         id="g32"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,472.5,368.5)"><path
+           d="M 0,0 H 914400 V 1143000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path34" /></g><path
+         d="m 608.5,369.5 h 72 v -90 h -72 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path36" /><g
+         id="g38"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,608.5,369.5)"><path
+           d="M 0,0 H 914400 V 1143000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path40" /></g><path
+         d="m 337.5,368.5 h 73 v -90 h -73 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path42" /><g
+         id="g44"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,337.5,368.5)"><path
+           d="M 0,0 H 927100 V 1143000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path46" /></g><path
+         d="m 106.5,365.5 h 113 v -86 h -113 z"
+         style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path48" /><g
+         id="g50"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,106.5,365.5)"><path
+           d="M 0,0 H 1435100 V 1092200 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path52" /></g><g
+         id="g54"
+         transform="translate(351.0344,-97)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text58"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan56">算⼦</tspan></text></g><g
+         id="g60"
+         transform="translate(387.0344,-97)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0"
+             y="0"
+             id="tspan62">1</tspan></text></g><g
+         id="g66"
+         transform="translate(485.2252,-97)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text70"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan68">算⼦</tspan></text></g><g
+         id="g72"
+         transform="translate(521.2252,-97)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text76"><tspan
+             x="0"
+             y="0"
+             id="tspan74">2</tspan></text></g><g
+         id="g78"
+         transform="translate(620.965,-97)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text82"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan80">算⼦</tspan></text></g><g
+         id="g84"
+         transform="translate(656.965,-97)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text88"><tspan
+             x="0"
+             y="0"
+             id="tspan86">3</tspan></text></g><g
+         id="g90"
+         transform="translate(12.20717,-118)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text94"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan92">设备</tspan></text></g><g
+         id="g96"
+         transform="translate(48.20716,-118)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text100"><tspan
+             x="0"
+             y="0"
+             id="tspan98">1</tspan></text></g><path
+         d="m 328.5,271.5 h 95 v -36 h -95 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path102" /><g
+         id="g104"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,328.5,271.5)"><path
+           d="M 0,0 H 1206500 V 457200 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path106" /></g><g
+         id="g108"
+         transform="translate(352.6288,-167)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text112"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan110">梯度</tspan></text></g><g
+         id="g114"
+         transform="translate(388.6288,-167)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text118"><tspan
+             x="0"
+             y="0"
+             id="tspan116">1</tspan></text></g><path
+         d="m 462.5,270.5 h 96 v -37 h -96 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path120" /><g
+         id="g122"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,462.5,270.5)"><path
+           d="M 0,0 H 1219200 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path124" /></g><g
+         id="g126"
+         transform="translate(487.2839,-167)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text130"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan128">梯度</tspan></text></g><g
+         id="g132"
+         transform="translate(523.2839,-167)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text136"><tspan
+             x="0"
+             y="0"
+             id="tspan134">2</tspan></text></g><path
+         d="m 595.5,269.5 h 95 v -37 h -95 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path138" /><g
+         id="g140"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,595.5,269.5)"><path
+           d="M 0,0 H 1206500 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path142" /></g><g
+         id="g144"
+         transform="translate(620.9015,-169)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text148"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan146">梯度</tspan></text></g><g
+         id="g150"
+         transform="translate(656.9015,-169)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text154"><tspan
+             x="0"
+             y="0"
+             id="tspan152">3</tspan></text></g><path
+         d="m 106.5,366.5 h 113 v -42 h -113 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path156" /><g
+         id="g158"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,106.5,366.5)"><path
+           d="M 0,0 H 1435100 V 533400 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path160" /></g><g
+         id="g162"
+         transform="translate(124.4777,-74)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text166"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan164">数据分区</tspan></text></g><g
+         id="g168"
+         transform="translate(196.4777,-74)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text172"><tspan
+             x="0"
+             y="0"
+             id="tspan170">1</tspan></text></g><path
+         d="m 221.5,344.75 105.4246,-10e-5 v -4.5 L 221.5,340.25 Z m 103.1746,4.4999 13.5,-6.75 -13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path174" /><path
+         d="m 410.5,344.75 51.1478,-10e-5 v -4.5 L 410.5,340.25 Z m 48.8978,4.4999 13.5,-6.75 -13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path176" /><path
+         d="m 544.5,344.75 51.1478,-10e-5 v -4.5 L 544.5,340.25 Z m 48.8978,4.4999 13.5,-6.75 -13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path178" /><path
+         d="M 604.8978,300.2499 553.75,300.25 v 4.5 l 51.1478,-10e-5 z M 556,295.75 l -13.5,6.75 13.5,6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path180" /><path
+         d="M 469.8978,298.2499 418.75,298.25 v 4.5 l 51.1478,-10e-5 z M 421,293.75 l -13.5,6.75 13.5,6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path182" /><path
+         d="m 83.5,164.166 c 0,13.9916 11.3424,25.334 25.334,25.334 h 573.3321 c 13.9915,0 25.3339,-11.3424 25.3339,-25.334 V 62.83402 c 0,-13.99159 -11.3424,-25.33401 -25.3339,-25.33401 H 108.834 c -13.9916,0 -25.334,11.34242 -25.334,25.33401 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path184" /><g
+         id="g186"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,83.5,189.5)"><path
+           d="M 0,321742 C 0,144048.8 144048.5,0 321741.3,0 H 7603059 c 177693,0 321741,144048.8 321741,321742 v 1286916 c 0,177693 -144048,321742 -321741,321742 H 321741.3 C 144048.5,1930400 0,1786351 0,1608658 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path188" /></g><path
+         d="m 471.5,137.5 h 72 v -90 h -72 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path190" /><g
+         id="g192"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,471.5,137.5)"><path
+           d="M 0,0 H 914400 V 1143000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path194" /></g><path
+         d="m 607.5,137.5 h 72 v -89 h -72 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path196" /><g
+         id="g198"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,607.5,137.5)"><path
+           d="M 0,0 H 914400 V 1130300 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path200" /></g><path
+         d="m 336.5,137.5 h 73 v -90 h -73 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path202" /><g
+         id="g204"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,336.5,137.5)"><path
+           d="M 0,0 H 927100 V 1143000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path206" /></g><path
+         d="m 105.5,134.5 h 113 v -86 h -113 z"
+         style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path208" /><g
+         id="g210"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,105.5,134.5)"><path
+           d="M 0,0 H 1435100 V 1092200 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path212" /></g><g
+         id="g214"
+         transform="translate(348.7747,-328)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text218"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan216">算⼦</tspan></text></g><g
+         id="g220"
+         transform="translate(384.7747,-328)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text224"><tspan
+             x="0"
+             y="0"
+             id="tspan222">1</tspan></text></g><g
+         id="g226"
+         transform="translate(485.3655,-328)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text230"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan228">算⼦</tspan></text></g><g
+         id="g232"
+         transform="translate(521.3655,-328)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text236"><tspan
+             x="0"
+             y="0"
+             id="tspan234">2</tspan></text></g><g
+         id="g238"
+         transform="translate(619.9053,-328)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text242"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan240">算⼦</tspan></text></g><g
+         id="g244"
+         transform="translate(655.9053,-328)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text248"><tspan
+             x="0"
+             y="0"
+             id="tspan246">3</tspan></text></g><path
+         d="m 328.5,183.5 h 95 v -37 h -95 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path250" /><g
+         id="g252"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,328.5,183.5)"><path
+           d="M 0,0 H 1206500 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path254" /></g><g
+         id="g256"
+         transform="translate(352.6288,-254)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text260"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan258">梯度</tspan></text></g><g
+         id="g262"
+         transform="translate(388.6288,-254)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text266"><tspan
+             x="0"
+             y="0"
+             id="tspan264">1</tspan></text></g><path
+         d="m 462.5,182.5 h 96 v -37 h -96 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path268" /><g
+         id="g270"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,462.5,182.5)"><path
+           d="M 0,0 H 1219200 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path272" /></g><g
+         id="g274"
+         transform="translate(488.4839,-256)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text278"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan276">梯度</tspan></text></g><g
+         id="g280"
+         transform="translate(524.4839,-256)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text284"><tspan
+             x="0"
+             y="0"
+             id="tspan282">2</tspan></text></g><path
+         d="m 595.5,180.5 h 95 v -37 h -95 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path286" /><g
+         id="g288"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,595.5,180.5)"><path
+           d="M 0,0 H 1206500 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path290" /></g><g
+         id="g292"
+         transform="translate(620.9015,-257)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text296"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan294">梯度</tspan></text></g><g
+         id="g298"
+         transform="translate(656.9015,-257)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text302"><tspan
+             x="0"
+             y="0"
+             id="tspan300">3</tspan></text></g><path
+         d="m 105.5,90.5 h 113 v -42 h -113 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path304" /><g
+         id="g306"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,105.5,90.5)"><path
+           d="M 0,0 H 1435100 V 533400 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path308" /></g><g
+         id="g310"
+         transform="translate(121.0906,-349)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text314"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan312">数据分区</tspan></text></g><g
+         id="g316"
+         transform="translate(193.0906,-349)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text320"><tspan
+             x="0"
+             y="0"
+             id="tspan318">2</tspan></text></g><path
+         d="m 220.5,66.75 105.4246,-7e-5 v -4.5 L 220.5,62.25 Z m 103.1746,4.49993 13.5,-6.75001 -13.5,-6.74999 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path322" /><path
+         d="m 409.5,67.75 51.1478,-6e-5 v -4.5 L 409.5,63.25 Z m 48.8978,4.49994 13.5,-6.75002 -13.5,-6.74998 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path324" /><path
+         d="m 543.5,66.75 51.1478,-6e-5 v -4.5 L 543.5,62.25 Z m 48.8978,4.49994 13.5,-6.75002 -13.5,-6.74998 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path326" /><path
+         d="m 605.8978,109.2499 -51.1478,1e-4 v 4.5 l 51.1478,-1e-4 z M 557,104.75 l -13.5,6.75 13.5,6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path328" /><path
+         d="m 470.8978,108.2499 -51.1478,1e-4 v 4.5 l 51.1478,-1e-4 z M 422,103.75 l -13.5,6.75 13.5,6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path330" /><g
+         id="g332"
+         transform="translate(11.22134,-300)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text336"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan334">设备</tspan></text></g><g
+         id="g338"
+         transform="translate(47.22134,-300)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text342"><tspan
+             x="0"
+             y="0"
+             id="tspan340">2</tspan></text></g><path
+         d="M 377,191.4653 377.0001,228 h -3 L 374,191.4653 Z m -6,1.5 4.5,-9 4.5,9 z m 9.0001,33.5347 -4.5,9 -4.5,-9 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path344" /><path
+         d="M 512,189.4653 512.0001,226 h -3 L 509,189.4653 Z m -6,1.5 4.5,-9 4.5,9 z m 9.0001,33.5347 -4.5,9 -4.5,-9 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path346" /><path
+         d="M 644,188.4653 644.0001,225 h -3 L 641,188.4653 Z m -6,1.5 4.5,-9 4.5,9 z m 9.0001,33.5347 -4.5,9 -4.5,-9 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path348" /><g
+         id="g350"
+         transform="translate(233.9832,-202)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text354"><tspan
+             x="0 12.0006 16.999201 21.997801 28.9944 36.999001 45.993599 54.988201 61.984798"
+             y="0"
+             sodipodi:role="line"
+             id="tspan352">Allreduce</tspan></text></g><g
+         id="g356"
+         transform="translate(205.9832,-224)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text360"><tspan
+             x="0 18 36 54 72 90 108"
+             y="0"
+             sodipodi:role="line"
+             id="tspan358">（计算平均值）</tspan></text></g><g
+         id="g362"
+         transform="translate(466.7052,-23)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text366"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan364">程序副本</tspan></text></g><g
+         id="g368"
+         transform="translate(464.2724,-397)"><text
+           transform="matrix(1,0,0,-1,0,414)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text372"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan370">程序副本</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-datacentre.pdf
+++ b/img/ch09/ch10-datacentre.pdf
--- a/img/ch09/ch10-datacentre.svg
+++ b/img/ch09/ch10-datacentre.svg
--- a/img/ch09/ch10-hybrid-parallel.pdf
+++ b/img/ch09/ch10-hybrid-parallel.pdf
--- a/img/ch09/ch10-hybrid-parallel.svg
+++ b/img/ch09/ch10-hybrid-parallel.svg
@@ -0,0 +1,560 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="1032.3997"
+   height="474.17447"
+   viewBox="0 0 1032.3997 474.17447"
+   sodipodi:docname="ch10-hybrid-parallel.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 799 V 379 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.86352878"
+     inkscape:cx="424.42129"
+     inkscape:cy="372.88856"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-hybrid-parallel"
+     transform="matrix(1.3333333,0,0,-1.3333333,-16.464185,493.54816)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="m 448.5,162.1663 c 0,13.4391 10.8946,24.3337 24.3337,24.3337 h 231.3326 c 13.4391,0 24.3337,-10.8946 24.3337,-24.3337 V 64.83374 C 728.5,51.39459 717.6054,40.5 704.1663,40.5 H 472.8337 C 459.3946,40.5 448.5,51.39459 448.5,64.83374 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,448.5,186.5)"><path
+           d="M 0,309038.5 C 0,138361.3 138361.2,0 309038.3,0 H 3246962 c 170677,0 309038,138361.3 309038,309038.5 V 1545161 c 0,170678 -138361,309039 -309038,309039 H 309038.3 C 138361.2,1854200 0,1715839 0,1545161 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path28" /></g><path
+         d="m 451.5,322.4996 c 0,11.5982 9.4022,21.0004 21.0003,21.0004 h 237.9994 c 11.5981,0 21.0003,-9.4022 21.0003,-21.0004 V 238.5004 C 731.5,226.9022 722.0978,217.5 710.4997,217.5 H 472.5003 c -11.5981,0 -21.0003,9.4022 -21.0003,21.0004 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path30" /><g
+         id="g32"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,451.5,343.5)"><path
+           d="M 0,266704.5 C 0,119407.7 119407.4,0 266703.9,0 H 3289296 c 147297,0 266704,119407.7 266704,266704.5 V 1333495 c 0,147297 -119407,266705 -266704,266705 H 266703.9 C 119407.4,1600200 0,1480792 0,1333495 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path34" /></g><path
+         d="m 83.5,322.4994 c 0,11.5983 9.40227,21.0006 21.0006,21.0006 h 274.9988 c 11.5983,0 21.0006,-9.4023 21.0006,-21.0006 V 238.5006 C 400.5,226.9023 391.0977,217.5 379.4994,217.5 H 104.5006 C 92.90227,217.5 83.5,226.9023 83.5,238.5006 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path36" /><g
+         id="g38"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,83.5,343.5)"><path
+           d="M 0,266707.4 C 0,119408.9 119408.8,0 266707.1,0 H 3759193 c 147298,0 266707,119408.9 266707,266707.4 V 1333493 c 0,147298 -119409,266707 -266707,266707 H 266707.1 C 119408.8,1600200 0,1480791 0,1333493 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path40" /></g><g
+         id="g42"
+         transform="translate(11.96142,-90)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text46"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan44">设备</tspan></text></g><g
+         id="g48"
+         transform="translate(47.96142,-90)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text52"><tspan
+             x="0"
+             y="0"
+             id="tspan50">1</tspan></text></g><path
+         d="m 83.5,163.8329 c 0,13.6233 11.04383,24.6671 24.6671,24.6671 h 265.6658 c 13.6233,0 24.6671,-11.0438 24.6671,-24.6671 V 65.16713 C 398.5,51.54385 387.4562,40.5 373.8329,40.5 H 108.1671 C 94.54383,40.5 83.5,51.54385 83.5,65.16713 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path54" /><g
+         id="g56"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,83.5,188.5)"><path
+           d="M 0,313272.5 C 0,140256.9 140256.6,0 313271.9,0 H 3687228 c 173016,0 313272,140256.9 313272,313272.5 V 1566327 c 0,173016 -140256,313273 -313272,313273 H 313271.9 C 140256.6,1879600 0,1739343 0,1566327 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path58" /></g><g
+         id="g60"
+         transform="translate(12.3048,-283)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan62">设备</tspan></text></g><g
+         id="g66"
+         transform="translate(48.3048,-283)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text70"><tspan
+             x="0"
+             y="0"
+             id="tspan68">3</tspan></text></g><path
+         d="m 316.1082,180.3532 0.5873,40.6259 -2.9997,0.0434 -0.5872,-40.6259 z m -5.9777,1.5866 4.3695,-9.0641 4.6296,8.934 z m 9.543,37.4961 -4.3695,9.0641 -4.6296,-8.934 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path72" /><path
+         d="m 589.0037,181.062 0.0188,37.9373 -3,0.001 -0.0188,-37.9372 3,-0.001 z m -5.9992,1.503 4.4955,-9.0022 4.5045,8.9977 -9,0.005 z m 9.0173,34.9328 -4.4956,9.0022 -4.5044,-8.9978 9,-0.004 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path74" /><g
+         id="g76"
+         transform="translate(320.3829,-183)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text80"><tspan
+             x="0 12.0006 16.0002 19.9998 25.999201 36 46.000801 56.001598 65.001602"
+             y="0"
+             sodipodi:role="line"
+             id="tspan78">Allreduce</tspan></text></g><path
+         d="m 225.5,320.75 26.5683,-10e-5 v -4.5 L 225.5,316.25 Z m 24.3183,4.4999 13.5,-6.75 -13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path82" /><path
+         d="m 367.5,316.75 120.4126,-10e-5 v -4.5 L 367.5,312.25 Z m 118.1626,4.4999 13.5,-6.75 -13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path84" /><path
+         d="m 221.5,72.75 25.147,-5e-5 v -4.5 L 221.5,68.25 Z m 22.897,4.49995 13.5,-6.75003 -13.5,-6.74997 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path86" /><path
+         d="m 363.5,73.75 118.9914,-7e-5 v -4.5 L 363.5,69.25 Z m 116.7413,4.49993 13.5,-6.75001 -13.5,-6.74999 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path88" /><path
+         d="m 263.5,265.5 h 104 v -37 h -104 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path90" /><g
+         id="g92"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,263.5,265.5)"><path
+           d="M 0,0 H 1320800 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path94" /></g><g
+         id="g96"
+         transform="translate(289.9471,-138)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text100"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan98">梯度</tspan></text></g><g
+         id="g102"
+         transform="translate(330.9471,-138)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text106"><tspan
+             x="0"
+             y="0"
+             id="tspan104">1</tspan></text></g><path
+         d="m 528.5,263.5 h 117 v -37 h -117 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path108" /><g
+         id="g110"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,528.5,263.5)"><path
+           d="M 0,0 H 1485900 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path112" /></g><g
+         id="g114"
+         transform="translate(565.3602,-140)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text118"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan116">梯度</tspan></text></g><g
+         id="g120"
+         transform="translate(601.3602,-140)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text124"><tspan
+             x="0"
+             y="0"
+             id="tspan122">2</tspan></text></g><path
+         d="m 264.5,173.5 h 100 v -37 h -100 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path126" /><g
+         id="g128"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,264.5,173.5)"><path
+           d="M 0,0 H 1270000 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path130" /></g><g
+         id="g132"
+         transform="translate(290.6458,-232)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text136"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan134">梯度</tspan></text></g><g
+         id="g138"
+         transform="translate(326.6458,-232)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text142"><tspan
+             x="0"
+             y="0"
+             id="tspan140">1</tspan></text></g><path
+         d="m 528.5,173.5 h 117 v -37 h -117 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path144" /><g
+         id="g146"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,528.5,173.5)"><path
+           d="M 0,0 H 1485900 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path148" /></g><g
+         id="g150"
+         transform="translate(565.3339,-231)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text154"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan152">梯度</tspan></text></g><g
+         id="g156"
+         transform="translate(601.3339,-231)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text160"><tspan
+             x="0"
+             y="0"
+             id="tspan158">2</tspan></text></g><path
+         d="M 499.1625,294.75 378.75,294.7499 v -4.5 l 120.4125,10e-5 z M 381,299.2499 l -13.5,-6.75 13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path162" /><path
+         d="m 493.8602,97.75 -123.1103,-7e-5 v -4.5 l 123.1103,7e-5 z M 373,102.2499 359.5,95.49992 373,88.74993 Z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path164" /><g
+         id="g166"
+         transform="translate(741.5074,-90)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text170"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan168">设备</tspan></text></g><g
+         id="g172"
+         transform="translate(777.5074,-90)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text176"><tspan
+             x="0"
+             y="0"
+             id="tspan174">2</tspan></text></g><g
+         id="g178"
+         transform="translate(741.5073,-283)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text182"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan180">设备</tspan></text></g><g
+         id="g184"
+         transform="translate(777.5073,-283)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text188"><tspan
+             x="0"
+             y="0"
+             id="tspan186">4</tspan></text></g><path
+         d="m 109.5,335.5 h 116 v -63 h -116 z"
+         style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path190" /><g
+         id="g192"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,109.5,335.5)"><path
+           d="M 0,0 H 1473200 V 800100 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path194" /></g><path
+         d="m 109.5,122.5 h 111 v -66 h -111 z"
+         style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path196" /><g
+         id="g198"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,109.5,122.5)"><path
+           d="M 0,0 H 1409700 V 838200 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path200" /></g><path
+         d="m 109.5,86.5 h 111 v -31 h -111 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path202" /><g
+         id="g204"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,109.5,86.5)"><path
+           d="M 0,0 H 1409700 V 393700 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path206" /></g><g
+         id="g208"
+         transform="translate(125.1799,-313)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text212"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan210">数据分区</tspan></text></g><g
+         id="g214"
+         transform="translate(197.1799,-313)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text218"><tspan
+             x="0"
+             y="0"
+             id="tspan216">2</tspan></text></g><path
+         d="m 109.5,335.5 h 116 v -33 h -116 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path220" /><g
+         id="g222"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,109.5,335.5)"><path
+           d="M 0,0 H 1473200 V 419100 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path224" /></g><g
+         id="g226"
+         transform="translate(128.6735,-69)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text230"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan228">数据分区</tspan></text></g><g
+         id="g232"
+         transform="translate(205.6735,-69)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text236"><tspan
+             x="0"
+             y="0"
+             id="tspan234">1</tspan></text></g><g
+         id="g238"
+         transform="translate(406.0215,-42)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text242"><tspan
+             x="0 9 19.000799 29.0016"
+             y="0"
+             sodipodi:role="line"
+             id="tspan240">send</tspan></text></g><path
+         d="m 263.5,335.5 h 104 v -63 h -104 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path244" /><g
+         id="g246"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,263.5,335.5)"><path
+           d="M 0,0 H 1320800 V 800100 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path248" /></g><g
+         id="g250"
+         transform="translate(289.1145,-81)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text254"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan252">算⼦</tspan></text></g><g
+         id="g256"
+         transform="translate(325.1145,-81)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text260"><tspan
+             x="0"
+             y="0"
+             id="tspan258">1</tspan></text></g><path
+         d="m 262.5,122.5 h 100 v -66 h -100 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path262" /><g
+         id="g264"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,262.5,122.5)"><path
+           d="M 0,0 H 1270000 V 838200 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path266" /></g><g
+         id="g268"
+         transform="translate(286.2876,-297)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text272"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan270">算⼦</tspan></text></g><g
+         id="g274"
+         transform="translate(322.2876,-297)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text278"><tspan
+             x="0"
+             y="0"
+             id="tspan276">1</tspan></text></g><path
+         d="m 499.5,334.5 h 178 v -62 h -178 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path280" /><g
+         id="g282"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,499.5,334.5)"><path
+           d="M 0,0 H 2260600 V 787400 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path284" /></g><g
+         id="g286"
+         transform="translate(560.4364,-81)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text290"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan288">算⼦</tspan></text></g><g
+         id="g292"
+         transform="translate(596.4364,-81)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text296"><tspan
+             x="0"
+             y="0"
+             id="tspan294">2</tspan></text></g><path
+         d="m 494.5,122.5 h 178 v -64 h -178 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path298" /><g
+         id="g300"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,494.5,122.5)"><path
+           d="M 0,0 H 2260600 V 812800 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path302" /></g><g
+         id="g304"
+         transform="translate(562.7028,-293)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text308"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan306">算⼦</tspan></text></g><g
+         id="g310"
+         transform="translate(598.7028,-293)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text314"><tspan
+             x="0"
+             y="0"
+             id="tspan312">2</tspan></text></g><g
+         id="g316"
+         transform="translate(257.4024,-24)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text320"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan318">程序分区</tspan></text></g><g
+         id="g322"
+         transform="translate(329.4024,-24)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text326"><tspan
+             x="0"
+             y="0"
+             id="tspan324">1</tspan></text></g><g
+         id="g328"
+         transform="translate(339.4024,-24)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text332"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan330">副本</tspan></text></g><g
+         id="g334"
+         transform="translate(524.6193,-28)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text338"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan336">程序分区</tspan></text></g><g
+         id="g340"
+         transform="translate(596.6193,-28)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text344"><tspan
+             x="0"
+             y="0"
+             id="tspan342">2</tspan></text></g><g
+         id="g346"
+         transform="translate(606.6193,-28)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text350"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan348">副本</tspan></text></g><g
+         id="g352"
+         transform="translate(525.8193,-362)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text356"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan354">程序分区</tspan></text></g><g
+         id="g358"
+         transform="translate(597.8193,-362)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text362"><tspan
+             x="0"
+             y="0"
+             id="tspan360">2</tspan></text></g><g
+         id="g364"
+         transform="translate(607.8193,-362)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text368"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan366">副本</tspan></text></g><g
+         id="g370"
+         transform="translate(258.384,-362)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text374"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan372">程序分区</tspan></text></g><g
+         id="g376"
+         transform="translate(330.384,-362)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text380"><tspan
+             x="0"
+             y="0"
+             id="tspan378">1</tspan></text></g><g
+         id="g382"
+         transform="translate(340.384,-362)"><text
+           transform="matrix(1,0,0,-1,0,379)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text386"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan384">副本</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-model-parallel-inter-op.pdf
+++ b/img/ch09/ch10-model-parallel-inter-op.pdf
--- a/img/ch09/ch10-model-parallel-inter-op.svg
+++ b/img/ch09/ch10-model-parallel-inter-op.svg
@@ -0,0 +1,249 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="997.37134"
+   height="451.50781"
+   viewBox="0 0 997.37134 451.50781"
+   sodipodi:docname="ch10-model-parallel-inter-op.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 783 V 361 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.64319923"
+     inkscape:cx="647.5443"
+     inkscape:cy="250.31124"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-model-parallel-inter-op"
+     transform="matrix(1.3333333,0,0,-1.3333333,-15.269851,470.8815)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="m 77.5,304.3327 c 0,11.6904 9.4769,21.1673 21.16725,21.1673 H 628.3328 c 11.6903,0 21.1672,-9.4769 21.1672,-21.1673 V 219.6673 C 649.5,207.9769 640.0231,198.5 628.3328,198.5 H 98.66725 C 86.9769,198.5 77.5,207.9769 77.5,219.6673 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,77.5,325.5)"><path
+           d="M 0,268824.6 C 0,120356.9 120356.7,0 268824.1,0 H 6995576 c 148467,0 268824,120356.9 268824,268824.6 V 1344075 c 0,148468 -120357,268825 -268824,268825 H 268824.1 C 120356.7,1612900 0,1492543 0,1344075 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path28" /></g><path
+         d="m 100.5,308.5 h 72 v -93 h -72 z"
+         style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path30" /><g
+         id="g32"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,100.5,308.5)"><path
+           d="M 0,0 H 914400 V 1181100 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path34" /></g><g
+         id="g36"
+         transform="translate(11.06567,-104)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text40"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan38">设备</tspan></text></g><g
+         id="g42"
+         transform="translate(47.06567,-104)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text46"><tspan
+             x="0"
+             y="0"
+             id="tspan44">1</tspan></text></g><path
+         d="m 77.5,138.8331 c 0,11.414 9.25285,20.6669 20.66682,20.6669 H 628.8332 c 11.414,0 20.6668,-9.2529 20.6668,-20.6669 V 56.16687 C 649.5,44.75287 640.2472,35.5 628.8332,35.5 H 98.16682 C 86.75285,35.5 77.5,44.75287 77.5,56.16687 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path48" /><g
+         id="g50"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,77.5,159.5)"><path
+           d="M 0,262469.3 C 0,117511.5 117511.2,0 262468.6,0 H 7001932 c 144957,0 262468,117511.5 262468,262469.3 V 1312331 c 0,144958 -117511,262469 -262468,262469 H 262468.6 C 117511.2,1574800 0,1457289 0,1312331 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path52" /></g><g
+         id="g54"
+         transform="translate(11.06567,-267)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text58"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan56">设备</tspan></text></g><g
+         id="g60"
+         transform="translate(47.06567,-267)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0"
+             y="0"
+             id="tspan62">2</tspan></text></g><path
+         d="m 100.5,309.5 h 72 v -94 h -72 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path66" /><g
+         id="g68"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,100.5,309.5)"><path
+           d="M 0,0 H 914400 V 1193800 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path70" /></g><g
+         id="g72"
+         transform="translate(117.6847,-103)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text76"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan74">数据</tspan></text></g><path
+         d="m 259.5,308.5 h 357 v -93 h -357 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path78" /><g
+         id="g80"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,259.5,308.5)"><path
+           d="M 0,0 H 4533900 V 1181100 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path82" /></g><g
+         id="g84"
+         transform="translate(415.6091,-103)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text88"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan86">算⼦</tspan></text></g><g
+         id="g90"
+         transform="translate(451.6091,-103)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text94"><tspan
+             x="0"
+             y="0"
+             id="tspan92">1</tspan></text></g><path
+         d="m 259.5,145.5 h 359 v -91 h -359 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path96" /><g
+         id="g98"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,259.5,145.5)"><path
+           d="M 0,0 H 4559300 V 1155700 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path100" /></g><g
+         id="g102"
+         transform="translate(414.8385,-270)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text106"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan104">算⼦</tspan></text></g><g
+         id="g108"
+         transform="translate(450.8385,-270)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text112"><tspan
+             x="0"
+             y="0"
+             id="tspan110">2</tspan></text></g><path
+         d="m 172.5062,264.75 75.6872,-0.2098 -0.0125,-4.5 -75.6871,0.2098 z m 73.4497,4.2964 13.4813,-6.7874 -13.5187,-6.7126 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path114" /><path
+         d="M 618.1843,265.75 H 670.75 V 172.5607 H 207.5 l 2.2497,2.2891 1.3598,-78.31075 -2.2497,2.21095 h 37.7058 v -4.5 h -39.9171 l -1.4379,82.8107 H 668.5 l -2.25,-2.25 V 263.5 l 2.25,-2.25 h -50.3157 z m -373.8687,-162.5 13.5,-6.75 -13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path116" /><g
+         id="g118"
+         transform="translate(699.2319,-84)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text122"><tspan
+             x="0 9 16.992001 27"
+             y="0"
+             sodipodi:role="line"
+             id="tspan120">Send</tspan></text></g><g
+         id="g124"
+         transform="translate(663.2319,-106)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text128"><tspan
+             x="0 18 36 54 72 90"
+             y="0"
+             sodipodi:role="line"
+             id="tspan126">（发送数据）</tspan></text></g><g
+         id="g130"
+         transform="translate(130.2561,-195)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text134"><tspan
+             x="0 10.9998 18.993601 25.9974 33.991199 38.997002 46.990799"
+             y="0"
+             sodipodi:role="line"
+             id="tspan132">Receive</tspan></text></g><g
+         id="g136"
+         transform="translate(103.2561,-217)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text140"><tspan
+             x="0 18 36 54 72 90"
+             y="0"
+             sodipodi:role="line"
+             id="tspan138">（接收数据）</tspan></text></g><g
+         id="g142"
+         transform="translate(358.2626,-344)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text146"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan144">程序分区</tspan></text></g><g
+         id="g148"
+         transform="translate(430.2626,-344)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text152"><tspan
+             x="0"
+             y="0"
+             id="tspan150">2</tspan></text></g><g
+         id="g154"
+         transform="translate(358.2625,-23)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text158"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan156">程序分区</tspan></text></g><g
+         id="g160"
+         transform="translate(430.2625,-23)"><text
+           transform="matrix(1,0,0,-1,0,361)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text164"><tspan
+             x="0"
+             y="0"
+             id="tspan162">1</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-model-parallel-intra-op.pdf
+++ b/img/ch09/ch10-model-parallel-intra-op.pdf
--- a/img/ch09/ch10-model-parallel-intra-op.svg
+++ b/img/ch09/ch10-model-parallel-intra-op.svg
@@ -0,0 +1,346 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="272"
+   height="108"
+   viewBox="0 0 272 108"
+   sodipodi:docname="ch10-model-parallel-intra-op.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 699 V 365 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.72103004"
+     inkscape:cx="-35.366071"
+     inkscape:cy="188.61905"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g22" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-model-parallel-intra-op"
+     transform="matrix(1.3333333,0,0,-1.3333333,-376.66666,387.33332)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="M 76.5,304.4994 C 76.5,317.2023 86.79769,327.5 99.50053,327.5 H 675.4995 c 12.7028,0 23.0005,-10.2977 23.0005,-23.0006 V 212.5006 C 698.5,199.7977 688.2023,189.5 675.4995,189.5 H 99.50053 C 86.79769,189.5 76.5,199.7977 76.5,212.5006 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,76.5,327.5)"><path
+           d="M 0,292107.2 C 0,130780.8 130780.7,0 292106.8,0 H 7607294 c 161325,0 292106,130780.8 292106,292107.2 V 1460493 c 0,161326 -130781,292107 -292106,292107 H 292106.8 C 130780.7,1752600 0,1621819 0,1460493 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path28" /></g><path
+         d="m 599.5,291.5 h 72 v -82 h -72 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path30" /><g
+         id="g32"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,599.5,291.5)"><path
+           d="M 0,0 H 914400 V 1041400 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path34" /></g><path
+         d="m 282.5,290.5 h 204 v -81 h -204 z"
+         style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path36" /><g
+         id="g38"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,282.5,290.5)"><path
+           d="M 0,0 H 2590800 V 1028700 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path40" /></g><path
+         d="m 98.5,290.5 h 72 v -81 h -72 z"
+         style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path42" /><g
+         id="g44"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,98.5,290.5)"><path
+           d="M 0,0 H 914400 V 1028700 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path46" /></g><g
+         id="g48"
+         transform="translate(611.9691,-118)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text52"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan50">算⼦</tspan></text></g><g
+         id="g54"
+         transform="translate(647.9691,-118)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text58"><tspan
+             x="0"
+             y="0"
+             id="tspan56">2</tspan></text></g><g
+         id="g60"
+         transform="translate(11.91087,-120)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan62">设备</tspan></text></g><g
+         id="g66"
+         transform="translate(47.91087,-120)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text70"><tspan
+             x="0"
+             y="0"
+             id="tspan68">1</tspan></text></g><path
+         d="M 76.5,151.4994 C 76.5,164.2023 86.79769,174.5 99.50053,174.5 H 675.4995 c 12.7028,0 23.0005,-10.2977 23.0005,-23.0006 V 59.50057 c 0,-12.70286 -10.2977,-23.00056 -23.0005,-23.00056 H 99.50053 C 86.79769,36.50001 76.5,46.79771 76.5,59.50057 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path72" /><g
+         id="g74"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,76.5,174.5)"><path
+           d="M 0,292107.2 C 0,130780.8 130780.7,0 292106.8,0 H 7607294 c 161325,0 292106,130780.8 292106,292107.2 V 1460493 c 0,161326 -130781,292107 -292106,292107 H 292106.8 C 130780.7,1752600 0,1621819 0,1460493 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path76" /></g><g
+         id="g78"
+         transform="translate(14.94685,-272)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text82"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan80">设备</tspan></text></g><g
+         id="g84"
+         transform="translate(50.94685,-272)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text88"><tspan
+             x="0"
+             y="0"
+             id="tspan86">2</tspan></text></g><path
+         d="m 98.5,291.5 h 72 v -82 h -72 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path90" /><g
+         id="g92"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,98.5,291.5)"><path
+           d="M 0,0 H 914400 V 1041400 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path94" /></g><g
+         id="g96"
+         transform="translate(115.8845,-119)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text100"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan98">数据</tspan></text></g><path
+         d="m 282.5,290.5 h 102 v -81 h -102 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path102" /><g
+         id="g104"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,282.5,290.5)"><path
+           d="M 0,0 H 1295400 V 1028700 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path106" /></g><g
+         id="g108"
+         transform="translate(311.0483,-119)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text112"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan110">分区</tspan></text></g><g
+         id="g114"
+         transform="translate(347.0483,-119)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text118"><tspan
+             x="0"
+             y="0"
+             id="tspan116">1</tspan></text></g><path
+         d="m 281.5,134.5 h 204 v -78 h -204 z"
+         style="fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path120" /><g
+         id="g122"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,281.5,134.5)"><path
+           d="M 0,0 H 2590800 V 990600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path124" /></g><path
+         d="m 383.5,134.5 h 102 v -78 h -102 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path126" /><g
+         id="g128"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,383.5,134.5)"><path
+           d="M 0,0 H 1295400 V 990600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path130" /></g><g
+         id="g132"
+         transform="translate(413.9496,-276)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text136"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan134">分区</tspan></text></g><g
+         id="g138"
+         transform="translate(449.9496,-276)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text142"><tspan
+             x="0"
+             y="0"
+             id="tspan140">2</tspan></text></g><g
+         id="g144"
+         transform="translate(194.7052,-138)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text148"><tspan
+             x="0 10.9998 18.003599 27.0054 36.007198 45.008999 52.012798 61.014599 68.018402"
+             y="0"
+             sodipodi:role="line"
+             id="tspan146">Broadcast</tspan></text></g><g
+         id="g150"
+         transform="translate(195.7052,-160)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text154"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan152">（⼴播）</tspan></text></g><path
+         d="m 170.5101,252.75 100.1402,-0.449 -0.0202,-4.5 -100.1402,0.449 z m 97.9104,4.061 13.4696,-6.8104 -13.5301,-6.6895 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path156" /><path
+         d="m 172.3316,251.8069 104.1661,-145.989 -3.6632,-2.6137 -104.1661,145.9889 z m 106.5224,-141.5438 2.3465,-14.90995 -13.3359,7.06875 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path158" /><path
+         d="M 487.3168,94.33215 593.8032,240.0888 590.1696,242.7434 483.6832,96.98676 Z m 108.7926,141.28515 2.5135,14.8827 -13.4142,-6.9188 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path160" /><path
+         d="m 486.5148,247.5079 101.6879,0.6682 -0.0296,4.4999 -101.6879,-0.6682 z m 99.4675,-3.8465 13.4553,6.8386 -13.544,6.6611 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path162" /><g
+         id="g164"
+         transform="translate(505.24,-137)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text168"><tspan
+             x="0 14.0004 22.996799 29.005199 39.009602 47.015999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan166">Gather</tspan></text></g><g
+         id="g170"
+         transform="translate(508.24,-159)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text174"><tspan
+             x="0"
+             y="0"
+             id="tspan172">(</tspan></text></g><g
+         id="g176"
+         transform="translate(513.24,-159)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text180"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan178">合并</tspan></text></g><g
+         id="g182"
+         transform="translate(549.24,-159)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text186"><tspan
+             x="0"
+             y="0"
+             id="tspan184">)</tspan></text></g><g
+         id="g188"
+         transform="translate(368.2626,-62)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text192"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan190">算⼦</tspan></text></g><g
+         id="g194"
+         transform="translate(404.2626,-62)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text198"><tspan
+             x="0"
+             y="0"
+             id="tspan196">1</tspan></text></g><g
+         id="g200"
+         transform="translate(368.4133,-216)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text204"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan202">算⼦</tspan></text></g><g
+         id="g206"
+         transform="translate(404.4133,-216)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text210"><tspan
+             x="0"
+             y="0"
+             id="tspan208">1</tspan></text></g><g
+         id="g212"
+         transform="translate(380.2,-23)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text216"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan214">程序分区</tspan></text></g><g
+         id="g218"
+         transform="translate(452.2,-23)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text222"><tspan
+             x="0"
+             y="0"
+             id="tspan220">1</tspan></text></g><g
+         id="g224"
+         transform="translate(381.4626,-348)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text228"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan226">程序分区</tspan></text></g><g
+         id="g230"
+         transform="translate(453.4626,-348)"><text
+           transform="matrix(1,0,0,-1,0,365)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text234"><tspan
+             x="0"
+             y="0"
+             id="tspan232">2</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-parameter-server-replication.pdf
+++ b/img/ch09/ch10-parameter-server-replication.pdf
--- a/img/ch09/ch10-parameter-server-replication.svg
+++ b/img/ch09/ch10-parameter-server-replication.svg
@@ -0,0 +1,224 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="779.24609"
+   height="442.80597"
+   viewBox="0 0 779.24609 442.80597"
+   sodipodi:docname="ch10-parameter-server-replication.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 609 V 342 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.78837719"
+     inkscape:cx="331.69402"
+     inkscape:cy="273.98053"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-parameter-server-replication"
+     transform="matrix(1.3333333,0,0,-1.3333333,-16.622627,442.80598)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="M 16.5,278.3332 C 16.5,294.9939 30.00611,308.5 46.66674,308.5 H 532.3333 c 16.6606,0 30.1667,-13.5061 30.1667,-30.1668 V 157.6668 C 562.5,141.0061 548.9939,127.5 532.3333,127.5 H 46.66674 C 30.00611,127.5 16.5,141.0061 16.5,157.6668 Z"
+         style="fill:#ededed;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,16.5,308.5)"><path
+           d="M 0,383117.9 C 0,171527.7 171527.6,0 383117.6,0 H 6551083 c 211590,0 383117,171527.7 383117,383117.9 V 1915582 c 0,211590 -171527,383118 -383117,383118 H 383117.6 C 171527.6,2298700 0,2127172 0,1915582 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path28" /></g><path
+         d="M 124.5,56.33316 C 124.5,62.50043 129.4996,67.5 135.6668,67.5 H 252.3332 C 258.5004,67.5 263.5,62.50043 263.5,56.33316 V 11.66684 C 263.5,5.499564 258.5004,0.4999987 252.3332,0.4999987 H 135.6668 c -6.1672,0 -11.1668,4.9995653 -11.1668,11.1668413 z"
+         style="fill:#e2f0d9;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path30" /><g
+         id="g32"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,124.5,67.5)"><path
+           d="M 0,141818.9 C 0,63494.5 63494.36,0 141818.6,0 H 1623481 c 78325,0 141819,63494.5 141819,141818.9 v 567262.2 c 0,78324.4 -63494,141818.9 -141819,141818.9 H 141818.6 C 63494.36,850900 0,787405.5 0,709081.1 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path34" /></g><g
+         id="g36"
+         transform="translate(159.4874,-313)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text40"><tspan
+             x="0 16.0002 24.9984 30.990601 38.980801 47.979 53.971199 58.9734"
+             y="0"
+             sodipodi:role="line"
+             id="tspan38">Worker 1</tspan></text></g><path
+         d="m 224.5,168.8333 c 0,3.1296 2.5371,5.6667 5.6667,5.6667 h 127.6666 c 3.1296,0 5.6667,-2.5371 5.6667,-5.6667 v -22.6666 c 0,-3.1296 -2.5371,-5.6667 -5.6667,-5.6667 H 230.1667 c -3.1296,0 -5.6667,2.5371 -5.6667,5.6667 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path42" /><g
+         id="g44"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,224.5,174.5)"><path
+           d="M 0,71967.31 C 0,32220.86 32220.84,0 71967.25,0 H 1693333 c 39746,0 71967,32220.86 71967,71967.31 V 359832.7 c 0,39746.4 -32221,71967.3 -71967,71967.3 H 71967.25 C 32220.84,431800 0,399579.1 0,359832.7 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path46" /></g><g
+         id="g48"
+         transform="translate(242.4272,-190)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text52"><tspan
+             x="0 7.9991999 17.006399 26.013599 35.020802 39.023998 48.0312 57.038399 61.041599 70.048798 79.056 87.055199 96.062401"
+             y="0"
+             sodipodi:role="line"
+             id="tspan50">Load balancer</tspan></text></g><path
+         d="m 61.5,276.1665 c 0,5.1548 4.17874,9.3335 9.33349,9.3335 H 192.1665 c 5.1548,0 9.3335,-4.1787 9.3335,-9.3335 v -37.333 c 0,-5.1548 -4.1787,-9.3335 -9.3335,-9.3335 H 70.83349 c -5.15475,0 -9.33349,4.1787 -9.33349,9.3335 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path54" /><g
+         id="g56"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,61.5,285.5)"><path
+           d="M 0,118535.3 C 0,53070.05 53070.05,0 118535.3,0 H 1659465 c 65465,0 118535,53070.05 118535,118535.3 v 474129.4 c 0,65465.3 -53070,118535.3 -118535,118535.3 H 118535.3 C 53070.05,711200 0,658130 0,592664.7 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path58" /></g><g
+         id="g60"
+         transform="translate(104.388,-90)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0 10.0008 18.993601 27.986401 31.9932 36 44.0028"
+             y="0"
+             sodipodi:role="line"
+             id="tspan62">Replica</tspan></text></g><path
+         d="m 224.5,274.1664 c 0,5.1548 4.1788,9.3336 9.3336,9.3336 h 120.3329 c 5.1547,0 9.3335,-4.1788 9.3335,-9.3336 v -37.3328 c 0,-5.1548 -4.1788,-9.3336 -9.3335,-9.3336 H 233.8336 c -5.1548,0 -9.3336,4.1788 -9.3336,9.3336 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path66" /><g
+         id="g68"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,224.5,283.5)"><path
+           d="M 0,118536.2 C 0,53070.48 53070.43,0 118536.1,0 H 1646764 c 65466,0 118536,53070.48 118536,118536.2 v 474127.6 c 0,65465.7 -53070,118536.2 -118536,118536.2 H 118536.1 C 53070.43,711200 0,658129.5 0,592663.8 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path70" /></g><g
+         id="g72"
+         transform="translate(266.4898,-92)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text76"><tspan
+             x="0 10.0008 18.993601 27.986401 31.9932 36 44.0028"
+             y="0"
+             sodipodi:role="line"
+             id="tspan74">Replica</tspan></text></g><path
+         d="m 386.5,274.1664 c 0,5.1548 4.1788,9.3336 9.3336,9.3336 h 120.3329 c 5.1547,0 9.3335,-4.1788 9.3335,-9.3336 v -37.3328 c 0,-5.1548 -4.1788,-9.3336 -9.3335,-9.3336 H 395.8336 c -5.1548,0 -9.3336,4.1788 -9.3336,9.3336 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path78" /><g
+         id="g80"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,386.5,283.5)"><path
+           d="M 0,118536.2 C 0,53070.48 53070.43,0 118536.1,0 H 1646764 c 65466,0 118536,53070.48 118536,118536.2 v 474127.6 c 0,65465.7 -53070,118536.2 -118536,118536.2 H 118536.1 C 53070.43,711200 0,658129.5 0,592663.8 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path82" /></g><g
+         id="g84"
+         transform="translate(428.5915,-92)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text88"><tspan
+             x="0 10.0008 18.993601 27.986401 31.9932 36 44.0028"
+             y="0"
+             sodipodi:role="line"
+             id="tspan86">Replica</tspan></text></g><path
+         d="m 293.1229,141.6059 -4.8472,-3.5362 0.884,-1.2118 4.8472,3.5362 z m -8.4826,-6.1884 -4.8472,-3.5363 0.8841,-1.2118 4.8472,3.5363 z m -8.4825,-6.1884 -4.8472,-3.5363 0.8841,-1.2118 4.8471,3.5363 z m -8.4826,-6.1885 -4.8471,-3.5362 0.884,-1.2118 4.8472,3.5363 z m -8.4825,-6.1884 -4.8472,-3.5362 0.8841,-1.2118 4.8471,3.5362 z m -8.4825,-6.1884 -4.8472,-3.5362 0.884,-1.2118 4.8472,3.5362 z m -8.4826,-6.1884 -4.8472,-3.5363 0.8841,-1.21174 4.8472,3.53624 z m -8.4825,-6.18843 -4.8472,-3.53624 0.8841,-1.21179 4.8471,3.53624 z m -8.4826,-6.18842 -4.8471,-3.53624 0.884,-1.21179 4.8472,3.53624 z m -8.4825,-6.18842 -4.8472,-3.53624 0.8841,-1.21179 4.8471,3.53624 z m -8.4826,-6.18842 -4.8471,-3.53623 0.884,-1.21179 4.8472,3.53623 z m -8.4825,-6.18841 -2.2178,-1.61798 0.8841,-1.21179 2.2178,1.61797 z m -2.7359,0.78919 -3.079,-5.95981 6.6153,1.11265 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path90" /><path
+         d="m 293.415,141.6247 4.9979,-3.3198 -0.83,-1.2495 -4.9979,3.3199 z m 8.7463,-5.8096 4.9978,-3.3199 -0.8299,-1.2494 -4.9979,3.3198 z m 8.7463,-5.8097 4.9978,-3.3198 -0.8299,-1.2495 -4.9979,3.3198 z m 8.7462,-5.8097 4.9979,-3.3198 -0.8299,-1.2495 -4.9979,3.3198 z m 8.7463,-5.8097 4.9979,-3.3198 -0.8299,-1.2495 -4.9979,3.3198 z m 8.7463,-5.8097 4.9979,-3.3198 -0.8299,-1.2495 -4.9979,3.3199 z m 8.7463,-5.8097 4.9979,-3.3198 -0.83,-1.2494 -4.9978,3.3198 z m 8.7463,-5.8096 4.9979,-3.31986 -0.83,-1.24947 -4.9978,3.31981 z m 8.7463,-5.80973 4.9979,-3.31982 -0.83,-1.24946 -4.9979,3.31981 z m 8.7463,-5.80968 4.9979,-3.31982 -0.83,-1.24946 -4.9979,3.31981 z m 8.7463,-5.80968 4.9979,-3.31982 -0.83,-1.24946 -4.9979,3.31981 z m 8.7463,-5.80968 4.9978,-3.31982 -0.8299,-1.24947 -4.9979,3.31982 z m 8.7463,-5.80969 0.2326,-0.15454 -0.8299,-1.24947 -0.2327,0.15454 z m 0.6447,2.2729 3.338,-5.81876 -6.6578,0.82088 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path92" /><path
+         d="M 333.5,56.33316 C 333.5,62.50043 338.4996,67.5 344.6668,67.5 H 461.3332 C 467.5004,67.5 472.5,62.50043 472.5,56.33316 V 11.66684 C 472.5,5.499564 467.5004,0.4999987 461.3332,0.4999987 H 344.6668 c -6.1672,0 -11.1668,4.9995653 -11.1668,11.1668413 z"
+         style="fill:#e2f0d9;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path94" /><g
+         id="g96"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,333.5,67.5)"><path
+           d="M 0,141818.9 C 0,63494.5 63494.36,0 141818.6,0 H 1623481 c 78325,0 141819,63494.5 141819,141818.9 v 567262.2 c 0,78324.4 -63494,141818.9 -141819,141818.9 H 141818.6 C 63494.36,850900 0,787405.5 0,709081.1 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path98" /></g><g
+         id="g100"
+         transform="translate(368.4054,-313)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text104"><tspan
+             x="0 16.0002 24.9984 30.990601 38.980801 47.979 53.971199 58.9734"
+             y="0"
+             sodipodi:role="line"
+             id="tspan102">Worker 2</tspan></text></g><path
+         d="M 132.8991,230.041 192.7311,75.30161 189.933,74.21968 130.1009,228.959 Z m 62.0891,-152.25834 -0.9513,-10.01723 -7.443,6.77145 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path106" /><path
+         d="M 454.8242,227.9703 404.4274,75.3497 407.2761,74.40903 457.673,227.0297 Z M 402.049,77.71454 403.5,67.7574 l 7.0951,7.13512 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path108" /><g
+         id="g110"
+         transform="translate(11.1574,-258)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:bold;font-size:18px;font-family:Arial;-inkscape-font-specification:Arial-BoldMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text114"><tspan
+             x="0 12.0006 22.9932 27.9918"
+             y="0"
+             sodipodi:role="line"
+             id="tspan112">Pull</tspan></text></g><g
+         id="g116"
+         transform="translate(49.1574,-258)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text120"><tspan
+             x="0 10.0008 25.0056 35.006401 45.007198 55.007999 65.008797 69.015602 79.016403 89.017197 94.014 99.010803 109.0116 119.0124 123.0192"
+             y="0"
+             sodipodi:role="line"
+             id="tspan118">embedding table</tspan></text></g><g
+         id="g122"
+         transform="translate(426.6187,-256)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:bold;font-size:18px;font-family:Arial;-inkscape-font-specification:Arial-BoldMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text126"><tspan
+             x="0 12.0006 22.9932 27.9918"
+             y="0"
+             sodipodi:role="line"
+             id="tspan124">Pull</tspan></text></g><g
+         id="g128"
+         transform="translate(464.6187,-256)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text132"><tspan
+             x="0 10.0008 25.0056 35.006401 45.007198 55.007999 65.008797 69.015602 79.016403 89.017197 94.014 99.010803 109.0116 119.0124 123.0192"
+             y="0"
+             sodipodi:role="line"
+             id="tspan130">embedding table</tspan></text></g><g
+         id="g134"
+         transform="translate(193.866,-158)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text138"><tspan
+             x="0 12.9996 21.999599 32.000401 42.001202 57.000599 61.000198 70.000198 80.000999 84.000603 88.000198 97.000198 102.0006 111.0006 121.0014 131.0022 141.00301 150.00301 160.0038 165.0042 171.0036 181.00439 191.0052 195.00481 199.00439 208.00439 218.0052"
+             y="0"
+             sodipodi:role="line"
+             id="tspan136">Dynamically choose replica </tspan></text></g><g
+         id="g140"
+         transform="translate(162.2196,-23)"><text
+           transform="matrix(1,0,0,-1,0,342)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text144"><tspan
+             x="0 12.0006 22.0014 28.000799 38.001598 53.000999 63.001801 68.002197 78.002998 84.002403 89.0028 101.0034 111.0042 117.0036 126.0036 136.00439 142.0038 147.0042 160.0038 170.00459 180.0054 184.005 188.00459 197.00459 207.0054 212.0058 216.0054 226.0062 236.007 241.0074 255.0078 261.0072 271.008 281.00879"
+             y="0"
+             sodipodi:role="line"
+             id="tspan142">Parameter Server Replication Group</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-parameter-servers.pdf
+++ b/img/ch09/ch10-parameter-servers.pdf
--- a/img/ch09/ch10-parameter-servers.svg
+++ b/img/ch09/ch10-parameter-servers.svg
@@ -0,0 +1,202 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="560.59259"
+   height="321.81696"
+   viewBox="0 0 560.59259 321.81696"
+   sodipodi:docname="ch10-parameter-servers.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0.1090903 H 444 V 257.99999 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="1.0450581"
+     inkscape:cx="157.40751"
+     inkscape:cy="152.14465"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-parameter-servers"
+     transform="matrix(1.3333333,0,0,-1.3333333,-15.941313,343.12426)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="m 57.36361,245.0179 c 0,6.1858 5.01452,11.2003 11.20025,11.2003 H 185.3634 c 6.1857,0 11.2002,-5.0145 11.2002,-11.2003 v -44.7994 c 0,-6.1858 -5.0145,-11.2003 -11.2002,-11.2003 H 68.56386 c -6.18573,0 -11.20025,5.0145 -11.20025,11.2003 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:evenodd;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,57.36361,256.2182)"><path
+           d="M 0,142243.2 C 0,63684.46 63684.39,0 142243.1,0 H 1625597 c 78559,0 142243,63684.46 142243,142243.2 v 568953.6 c 0,78558.8 -63684,142243.2 -142243,142243.2 H 142243.1 C 63684.39,853440 0,789755.6 0,711196.8 Z"
+           style="fill:none;stroke:#000000;stroke-width:28575;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path28" /></g><g
+         id="g30"
+         transform="matrix(0.2181818,0,0,0.2181818,77.67966,-40.69089)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text34"><tspan
+             x="0 82.501991 165.00398 247.50597 330.00797"
+             y="0"
+             sodipodi:role="line"
+             id="tspan32">参数服务器</tspan></text></g><g
+         id="g36"
+         transform="matrix(0.2181818,0,0,0.2181818,167.6797,-40.69089)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text40"><tspan
+             x="0"
+             y="0"
+             id="tspan38">1</tspan></text></g><path
+         d="m 253.5091,245.0179 c 0,6.1858 5.0145,11.2003 11.2002,11.2003 H 381.727 c 6.1857,0 11.2002,-5.0145 11.2002,-11.2003 v -44.7994 c 0,-6.1858 -5.0145,-11.2003 -11.2002,-11.2003 H 264.7093 c -6.1857,0 -11.2002,5.0145 -11.2002,11.2003 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:evenodd;stroke:none"
+         id="path42" /><g
+         id="g44"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,253.5091,256.2182)"><path
+           d="M 0,142243.6 C 0,63684.61 63684.51,0 142243.3,0 H 1628368 c 78559,0 142243,63684.61 142243,142243.6 v 568952.9 c 0,78558.9 -63684,142243.5 -142243,142243.5 H 142243.3 C 63684.51,853440 0,789755.4 0,711196.5 Z"
+           style="fill:none;stroke:#000000;stroke-width:28575;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path46" /></g><g
+         id="g48"
+         transform="matrix(0.2181818,0,0,0.2181818,273.9655,-40.69089)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text52"><tspan
+             x="0 82.501991 165.00398 247.50597 330.00797"
+             y="0"
+             sodipodi:role="line"
+             id="tspan50">参数服务器</tspan></text></g><g
+         id="g54"
+         transform="matrix(0.2181818,0,0,0.2181818,363.9655,-40.69089)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text58"><tspan
+             x="0"
+             y="0"
+             id="tspan56">2</tspan></text></g><path
+         d="m 57.36361,115.1634 c 0,6.2058 5.03081,11.2366 11.23663,11.2366 H 185.327 c 6.2058,0 11.2366,-5.0308 11.2366,-11.2366 V 70.21846 c 0,-6.20582 -5.0308,-11.23664 -11.2366,-11.23664 H 68.60024 c -6.20582,0 -11.23663,5.03082 -11.23663,11.23664 z"
+         style="fill:#e2f0d9;fill-opacity:1;fill-rule:evenodd;stroke:none"
+         id="path60" /><g
+         id="g62"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,57.36361,126.4)"><path
+           d="M 0,142705.4 C 0,63891.38 63891.27,0 142705.1,0 H 1625135 c 78814,0 142705,63891.38 142705,142705.4 v 570800.2 c 0,78814 -63891,142705.4 -142705,142705.4 H 142705.1 C 63891.27,856211 0,792319.6 0,713505.6 Z"
+           style="fill:none;stroke:#000000;stroke-width:28575;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path64" /></g><g
+         id="g66"
+         transform="matrix(0.2181818,0,0,0.2181818,77.67974,-170.7273)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text70"><tspan
+             x="0 82.501991 165.00398 247.50597 330.00797"
+             y="0"
+             sodipodi:role="line"
+             id="tspan68">训练服务器</tspan></text></g><g
+         id="g72"
+         transform="matrix(0.2181818,0,0,0.2181818,167.6797,-170.7273)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text76"><tspan
+             x="0"
+             y="0"
+             id="tspan74">1</tspan></text></g><path
+         d="m 253.5091,115.1634 c 0,6.2058 5.0307,11.2366 11.2365,11.2366 h 116.9451 c 6.2058,0 11.2365,-5.0308 11.2365,-11.2366 V 70.21839 c 0,-6.20579 -5.0307,-11.23657 -11.2365,-11.23657 H 264.7456 c -6.2058,0 -11.2365,5.03078 -11.2365,11.23657 z"
+         style="fill:#e2f0d9;fill-opacity:1;fill-rule:evenodd;stroke:none"
+         id="path78" /><g
+         id="g80"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,253.5091,126.4)"><path
+           d="M 0,142704.4 C 0,63890.94 63890.84,0 142704.2,0 H 1627907 c 78813,0 142704,63890.94 142704,142704.4 v 570802.2 c 0,78813.4 -63891,142704.4 -142704,142704.4 H 142704.2 C 63890.84,856211 0,792320 0,713506.6 Z"
+           style="fill:none;stroke:#000000;stroke-width:28575;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path82" /></g><g
+         id="g84"
+         transform="matrix(0.2181818,0,0,0.2181818,273.9655,-170.7273)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text88"><tspan
+             x="0 82.501991 165.00398 247.50597 330.00797"
+             y="0"
+             sodipodi:role="line"
+             id="tspan86">训练服务器</tspan></text></g><g
+         id="g90"
+         transform="matrix(0.2181818,0,0,0.2181818,363.9655,-170.7273)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text94"><tspan
+             x="0"
+             y="0"
+             id="tspan92">2</tspan></text></g><path
+         d="m 100.8159,189.4546 1e-4,-57.1161 h -2.24999 l -8e-5,57.1161 z m 2.2501,-55.991 -3.37499,-6.75 -3.37501,6.7499 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path96" /><path
+         d="m 141.1114,126.3258 v 57.5038 h 2.25 v -57.5038 z m -2.25,56.3788 3.375,6.75 3.375,-6.75 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path98" /><path
+         d="m 299.4611,190.5282 -191.0289,-59.7925 0.6721,-2.1473 191.0289,59.7925 z m -190.6274,-57.3093 -5.4337,-5.2372 7.45,-1.2046 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path100" /><path
+         d="m 142.5686,125.2512 198.839,61.4672 -0.6645,2.1497 -198.839,-61.4673 z m 198.4288,58.9854 5.4521,5.218 -7.4457,1.2309 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path102" /><path
+         d="m 301.325,189.4546 1e-4,-57.1161 h -2.25 l -1e-4,57.1161 z m 2.2501,-55.991 -3.375,-6.75 -3.375,6.7499 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path104" /><path
+         d="m 100.464,190.0916 189.3526,-59.4015 -0.6734,-2.1468 -189.3527,59.4015 z m 188.9528,-56.9179 5.4303,-5.2407 -7.4508,-1.1998 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path106" /><path
+         d="m 347.7977,126.7135 10e-5,57.1161 h -2.25 l -10e-5,-57.1161 z m 2.2501,55.9911 -3.375,6.75 -3.375,-6.75 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path108" /><path
+         d="m 345.959,125.6382 -198.6768,61.0879 0.6613,2.1507 198.6767,-61.088 z m -198.2626,58.6066 -5.46,5.2098 7.4438,1.2421 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path110" /><g
+         id="g112"
+         transform="matrix(0.2181818,0,0,0.2181818,11.7084,-104.6182)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text116"><tspan
+             x="0 82.501991 165.00398 247.50597"
+             y="0"
+             sodipodi:role="line"
+             id="tspan114">拉取参数</tspan></text></g><g
+         id="g118"
+         transform="matrix(0.2181818,0,0,0.2181818,360.3608,-106.8)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text122"><tspan
+             x="0 82.501991 165.00398 247.50597"
+             y="0"
+             sodipodi:role="line"
+             id="tspan120">推送参数</tspan></text></g><path
+         d="m 126.275,53.79316 v -13.5 h 198.5357 v 12.5 h -2.25 v -11.375 l 1.125,1.125 H 127.4 l 1.125,-1.125 v 12.375 z m 4.5,-1.125 -3.375,6.75 -3.375,-6.75 z m 196.2857,-1 -3.375,6.75 -3.375,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path124" /><g
+         id="g126"
+         transform="matrix(0.2181818,0,0,0.2181818,190.1605,-241.6363)"><text
+           transform="matrix(1,0,0,-1,0,1182)"
+           style="font-variant:normal;font-weight:normal;font-size:83px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text130"><tspan
+             x="0 56.149494 78.51799 100.88649 130.64198 166.12448 208.49597 251.44847 285.27097"
+             y="0"
+             sodipodi:role="line"
+             id="tspan128">Allreduce</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-pipeline-parallel.pdf
+++ b/img/ch09/ch10-pipeline-parallel.pdf
--- a/img/ch09/ch10-pipeline-parallel.svg
+++ b/img/ch09/ch10-pipeline-parallel.svg
@@ -0,0 +1,589 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="1090.656"
+   height="313.41534"
+   viewBox="0 0 1090.656 313.41534"
+   sodipodi:docname="ch10-pipeline-parallel.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 833 V 256 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.83401244"
+     inkscape:cx="489.80085"
+     inkscape:cy="182.25148"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-pipeline-parallel"
+     transform="matrix(1.3333333,0,0,-1.3333333,-16.010641,330.62369)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="m 186,90 h 56 V 57 h -56 z"
+         style="fill:#bdd7ee;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="translate(205.7981,-188)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text30"><tspan
+             x="0"
+             y="0"
+             id="tspan28">F</tspan></text></g><g
+         id="g32"
+         transform="translate(215.7981,-191)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text36"><tspan
+             x="0"
+             y="0"
+             id="tspan34">0</tspan></text></g><path
+         d="m 241,123 h 57 V 90 h -57 z"
+         style="fill:#bdd7ee;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path38" /><g
+         id="g40"
+         transform="translate(261.4467,-155)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text44"><tspan
+             x="0"
+             y="0"
+             id="tspan42">F</tspan></text></g><g
+         id="g46"
+         transform="translate(271.4467,-158)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text50"><tspan
+             x="0"
+             y="0"
+             id="tspan48">1</tspan></text></g><path
+         d="m 298,157 h 56 v -33 h -56 z"
+         style="fill:#bdd7ee;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path52" /><g
+         id="g54"
+         transform="translate(318.0303,-121)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text58"><tspan
+             x="0"
+             y="0"
+             id="tspan56">F</tspan></text></g><g
+         id="g60"
+         transform="translate(328.0303,-124)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0"
+             y="0"
+             id="tspan62">2</tspan></text></g><path
+         d="m 354,191 h 57 v -34 h -57 z"
+         style="fill:#bdd7ee;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path66" /><g
+         id="g68"
+         transform="translate(374.6138,-87)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text72"><tspan
+             x="0"
+             y="0"
+             id="tspan70">F</tspan></text></g><g
+         id="g74"
+         transform="translate(384.6138,-90)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text78"><tspan
+             x="0"
+             y="0"
+             id="tspan76">3</tspan></text></g><path
+         d="m 470,191 h 62 v -34 h -62 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path80" /><g
+         id="g82"
+         transform="translate(491.7736,-87)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text86"><tspan
+             x="0"
+             y="0"
+             id="tspan84">B</tspan></text></g><g
+         id="g88"
+         transform="translate(502.7736,-90)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text92"><tspan
+             x="0"
+             y="0"
+             id="tspan90">3</tspan></text></g><path
+         d="m 532,157 h 75 v -33 h -75 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path94" /><g
+         id="g96"
+         transform="translate(560.2103,-121)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text100"><tspan
+             x="0"
+             y="0"
+             id="tspan98">B</tspan></text></g><g
+         id="g102"
+         transform="translate(571.2103,-124)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text106"><tspan
+             x="0"
+             y="0"
+             id="tspan104">2</tspan></text></g><path
+         d="m 607,123 h 74 V 90 h -74 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path108" /><g
+         id="g110"
+         transform="translate(634.7817,-155)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text114"><tspan
+             x="0"
+             y="0"
+             id="tspan112">B</tspan></text></g><g
+         id="g116"
+         transform="translate(645.7817,-158)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text120"><tspan
+             x="0"
+             y="0"
+             id="tspan118">1</tspan></text></g><path
+         d="m 681,90 h 75 V 57 h -75 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path122" /><g
+         id="g124"
+         transform="translate(709.3531,-188)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text128"><tspan
+             x="0"
+             y="0"
+             id="tspan126">B</tspan></text></g><g
+         id="g130"
+         transform="translate(720.3531,-191)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text134"><tspan
+             x="0"
+             y="0"
+             id="tspan132">0</tspan></text></g><path
+         d="m 242,90 h 57 V 57 h -57 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path136" /><g
+         id="g138"
+         transform="translate(262.0973,-188)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text142"><tspan
+             x="0"
+             y="0"
+             id="tspan140">F</tspan></text></g><g
+         id="g144"
+         transform="translate(272.0973,-191)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text148"><tspan
+             x="0"
+             y="0"
+             id="tspan146">0</tspan></text></g><path
+         d="m 299,123 h 56 V 90 h -56 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path150" /><g
+         id="g152"
+         transform="translate(318.7902,-155)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text156"><tspan
+             x="0"
+             y="0"
+             id="tspan154">F</tspan></text></g><g
+         id="g158"
+         transform="translate(328.7902,-158)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text162"><tspan
+             x="0"
+             y="0"
+             id="tspan160">1</tspan></text></g><path
+         d="m 354,157 h 57 v -33 h -57 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path164" /><g
+         id="g166"
+         transform="translate(374.6138,-121)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text170"><tspan
+             x="0"
+             y="0"
+             id="tspan168">F</tspan></text></g><g
+         id="g172"
+         transform="translate(384.6138,-124)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text176"><tspan
+             x="0"
+             y="0"
+             id="tspan174">2</tspan></text></g><path
+         d="m 412,191 h 56 v -34 h -56 z"
+         style="fill:#deebf7;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path178" /><g
+         id="g180"
+         transform="translate(431.8127,-87)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text184"><tspan
+             x="0"
+             y="0"
+             id="tspan182">F</tspan></text></g><g
+         id="g186"
+         transform="translate(441.8127,-90)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text190"><tspan
+             x="0"
+             y="0"
+             id="tspan188">3</tspan></text></g><path
+         d="m 532,191 h 74 v -34 h -74 z"
+         style="fill:#d0cece;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path192" /><g
+         id="g194"
+         transform="translate(559.7794,-87)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text198"><tspan
+             x="0"
+             y="0"
+             id="tspan196">B</tspan></text></g><g
+         id="g200"
+         transform="translate(570.7794,-90)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text204"><tspan
+             x="0"
+             y="0"
+             id="tspan202">3</tspan></text></g><path
+         d="m 607,157 h 74 v -33 h -74 z"
+         style="fill:#d0cece;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path206" /><g
+         id="g208"
+         transform="translate(634.7817,-121)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text212"><tspan
+             x="0"
+             y="0"
+             id="tspan210">B</tspan></text></g><g
+         id="g214"
+         transform="translate(645.7817,-124)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text218"><tspan
+             x="0"
+             y="0"
+             id="tspan216">2</tspan></text></g><path
+         d="m 680,123 h 75 V 90 h -75 z"
+         style="fill:#d0cece;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path220" /><g
+         id="g222"
+         transform="translate(708.7025,-155)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text226"><tspan
+             x="0"
+             y="0"
+             id="tspan224">B</tspan></text></g><g
+         id="g228"
+         transform="translate(719.7025,-158)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text232"><tspan
+             x="0"
+             y="0"
+             id="tspan230">1</tspan></text></g><path
+         d="m 756,90 h 74 V 57 h -74 z"
+         style="fill:#d0cece;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path234" /><g
+         id="g236"
+         transform="translate(783.9246,-188)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text240"><tspan
+             x="0"
+             y="0"
+             id="tspan238">B</tspan></text></g><g
+         id="g242"
+         transform="translate(794.9246,-191)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:12px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text246"><tspan
+             x="0"
+             y="0"
+             id="tspan244">0</tspan></text></g><g
+         id="g248"
+         transform="translate(76.85417,-189)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text252"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan250">设备</tspan></text></g><g
+         id="g254"
+         transform="translate(112.8542,-189)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text258"><tspan
+             x="0"
+             y="0"
+             id="tspan256">1</tspan></text></g><g
+         id="g260"
+         transform="translate(76.85417,-155)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text264"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan262">设备</tspan></text></g><g
+         id="g266"
+         transform="translate(112.8542,-155)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text270"><tspan
+             x="0"
+             y="0"
+             id="tspan268">2</tspan></text></g><g
+         id="g272"
+         transform="translate(76.87843,-120)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text276"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan274">设备</tspan></text></g><g
+         id="g278"
+         transform="translate(112.8784,-120)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text282"><tspan
+             x="0"
+             y="0"
+             id="tspan280">3</tspan></text></g><g
+         id="g284"
+         transform="translate(76.85417,-86)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text288"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan286">设备</tspan></text></g><g
+         id="g290"
+         transform="translate(112.8542,-86)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text294"><tspan
+             x="0"
+             y="0"
+             id="tspan292">4</tspan></text></g><g
+         id="g296"
+         transform="translate(69.05582,-240)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text300"><tspan
+             x="0 18 36 54 72 90"
+             y="0"
+             sodipodi:role="line"
+             id="tspan298">将数据切分为</tspan></text></g><g
+         id="g302"
+         transform="translate(177.0558,-240)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:bold;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Bold;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text306"><tspan
+             x="0 18 36"
+             y="0"
+             sodipodi:role="line"
+             id="tspan304">微批次</tspan></text></g><g
+         id="g308"
+         transform="translate(231.0558,-240)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text312"><tspan
+             x="0"
+             y="0"
+             id="tspan310">（</tspan></text></g><g
+         id="g314"
+         transform="translate(249.0558,-240)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text318"><tspan
+             x="0 14.9994 20.008801 27.016199 34.023602"
+             y="0"
+             sodipodi:role="line"
+             id="tspan316">Micro</tspan></text></g><g
+         id="g320"
+         transform="translate(292.0558,-240)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text324"><tspan
+             x="0"
+             y="0"
+             id="tspan322">-</tspan></text></g><g
+         id="g326"
+         transform="translate(298.0558,-240)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text330"><tspan
+             x="0 9 18 23.993999 30.996"
+             y="0"
+             sodipodi:role="line"
+             id="tspan328">batch</tspan></text></g><g
+         id="g332"
+         transform="translate(339.0558,-240)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text336"><tspan
+             x="0 18 36 54 72 90 108"
+             y="0"
+             sodipodi:role="line"
+             id="tspan334">）来构建流⽔线</tspan></text></g><path
+         d="m 182,215.125 639.0713,-1e-4 v -2.25 L 182,212.875 Z m 637.9462,2.2499 6.75,-3.375 -6.75,-3.375 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path338" /><g
+         id="g340"
+         transform="translate(781.6303,-23)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text344"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan342">时间</tspan></text></g><g
+         id="g346"
+         transform="translate(182.8676,-77)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text350"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan348">前向计算</tspan></text></g><g
+         id="g352"
+         transform="translate(715.5527,-75)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text356"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan354">后向计算</tspan></text></g><path
+         d="m 36.5,183.5 h 25 v -16 h -25 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path358" /><g
+         id="g360"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,36.5,183.5)"><path
+           d="M 0,0 H 317500 V 203200 H 0 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path362" /></g><path
+         d="m 36.5,146.5 h 25 v -16 h -25 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path364" /><g
+         id="g366"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,36.5,146.5)"><path
+           d="M 0,0 H 317500 V 203200 H 0 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path368" /></g><path
+         d="m 36.5,114.5 h 25 v -16 h -25 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path370" /><g
+         id="g372"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,36.5,114.5)"><path
+           d="M 0,0 H 317500 V 203200 H 0 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path374" /></g><path
+         d="m 36.5,79.5 h 25 v -16 h -25 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path376" /><g
+         id="g378"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,36.5,79.5)"><path
+           d="M 0,0 H 317500 V 203200 H 0 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path380" /></g><g
+         id="g382"
+         transform="translate(11.8322,-49)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text386"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan384">模型分区</tspan></text></g><g
+         id="g388"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,168.5,98.5)"><path
+           d="M 0,105834.6 C 0,47383.77 47383.72,0 105834.5,0 H 1646765 c 58451,0 105835,47383.77 105835,105834.6 v 423330.8 c 0,58450.8 -47384,105834.6 -105835,105834.6 H 105834.5 C 47383.72,635000 0,587616.2 0,529165.4 Z"
+           style="fill:none;stroke:#c00000;stroke-width:38100;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:114300, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path390" /></g><path
+         d="m 49.125,79.82118 6e-5,13.55382 -2.25,1e-5 -6e-5,-13.55382 z M 51.37505,92.24999 48.00008,99 44.62505,92.25002 Z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path392" /><path
+         d="m 46.87508,114.5194 -5e-5,10.8556 h 2.25 l 5e-5,-10.8555 z M 44.62503,124.25 48,131 l 3.37503,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path394" /><path
+         d="m 49.125,147.0624 6e-5,15.3126 h -2.25 l -6e-5,-15.3126 z m 2.25005,14.1876 -3.37497,6.75 -3.37503,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path396" /><g
+         id="g398"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,453.5,201.5)"><path
+           d="M 0,107952 C 0,48331.76 48331.76,0 107952,0 h 1003296 c 59620,0 107952,48331.76 107952,107952 v 431796 c 0,59620.3 -48332,107952 -107952,107952 H 107952 C 48331.76,647700 0,599368.3 0,539748 Z"
+           style="fill:none;stroke:#c00000;stroke-width:38100;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:114300, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path400" /></g><g
+         id="g402"
+         transform="translate(362.4592,-24)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:bold;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Bold;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text406"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan404">重新计算</tspan></text></g><g
+         id="g408"
+         transform="translate(434.4592,-24)"><text
+           transform="matrix(1,0,0,-1,0,256)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text412"><tspan
+             x="0 18 36 54 72 90 108 126 144 162"
+             y="0"
+             sodipodi:role="line"
+             id="tspan410">激活值来减少内存开销</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-recommendation-model.pdf
+++ b/img/ch09/ch10-recommendation-model.pdf
--- a/img/ch09/ch10-recommendation-model.svg
+++ b/img/ch09/ch10-recommendation-model.svg
@@ -0,0 +1,454 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="1169.1067"
+   height="485.94922"
+   viewBox="0 0 1169.1067 485.94922"
+   sodipodi:docname="ch10-recommendation-model.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 903 V 387 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.79109039"
+     inkscape:cx="530.28074"
+     inkscape:cy="251.55153"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-recommendation-model"
+     transform="matrix(1.3333333,0,0,-1.3333333,-17.714845,501.17967)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="m 423.5,348.5 h 230 v -17 h -230 z"
+         style="fill:#fbe5d6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,423.5,348.5)"><path
+           d="M 0,0 H 2921000 V 215900 H 0 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path28" /></g><g
+         id="g30"
+         transform="translate(654.1775,-47)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text34"><tspan
+             x="0 12 16 24 28 37 42 50 54 63 67 72 76 85 89 98 103 112 120 124 128 137 146 154"
+             y="0"
+             sodipodi:role="line"
+             id="tspan32">History of Interactions </tspan></text></g><g
+         id="g36"
+         transform="translate(671.1775,-66)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text40"><tspan
+             x="0 4 9.0080004 21.007999 30.016001 39.023998 48.032001 52.032001 61.040001 70.047997 74.047997 83.056 86.047997 90.047997 94.047997 103.056 116.064"
+             y="0"
+             sodipodi:role="line"
+             id="tspan38">/ Candidate Items</tspan></text></g><path
+         d="m 544,327.5 10e-5,-13.022 -3.0001,-1e-4 V 327.5 Z m 3,-11.522 -4.4999,-9 -4.5001,8.9999 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path42" /><path
+         d="m 354.5,283.1662 c 0,13.9915 11.3423,25.3338 25.3338,25.3338 h 133.3324 c 13.9915,0 25.3338,-11.3423 25.3338,-25.3338 V 181.8338 C 538.5,167.8423 527.1577,156.5 513.1662,156.5 H 379.8338 c -13.9915,0 -25.3338,11.3423 -25.3338,25.3338 z"
+         style="fill:#f8cbad;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path44" /><g
+         id="g46"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,354.5,308.5)"><path
+           d="M 0,321739.7 C 0,144047.8 144047.6,0 321739.3,0 H 2015061 c 177691,0 321739,144047.8 321739,321739.7 V 1608660 c 0,177692 -144048,321740 -321739,321740 H 321739.3 C 144047.6,1930400 0,1786352 0,1608660 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path48" /></g><g
+         id="g50"
+         transform="translate(403.2841,-150)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text54"><tspan
+             x="0 8 16 24 32 46 50 57.007999 60 68 76"
+             y="0"
+             sodipodi:role="line"
+             id="tspan52">1000M x 100</tspan></text></g><g
+         id="g56"
+         transform="translate(392.2841,-169)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text60"><tspan
+             x="0 10 16 24 30 34 42 55.007999 63.007999 71.008003 79.008003 87.008003 91.008003 99.008003"
+             y="0"
+             sodipodi:role="line"
+             id="tspan58">User Embedding</tspan></text></g><path
+         d="m 555.5,302.1665 c 0,4.0502 3.2833,7.3335 7.3335,7.3335 h 157.333 c 4.0502,0 7.3335,-3.2833 7.3335,-7.3335 v -29.333 c 0,-4.0502 -3.2833,-7.3335 -7.3335,-7.3335 h -157.333 c -4.0502,0 -7.3335,3.2833 -7.3335,7.3335 z"
+         style="fill:#f8cbad;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path62" /><g
+         id="g64"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,555.5,309.5)"><path
+           d="M 0,93135.72 C 0,41698.28 41698.23,0 93135.61,0 H 2091264 c 51438,0 93136,41698.28 93136,93135.72 V 465664.3 c 0,51437.4 -41698,93135.7 -93136,93135.7 H 93135.61 C 41698.23,558800 0,517101.7 0,465664.3 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path66" /></g><g
+         id="g68"
+         transform="translate(606.428,-95)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text72"><tspan
+             x="0 8 16 30 34 41.007999 44 52 60"
+             y="0"
+             sodipodi:role="line"
+             id="tspan70">10M x 100</tspan></text></g><g
+         id="g74"
+         transform="translate(587.428,-114)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text78"><tspan
+             x="0 4 8.9919996 16.992001 30 34 42 55.007999 63.007999 71.008003 79.008003 87.008003 91.008003 99.008003"
+             y="0"
+             sodipodi:role="line"
+             id="tspan76">Item Embedding</tspan></text></g><path
+         d="m 555.5,249.1665 c 0,4.0502 3.2833,7.3335 7.3335,7.3335 h 157.333 c 4.0502,0 7.3335,-3.2833 7.3335,-7.3335 v -29.333 c 0,-4.0502 -3.2833,-7.3335 -7.3335,-7.3335 h -157.333 c -4.0502,0 -7.3335,3.2833 -7.3335,7.3335 z"
+         style="fill:#f8cbad;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path80" /><g
+         id="g82"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,555.5,256.5)"><path
+           d="M 0,93135.72 C 0,41698.28 41698.23,0 93135.61,0 H 2091264 c 51438,0 93136,41698.28 93136,93135.72 V 465664.3 c 0,51437.4 -41698,93135.7 -93136,93135.7 H 93135.61 C 41698.23,558800 0,517101.7 0,465664.3 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path84" /></g><g
+         id="g86"
+         transform="translate(598.4281,-148)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text90"><tspan
+             x="0 10 16 24 30 34 43.007999 51.007999 59.007999 64 72 79.008003 84"
+             y="0"
+             sodipodi:role="line"
+             id="tspan88">User Context </tspan></text></g><g
+         id="g92"
+         transform="translate(604.4281,-167)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text96"><tspan
+             x="0 8 21.007999 29.007999 37.007999 45.007999 53.007999 57.007999 65.008003"
+             y="0"
+             sodipodi:role="line"
+             id="tspan94">Embedding</tspan></text></g><path
+         d="m 555.5,193.1665 c 0,4.0502 3.2833,7.3335 7.3335,7.3335 h 157.333 c 4.0502,0 7.3335,-3.2833 7.3335,-7.3335 v -29.333 c 0,-4.0502 -3.2833,-7.3335 -7.3335,-7.3335 h -157.333 c -4.0502,0 -7.3335,3.2833 -7.3335,7.3335 z"
+         style="fill:#f8cbad;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path98" /><g
+         id="g100"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,555.5,200.5)"><path
+           d="M 0,93135.72 C 0,41698.28 41698.23,0 93135.61,0 H 2091264 c 51438,0 93136,41698.28 93136,93135.72 V 465664.3 c 0,51437.4 -41698,93135.7 -93136,93135.7 H 93135.61 C 41698.23,558800 0,517101.7 0,465664.3 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path102" /></g><g
+         id="g104"
+         transform="translate(598.4281,-203)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text108"><tspan
+             x="0 4 8.9919996 16.992001 30 34 43.007999 51.007999 59.007999 64 72 79.008003 84"
+             y="0"
+             sodipodi:role="line"
+             id="tspan106">Item Context </tspan></text></g><g
+         id="g110"
+         transform="translate(604.4281,-222)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text114"><tspan
+             x="0 8 21.007999 29.007999 37.007999 45.007999 53.007999 57.007999 65.008003"
+             y="0"
+             sodipodi:role="line"
+             id="tspan112">Embedding</tspan></text></g><path
+         d="m 361.5,138.1665 c 0,4.6025 3.731,8.3335 8.3334,8.3335 h 345.3332 c 4.6024,0 8.3334,-3.731 8.3334,-8.3335 v -33.333 C 723.5,100.231 719.769,96.5 715.1666,96.5 H 369.8334 c -4.6024,0 -8.3334,3.731 -8.3334,8.3335 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path116" /><g
+         id="g118"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,361.5,146.5)"><path
+           d="M 0,105835 C 0,47383.95 47383.82,0 105834.7,0 H 4491565 c 58451,0 105835,47383.95 105835,105835 v 423330 c 0,58451.1 -47384,105835 -105835,105835 H 105834.7 C 47383.82,635000 0,587616.1 0,529165 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path120" /></g><path
+         d="m 392.5,139.5 h 300 v -11 h -300 z"
+         style="fill:#ed7d31;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path122" /><g
+         id="g124"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,392.5,139.5)"><path
+           d="M 0,0 H 3810000 V 139700 H 0 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path126" /></g><path
+         d="m 424.5,118.5 h 239 v -11 h -239 z"
+         style="fill:#ed7d31;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path128" /><g
+         id="g130"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,424.5,118.5)"><path
+           d="M 0,0 H 3035300 V 139700 H 0 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path132" /></g><path
+         d="m 463.5,78.5 h 157 v -15 h -157 z"
+         style="fill:#fbe5d6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path134" /><g
+         id="g136"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,463.5,78.5)"><path
+           d="M 0,0 H 1993900 V 190500 H 0 Z"
+           style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path138" /></g><path
+         d="m 543.9996,99.53309 0.3528,-15.98875 -2.9992,-0.06618 -0.3528,15.98875 z m 3.319,-14.42294 -4.3003,-9.09708 -4.6975,8.89854 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path140" /><g
+         id="g142"
+         transform="translate(630.2726,-322)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text146"><tspan
+             x="0 12 21 29 38 51 64 73 82 91 100 104 108 117 126 134 138 142 147 159 168 177"
+             y="0"
+             sodipodi:role="line"
+             id="tspan144">Recommendations / Rank</tspan></text></g><g
+         id="g148"
+         transform="translate(741.0651,-156)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text152"><tspan
+             x="0 10.0008 20.0016 30.002399 39.002399 44.0028 58.003201 70.003799"
+             y="0"
+             sodipodi:role="line"
+             id="tspan150">100s GB </tspan></text></g><g
+         id="g154"
+         transform="translate(816.0651,-156)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text158"><tspan
+             x="0"
+             y="0"
+             id="tspan156">–</tspan></text></g><g
+         id="g160"
+         transform="translate(831.0651,-156)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text164"><tspan
+             x="0 10.0008 20.0016 29.0016 34.001999 45.001801"
+             y="0"
+             sodipodi:role="line"
+             id="tspan162">10s TB</tspan></text></g><g
+         id="g166"
+         transform="translate(741.0651,-273)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text170"><tspan
+             x="0 10.0008 20.0016 29.0016 34.001999 48.002399 60.002998"
+             y="0"
+             sodipodi:role="line"
+             id="tspan168">10s GB </tspan></text></g><g
+         id="g172"
+         transform="translate(806.0651,-273)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text176"><tspan
+             x="0"
+             y="0"
+             id="tspan174">–</tspan></text></g><g
+         id="g178"
+         transform="translate(821.0651,-273)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text182"><tspan
+             x="0 10.0008 20.0016 30.002399 39.002399 44.0028 58.003201"
+             y="0"
+             sodipodi:role="line"
+             id="tspan180">100s GB</tspan></text></g><g
+         id="g184"
+         transform="translate(447.3055,-351)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text188"><tspan
+             x="0"
+             y="0"
+             id="tspan186">•</tspan></text></g><g
+         id="g190"
+         transform="translate(470.3055,-351)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text194"><tspan
+             x="0 10.9998 16.999201 27 41.999401 52.000198 62.000999 72.001801 82.002602 92.003403 101.0034 106.0038 121.0032 131.004 146.0034 156.0042 162.0036 171.0036 176.004 185.004 195.00481 204.00481"
+             y="0"
+             sodipodi:role="line"
+             id="tspan192">Tremendous memory cost</tspan></text></g><g
+         id="g196"
+         transform="translate(447.3055,-372)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text200"><tspan
+             x="0"
+             y="0"
+             id="tspan198">•</tspan></text></g><g
+         id="g202"
+         transform="translate(470.3055,-372)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text206"><tspan
+             x="0 12.0006 27 37.000801 41.000401 45 50.000401 59.000401 69.001198 84.000603 94.001404 104.0022 109.0026 119.0034 124.0038 128.0034 138.0042 148.005 153.0054 162.0054 172.0062 181.0062"
+             y="0"
+             sodipodi:role="line"
+             id="tspan204">Small computation cost</tspan></text></g><path
+         d="m 78.5,344.5 h 84 v -33 h -84 z"
+         style="fill:#e2f0d9;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path208" /><g
+         id="g210"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,78.5,344.5)"><path
+           d="M 0,0 H 1066800 V 419100 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path212" /></g><g
+         id="g214"
+         transform="translate(99.07492,-64)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text218"><tspan
+             x="0 4 17.007999 25.007999 33.007999"
+             y="0"
+             sodipodi:role="line"
+             id="tspan216">Image</tspan></text></g><path
+         d="M 21,260.166 C 21,277.1951 34.80485,291 51.83399,291 H 184.166 C 201.1952,291 215,277.1951 215,260.166 V 136.834 C 215,119.8049 201.1952,106 184.166,106 H 51.83399 C 34.80485,106 21,119.8049 21,136.834 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path220" /><g
+         id="g222"
+         transform="translate(76.05567,-314)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:16px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text226"><tspan
+             x="0 12 16 25 33 41 45 49 53 61 70 74 78 87"
+             y="0"
+             sodipodi:role="line"
+             id="tspan224">Classification</tspan></text></g><path
+         d="m 77.5,277.5 h 84 v -17 h -84 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path228" /><g
+         id="g230"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,77.5,277.5)"><path
+           d="M 0,0 H 1066800 V 215900 H 0 Z"
+           style="fill:none;stroke:#70ad47;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path232" /></g><path
+         d="m 90.5,253.5 h 60 v -16 h -60 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path234" /><g
+         id="g236"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,90.5,253.5)"><path
+           d="M 0,0 H 762000 V 203200 H 0 Z"
+           style="fill:none;stroke:#70ad47;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path238" /></g><path
+         d="m 36.5,157.5 h 165 v -17 h -165 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path240" /><g
+         id="g242"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,36.5,157.5)"><path
+           d="M 0,0 H 2095500 V 215900 H 0 Z"
+           style="fill:none;stroke:#70ad47;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path244" /></g><path
+         d="m 90.5,230.5 h 60 v -17 h -60 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path246" /><g
+         id="g248"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,90.5,230.5)"><path
+           d="M 0,0 H 762000 V 215900 H 0 Z"
+           style="fill:none;stroke:#70ad47;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path250" /></g><path
+         d="m 77.5,207.5 h 84 v -17 h -84 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path252" /><g
+         id="g254"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,77.5,207.5)"><path
+           d="M 0,0 H 1066800 V 215900 H 0 Z"
+           style="fill:none;stroke:#70ad47;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path256" /></g><path
+         d="m 63.5,181.5 h 111 v -17 h -111 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path258" /><g
+         id="g260"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,63.5,181.5)"><path
+           d="M 0,0 H 1409700 V 215900 H 0 Z"
+           style="fill:none;stroke:#70ad47;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path262" /></g><path
+         d="m 64.5,132.5 h 111 v -17 h -111 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path264" /><g
+         id="g266"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,64.5,132.5)"><path
+           d="M 0,0 H 1409700 V 215900 H 0 Z"
+           style="fill:none;stroke:#70ad47;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path268" /></g><path
+         d="m 119.75,309 1e-4,-21.637 h -1.5 L 118.25,309 Z m 2.2501,-20.637 -3,-6 -3,6 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path270" /><path
+         d="m 117.75,116 1e-4,-18.38685 h -1.5 L 116.25,116 Z m 2.2501,-17.38684 -3,-6.00001 -3,5.99999 z"
+         style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path272" /><g
+         id="g274"
+         transform="translate(20.75323,-349)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text278"><tspan
+             x="0"
+             y="0"
+             id="tspan276">•</tspan></text></g><g
+         id="g280"
+         transform="translate(43.75323,-349)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text284"><tspan
+             x="0 14.9994 25.0002 35.000999 45.001801 51.001202 61.001999 66.002403 76.003197 81.003601 96.002998 106.0038 121.0032 131.004 137.0034 146.0034 151.0038 160.0038 170.00459 179.00459"
+             y="0"
+             sodipodi:role="line"
+             id="tspan282">Moderate memory cost</tspan></text></g><g
+         id="g286"
+         transform="translate(20.75323,-370)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text290"><tspan
+             x="0"
+             y="0"
+             id="tspan288">•</tspan></text></g><g
+         id="g292"
+         transform="translate(43.75323,-370)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text296"><tspan
+             x="0 10.0008 20.0016 26.000999 36.001801 46.002602 51.002998 60.002998 70.003799 85.003197 95.003998 105.0048 110.0052 120.006 125.0064 129.006 139.00681 149.0076 154.008 163.008 173.0088 182.0088"
+             y="0"
+             sodipodi:role="line"
+             id="tspan294">Large computation cost</tspan></text></g><g
+         id="g298"
+         transform="translate(11.91504,-28)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text302"><tspan
+             x="0 12.9996 23.000401 33.001202 39.000599 49.0014 53.000999 58.0014 68.002197 78.002998 83.003403 96.002998 106.0038 112.0032 121.0032 126.0036 131.004 137.0034 147.0042 151.0038 161.00459 165.0042 175.005 185.0058 190.0062 199.0062 208.0062 217.0062 222.00661 232.0074"
+             y="0"
+             sodipodi:role="line"
+             id="tspan300">Neural network training system</tspan></text></g><g
+         id="g304"
+         transform="translate(449.7736,-24)"><text
+           transform="matrix(1,0,0,-1,0,387)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Arial;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text308"><tspan
+             x="0 12.9996 23.000401 32.000401 42.001202 57.000599 72 82.000801 92.001602 102.0024 112.0032 117.0036 121.0032 131.004 141.00481 146.0052 155.0052 164.0052 173.0052 178.0056 188.00639"
+             y="0"
+             sodipodi:role="line"
+             id="tspan306">Recommendation system</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-single-node.pdf
+++ b/img/ch09/ch10-single-node.pdf
--- a/img/ch09/ch10-single-node.svg
+++ b/img/ch09/ch10-single-node.svg
@@ -0,0 +1,309 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="1052.9974"
+   height="346.66837"
+   viewBox="0 0 1052.9975 346.66837"
+   sodipodi:docname="ch10-single-node.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 802 V 260 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     inkscape:zoom="0.85170225"
+     inkscape:cx="554.77134"
+     inkscape:cy="187.85908"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-single-node"
+     transform="matrix(1.3333333,0,0,-1.3333333,-16.336198,346.66818)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="m 82.5,216.3324 c 0,23.8408 19.3268,43.1676 43.1675,43.1676 h 632.665 c 23.8407,0 43.1675,-19.3268 43.1675,-43.1676 V 43.66758 C 801.5,19.82678 782.1732,0.4999909 758.3325,0.4999909 H 125.6675 C 101.8268,0.4999909 82.5,19.82678 82.5,43.66758 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,82.5,259.5)"><path
+           d="M 0,548228.3 C 0,245450.2 245449.8,0 548227.5,0 H 8583072 c 302778,0 548228,245450.2 548228,548228.3 V 2741072 c 0,302778 -245450,548228 -548228,548228 H 548227.5 C 245449.8,3289300 0,3043850 0,2741072 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path28" /></g><path
+         d="m 491.5,204.5 h 72 v -138 h -72 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path30" /><g
+         id="g32"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,491.5,204.5)"><path
+           d="M 0,0 H 914400 V 1752600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path34" /></g><path
+         d="m 653.5,205.5 h 72 v -138 h -72 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path36" /><g
+         id="g38"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,653.5,205.5)"><path
+           d="M 0,0 H 914400 V 1752600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path40" /></g><path
+         d="m 330.5,204.5 h 72 v -138 h -72 z"
+         style="fill:#a9d18e;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path42" /><g
+         id="g44"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,330.5,204.5)"><path
+           d="M 0,0 H 914400 V 1752600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path46" /></g><path
+         d="m 108.5,204.5 h 72 v -138 h -72 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path48" /><g
+         id="g50"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,108.5,204.5)"><path
+           d="M 0,0 H 914400 V 1752600 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path52" /></g><g
+         id="g54"
+         transform="translate(345.5718,-125)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text58"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan56">算⼦</tspan></text></g><g
+         id="g60"
+         transform="translate(381.5718,-125)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0"
+             y="0"
+             id="tspan62">1</tspan></text></g><g
+         id="g66"
+         transform="translate(331.5718,-147)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text70"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan68">（参数）</tspan></text></g><g
+         id="g72"
+         transform="translate(505.8641,-125)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text76"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan74">算⼦</tspan></text></g><g
+         id="g78"
+         transform="translate(541.8641,-125)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text82"><tspan
+             x="0"
+             y="0"
+             id="tspan80">2</tspan></text></g><g
+         id="g84"
+         transform="translate(491.8641,-147)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text88"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan86">（参数）</tspan></text></g><g
+         id="g90"
+         transform="translate(665.4562,-125)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text94"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan92">算⼦</tspan></text></g><g
+         id="g96"
+         transform="translate(701.4562,-125)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text100"><tspan
+             x="0"
+             y="0"
+             id="tspan98">3</tspan></text></g><g
+         id="g102"
+         transform="translate(651.4562,-147)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text106"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan104">（参数）</tspan></text></g><g
+         id="g108"
+         transform="translate(488.5967,-34)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:bold;font-size:20px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Bold;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text112"><tspan
+             x="0 20"
+             y="0"
+             sodipodi:role="line"
+             id="tspan110">程序</tspan></text></g><g
+         id="g114"
+         transform="translate(210.1814,-64)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text118"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan116">前向</tspan></text></g><g
+         id="g120"
+         transform="translate(601.0888,-184)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text124"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan122">反向</tspan></text></g><g
+         id="g126"
+         transform="translate(123.1056,-132)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:bold;font-size:20px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Bold;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text130"><tspan
+             x="0 20"
+             y="0"
+             sodipodi:role="line"
+             id="tspan128">数据</tspan></text></g><g
+         id="g132"
+         transform="translate(11.86543,-140)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text136"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan134">设备</tspan></text></g><g
+         id="g138"
+         transform="translate(47.86543,-140)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text142"><tspan
+             x="0"
+             y="0"
+             id="tspan140">1</tspan></text></g><path
+         d="m 180.5,181.75 129.4448,-1e-4 v -4.5 L 180.5,177.25 Z m 127.1948,4.4999 13.5,-6.75 -13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path144" /><path
+         d="m 402.5,181.75 75.7035,-1e-4 v -4.5 L 402.5,177.25 Z m 73.4535,4.4999 13.5,-6.75 -13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path146" /><path
+         d="m 563.5,180.75 78.6561,-1e-4 v -4.5 L 563.5,176.25 Z m 76.4061,4.4999 13.5,-6.75 -13.5,-6.75 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path148" /><path
+         d="m 653.3761,100.7499 -77.6501,-0.80835 0.0469,-4.49976 77.65,0.80833 z m -75.4471,3.7148 -13.429,-6.89013 13.5696,-6.60911 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path150" /><path
+         d="m 491.2052,99.75 -77.4552,-7e-5 v -4.5 l 77.4552,7e-5 z M 416,104.2499 402.5,97.49992 416,90.74993 Z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path152" /><path
+         d="m 319.5,56.5 h 93 v -37 h -93 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path154" /><g
+         id="g156"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,319.5,56.5)"><path
+           d="M 0,0 H 1181100 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path158" /></g><g
+         id="g160"
+         transform="translate(343.4833,-226)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text164"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan162">梯度</tspan></text></g><g
+         id="g166"
+         transform="translate(379.4833,-226)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text170"><tspan
+             x="0"
+             y="0"
+             id="tspan168">1</tspan></text></g><path
+         d="m 478.5,56.5 h 93 v -37 h -93 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path172" /><g
+         id="g174"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,478.5,56.5)"><path
+           d="M 0,0 H 1181100 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path176" /></g><g
+         id="g178"
+         transform="translate(504.7962,-227)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text182"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan180">梯度</tspan></text></g><g
+         id="g184"
+         transform="translate(540.7962,-227)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text188"><tspan
+             x="0"
+             y="0"
+             id="tspan186">2</tspan></text></g><path
+         d="m 640.5,57.5 h 93 v -37 h -93 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path190" /><g
+         id="g192"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,640.5,57.5)"><path
+           d="M 0,0 H 1181100 V 469900 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path194" /></g><g
+         id="g196"
+         transform="translate(664.2755,-227)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text200"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan198">梯度</tspan></text></g><g
+         id="g202"
+         transform="translate(700.2755,-227)"><text
+           transform="matrix(1,0,0,-1,0,260)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-SC;-inkscape-font-specification:STSongti-SC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text206"><tspan
+             x="0"
+             y="0"
+             id="tspan204">3</tspan></text></g></g></g></svg>
--- a/img/ch09/ch10-single-vs-multi.pdf
+++ b/img/ch09/ch10-single-vs-multi.pdf
--- a/img/ch09/ch10-single-vs-multi.svg
+++ b/img/ch09/ch10-single-vs-multi.svg
@@ -0,0 +1,339 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   version="1.1"
+   id="svg2"
+   xml:space="preserve"
+   width="1210.6666"
+   height="389.91666"
+   viewBox="0 0 1210.6666 389.91665"
+   sodipodi:docname="ch10-single-vs-multi.svg"
+   inkscape:version="1.1.1 (3bf5ae0d25, 2021-09-20)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+     id="defs6"><clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath18"><path
+         d="M 0,0 H 914 V 310 H 0 Z"
+         id="path16" /></clipPath></defs><sodipodi:namedview
+     id="namedview4"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.53460062"
+     inkscape:cx="416.19854"
+     inkscape:cy="192.66719"
+     inkscape:window-width="1920"
+     inkscape:window-height="1137"
+     inkscape:window-x="1912"
+     inkscape:window-y="-8"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g8" /><g
+     id="g8"
+     inkscape:groupmode="layer"
+     inkscape:label="ch10-single-vs-multi"
+     transform="matrix(1.3333333,0,0,-1.3333333,-7.999998,407.99999)"><g
+       id="g10" /><g
+       id="g22"><path
+         d="m 6.5,224.6663 c 0,7.0879 5.74582,12.8337 12.83364,12.8337 H 167.6664 c 7.0878,0 12.8336,-5.7458 12.8336,-12.8337 V 173.3336 C 180.5,166.2458 174.7542,160.5 167.6664,160.5 H 19.33364 C 12.24582,160.5 6.5,166.2458 6.5,173.3336 Z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path24" /><g
+         id="g26"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,6.5,237.5)"><path
+           d="M 0,162987.4 C 0,72971.93 72971.88,0 162987.3,0 H 2046813 c 90015,0 162987,72971.93 162987,162987.4 v 651925.3 c 0,90015.4 -72972,162987.3 -162987,162987.3 H 162987.3 C 72971.88,977900 0,904928.1 0,814912.7 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path28" /></g><path
+         d="m 6.5,305.5 h 174 v -30 H 6.5 Z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path30" /><g
+         id="g32"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,6.5,305.5)"><path
+           d="M 0,0 H 2209800 V 381000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path34" /></g><path
+         d="m 544.5,305.5 h 57 v -31 h -57 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path36" /><g
+         id="g38"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,544.5,305.5)"><path
+           d="M 0,0 H 723900 V 393700 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path40" /></g><path
+         d="m 605.5,305.5 h 57 v -31 h -57 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path42" /><g
+         id="g44"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,605.5,305.5)"><path
+           d="M 0,0 H 723900 V 393700 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path46" /></g><path
+         d="m 6.5,119.5 h 174 v -30 H 6.5 Z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path48" /><g
+         id="g50"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,6.5,119.5)"><path
+           d="M 0,0 H 2209800 V 381000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path52" /></g><g
+         id="g54"
+         transform="translate(53.74378,-116)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text58"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan56">计算节点</tspan></text></g><g
+         id="g60"
+         transform="translate(72.88779,-23)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text64"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan62">输入</tspan></text></g><g
+         id="g66"
+         transform="translate(66.13496,-210)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text70"><tspan
+             x="0 18"
+             y="0"
+             sodipodi:role="line"
+             id="tspan68">输出</tspan></text></g><path
+         d="M 97.25,275 V 262.0899 H 102 L 92.5,237 83,262.0899 h 4.75 V 275 Z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path72" /><path
+         d="M 97.25,160 V 146.0898 H 102 L 92.5,121 83,146.0898 h 4.75 V 160 Z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path74" /><g
+         id="g76"
+         transform="translate(12.7774,-293)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:20px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text80"><tspan
+             x="0"
+             y="0"
+             id="tspan78">（</tspan></text></g><g
+         id="g82"
+         transform="translate(32.7774,-293)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:20px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text86"><tspan
+             x="0"
+             y="0"
+             id="tspan84">a</tspan></text></g><g
+         id="g88"
+         transform="translate(42.7774,-293)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:20px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text92"><tspan
+             x="0"
+             y="0"
+             id="tspan90">）</tspan></text></g><g
+         id="g94"
+         transform="translate(62.7774,-293)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:20px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text98"><tspan
+             x="0 20 40 60 80"
+             y="0"
+             sodipodi:role="line"
+             id="tspan96">单节点执⾏</tspan></text></g><path
+         d="m 357.5,222.6663 c 0,7.0879 5.7458,12.8337 12.8336,12.8337 h 148.3328 c 7.0878,0 12.8336,-5.7458 12.8336,-12.8337 V 171.3336 C 531.5,164.2458 525.7542,158.5 518.6664,158.5 H 370.3336 c -7.0878,0 -12.8336,5.7458 -12.8336,12.8336 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path100" /><g
+         id="g102"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,357.5,235.5)"><path
+           d="M 0,162987.4 C 0,72971.93 72971.88,0 162987.3,0 H 2046813 c 90015,0 162987,72971.93 162987,162987.4 v 651925.3 c 0,90015.4 -72972,162987.3 -162987,162987.3 H 162987.3 C 72971.88,977900 0,904928.1 0,814912.7 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path104" /></g><g
+         id="g106"
+         transform="translate(400.5202,-118)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text110"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan108">计算节点</tspan></text></g><g
+         id="g112"
+         transform="translate(477.5202,-118)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text116"><tspan
+             x="0"
+             y="0"
+             id="tspan114">1</tspan></text></g><path
+         d="m 549.5,222.6663 c 0,7.0879 5.7458,12.8337 12.8336,12.8337 h 148.3328 c 7.0878,0 12.8336,-5.7458 12.8336,-12.8337 V 171.3336 C 723.5,164.2458 717.7542,158.5 710.6664,158.5 H 562.3336 c -7.0878,0 -12.8336,5.7458 -12.8336,12.8336 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path118" /><g
+         id="g120"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,549.5,235.5)"><path
+           d="M 0,162987.4 C 0,72971.93 72971.88,0 162987.3,0 H 2046813 c 90015,0 162987,72971.93 162987,162987.4 v 651925.3 c 0,90015.4 -72972,162987.3 -162987,162987.3 H 162987.3 C 72971.88,977900 0,904928.1 0,814912.7 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path122" /></g><path
+         d="m 666.5,305.5 h 58 v -30 h -58 z"
+         style="fill:#9dc3e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path124" /><g
+         id="g126"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,666.5,305.5)"><path
+           d="M 0,0 H 736600 V 381000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path128" /></g><g
+         id="g130"
+         transform="translate(591.9943,-119)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text134"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan132">计算节点</tspan></text></g><g
+         id="g136"
+         transform="translate(668.9943,-119)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text140"><tspan
+             x="0"
+             y="0"
+             id="tspan138">2</tspan></text></g><path
+         d="m 739.5,222.6663 c 0,7.0879 5.7458,12.8337 12.8336,12.8337 h 148.3328 c 7.0878,0 12.8336,-5.7458 12.8336,-12.8337 V 171.3336 C 913.5,164.2458 907.7542,158.5 900.6664,158.5 H 752.3336 c -7.0878,0 -12.8336,5.7458 -12.8336,12.8336 z"
+         style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path142" /><g
+         id="g144"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,739.5,235.5)"><path
+           d="M 0,162987.4 C 0,72971.93 72971.88,0 162987.3,0 H 2046813 c 90015,0 162987,72971.93 162987,162987.4 v 651925.3 c 0,90015.4 -72972,162987.3 -162987,162987.3 H 162987.3 C 72971.88,977900 0,904928.1 0,814912.7 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:50800, 38100;stroke-dashoffset:0;stroke-opacity:1"
+           id="path146" /></g><g
+         id="g148"
+         transform="translate(784.5895,-120)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text152"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan150">计算节点</tspan></text></g><g
+         id="g154"
+         transform="translate(861.5895,-120)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text158"><tspan
+             x="0"
+             y="0"
+             id="tspan156">3</tspan></text></g><path
+         d="m 541.5,109.5 h 58 v -30 h -58 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path160" /><g
+         id="g162"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,541.5,109.5)"><path
+           d="M 0,0 H 736600 V 381000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path164" /></g><path
+         d="m 603.5,109.5 h 57 v -30 h -57 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path166" /><g
+         id="g168"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,603.5,109.5)"><path
+           d="M 0,0 H 723900 V 381000 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path170" /></g><path
+         d="m 664.5,110.5 h 57 v -31 h -57 z"
+         style="fill:#afabab;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path172" /><g
+         id="g174"
+         transform="matrix(7.874016e-5,0,0,-7.874016e-5,664.5,110.5)"><path
+           d="M 0,0 H 723900 V 393700 H 0 Z"
+           style="fill:none;stroke:#000000;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+           id="path176" /></g><path
+         d="m 543.5989,265.8866 -16.202,-13.1273 3.0251,-3.7336 -25.7713,-8.5115 13.671,23.446 3.0251,-3.7336 16.202,13.1273 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path178" /><path
+         d="M 637.25,273 V 259.0898 H 642 L 632.5,234 623,259.0898 h 4.75 V 273 Z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path180" /><path
+         d="m 729.2845,272.8375 15.5861,-12.7986 3.0495,3.7137 13.517,-23.5351 -25.715,8.6803 3.0495,3.7137 -15.5861,12.7986 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path182" /><g
+         id="g184"
+         transform="translate(456.0262,-26)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text188"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan186">切分输入</tspan></text></g><path
+         d="m 512.5196,152.6016 14.7431,-14.7469 3.3983,3.3974 11.1488,-24.745 -24.742,11.1553 3.3983,3.3974 -14.7431,14.747 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path190" /><path
+         d="M 636.25,154 V 141.0899 H 641 L 631.5,116 622,141.0899 h 4.75 V 154 Z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path192" /><path
+         d="m 759.1565,146.595 -13.9554,-14.5595 3.4691,-3.3251 -24.5018,-11.6736 10.6255,24.9742 3.4691,-3.3252 13.9554,14.5594 z"
+         style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+         id="path194" /><g
+         id="g196"
+         transform="translate(454.8528,-223)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text200"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan198">合并输出</tspan></text></g><g
+         id="g202"
+         transform="translate(500.6803,-293)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:20px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text206"><tspan
+             x="0"
+             y="0"
+             id="tspan204">（</tspan></text></g><g
+         id="g208"
+         transform="translate(520.6803,-293)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:20px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text212"><tspan
+             x="0"
+             y="0"
+             id="tspan210">b</tspan></text></g><g
+         id="g214"
+         transform="translate(530.6803,-293)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:20px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text218"><tspan
+             x="0"
+             y="0"
+             id="tspan216">）</tspan></text></g><g
+         id="g220"
+         transform="translate(550.6803,-293)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:20px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text224"><tspan
+             x="0 20 40 60 80 100 120 140"
+             y="0"
+             sodipodi:role="line"
+             id="tspan222">分布式多节点执⾏</tspan></text></g><g
+         id="g226"
+         transform="translate(269.4391,-117)"><text
+           transform="matrix(1,0,0,-1,0,310)"
+           style="font-variant:normal;font-weight:normal;font-size:18px;font-family:STSongti-TC;-inkscape-font-specification:STSongti-TC-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="text230"><tspan
+             x="0 18 36 54"
+             y="0"
+             sodipodi:role="line"
+             id="tspan228">并⾏计算</tspan></text></g></g></g></svg>
--- a/img/ch10/ch10-abstract-recommendation-systems.svg
+++ b/img/ch10/ch10-abstract-recommendation-systems.svg
@@ -0,0 +1,378 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   width="245.84119mm"
+   height="115.22363mm"
+   viewBox="0 0 245.84119 115.22364"
+   version="1.1"
+   id="svg5"
+   inkscape:version="1.1.2 (1:1.1+202202050950+0a00cf5339)"
+   sodipodi:docname="ch10-abstract-recommendation-systems.svg"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg">
+  <sodipodi:namedview
+     id="namedview7"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     inkscape:document-units="mm"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.90583672"
+     inkscape:cx="809.74858"
+     inkscape:cy="397.97459"
+     inkscape:window-width="1848"
+     inkscape:window-height="1136"
+     inkscape:window-x="72"
+     inkscape:window-y="27"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g1236" />
+  <defs
+     id="defs2">
+    <clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath25">
+      <path
+         d="M 0,0.1636355 H 536.9455 V 351.00004 H 0 Z"
+         id="path23" />
+    </clipPath>
+    <clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath905">
+      <path
+         d="M 0,1.206994e-5 H 959.76 V 540.00001 H 0 Z"
+         id="path903" />
+    </clipPath>
+    <clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath1246">
+      <path
+         d="M 0,0.07272646 H 698.8364 V 328.00003 H 0 Z"
+         id="path1244" />
+    </clipPath>
+  </defs>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-7.4951336,-56.575409)">
+    <g
+       id="g15"
+       inkscape:label="ch11-recommendation-models(1)"
+       transform="matrix(0.35277777,0,0,-0.35277777,44.08155,178.42815)">
+      <g
+         id="g17" />
+      <g
+         id="g29">
+        <g
+           id="g1236"
+           inkscape:label="ch11-abstract-recommendation-systems(1)"
+           transform="translate(-314.29928,7.6998617)">
+          <g
+             id="g1238" />
+          <g
+             id="g1250"
+             transform="translate(-74.216339)">
+            <path
+               d="M 284.80609,337.70949 H 981.67874 V 74.364063 H 284.80609 Z"
+               style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+               id="path1252" />
+            <path
+               d="m 658.55154,246.14539 c 0,12.1305 9.8336,21.9641 21.9641,21.9641 h 87.8536 c 12.1305,0 21.9641,-9.8336 21.9641,-21.9641 v -132.3628 c 0,-12.1304 -9.8336,-21.964037 -21.9641,-21.964037 h -87.8536 c -12.1305,0 -21.9641,9.833637 -21.9641,21.964037 z"
+               style="fill:#deebf7;fill-opacity:1;fill-rule:evenodd;stroke:none"
+               id="path1258" />
+            <g
+               id="g1260"
+               transform="matrix(7.874016e-5,0,0,-7.874016e-5,658.55154,268.10949)">
+              <path
+                 d="M 0,278943.7 C 0,124887.3 124887.4,0 278943.9,0 H 1394685 c 154057,0 278944,124887.3 278944,278943.7 V 1959951 c 0,154057 -124887,278944 -278944,278944 H 278943.9 C 124887.4,2238895 0,2114008 0,1959951 Z"
+                 style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+                 id="path1262" />
+            </g>
+            <g
+               id="g1264"
+               transform="matrix(0.2181818,0,0,0.2181818,688.28324,-153.92681)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1268"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1266">模型</tspan></text>
+            </g>
+            <g
+               id="g1270"
+               transform="matrix(0.2181818,0,0,0.2181818,724.28324,-153.92681)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1274"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1272">训练</tspan></text>
+            </g>
+            <path
+               d="m 315.7879,147.81839 c 0,6.1857 5.01448,11.2002 11.20016,11.2002 h 109.38148 c 6.1857,0 11.2002,-5.0145 11.2002,-11.2002 v -44.79967 c 0,-6.185677 -5.0145,-11.200167 -11.2002,-11.200167 H 326.98806 c -6.18568,0 -11.20016,5.01449 -11.20016,11.200167 z"
+               style="fill:#e2f0d9;fill-opacity:1;fill-rule:evenodd;stroke:none"
+               id="path1276" />
+            <g
+               id="g1278"
+               transform="matrix(7.874016e-5,0,0,-7.874016e-5,315.7879,159.01859)">
+              <path
+                 d="M 0,142242.1 C 0,63683.97 63683.91,0 142242,0 h 1389145 c 78558,0 142242,63683.97 142242,142242.1 v 568955.8 c 0,78558.1 -63684,142242.1 -142242,142242.1 H 142242 C 63683.91,853440 0,789756 0,711197.9 Z"
+                 style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+                 id="path1280" />
+            </g>
+            <g
+               id="g1282"
+               transform="matrix(0.2181818,0,0,0.2181818,345.51482,-208.25411)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1286"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1284">数据</tspan></text>
+            </g>
+            <g
+               id="g1288"
+               transform="matrix(0.2181818,0,0,0.2181818,381.51482,-208.25411)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1292"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1290">收集</tspan></text>
+            </g>
+            <g
+               id="g1294"
+               transform="matrix(0.2181818,0,0,0.2181818,593.13304,-32.399577)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:92px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1298"><tspan
+                   x="0 91.668793 183.33759 275.00638"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1296">推荐系统</tspan></text>
+            </g>
+            <path
+               d="M 284.80609,58.000433 H 981.67874 V 11.091303 H 284.80609 Z"
+               style="fill:#fff2cc;fill-opacity:1;fill-rule:nonzero;stroke:none"
+               id="path1300" />
+            <g
+               id="g1306"
+               transform="matrix(0.2181818,0,0,0.2181818,552.13314,-299.23591)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1310"><tspan
+                   x="0 82.501991 165.00398 247.50597 330.00797"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1308">推荐服务的</tspan></text>
+            </g>
+            <g
+               id="g1312"
+               transform="matrix(0.2181818,0,0,0.2181818,642.13314,-299.23591)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1316"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1314">全球</tspan></text>
+            </g>
+            <g
+               id="g1318"
+               transform="matrix(0.2181818,0,0,0.2181818,678.13314,-299.23591)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1322"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1320">用户</tspan></text>
+            </g>
+            <path
+               d="m 383.06971,58.068003 6e-5,26.35969 h -3 l -6e-5,-26.35968 z m 3.00003,24.85968 -4.49995,9.00001 -4.50002,-8.99999 z"
+               style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+               id="path1324" />
+            <path
+               d="m 315.7879,256.90929 c 0,6.1857 5.01448,11.2002 11.20016,11.2002 h 109.38148 c 6.1857,0 11.2002,-5.0145 11.2002,-11.2002 v -44.7996 c 0,-6.1857 -5.0145,-11.2002 -11.2002,-11.2002 H 326.98806 c -6.18568,0 -11.20016,5.0145 -11.20016,11.2002 z"
+               style="fill:#e2f0d9;fill-opacity:1;fill-rule:evenodd;stroke:none"
+               id="path1326" />
+            <g
+               id="g1328"
+               transform="matrix(7.874016e-5,0,0,-7.874016e-5,315.7879,268.10949)">
+              <path
+                 d="M 0,142242.1 C 0,63683.97 63683.91,0 142242,0 h 1389145 c 78558,0 142242,63683.97 142242,142242.1 v 568955.8 c 0,78558.1 -63684,142242.1 -142242,142242.1 H 142242 C 63683.91,853440 0,789756 0,711197.9 Z"
+                 style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+                 id="path1330" />
+            </g>
+            <g
+               id="g1332"
+               transform="matrix(0.2181818,0,0,0.2181818,345.51482,-99.381407)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1336"><tspan
+                   x="0 82.501991 165.00398 247.50597"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1334">数据处理</tspan></text>
+            </g>
+            <path
+               d="m 487.27874,246.18189 c 0,12.1103 9.8174,21.9276 21.9277,21.9276 h 87.7083 c 12.1103,0 21.9276,-9.8173 21.9276,-21.9276 v -132.4357 c 0,-12.11031 -9.8173,-21.927637 -21.9276,-21.927637 h -87.7083 c -12.1103,0 -21.9277,9.817327 -21.9277,21.927637 z"
+               style="fill:#e2f0d9;fill-opacity:1;fill-rule:evenodd;stroke:none"
+               id="path1338" />
+            <g
+               id="g1340"
+               transform="matrix(7.874016e-5,0,0,-7.874016e-5,487.27874,268.10949)">
+              <path
+                 d="M 0,278480.9 C 0,124680.2 124680.3,0 278481.2,0 H 1392377 c 153801,0 278481,124680.2 278481,278480.9 V 1960414 c 0,153801 -124680,278481 -278481,278481 H 278481.2 C 124680.3,2238895 0,2114215 0,1960414 Z"
+                 style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+                 id="path1342" />
+            </g>
+            <g
+               id="g1344"
+               transform="matrix(0.2181818,0,0,0.2181818,516.89904,-153.92681)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1348"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1346">数据</tspan></text>
+            </g>
+            <g
+               id="g1350"
+               transform="matrix(0.2181818,0,0,0.2181818,552.89904,-153.92681)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1354"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1352">存储</tspan></text>
+            </g>
+            <path
+               d="m 830.04244,256.90899 c 0,6.1859 5.0146,11.2005 11.2004,11.2005 h 109.1628 c 6.1859,0 11.2005,-5.0146 11.2005,-11.2005 v -44.8006 c 0,-6.1859 -5.0146,-11.2005 -11.2005,-11.2005 h -109.1628 c -6.1858,0 -11.2004,5.0146 -11.2004,11.2005 z"
+               style="fill:#deebf7;fill-opacity:1;fill-rule:evenodd;stroke:none"
+               id="path1356" />
+            <g
+               id="g1358"
+               transform="matrix(7.874016e-5,0,0,-7.874016e-5,830.04244,268.10949)">
+              <path
+                 d="M 0,142246.1 C 0,63685.74 63685.67,0 142245.9,0 H 1528613 c 78560,0 142246,63685.74 142246,142246.1 V 711214 c 0,78560.3 -63686,142246.1 -142246,142246.1 H 142245.9 C 63685.67,853460.1 0,789774.3 0,711214 Z"
+                 style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+                 id="path1360" />
+            </g>
+            <g
+               id="g1362"
+               transform="matrix(0.2181818,0,0,0.2181818,859.66754,-99.381407)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1366"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1364">模型</tspan></text>
+            </g>
+            <g
+               id="g1368"
+               transform="matrix(0.2181818,0,0,0.2181818,895.66754,-99.381407)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1372"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1370">存储</tspan></text>
+            </g>
+            <path
+               d="m 830.04244,147.81809 c 0,6.1858 5.0146,11.2005 11.2004,11.2005 h 109.1628 c 6.1859,0 11.2005,-5.0147 11.2005,-11.2005 v -44.80064 c 0,-6.185857 -5.0146,-11.200477 -11.2005,-11.200477 h -109.1628 c -6.1858,0 -11.2004,5.01462 -11.2004,11.200477 z"
+               style="fill:#deebf7;fill-opacity:1;fill-rule:evenodd;stroke:none"
+               id="path1374" />
+            <g
+               id="g1376"
+               transform="matrix(7.874016e-5,0,0,-7.874016e-5,830.04244,159.01859)">
+              <path
+                 d="M 0,142246.1 C 0,63685.74 63685.67,0 142245.9,0 H 1528613 c 78560,0 142246,63685.74 142246,142246.1 V 711214 c 0,78560.3 -63686,142246.1 -142246,142246.1 H 142245.9 C 63685.67,853460.1 0,789774.3 0,711214 Z"
+                 style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+                 id="path1378" />
+            </g>
+            <g
+               id="g1380"
+               transform="matrix(0.2181818,0,0,0.2181818,859.66754,-208.25411)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1384"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1382">推理</tspan></text>
+            </g>
+            <g
+               id="g1386"
+               transform="matrix(0.2181818,0,0,0.2181818,895.66754,-208.25411)">
+              <text
+                 transform="matrix(1,0,0,-1,0,1503)"
+                 style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+                 id="text1390"><tspan
+                   x="0 82.501991"
+                   y="0"
+                   sodipodi:role="line"
+                   id="tspan1388">服务</tspan></text>
+            </g>
+            <path
+               d="m 383.06971,159.21889 7e-5,34.2997 h -3 l -7e-5,-34.2997 z m 3.00003,32.7996 -4.49995,9.0001 -4.50002,-9 z"
+               style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+               id="path1392" />
+            <path
+               d="m 447.46054,236.11859 32.2079,-1e-4 v -3 l -32.2079,1e-4 z m 30.7079,2.9999 9,-4.5 -9,-4.5 z"
+               style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+               id="path1394" />
+            <path
+               d="m 618.73334,236.99129 32.2079,-1e-4 v -3 l -32.2079,1e-4 z m 30.7079,3 9,-4.5001 -9.0001,-4.4999 z"
+               style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+               id="path1396" />
+            <path
+               d="m 790.22424,236.11859 32.2079,-1e-4 v -3 l -32.2079,1e-4 z m 30.7079,2.9999 9,-4.5 -9,-4.5 z"
+               style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+               id="path1398" />
+            <path
+               d="m 898.85154,201.01859 v -34.2997 h -3 v 34.2997 z m 3,-32.7997 -4.5,-9 -4.5,9 z"
+               style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+               id="path1400" />
+            <path
+               d="m 898.85154,91.927683 v -26.35968 l -3,-1e-5 v 26.35969 z m 3,-24.85968 -4.5,-9 -4.5,8.99999 z"
+               style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+               id="path1402" />
+          </g>
+        </g>
+      </g>
+    </g>
+  </g>
+</svg>
--- a/img/ch10/ch10-federated-learning-architecture.png
+++ b/img/ch10/ch10-federated-learning-architecture.png
--- a/img/ch10/ch10-federated-learning-different-connection.png
+++ b/img/ch10/ch10-federated-learning-different-connection.png
--- a/img/ch10/ch10-federated-learning-fedavg.png
+++ b/img/ch10/ch10-federated-learning-fedavg.png
--- a/img/ch10/ch10-federated-learning-flow.png
+++ b/img/ch10/ch10-federated-learning-flow.png
--- a/img/ch10/ch10-recommendation-models.svg
+++ b/img/ch10/ch10-recommendation-models.svg
@@ -0,0 +1,406 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   width="185.10585mm"
+   height="116.63811mm"
+   viewBox="0 0 185.10585 116.63811"
+   version="1.1"
+   id="svg5"
+   inkscape:version="1.1.2 (1:1.1+202202050950+0a00cf5339)"
+   sodipodi:docname="ch10-recommendation-models.svg"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg">
+  <sodipodi:namedview
+     id="namedview7"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     inkscape:document-units="mm"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="0.64052329"
+     inkscape:cx="501.93335"
+     inkscape:cy="434.80074"
+     inkscape:window-width="1848"
+     inkscape:window-height="1136"
+     inkscape:window-x="72"
+     inkscape:window-y="27"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g15" />
+  <defs
+     id="defs2">
+    <clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath25">
+      <path
+         d="M 0,0.1636355 H 536.9455 V 351.00004 H 0 Z"
+         id="path23" />
+    </clipPath>
+  </defs>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(27.764827,-33.462145)">
+    <g
+       id="g15"
+       inkscape:label="ch11-recommendation-models(1)"
+       transform="matrix(0.35277777,0,0,-0.35277777,-27.90913,154.3432)">
+      <g
+         id="g17" />
+      <g
+         id="g29">
+        <path
+           d="M 70.29094,297.6909 H 300.6909 V 280.4546 H 70.29094 Z"
+           style="fill:#fbe5d6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path31" />
+        <g
+           id="g33"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,70.29094,297.6909)">
+          <path
+             d="M 0,0 H 2926080 V 218902 H 0 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path35" />
+        </g>
+        <g
+           id="g37"
+           transform="matrix(0.2181818,0,0,0.2181818,383.0586,-67.03634)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text41"><tspan
+               x="0 73.335793 146.67159 220.00737"
+               y="0"
+               sodipodi:role="line"
+               id="tspan39">交互历史</tspan></text>
+        </g>
+        <g
+           id="g43"
+           transform="matrix(0.2181818,0,0,0.2181818,447.0585,-67.03634)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text47"><tspan
+               x="0"
+               y="0"
+               id="tspan45">/</tspan></text>
+        </g>
+        <g
+           id="g49"
+           transform="matrix(0.2181818,0,0,0.2181818,451.5585,-67.03634)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text53"><tspan
+               x="0 73.335793 146.67159"
+               y="0"
+               sodipodi:role="line"
+               id="tspan51">候选项</tspan></text>
+        </g>
+        <path
+           d="m 190.1546,276.6364 v -13.0221 h -3 v 13.0221 z m 3,-11.522 -4.5,-9.0001 -4.5,9 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path55" />
+        <path
+           d="m 0.9090492,232.0903 c 0,14.0585 11.3966408,25.4551 25.4551008,25.4551 H 160.0358 c 14.0585,0 25.4551,-11.3966 25.4551,-25.4551 v -101.817 c 0,-14.0585 -11.3966,-25.4551 -25.4551,-25.4551 H 26.36415 c -14.05846,0 -25.4551008,11.3966 -25.4551008,25.4551 z"
+           style="fill:#f8cbad;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path57" />
+        <g
+           id="g59"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,0.9090492,257.5454)">
+          <path
+             d="M 0,323279.9 C 0,144737.4 144737.3,0 323279.8,0 H 2020910 c 178543,0 323280,144737.4 323280,323279.9 V 1616356 c 0,178543 -144737,323280 -323280,323280 H 323279.8 C 144737.3,1939636 0,1794899 0,1616356 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path61" />
+        </g>
+        <g
+           id="g63"
+           transform="matrix(0.2181818,0,0,0.2181818,50.81908,-165.4363)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text67"><tspan
+               x="0 37.237293 74.474586 111.71188 148.94917 211.37148 227.94977 259.42007 275.99835 313.23566 350.47296"
+               y="0"
+               sodipodi:role="line"
+               id="tspan65">1000M x 100</tspan></text>
+        </g>
+        <g
+           id="g69"
+           transform="matrix(0.2181818,0,0,0.2181818,53.13408,-184.4182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text73"><tspan
+               x="0 73.335793 146.67159 220.00737 293.34317"
+               y="0"
+               sodipodi:role="line"
+               id="tspan71">用户嵌入表</tspan></text>
+        </g>
+        <path
+           d="m 202.2909,250.8908 c 0,4.0368 3.2725,7.3092 7.3093,7.3092 h 157.527 c 4.0367,0 7.3092,-3.2724 7.3092,-7.3092 v -29.2361 c 0,-4.0368 -3.2725,-7.3093 -7.3092,-7.3093 h -157.527 c -4.0368,0 -7.3093,3.2725 -7.3093,7.3093 z"
+           style="fill:#f8cbad;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path75" />
+        <g
+           id="g77"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,202.2909,258.2)">
+          <path
+             d="M 0,92827.1 C 0,41560.11 41560.08,0 92827.03,0 H 2093420 c 51267,0 92827,41560.11 92827,92827.1 v 371298.8 c 0,51267 -41560,92827.1 -92827,92827.1 H 92827.03 C 41560.08,556953 0,515392.9 0,464125.9 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path79" />
+        </g>
+        <g
+           id="g81"
+           transform="matrix(0.2181818,0,0,0.2181818,254.1505,-110.2363)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text85"><tspan
+               x="0 37.237293 74.474586 136.89688 153.47517 184.94548 201.52377 238.76106 275.99835"
+               y="0"
+               sodipodi:role="line"
+               id="tspan83">10M x 100</tspan></text>
+        </g>
+        <g
+           id="g87"
+           transform="matrix(0.2181818,0,0,0.2181818,248.3355,-129.2182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text91"><tspan
+               x="0 73.335793 146.67159 220.00737 293.34317"
+               y="0"
+               sodipodi:role="line"
+               id="tspan89">内容嵌入表</tspan></text>
+        </g>
+        <path
+           d="m 202.2909,197.8726 c 0,4.0368 3.2725,7.3092 7.3093,7.3092 h 157.527 c 4.0367,0 7.3092,-3.2724 7.3092,-7.3092 v -29.2361 c 0,-4.0368 -3.2725,-7.3093 -7.3092,-7.3093 h -157.527 c -4.0368,0 -7.3093,3.2725 -7.3093,7.3093 z"
+           style="fill:#f8cbad;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path93" />
+        <g
+           id="g95"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,202.2909,205.1818)">
+          <path
+             d="M 0,92827.1 C 0,41560.11 41560.08,0 92827.03,0 H 2093420 c 51267,0 92827,41560.11 92827,92827.1 v 371298.8 c 0,51267 -41560,92827.1 -92827,92827.1 H 92827.03 C 41560.08,556953 0,515392.9 0,464125.9 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path97" />
+        </g>
+        <g
+           id="g99"
+           transform="matrix(0.2181818,0,0,0.2181818,224.3355,-172.8545)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text103"><tspan
+               x="0 73.335793 146.67159 220.00737 293.34317 366.67896 440.01474 513.35052"
+               y="0"
+               sodipodi:role="line"
+               id="tspan101">用户上下文嵌入表</tspan></text>
+        </g>
+        <path
+           d="m 202.2909,142.2362 c 0,4.0368 3.2725,7.3092 7.3093,7.3092 h 157.527 c 4.0367,0 7.3092,-3.2724 7.3092,-7.3092 V 113 c 0,-4.0368 -3.2725,-7.3092 -7.3092,-7.3092 h -157.527 c -4.0368,0 -7.3093,3.2724 -7.3093,7.3092 z"
+           style="fill:#f8cbad;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path105" />
+        <g
+           id="g107"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,202.2909,149.5454)">
+          <path
+             d="M 0,92827.33 C 0,41560.21 41560.08,0 92827.03,0 H 2093420 c 51267,0 92827,41560.21 92827,92827.33 V 464127.1 c 0,51267.1 -41560,92827.3 -92827,92827.3 H 92827.03 C 41560.08,556954.4 0,515394.2 0,464127.1 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path109" />
+        </g>
+        <g
+           id="g111"
+           transform="matrix(0.2181818,0,0,0.2181818,224.3355,-228.2727)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text115"><tspan
+               x="0 73.335793 146.67159 220.00737 293.34317 366.67896 440.01474 513.35052"
+               y="0"
+               sodipodi:role="line"
+               id="tspan113">内容上下文嵌入表</tspan></text>
+        </g>
+        <path
+           d="m 8.327238,87.50894 c 0,4.61922 3.744592,8.36383 8.363772,8.36383 H 362.1453 c 4.6192,0 8.3638,-3.74461 8.3638,-8.36383 V 54.05479 c 0,-4.61922 -3.7446,-8.36383 -8.3638,-8.36383 H 16.69101 c -4.61918,0 -8.363772,3.74461 -8.363772,8.36383 z"
+           style="fill:#e7e6e6;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path117" />
+        <g
+           id="g119"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,8.327238,95.87277)">
+          <path
+             d="M 0,106220.7 C 0,47556.62 47556.26,0 106219.9,0 H 4493489 c 58664,0 106220,47556.62 106220,106220.7 v 424867.6 c 0,58664.1 -47556,106220.7 -106220,106220.7 H 106219.9 C 47556.26,637309 0,589752.4 0,531088.3 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path121" />
+        </g>
+        <path
+           d="M 39.09094,88.0182 H 339.7454 V 77.32725 H 39.09094 Z"
+           style="fill:#ed7d31;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path123" />
+        <g
+           id="g125"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,39.09094,88.0182)">
+          <path
+             d="M 0,0 H 3818312 V 135775 H 0 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path127" />
+        </g>
+        <path
+           d="M 71.59999,67.07277 H 309.8546 V 56.38183 H 71.59999 Z"
+           style="fill:#ed7d31;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path129" />
+        <g
+           id="g131"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,71.59999,67.07277)">
+          <path
+             d="M 0,0 H 3025833 V 135775 H 0 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path133" />
+        </g>
+        <path
+           d="m 110.8728,27.36363 h 156 V 12.52725 h -156 z"
+           style="fill:#fbe5d6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path135" />
+        <g
+           id="g137"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,110.8728,27.36363)">
+          <path
+             d="M 0,0 H 1981200 V 188422 H 0 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path139" />
+        </g>
+        <path
+           d="m 190.1542,48.66947 0.3528,-15.98875 -2.9993,-0.06618 -0.3528,15.98875 z m 3.319,-14.42294 -4.3004,-9.09708 -4.6974,8.89854 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path141" />
+        <g
+           id="g143"
+           transform="matrix(0.2181818,0,0,0.2181818,408.9256,-336.4909)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:73px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text147"><tspan
+               x="0 73.335793 146.67159 220.00737"
+               y="0"
+               sodipodi:role="line"
+               id="tspan145">推荐结果</tspan></text>
+        </g>
+        <g
+           id="g149"
+           transform="matrix(0.2181818,0,0,0.2181818,422.6449,-165.2182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text153"><tspan
+               x="0 82.501991 165.00398"
+               y="0"
+               sodipodi:role="line"
+               id="tspan151">嵌入表</tspan></text>
+        </g>
+        <g
+           id="g155"
+           transform="matrix(0.2181818,0,0,0.2181818,376.1449,-186.8182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text159"><tspan
+               x="0 45.832596 91.665192 137.49779 178.76538 201.68997 265.86557 320.82816"
+               y="0"
+               sodipodi:role="line"
+               id="tspan157">100s GB</tspan></text>
+        </g>
+        <g
+           id="g161"
+           transform="matrix(0.2181818,0,0,0.2181818,451.1449,-186.8182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text165"><tspan
+               x="0"
+               y="0"
+               id="tspan163">–</tspan></text>
+        </g>
+        <g
+           id="g167"
+           transform="matrix(0.2181818,0,0,0.2181818,466.1449,-186.8182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text171"><tspan
+               x="0 45.832596 91.665192 132.93279 155.85738 206.25497"
+               y="0"
+               sodipodi:role="line"
+               id="tspan169">10s TB</tspan></text>
+        </g>
+        <g
+           id="g173"
+           transform="matrix(0.2181818,0,0,0.2181818,396.9702,-275.1818)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text177"><tspan
+               x="0 82.501991 165.00398 247.50597 330.00797 412.50995"
+               y="0"
+               sodipodi:role="line"
+               id="tspan175">深度神经网络</tspan></text>
+        </g>
+        <g
+           id="g179"
+           transform="matrix(0.2181818,0,0,0.2181818,375.9702,-296.7818)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text183"><tspan
+               x="0 45.832596 91.665192 132.93279 155.85738 220.03297 274.99557"
+               y="0"
+               sodipodi:role="line"
+               id="tspan181">10s GB</tspan></text>
+        </g>
+        <g
+           id="g185"
+           transform="matrix(0.2181818,0,0,0.2181818,440.9702,-296.7818)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text189"><tspan
+               x="0"
+               y="0"
+               id="tspan187">–</tspan></text>
+        </g>
+        <g
+           id="g191"
+           transform="matrix(0.2181818,0,0,0.2181818,455.9702,-296.7818)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text195"><tspan
+               x="0 45.832596 91.665192 137.49779 178.76538 201.68997 265.86557"
+               y="0"
+               sodipodi:role="line"
+               id="tspan193">100s GB</tspan></text>
+        </g>
+        <g
+           id="g197"
+           transform="matrix(0.2181818,0,0,0.2181818,109.7371,-25.36362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1608)"
+             style="font-variant:normal;font-weight:bold;font-size:92px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei-Bold;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text201"><tspan
+               x="0 91.668793 183.33759 275.00638 366.67517 458.34396 550.01276 641.68152"
+               y="0"
+               sodipodi:role="line"
+               id="tspan199">深度学习推荐模型</tspan></text>
+        </g>
+      </g>
+    </g>
+  </g>
+</svg>
--- a/img/ch10/ch10-recommendation-systems.svg
+++ b/img/ch10/ch10-recommendation-systems.svg
@@ -0,0 +1,628 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   width="189.62125mm"
+   height="116.03821mm"
+   viewBox="0 0 189.62125 116.03821"
+   version="1.1"
+   id="svg5"
+   inkscape:version="1.1.2 (1:1.1+202202050950+0a00cf5339)"
+   sodipodi:docname="ch10-recommendation-systems.svg"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg">
+  <sodipodi:namedview
+     id="namedview7"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     inkscape:document-units="mm"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     inkscape:zoom="1.2810466"
+     inkscape:cx="346.98192"
+     inkscape:cy="186.95651"
+     inkscape:window-width="1848"
+     inkscape:window-height="1136"
+     inkscape:window-x="72"
+     inkscape:window-y="27"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="g31" />
+  <defs
+     id="defs2">
+    <clipPath
+       clipPathUnits="userSpaceOnUse"
+       id="clipPath41">
+      <path
+         d="M 0,0.1090901 H 538.9091 V 329.99999 H 0 Z"
+         id="path39" />
+    </clipPath>
+  </defs>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(61.779345,-53.24402)">
+    <g
+       id="g31"
+       inkscape:label="ch11-recommendation-systems(1)"
+       transform="matrix(0.35277777,0,0,-0.35277777,-72.3985,168.34707)">
+      <g
+         id="g33" />
+      <g
+         id="g45"
+         transform="translate(29.128792,-2.9690309)">
+        <path
+           d="M 1.47275,327.6546 H 199.3636 V 64.09095 H 1.47275 Z"
+           style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path47" />
+        <g
+           id="g49"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,1.47275,327.6546)">
+          <path
+             d="M 0,0 H 2513214 V 3347258 H 0 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path51" />
+        </g>
+        <path
+           d="M 220.0909,328.7454 H 363.6546 V 65.18181 H 220.0909 Z"
+           style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path53" />
+        <g
+           id="g55"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,220.0909,328.7454)">
+          <path
+             d="M 0,0 H 1823258 V 3347258 H 0 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path57" />
+        </g>
+        <path
+           d="M 394.4182,328.7454 H 537.9818 V 65.18181 H 394.4182 Z"
+           style="fill:#e7e6e6;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path59" />
+        <g
+           id="g61"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,394.4182,328.7454)">
+          <path
+             d="M 0,0 H 1823258 V 3347258 H 0 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path63" />
+        </g>
+        <path
+           d="m 31.36362,281.7634 c 0,6.1857 5.01448,11.2002 11.20017,11.2002 H 151.7271 c 6.1857,0 11.2001,-5.0145 11.2001,-11.2002 v -44.7996 c 0,-6.1857 -5.0144,-11.2002 -11.2001,-11.2002 H 42.56379 c -6.18569,0 -11.20017,5.0145 -11.20017,11.2002 z"
+           style="fill:#deebf7;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path65" />
+        <g
+           id="g67"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,31.36362,292.9636)">
+          <path
+             d="M 0,142242.4 C 0,63684.11 63684,0 142242.2,0 H 1528616 c 78558,0 142242,63684.11 142242,142242.4 v 568955.2 c 0,78558.3 -63684,142242.4 -142242,142242.4 H 142242.2 C 63684,853440 0,789755.9 0,711197.6 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path69" />
+        </g>
+        <g
+           id="g71"
+           transform="matrix(0.2181818,0,0,0.2181818,52.0736,-54.87271)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text75"><tspan
+               x="0 82.501991 165.00398 247.50597 330.00797"
+               y="0"
+               sodipodi:role="line"
+               id="tspan73">参数服务器</tspan></text>
+        </g>
+        <g
+           id="g77"
+           transform="matrix(0.2181818,0,0,0.2181818,79.0736,-76.47271)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text81"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan79">集群</tspan></text>
+        </g>
+        <g
+           id="g83"
+           transform="matrix(0.2181818,0,0,0.2181818,45.3236,-98.07271)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text87"><tspan
+               x="0 25.207096 67.014191 108.82129 150.62839 182.72447"
+               y="0"
+               sodipodi:role="line"
+               id="tspan85">(100s</tspan></text>
+        </g>
+        <g
+           id="g89"
+           transform="matrix(0.2181818,0,0,0.2181818,89.3236,-98.07271)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text93"><tspan
+               x="0 82.501991 165.00398"
+               y="0"
+               sodipodi:role="line"
+               id="tspan91">服务器</tspan></text>
+        </g>
+        <g
+           id="g95"
+           transform="matrix(0.2181818,0,0,0.2181818,143.3236,-98.07271)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text99"><tspan
+               x="0"
+               y="0"
+               id="tspan97">)</tspan></text>
+        </g>
+        <path
+           d="m 32.45456,137.7635 c 0,6.1856 5.01448,11.2001 11.20016,11.2001 H 153.0362 c 6.1857,0 11.2002,-5.0145 11.2002,-11.2001 V 92.96379 c 0,-6.18568 -5.0145,-11.20017 -11.2002,-11.20017 H 43.65472 c -6.18568,0 -11.20016,5.01449 -11.20016,11.20017 z"
+           style="fill:#e2f0d9;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path101" />
+        <g
+           id="g103"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,32.45456,148.9636)">
+          <path
+             d="M 0,142242.1 C 0,63683.97 63683.91,0 142242,0 h 1389145 c 78558,0 142242,63683.97 142242,142242.1 v 568955.8 c 0,78558.1 -63684,142242.1 -142242,142242.1 H 142242 C 63683.91,853440 0,789756 0,711197.9 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path105" />
+        </g>
+        <g
+           id="g107"
+           transform="matrix(0.2181818,0,0,0.2181818,53.21227,-209.5636)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text111"><tspan
+               x="0 82.501991 165.00398 247.50597 330.00797"
+               y="0"
+               sodipodi:role="line"
+               id="tspan109">训练服务器</tspan></text>
+        </g>
+        <g
+           id="g113"
+           transform="matrix(0.2181818,0,0,0.2181818,80.21227,-231.1636)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text117"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan115">集群</tspan></text>
+        </g>
+        <path
+           d="m 226.2,224.818 c 0,6.1857 5.0145,11.2002 11.2002,11.2002 h 109.1632 c 6.1857,0 11.2002,-5.0145 11.2002,-11.2002 v -44.7996 c 0,-6.1857 -5.0145,-11.2002 -11.2002,-11.2002 H 237.4002 c -6.1857,0 -11.2002,5.0145 -11.2002,11.2002 z"
+           style="fill:#deebf7;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path119" />
+        <g
+           id="g121"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,226.2,236.0182)">
+          <path
+             d="M 0,142242.4 C 0,63684.11 63684,0 142242.2,0 H 1528616 c 78558,0 142242,63684.11 142242,142242.4 v 568955.2 c 0,78558.3 -63684,142242.4 -142242,142242.4 H 142242.2 C 63684,853440 0,789755.9 0,711197.6 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path123" />
+        </g>
+        <g
+           id="g125"
+           transform="matrix(0.2181818,0,0,0.2181818,246.8299,-111.8182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text129"><tspan
+               x="0 82.501991 165.00398 247.50597"
+               y="0"
+               sodipodi:role="line"
+               id="tspan127">参数服务</tspan></text>
+        </g>
+        <g
+           id="g131"
+           transform="matrix(0.2181818,0,0,0.2181818,318.8299,-111.8182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text135"><tspan
+               x="0"
+               y="0"
+               id="tspan133">器</tspan></text>
+        </g>
+        <g
+           id="g137"
+           transform="matrix(0.2181818,0,0,0.2181818,273.8299,-133.4182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text141"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan139">副本</tspan></text>
+        </g>
+        <g
+           id="g143"
+           transform="matrix(0.2181818,0,0,0.2181818,240.0799,-155.0182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text147"><tspan
+               x="0"
+               y="0"
+               id="tspan145">(</tspan></text>
+        </g>
+        <g
+           id="g149"
+           transform="matrix(0.2181818,0,0,0.2181818,245.5799,-155.0182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text153"><tspan
+               x="0 41.823696 83.647392 125.47108"
+               y="0"
+               sodipodi:role="line"
+               id="tspan151">100s</tspan></text>
+        </g>
+        <g
+           id="g155"
+           transform="matrix(0.2181818,0,0,0.2181818,284.0799,-155.0182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text159"><tspan
+               x="0 82.501991 165.00398"
+               y="0"
+               sodipodi:role="line"
+               id="tspan157">服务器</tspan></text>
+        </g>
+        <g
+           id="g161"
+           transform="matrix(0.2181818,0,0,0.2181818,338.0799,-155.0182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text165"><tspan
+               x="0"
+               y="0"
+               id="tspan163">)</tspan></text>
+        </g>
+        <path
+           d="m 400.309,224.818 c 0,6.1857 5.0145,11.2002 11.2002,11.2002 h 109.3816 c 6.1856,0 11.2001,-5.0145 11.2001,-11.2002 v -44.7996 c 0,-6.1857 -5.0145,-11.2002 -11.2001,-11.2002 H 411.5092 c -6.1857,0 -11.2002,5.0145 -11.2002,11.2002 z"
+           style="fill:#deebf7;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path167" />
+        <g
+           id="g169"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,400.309,236.0182)">
+          <path
+             d="M 0,142242.1 C 0,63683.97 63683.95,0 142242.1,0 H 1531388 c 78558,0 142242,63683.97 142242,142242.1 v 568955.8 c 0,78558.1 -63684,142242.1 -142242,142242.1 H 142242.1 C 63683.95,853440 0,789756 0,711197.9 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path171" />
+        </g>
+        <g
+           id="g173"
+           transform="matrix(0.2181818,0,0,0.2181818,421.0908,-111.8182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text177"><tspan
+               x="0 82.501991 165.00398 247.50597"
+               y="0"
+               sodipodi:role="line"
+               id="tspan175">参数服务</tspan></text>
+        </g>
+        <g
+           id="g179"
+           transform="matrix(0.2181818,0,0,0.2181818,493.0908,-111.8182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text183"><tspan
+               x="0"
+               y="0"
+               id="tspan181">器</tspan></text>
+        </g>
+        <g
+           id="g185"
+           transform="matrix(0.2181818,0,0,0.2181818,448.0908,-133.4182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text189"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan187">副本</tspan></text>
+        </g>
+        <g
+           id="g191"
+           transform="matrix(0.2181818,0,0,0.2181818,414.3408,-155.0182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text195"><tspan
+               x="0"
+               y="0"
+               id="tspan193">(</tspan></text>
+        </g>
+        <g
+           id="g197"
+           transform="matrix(0.2181818,0,0,0.2181818,419.8408,-155.0182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text201"><tspan
+               x="0 41.823696 83.647392 125.47108 157.58379"
+               y="0"
+               sodipodi:role="line"
+               id="tspan199">100s</tspan></text>
+        </g>
+        <g
+           id="g203"
+           transform="matrix(0.2181818,0,0,0.2181818,458.3408,-155.0182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text207"><tspan
+               x="0 82.501991 165.00398"
+               y="0"
+               sodipodi:role="line"
+               id="tspan205">服务器</tspan></text>
+        </g>
+        <g
+           id="g209"
+           transform="matrix(0.2181818,0,0,0.2181818,512.3408,-155.0182)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text213"><tspan
+               x="0"
+               y="0"
+               id="tspan211">)</tspan></text>
+        </g>
+        <path
+           d="m 162.8182,260.9728 h 130.4181 v -17.2553 h -3 v 15.7553 l 1.5,-1.5 H 162.8182 Z m 133.4181,-15.7553 -4.5,-9 -4.5,9 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path215" />
+        <path
+           d="m 162.8182,260.9728 h 304.679 v -17.2552 h -3 v 15.7552 l 1.5,-1.5 h -303.179 z m 307.679,-15.7552 -4.5,-9 -4.5,9 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path217" />
+        <path
+           d="m 226.4182,136.2362 c 0,6.1857 5.0145,11.2002 11.2002,11.2002 h 109.1632 c 6.1857,0 11.2002,-5.0145 11.2002,-11.2002 V 91.43657 c 0,-6.1857 -5.0145,-11.20019 -11.2002,-11.20019 H 237.6184 c -6.1857,0 -11.2002,5.01449 -11.2002,11.20019 z"
+           style="fill:#e2f0d9;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path219" />
+        <g
+           id="g221"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,226.4182,147.4364)">
+          <path
+             d="M 0,142242.4 C 0,63684.11 63684,0 142242.2,0 H 1528616 c 78558,0 142242,63684.11 142242,142242.4 v 568955.2 c 0,78558.3 -63684,142242.4 -142242,142242.4 H 142242.2 C 63684,853440 0,789755.9 0,711197.6 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path223" />
+        </g>
+        <g
+           id="g225"
+           transform="matrix(0.2181818,0,0,0.2181818,247.0946,-211.3091)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text229"><tspan
+               x="0 82.501991 165.00398 247.50597 330.00797"
+               y="0"
+               sodipodi:role="line"
+               id="tspan227">推理服务器</tspan></text>
+        </g>
+        <g
+           id="g231"
+           transform="matrix(0.2181818,0,0,0.2181818,274.0946,-232.9091)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text235"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan233">集群</tspan></text>
+        </g>
+        <path
+           d="m 400.309,136.6726 c 0,6.1857 5.0145,11.2002 11.2002,11.2002 h 109.3816 c 6.1856,0 11.2001,-5.0145 11.2001,-11.2002 V 91.87292 c 0,-6.18568 -5.0145,-11.20017 -11.2001,-11.20017 H 411.5092 c -6.1857,0 -11.2002,5.01449 -11.2002,11.20017 z"
+           style="fill:#e2f0d9;fill-opacity:1;fill-rule:evenodd;stroke:none"
+           id="path237" />
+        <g
+           id="g239"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,400.309,147.8728)">
+          <path
+             d="M 0,142242.1 C 0,63683.97 63683.95,0 142242.1,0 H 1531388 c 78558,0 142242,63683.97 142242,142242.1 v 568955.8 c 0,78558.1 -63684,142242.1 -142242,142242.1 H 142242.1 C 63683.95,853440 0,789756 0,711197.9 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path241" />
+        </g>
+        <g
+           id="g243"
+           transform="matrix(0.2181818,0,0,0.2181818,421.0908,-210.6545)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text247"><tspan
+               x="0 82.501991 165.00398 247.50597 330.00797"
+               y="0"
+               sodipodi:role="line"
+               id="tspan245">推理服务器</tspan></text>
+        </g>
+        <g
+           id="g249"
+           transform="matrix(0.2181818,0,0,0.2181818,448.0908,-232.2545)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text253"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan251">集群</tspan></text>
+        </g>
+        <path
+           d="m 98.75653,218.3959 0.91612,-61.7416 -2.99967,-0.0445 -0.91611,61.7416 z m -5.97708,-1.5889 4.36598,9.0658 4.63307,-8.9323 z m 9.87065,-58.6083 -4.36601,-9.0658 -4.63303,8.9323 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path255" />
+        <path
+           d="m 293.2635,168.9458 0.1719,-13.8983 -2.9997,-0.0371 -0.172,13.8983 z m 3.1532,-12.3613 -4.3884,-9.055 -4.611,8.9437 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path257" />
+        <path
+           d="m 467.591,168.9273 v -13.3886 h -3 v 13.3885 z m 3,-11.8885 -4.5,-9.0001 -4.5,9 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path259" />
+        <g
+           id="g261"
+           transform="matrix(0.2181818,0,0,0.2181818,56.79727,-24.76362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text265"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan263">数据</tspan></text>
+        </g>
+        <g
+           id="g267"
+           transform="matrix(0.2181818,0,0,0.2181818,92.79727,-24.76362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text271"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan269">中心</tspan></text>
+        </g>
+        <g
+           id="g273"
+           transform="matrix(0.2181818,0,0,0.2181818,133.7973,-24.76362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text277"><tspan
+               x="0"
+               y="0"
+               id="tspan275">1</tspan></text>
+        </g>
+        <g
+           id="g279"
+           transform="matrix(0.2181818,0,0,0.2181818,248.3299,-24.76362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text283"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan281">数据</tspan></text>
+        </g>
+        <g
+           id="g285"
+           transform="matrix(0.2181818,0,0,0.2181818,284.3299,-24.76362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text289"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan287">中心</tspan></text>
+        </g>
+        <g
+           id="g291"
+           transform="matrix(0.2181818,0,0,0.2181818,325.3299,-24.76362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text295"><tspan
+               x="0"
+               y="0"
+               id="tspan293">2</tspan></text>
+        </g>
+        <g
+           id="g297"
+           transform="matrix(0.2181818,0,0,0.2181818,422.5908,-24.76362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text301"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan299">数据</tspan></text>
+        </g>
+        <g
+           id="g303"
+           transform="matrix(0.2181818,0,0,0.2181818,458.5908,-24.76362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:MicrosoftYaHei;-inkscape-font-specification:MicrosoftYaHei;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text307"><tspan
+               x="0 82.501991"
+               y="0"
+               sodipodi:role="line"
+               id="tspan305">中心</tspan></text>
+        </g>
+        <g
+           id="g309"
+           transform="matrix(0.2181818,0,0,0.2181818,499.5908,-24.76362)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text313"><tspan
+               x="0"
+               y="0"
+               id="tspan311">3</tspan></text>
+        </g>
+        <path
+           d="M 1.47275,47.72725 H 537.9818 V 0.8181903 H 1.47275 Z"
+           style="fill:#fff2cc;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path315" />
+        <g
+           id="g317"
+           transform="matrix(7.874016e-5,0,0,-7.874016e-5,1.47275,47.72725)">
+          <path
+             d="M 0,0 H 6813665 V 595745 H 0 Z"
+             style="fill:none;stroke:#2f528f;stroke-width:12700;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:8;stroke-dasharray:none;stroke-opacity:1"
+             id="path319" />
+        </g>
+        <g
+           id="g321"
+           transform="matrix(0.2181818,0,0,0.2181818,188.5966,-311.4545)">
+          <text
+             transform="matrix(1,0,0,-1,0,1512)"
+             style="font-variant:normal;font-weight:normal;font-size:83px;font-family:DengXian;-inkscape-font-specification:DengXian-Regular;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
+             id="text325"><tspan
+               x="0 82.501991 165.00398 247.50597 330.00797 412.50995 495.01193 577.51392 660.01593"
+               y="0"
+               sodipodi:role="line"
+               id="tspan323">推荐服务的全球用户</tspan></text>
+        </g>
+        <path
+           d="m 292.609,79.25458 1e-4,-21.70339 -3,-10e-6 -1e-4,21.70339 z m 3.0001,-20.20338 -4.5,-9.00001 -4.5,8.99998 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path327" />
+        <path
+           d="M 467.591,79.25458 V 57.55119 l -3,-10e-6 v 21.70339 z m 3,-20.20338 -4.5,-9.00001 -4.5,8.99998 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path329" />
+        <path
+           d="m 99.73637,50.58244 6e-5,23.79031 -3,1e-5 -6e-5,-23.79032 z m 3.00003,22.2903 -4.49995,9.00001 -4.50002,-8.99999 z"
+           style="fill:#4472c4;fill-opacity:1;fill-rule:nonzero;stroke:none"
+           id="path331" />
+      </g>
+    </g>
+  </g>
+</svg>
--- a/index.md
+++ b/index.md
@@ -20,10 +20,10 @@ chapter_distributed_training/index

 chapter_preface_extension/index

+chapter_recommender_system/index
 chapter_federated_learning/index
 chapter_reinforcement_learning/index
 chapter_explainable_AI/index
-chapter_online_machine_learning/index

 ```

--- a/mlsys.bib
+++ b/mlsys.bib
@@ -279,3 +279,384 @@
  year={2021},
  organization={PMLR}
 }
+
+@inproceedings{MLSYS2021_979d472a,
+ author = {Yin, Chunxing and Acun, Bilge and Wu, Carole-Jean and Liu, Xing},
+ booktitle = {Proceedings of Machine Learning and Systems},
+ editor = {A. Smola and A. Dimakis and I. Stoica},
+ pages = {448--462},
+ title = {TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models},
+ url = {https://proceedings.mlsys.org/paper/2021/file/979d472a84804b9f647bc185a877a8b5-Paper.pdf},
+ volume = {3},
+ year = {2021}
+}
+
+@inproceedings{MLSYS2020_f7e6c855,
+ author = {Zhao, Weijie and Xie, Deping and Jia, Ronglai and Qian, Yulei and Ding, Ruiquan and Sun, Mingming and Li, Ping},
+ booktitle = {Proceedings of Machine Learning and Systems},
+ editor = {I. Dhillon and D. Papailiopoulos and V. Sze},
+ pages = {412--428},
+ title = {Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems},
+ url = {https://proceedings.mlsys.org/paper/2020/file/f7e6c85504ce6e82442c770f7c8606f0-Paper.pdf},
+ volume = {2},
+ year = {2020}
+}
+
+@article{zionex,
+  title={Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models},
+  author={Mudigere, Dheevatsa and Hao, Yuchen and Huang, Jianyu and Jia, Zhihao and Tulloch, Andrew and Sridharan, Srinivas and Liu, Xing and Ozdal, Mustafa and Nie, Jade and Park, Jongsoo and others},
+  journal={arXiv preprint arXiv:2104.05158},
+  year={2021}
+}
+
+@inproceedings{gong2020edgerec,
+  title={EdgeRec: Recommender System on Edge in Mobile Taobao},
+  author={Gong, Yu and Jiang, Ziwen and Feng, Yufei and Hu, Binbin and Zhao, Kaiqi and Liu, Qingwen and Ou, Wenwu},
+  booktitle={Proceedings of the 29th ACM International Conference on Information \& Knowledge Management},
+  pages={2477--2484},
+  year={2020}
+}
+
+@inproceedings{NEURIPS2020_a1d4c20b,
+ author = {He, Chaoyang and Annavaram, Murali and Avestimehr, Salman},
+ booktitle = {Advances in Neural Information Processing Systems},
+ editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
+ pages = {14068--14080},
+ publisher = {Curran Associates, Inc.},
+ title = {Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge},
+ url = {https://proceedings.neurips.cc/paper/2020/file/a1d4c20b182ad7137ab3606f0e3fc8a4-Paper.pdf},
+ volume = {33},
+ year = {2020}
+}
+
+@INPROCEEDINGS{9355295,  
+  author={Xie, Minhui and Ren, Kai and Lu, Youyou and Yang, Guangxu and Xu, Qingxing and Wu, Bihai and Lin, Jiazhen and Ao, Hongbo and Xu, Wanhong and Shu, Jiwu},  
+  booktitle={SC20: International Conference for High Performance Computing, Networking, Storage and Analysis},   
+  title={Kraken: Memory-Efficient Continual Learning for Large-Scale Real-Time Recommendations},   
+  year={2020},  
+  volume={},  
+  number={},  
+  pages={1-17},  
+  doi={10.1109/SC41405.2020.00025}
+}
+
+@inproceedings{MLSYS2021_ec895663,
+ author = {Jiang, Wenqi and He, Zhenhao and Zhang, Shuai and Preu\ss er, Thomas B. and Zeng, Kai and Feng, Liang and Zhang, Jiansong and Liu, Tongxuan and Li , Yong and Zhou, Jingren and Zhang, Ce and Alonso, Gustavo},
+ booktitle = {Proceedings of Machine Learning and Systems},
+ editor = {A. Smola and A. Dimakis and I. Stoica},
+ pages = {845--859},
+ title = {MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions},
+ url = {https://proceedings.mlsys.org/paper/2021/file/ec8956637a99787bd197eacd77acce5e-Paper.pdf},
+ volume = {3},
+ year = {2021}
+}
+
+@inproceedings{10.1145/3394486.3403059,
+author = {Shi, Hao-Jun Michael and Mudigere, Dheevatsa and Naumov, Maxim and Yang, Jiyan},
+title = {Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems},
+year = {2020},
+isbn = {9781450379984},
+publisher = {Association for Computing Machinery},
+address = {New York, NY, USA},
+url = {https://doi.org/10.1145/3394486.3403059},
+doi = {10.1145/3394486.3403059},
+abstract = {},
+booktitle = {Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining},
+pages = {165–175},
+numpages = {11},
+keywords = {model compression, recommendation systems, embeddings},
+location = {Virtual Event, CA, USA},
+series = {KDD '20}
+}
+
+@misc{ginart2021mixed,
+      title={Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems}, 
+      author={Antonio Ginart and Maxim Naumov and Dheevatsa Mudigere and Jiyan Yang and James Zou},
+      year={2021},
+      eprint={1909.11810},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+
+@inproceedings{10.1145/2020408.2020444,
+author = {Chu, Wei and Zinkevich, Martin and Li, Lihong and Thomas, Achint and Tseng, Belle},
+title = {Unbiased Online Active Learning in Data Streams},
+year = {2011},
+isbn = {9781450308137},
+publisher = {Association for Computing Machinery},
+address = {New York, NY, USA},
+url = {https://doi.org/10.1145/2020408.2020444},
+doi = {10.1145/2020408.2020444},
+abstract = {Unlabeled samples can be intelligently selected for labeling to minimize classification error. In many real-world applications, a large number of unlabeled samples arrive in a streaming manner, making it impossible to maintain all the data in a candidate pool. In this work, we focus on binary classification problems and study selective labeling in data streams where a decision is required on each sample sequentially. We consider the unbiasedness property in the sampling process, and design optimal instrumental distributions to minimize the variance in the stochastic process. Meanwhile, Bayesian linear classifiers with weighted maximum likelihood are optimized online to estimate parameters. In empirical evaluation, we collect a data stream of user-generated comments on a commercial news portal in 30 consecutive days, and carry out offline evaluation to compare various sampling strategies, including unbiased active learning, biased variants, and random sampling. Experimental results verify the usefulness of online active learning, especially in the non-stationary situation with concept drift.},
+booktitle = {Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},
+pages = {195–203},
+numpages = {9},
+keywords = {unbiasedness, bayesian online learning, active learning, data streaming, adaptive importance sampling},
+location = {San Diego, California, USA},
+series = {KDD '11}
+}
+
+@inproceedings{10.1145/3267809.3267817,
+author = {Tian, Huangshi and Yu, Minchen and Wang, Wei},
+title = {Continuum: A Platform for Cost-Aware, Low-Latency Continual Learning},
+year = {2018},
+isbn = {9781450360111},
+publisher = {Association for Computing Machinery},
+address = {New York, NY, USA},
+url = {https://doi.org/10.1145/3267809.3267817},
+doi = {10.1145/3267809.3267817},
+abstract = {Many machine learning applications operate in dynamic environments that change over time, in which models must be continually updated to capture the recent trend in data. However, most of today's learning frameworks perform training offline, without a system support for continual model updating.In this paper, we design and implement Continuum, a general-purpose platform that streamlines the implementation and deployment of continual model updating across existing learning frameworks. In pursuit of fast data incorporation, we further propose two update policies, cost-aware and best-effort, that judiciously determine when to perform model updating, with and without accounting for the training cost (machine-time), respectively. Theoretical analysis shows that cost-aware policy is 2-competitive. We implement both polices in Continuum, and evaluate their performance through EC2 deployment and trace-driven simulations. The evaluation shows that Continuum results in reduced data incorporation latency, lower training cost, and improved model quality in a number of popular online learning applications that span multiple application domains, programming languages, and frameworks.},
+booktitle = {Proceedings of the ACM Symposium on Cloud Computing},
+pages = {26–40},
+numpages = {15},
+keywords = {Competitive Analysis, Continual Learning System, Online Algorithm},
+location = {Carlsbad, CA, USA},
+series = {SoCC '18}
+}
+
+@inproceedings{10.1145/2648584.2648589,
+author = {He, Xinran and Pan, Junfeng and Jin, Ou and Xu, Tianbing and Liu, Bo and Xu, Tao and Shi, Yanxin and Atallah, Antoine and Herbrich, Ralf and Bowers, Stuart and Candela, Joaquin Qui\~{n}onero},
+title = {Practical Lessons from Predicting Clicks on Ads at Facebook},
+year = {2014},
+isbn = {9781450329996},
+publisher = {Association for Computing Machinery},
+address = {New York, NY, USA},
+url = {https://doi.org/10.1145/2648584.2648589},
+doi = {10.1145/2648584.2648589},
+abstract = {Online advertising allows advertisers to only bid and pay for measurable user responses, such as clicks on ads. As a consequence, click prediction systems are central to most online advertising systems. With over 750 million daily active users and over 1 million active advertisers, predicting clicks on Facebook ads is a challenging machine learning task. In this paper we introduce a model which combines decision trees with logistic regression, outperforming either of these methods on its own by over 3%, an improvement with significant impact to the overall system performance. We then explore how a number of fundamental parameters impact the final prediction performance of our system. Not surprisingly, the most important thing is to have the right features: those capturing historical information about the user or ad dominate other types of features. Once we have the right features and the right model (decisions trees plus logistic regression), other factors play small roles (though even small improvements are important at scale). Picking the optimal handling for data freshness, learning rate schema and data sampling improve the model slightly, though much less than adding a high-value feature, or picking the right model to begin with.},
+booktitle = {Proceedings of the Eighth International Workshop on Data Mining for Online Advertising},
+pages = {1–9},
+numpages = {9},
+location = {New York, NY, USA},
+series = {ADKDD'14}
+}
+
+@misc{2017NVIDIA,
+  author={NVIDIA},
+  title={NVIDIA Tesla V100 GPU Architecture: The World's Most Advanced Datacenter GPU},
+  year={2017},
+  howpublished = "Website",
+  note = {\url{http://www.nvidia.com/object/volta-architecture-whitepaper.html}}
+}
+
+@article{2018Modeling,
+  title={Modeling Deep Learning Accelerator Enabled GPUs},
+  author={Raihan, M. A. and Goli, N. and Aamodt, T.},
+  journal={arXiv e-prints arXiv:1811.08309},
+  year={2018}
+}
+
+@misc{2020MLIR,
+  title={MLIR: A Compiler Infrastructure for the End of Moore's Law},
+  author={ Lattner, C.  and  Amini, M.  and  Bondhugula, U.  and  Cohen, A.  and  Davis, A.  and  Pienaar, J.  and  Riddle, R.  and  Shpeisman, T.  and  Vasilache, N.  and  Zinenko, O. },
+  year={2020},
+}
+
+@book{2007Engineering,
+  title={Engineering a Compiler},
+  author={ Cooper, Keith D.  and  Torczon, Linda },
+  publisher={Engineering A Compiler},
+  year={2007},
+}
+
+@misc{2007Compilers,
+  title={Compilers: Principles, Techniques, and Tools (Rental), 2nd Edition},
+  author={ Aho, A. V.  and  Lam, M. S.  and  Ullman, J. D.  and  Sethi, R. },
+  year={2007},
+}
+
+@inproceedings{2004LLVM,
+  title={LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation},
+  author={ Lattner, C.  and  Adve, V. },
+  booktitle={Code Generation and Optimization, 2004. CGO 2004. International Symposium on},
+  year={2004},
+}
+
+@article{Richard1995A,
+  title={A correspondence between continuation passing style and static single assignment form},
+  author={Richard and A. and Kelsey},
+  journal={Acm Sigplan Notices},
+  year={1995},
+}
+
+@article{2010C,
+  title={C++ lambda expressions and closures},
+  author={ Jaervi, Jaakko  and  Freeman, J. },
+  journal={Science of Computer Programming},
+  volume={75},
+  number={9},
+  pages={762-772},
+  year={2010},
+}
+
+@article{spuler1994compiler,
+  title={Compiler detection of function call side effects},
+  author={Spuler, David A and Sajeev, A Sayed Muhammed},
+  journal={Informatica},
+  volume={18},
+  number={2},
+  pages={219--227},
+  year={1994},
+  publisher={Citeseer}
+}
+
+@book{10.5555/1455489,
+  author = {Griewank, Andreas and Walther, Andrea},
+  title = {Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation},
+  year = {2008},
+  isbn = {0898716594},
+  publisher = {Society for Industrial and Applied Mathematics},
+  address = {USA},
+  edition = {Second},
+}
+
+@article{2015Automatic,
+  title={Automatic Differentiation in Machine Learning: a Survey},
+  author={ Pearlmutter, B. A. },
+  journal={computer science},
+  number={February},
+  year={2015},
+}
+
+@article{2015Numerical,
+  title={Numerical Analysis},
+  author={ Burden, R. L.  and  Faires, Jdd },
+  journal={Journal of the Royal Statistical Society},
+  volume={71},
+  number={1},
+  pages={48-50},
+  year={2015},
+}
+
+@book{2003Computer,
+  title={Computer Algebra Handbook: Foundations * Applications * Systems},
+  author={ Grabmeier, J.  and  Kaltofen, E.  and  Weispfenning, V. },
+  publisher={Computer algebra handbook : foundations, applications, systems},
+  year={2003},
+}
+
+@inbook{10.5555/60181.60188,
+author = {Corliss, George F.},
+title = {Applications of Differentiation Arithmetic},
+year = {1988},
+isbn = {0125056303},
+publisher = {Academic Press Professional, Inc.},
+address = {USA},
+booktitle = {Reliability in Computing: The Role of Interval Methods in Scientific Computing},
+pages = {127–148},
+numpages = {22}
+}
+
+@article{2000An,
+  title={An introduction to automatic differentiation},
+  author={ Verma, A. },
+  journal={Siam Computational Differentiation Techniques Applications & Tools},
+  volume={78},
+  number={7},
+  pages={804-807},
+  year={2000},
+}
+
+@inproceedings{2006The,
+  title={The Data-Flow Equations of Checkpointing in Reverse Automatic Differentiation},
+  author={ Dauvergne, B.  and L Hascoët},
+  booktitle={Computational Science-iccs, International Conference, Reading, Uk, May},
+  year={2006},
+}
+
+@article{2017Divide,
+  title={Divide-and-Conquer Checkpointing for Arbitrary Programs with No User Annotation},
+  author={ Siskind, Jeffrey Mark  and  Pearlmutter, Barak A. },
+  journal={Optimization Methods and Software},
+  volume={33},
+  number={4-6},
+  year={2017},
+}
+
+@article{1969The,
+  title={The Principal Type-Scheme of an Object in Combinatory Logic},
+  author={ Hindley, R. },
+  journal={Transactions of the American Mathematical Society},
+  volume={146},
+  pages={29-60},
+  year={1969},
+}
+
+@article{1978A,
+  title={A theory of type polymorphism in programming},
+  author={ Milner, R. },
+  journal={Journal of Computer and System Sciences},
+  volume={17},
+  number={3},
+  pages={348-375},
+  year={1978},
+}
+
+@article{ragan2013halide,
+  title={Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines},
+  author={Ragan-Kelley, Jonathan and Barnes, Connelly and Adams, Andrew and Paris, Sylvain and Durand, Fr{\'e}do and Amarasinghe, Saman},
+  journal={Acm Sigplan Notices},
+  volume={48},
+  number={6},
+  pages={519--530},
+  year={2013},
+  publisher={ACM New York, NY, USA}
+}
+
+@inproceedings{verdoolaege2010isl,
+  title={isl: An integer set library for the polyhedral model},
+  author={Verdoolaege, Sven},
+  booktitle={International Congress on Mathematical Software},
+  pages={299--302},
+  year={2010},
+  organization={Springer}
+}
+
+@article{chen2018tvm,
+  title={TVM: end-to-end optimization stack for deep learning},
+  author={Chen, Tianqi and Moreau, Thierry and Jiang, Ziheng and Shen, Haichen and Yan, Eddie Q and Wang, Leyuan and Hu, Yuwei and Ceze, Luis and Guestrin, Carlos and Krishnamurthy, Arvind},
+  journal={arXiv preprint arXiv:1802.04799},
+  volume={11},
+  pages={20},
+  year={2018},
+  publisher={CoRR}
+}
+
+@inproceedings{zheng2020ansor,
+  title={Ansor: Generating $\{$High-Performance$\}$ Tensor Programs for Deep Learning},
+  author={Zheng, Lianmin and Jia, Chengfan and Sun, Minmin and Wu, Zhao and Yu, Cody Hao and Haj-Ali, Ameer and Wang, Yida and Yang, Jun and Zhuo, Danyang and Sen, Koushik and others},
+  booktitle={14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)},
+  pages={863--879},
+  year={2020}
+}
+
+@inproceedings{zhao2021akg,
+  title={AKG: automatic kernel generation for neural processing units using polyhedral transformations},
+  author={Zhao, Jie and Li, Bojie and Nie, Wang and Geng, Zhen and Zhang, Renwei and Gao, Xiong and Cheng, Bin and Wu, Chen and Cheng, Yun and Li, Zheng and others},
+  booktitle={Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation},
+  pages={1233--1248},
+  year={2021}
+}
+
+@article{lattner2020mlir,
+  title={MLIR: A compiler infrastructure for the end of Moore's law},
+  author={Lattner, Chris and Amini, Mehdi and Bondhugula, Uday and Cohen, Albert and Davis, Andy and Pienaar, Jacques and Riddle, River and Shpeisman, Tatiana and Vasilache, Nicolas and Zinenko, Oleksandr},
+  journal={arXiv preprint arXiv:2002.11054},
+  year={2020}
+}
+
+@article{vasilache2022composable,
+  title={Composable and Modular Code Generation in MLIR: A Structured and Retargetable Approach to Tensor Compiler Construction},
+  author={Vasilache, Nicolas and Zinenko, Oleksandr and Bik, Aart JC and Ravishankar, Mahesh and Raoux, Thomas and Belyaev, Alexander and Springer, Matthias and Gysi, Tobias and Caballero, Diego and Herhut, Stephan and others},
+  journal={arXiv preprint arXiv:2202.03293},
+  year={2022}
+}
+
+@inproceedings{bastoul2004code,
+  title={Code generation in the polyhedral model is easier than you think},
+  author={Bastoul, C{\'e}dric},
+  booktitle={Proceedings. 13th International Conference on Parallel Architecture and Compilation Techniques, 2004. PACT 2004.},
+  pages={7--16},
+  year={2004},
+  organization={IEEE}
+}