build: migrate docs build and deploy to mdbook (#490)

* build: add mdbook support for zh chapters

Add mdBook configuration rooted at zh_chapters, generate and commit SUMMARY.md, rewrite d2l-specific directives through a Python preprocessor, refresh chapter resource symlinks from the build scripts, and ignore local build-only links and helper directories.

* feat: add raw HTML inline and frontpage layout support for mdbook preprocessor

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add dark mode image background for mdbook dark themes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add resource symlinks and repo root static fallback to mdbook build

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add BibTeX citation support with inline links and bibliography

Parse mlsys.bib to generate author-year inline citations linked to
per-page bibliography sections. Missing bib keys degrade gracefully
to plain text placeholders.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: switch citation display to footnote style

Use numbered superscript references [1] [2] inline with an ordered
list bibliography at page bottom. Each entry has a back-link (↩)
to the citation site.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: strip LaTeX escapes outside math mode in mdbook preprocessor

Remove \_, \%, \#, \& escapes from text outside $...$ math spans
while preserving them inside math mode for MathJax compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: set frontpage author grid to 6 columns and widen main content area

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: group mdbook toc by part titles

* fix: enable inline math rendering in mdbook

* build: migrate docs publishing to mdbook

Move the English root site to mdBook, keep the Chinese site as a sub-book, and update CI/deploy to publish .mdbook outputs to docs/ and docs/cn/. Also add regression coverage for placeholder skipping, publish-tree assembly, and shared resource setup.

* ci: use official pages deployment workflow

Switch the docs deployment workflow to the official GitHub Pages actions flow and verify it uses Pages action outputs for the deployment URL.

* feat: add homepage language switch links

Inject a homepage-only language switch into the mdBook frontpage wrapper so the English homepage links to the Chinese homepage and the Chinese homepage links back to the English homepage.

* fix: correct english homepage frontpage

Add an English-specific frontpage template so the default homepage no longer falls back to the Chinese frontpage, and clear homepage image backgrounds in the frontpage wrapper CSS.

* fix: align english homepage author grid

Top-align the English homepage author cards, enlarge the row gap, and normalize avatar sizing so author portraits line up consistently.

* fix: restore dark mode body image backgrounds

Apply light gray backgrounds to body images in dark themes for both English and Chinese mdBook themes while explicitly excluding homepage frontpage images.

* fix: restyle homepage language switch button

Move the homepage language switch below the GitHub star button and restyle it to match the same button family on both the English and Chinese homepages.

* fix: center homepage content container

Align the English and Chinese homepage frontpage wrapper with the main content container so homepage content is centered like normal body content.

* fix: stack english homepage footer copy

Keep the English homepage contributor and errata footer lines in normal block flow so each sentence stays on its own line instead of being laid out as author-grid columns.

* fix: widen centered homepage container

Keep the homepage frontpage wrapper centered while ensuring it uses at least 80% of the available content area, without changing normal body page layout.

* fix: widen homepage main content area

Apply a homepage-only override so mdbook-content > main uses at least 80% of the available content width while keeping normal body pages on the default layout.

* ci: use peaceiris action for mdbook

Replace manual mdBook installation in CI and Pages workflows with peaceiris/actions-mdbook@v2 and keep a regression test to ensure the action stays in use.

* fix: reduce homepage main width floor

Lower the homepage-only mdbook-content > main minimum width from 80% to 65% while leaving normal body pages unchanged.

* build: switch math rendering to mdbook-katex

Use mdbook-katex in pre-render mode for both books, pin mdBook to a compatible version, update build scripts and workflows, and replace the old MathJax regression tests with KaTeX coverage.

* Revert "build: switch math rendering to mdbook-katex"

This reverts commit b9cf38a5d1.

* build: switch math rendering from MathJax to mdbook-typst-math

* ci: deploy docs to openmlsys.github.io repo

* fix: convert pandoc tables to GFM pipe tables for mdbook

* feat: convert :eqlabel:/:eqref: to MathJax \tag/\label/\eqref

- Add process_equation_labels() to inject \tag{n}\label{name} into
  preceding $$ equations, replacing :eqlabel: directives
- Change :eqref: conversion from backtick code to $\eqref{name}$
  for clickable cross-references
- Add TeX.equationNumbers.autoNumber:"none" to MathJax config to
  prevent conflicts with manual \tag numbering
- Add tests for single-line, multi-line, and sequential numbering

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* ci: cache mdbook-typst-math binary in workflows

* feat: add LaTeX-to-Typst math converter with eqref/tag support

* feat: integrate LaTeX-to-Typst conversion into zh preprocessor

* fix: strip LaTeX escapes only outside math spans and code blocks

* fix: load references/*.bib so all citations render correctly

* fix: skip citations with no bib entry instead of rendering raw keys

* ci: remove redundant CI workflow, keep only deploy workflow

* ci: Add CI workflow for testing and building mdBook

* ci: remove concurrency settings from update_docs.yml

Removed concurrency settings from the update_docs workflow.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
anyin233
2026-03-11 16:17:37 +00:00
committed by GitHub
parent 09dd269236
commit 6c9673a659
44 changed files with 3654 additions and 1441 deletions

View File

@@ -6,8 +6,8 @@ on:
workflow_dispatch:
jobs:
build-en:
name: Build (English)
test:
name: Tests
runs-on: ubuntu-22.04
steps:
@@ -17,62 +17,30 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: 'pip'
- name: Install pandoc
run: |
wget -q https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-1-amd64.deb
sudo dpkg -i pandoc-2.19.2-1-amd64.deb
- name: Install d2lbook
run: |
git clone https://github.com/openmlsys/d2l-book.git
cd d2l-book
# Fix Python 3.10+ incompatibility: bibtex<2.0.0 depends on oset which
# uses collections.MutableSet removed in Python 3.10.
sed -i "s/'sphinxcontrib-bibtex<2.0.0'/'sphinxcontrib-bibtex>=2.5.0'/" setup.py
python3 -m pip install .
- name: Install Python dependencies
run: python3 -m pip install -r requirements.txt
- name: Build English HTML
run: bash build_html.sh
build-zh:
name: Build (Chinese)
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.10
uses: actions/setup-python@v5
- name: Setup mdBook
uses: peaceiris/actions-mdbook@v2
with:
python-version: '3.10'
cache: 'pip'
mdbook-version: 'latest'
- name: Install pandoc
- name: Cache mdbook-typst-math binary
uses: actions/cache@v4
with:
path: .mdbook-bin
key: mdbook-typst-math-v0.3.0-linux-x86_64
- name: Run mdBook regression tests
run: |
wget -q https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-1-amd64.deb
sudo dpkg -i pandoc-2.19.2-1-amd64.deb
python3 -m unittest discover -s tests -p 'test_prepare_mdbook.py'
python3 -m unittest discover -s tests -p 'test_prepare_mdbook_zh.py'
python3 -m unittest discover -s tests -p 'test_assemble_docs_publish_tree.py'
python3 -m unittest discover -s tests -p 'test_ensure_book_resources.py'
python3 -m unittest discover -s tests -p 'test_mdbook_typst_math.py'
python3 -m unittest discover -s tests -p 'test_ensure_mdbook_typst_math.py'
python3 -m unittest discover -s tests -p 'test_update_docs_workflow.py'
- name: Install d2lbook
run: |
git clone https://github.com/openmlsys/d2l-book.git
cd d2l-book
sed -i "s/'sphinxcontrib-bibtex<2.0.0'/'sphinxcontrib-bibtex>=2.5.0'/" setup.py
python3 -m pip install .
- name: Build English HTML with mdBook
run: bash build_mdbook.sh
- name: Install Python dependencies
run: python3 -m pip install -r requirements.txt
- name: Build Chinese HTML
run: bash build_html_zh.sh
build:
name: build
needs: [build-en, build-zh]
runs-on: ubuntu-22.04
steps:
- run: echo "All builds passed"
- name: Build Chinese HTML with mdBook
run: bash build_mdbook_zh.sh

View File

@@ -16,47 +16,53 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: 'pip'
- name: Install pandoc
- name: Cache mdbook-typst-math binary
uses: actions/cache@v4
with:
path: .mdbook-bin
key: mdbook-typst-math-v0.3.0-linux-x86_64
- name: Setup mdBook
uses: peaceiris/actions-mdbook@v2
with:
mdbook-version: 'latest'
- name: Run mdBook regression tests
run: |
wget -q https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-1-amd64.deb
sudo dpkg -i pandoc-2.19.2-1-amd64.deb
python3 -m unittest discover -s tests -p 'test_prepare_mdbook.py'
python3 -m unittest discover -s tests -p 'test_prepare_mdbook_zh.py'
python3 -m unittest discover -s tests -p 'test_assemble_docs_publish_tree.py'
python3 -m unittest discover -s tests -p 'test_ensure_book_resources.py'
python3 -m unittest discover -s tests -p 'test_mdbook_typst_math.py'
python3 -m unittest discover -s tests -p 'test_ensure_mdbook_typst_math.py'
- name: Install d2lbook
run: |
git clone https://github.com/openmlsys/d2l-book.git
cd d2l-book
# Fix Python 3.10+ incompatibility: bibtex<2.0.0 depends on oset which
# uses collections.MutableSet removed in Python 3.10.
sed -i "s/'sphinxcontrib-bibtex<2.0.0'/'sphinxcontrib-bibtex>=2.5.0'/" setup.py
python3 -m pip install .
- name: Build English HTML with mdBook
run: bash build_mdbook.sh
- name: Install Python dependencies
run: python3 -m pip install -r requirements.txt sphinx-mathjax-offline
- name: Build Chinese HTML with mdBook
run: bash build_mdbook_zh.sh
- name: Build English HTML
run: bash build_html.sh
- name: Build Chinese HTML
run: bash build_html_zh.sh
- name: Deploy to GitHub Pages
- name: Deploy to openmlsys.github.io
env:
DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
run: |
git clone https://x-access-token:${DEPLOY_TOKEN}@github.com/openmlsys/openmlsys.github.io.git
# English → root (default language)
cp -r en_chapters/_build/html/* openmlsys.github.io/docs/
# Chinese → /cn/ subdirectory
mkdir -p openmlsys.github.io/docs/cn
cp -r zh_chapters/_build/html/* openmlsys.github.io/docs/cn/
python3 tools/assemble_docs_publish_tree.py \
--destination-root openmlsys.github.io \
--docs-subdir docs \
--en-source .mdbook/book \
--zh-source .mdbook-zh/book
cd openmlsys.github.io
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add .
git commit -m "deploy: update docs (en+zh) from openmlsys-zh@${{ github.sha }}" || echo "No changes to commit"
git push
git add docs
if git diff --cached --quiet; then
echo "No changes to commit"
else
git commit -m "deploy: update docs from ${GITHUB_REPOSITORY}@${GITHUB_SHA}"
git push
fi

17
.gitignore vendored
View File

@@ -12,3 +12,20 @@ test*.md
run.sh
.idea
env
.mdbook/
.mdbook-zh/
.mdbook-zh-test/
.mdbook-bin/
task_plan.md
findings.md
progress.md
d2l-book/
docs/
en_chapters/img
en_chapters/references
en_chapters/static
en_chapters/mlsys.bib
zh_chapters/img
zh_chapters/references
zh_chapters/static
zh_chapters/mlsys.bib

19
book.toml Normal file
View File

@@ -0,0 +1,19 @@
[book]
authors = ["OpenMLSys Contributors"]
language = "en"
src = "en_chapters"
title = "Machine Learning Systems: Design and Implementation"
[build]
build-dir = ".mdbook/book"
create-missing = false
[preprocessor.openmlsys]
command = "python3 tools/mdbook_preprocessor.py"
[preprocessor.typst-math]
[output.html]
git-repository-url = "https://github.com/openmlsys/openmlsys-zh"
preferred-dark-theme = "navy"
additional-css = ["theme/dark-mode-images.css", "theme/typst.css"]

19
books/zh/book.toml Normal file
View File

@@ -0,0 +1,19 @@
[book]
authors = ["OpenMLSys Contributors"]
language = "zh-CN"
src = "../../zh_chapters"
title = "机器学习系统:设计和实现"
[build]
build-dir = "../../.mdbook-zh/book"
create-missing = false
[preprocessor.openmlsys-zh]
command = "python3 ../../tools/mdbook_zh_preprocessor.py"
[preprocessor.typst-math]
[output.html]
git-repository-url = "https://github.com/openmlsys/openmlsys-zh"
preferred-dark-theme = "navy"
additional-css = ["theme/dark-mode-images.css", "theme/typst.css"]

View File

@@ -0,0 +1,16 @@
/* 暗色模式下仅为正文图片添加浅灰色背景,提高透明背景图片的可读性 */
.navy .content main img,
.coal .content main img,
.ayu .content main img {
background-color: #e8e8e8;
border-radius: 4px;
padding: 8px;
}
/* 首页 frontpage 图片保持透明,不添加正文图像底色。 */
.navy .openmlsys-frontpage img,
.coal .openmlsys-frontpage img,
.ayu .openmlsys-frontpage img {
background-color: transparent !important;
padding: 0 !important;
}

16
books/zh/theme/typst.css Normal file
View File

@@ -0,0 +1,16 @@
.typst-inline {
display: inline-flex;
vertical-align: -0.2em;
}
.typst-display {
display: flex;
justify-content: center;
margin: 1rem 0;
overflow-x: auto;
}
.typst-doc {
color: var(--fg);
max-width: 100%;
}

View File

@@ -10,12 +10,7 @@ set -e
ROOT="$(cd "$(dirname "$0")" && pwd)"
# ── Create resource symlinks ──────────────────────────────────────────────────
for target in img references static mlsys.bib; do
link="$ROOT/en_chapters/$target"
if [ ! -e "$link" ]; then
ln -sf "$ROOT/$target" "$link"
fi
done
python3 "$ROOT/tools/ensure_book_resources.py" --chapter-dir "$ROOT/en_chapters"
# ── Build ─────────────────────────────────────────────────────────────────────
cd "$ROOT/en_chapters"

View File

@@ -10,12 +10,7 @@ set -e
ROOT="$(cd "$(dirname "$0")" && pwd)"
# ── Create resource symlinks ──────────────────────────────────────────────────
for target in img references static mlsys.bib; do
link="$ROOT/zh_chapters/$target"
if [ ! -e "$link" ]; then
ln -sf "$ROOT/$target" "$link"
fi
done
python3 "$ROOT/tools/ensure_book_resources.py" --chapter-dir "$ROOT/zh_chapters"
# ── Build ─────────────────────────────────────────────────────────────────────
cd "$ROOT/zh_chapters"

27
build_mdbook.sh Normal file
View File

@@ -0,0 +1,27 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PYTHON_BIN="$(command -v python3 || command -v python || true)"
if [[ -z "${PYTHON_BIN}" ]]; then
echo "Python is required to prepare the mdBook staging tree." >&2
exit 1
fi
if ! command -v mdbook >/dev/null 2>&1; then
echo "mdbook is not installed. Install it first, for example with: cargo install mdbook" >&2
exit 1
fi
MDBOOK_TYPST_MATH_BIN_DIR="${ROOT}/.mdbook-bin"
"${PYTHON_BIN}" "${ROOT}/tools/ensure_mdbook_typst_math.py" --output-dir "${MDBOOK_TYPST_MATH_BIN_DIR}" >/dev/null
export PATH="${MDBOOK_TYPST_MATH_BIN_DIR}:${PATH}"
"${PYTHON_BIN}" "${ROOT}/tools/ensure_book_resources.py" --chapter-dir "${ROOT}/en_chapters"
"${PYTHON_BIN}" "${ROOT}/tools/prepare_mdbook.py" \
--source "${ROOT}/en_chapters" \
--summary-output "${ROOT}/en_chapters/SUMMARY.md" \
--placeholder-prefix "[TODO: src = zh_chapters/"
mdbook build "${ROOT}"

29
build_mdbook_zh.sh Executable file
View File

@@ -0,0 +1,29 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PYTHON_BIN="$(command -v python3 || command -v python || true)"
if [[ -z "${PYTHON_BIN}" ]]; then
echo "Python is required to prepare the mdBook staging tree." >&2
exit 1
fi
if ! command -v mdbook >/dev/null 2>&1; then
echo "mdbook is not installed. Install it first, for example with: cargo install mdbook" >&2
exit 1
fi
MDBOOK_TYPST_MATH_BIN_DIR="${ROOT}/.mdbook-bin"
"${PYTHON_BIN}" "${ROOT}/tools/ensure_mdbook_typst_math.py" --output-dir "${MDBOOK_TYPST_MATH_BIN_DIR}" >/dev/null
export PATH="${MDBOOK_TYPST_MATH_BIN_DIR}:${PATH}"
# ── Create resource links ─────────────────────────────────────────────────────
"${PYTHON_BIN}" "${ROOT}/tools/ensure_book_resources.py" --chapter-dir "${ROOT}/zh_chapters"
# ── Build ─────────────────────────────────────────────────────────────────────
"${PYTHON_BIN}" "${ROOT}/tools/prepare_mdbook_zh.py" \
--source "${ROOT}/zh_chapters" \
--summary-output "${ROOT}/zh_chapters/SUMMARY.md"
mdbook build "${ROOT}/books/zh"

3
en_chapters/SUMMARY.md Normal file
View File

@@ -0,0 +1,3 @@
# Summary
[Machine Learning Systems: Design and Implementation](index.md)

507
en_chapters/frontpage.html Normal file
View File

@@ -0,0 +1,507 @@
<head>
<style>
h1, .side-doc-outline {
display: none;
}
.document .page-content {
width: 100%;
}
h2, h3, h4 {
letter-spacing: 2px;
}
.mdl-grid {
align-items: center;
justify-content: center;
text-align: center;
padding: 150px 0 0 0;
}
h2.toc {
padding: 150px 0 0 0;
}
.header h2 {
font-size: 2rem;
padding-bottom: 35px;
}
.header p {
font-size: 1.25rem;
letter-spacing: 0px;
line-height: 1.5rempx;
padding-bottom: 10px;
}
.header .mdl-button {
margin: 0px 10px;
letter-spacing: 2px;
font-size: 16px;
margin: 1rem 1.5rem;
height: 40px;
width: 120px;
}
.author-item {
max-width: 300px;
width: 100%;
display: flex;
flex-direction: column;
align-items: center;
}
.author-item h3 {
padding-top: 25px;
padding-bottom: 10px;
}
.author-item img {
border-radius: 50%;
width: 120px;
height: 120px;
object-fit: cover;
}
.authors.mdl-grid {
align-items: flex-start;
row-gap: calc(24px + 1.5em);
}
.authors .mdl-cell:not(.author-group-title) {
display: flex;
align-items: flex-start;
justify-content: center;
}
.author-group-title {
display: block;
width: 100%;
}
.author-group-title h3,
.author-group-title h4 {
width: 100%;
margin: 0 0 0.75em;
}
.author-group-title h4:last-child {
margin-bottom: 0;
}
.authors h4 {
width: 100%;
}
.authors h2 {
padding-bottom: 18px;
}
.running-item {
max-width: 300px;
}
.running-item img {
height: 40px;
}
.running-item p {
padding: 20px;
}
.running.mdl-grid {
padding: 20px 0 0 0;
}
.logos img {
display: inline;
max-height: 70px;
max-width: 100px;
margin-left: auto;
margin-right: auto;
}
.logos {
padding-top: 1em;
padding-bottom: 2em;
}
.logos .mdl-grid {
padding: 0;
}
.features .logos .mdl-cell {
padding: 10px 0;
}
.features .mdl-cell {
padding: 0 36px;
}
.features h2, .features-2 h2 {
padding-bottom: 24px;
}
.features-2 p {
padding: 0 200px;
}
.institute.mdl-grid {
padding: 24px;
}
.institute .mdl-cell {
text-align: left;
padding: 0;
}
#logoimg {
padding-top: 0em;
padding-bottom: 2em;
width: 100%;
}
#map {
height: 400px;
width: 100%;
}
#mapimg {
width: 100%;
}
a {
font-size: normal;
}
@media (max-width: 1000px) {
.mdl-grid, h2.toc {
padding: 50px 0 0 0;
}
.header img, .features-2 img {
max-width: 300px;
}
.header .mdl-cell--1-col {
display: none;
}
.features img {
max-width: 350px;
}
.features-2 p {
padding: 0 24px;
}
.features .mdl-cell {
padding: 0 8px;
}
}
.collapsible {
background-color: #EEE;
color: black;
cursor: pointer;
padding: 6px;
width: 100%;
border: none;
text-align: left;
outline: none;
font-size: 16px;
}
.collapsible-active, .collapsible:hover {
background-color: #DDD;
}
.expansion {
padding: 3px 6px;
max-height: 0;
overflow: hidden;
transition: max-height 0.2s ease-out;
}
.cover h2 {
font-size: 34px;
margin-bottom: 0px;
padding-bottom: 20px;
}
.cover p.version {
font-size: 18px;
padding: 0 0 9px 0;
}
.cover p {
margin: 0;
padding: 20px 0 0px 0;
font-size: 20px;
}
</style>
</head>
<div class="mdl-grid header">
<div class="mdl-cell mdl-cell--3-col">
<img src="_images/logo.png" class="img-fluid" alt="">
</div>
<div class="mdl-cell mdl-cell--1-col">
</div>
<div class="mdl-cell mdl-cell--5-col mdl-cell--middle cover">
<h2 class="mdl-color-text--primary">Machine Learning Systems: Design and Implementation</h2>
<p>The first open-source book to comprehensively cover machine learning systems</p>
<p><a class="github-button" href="https://github.com/openmlsys/openmlsys-zh" data-icon="octicon-star" data-size="large" data-show-count="true" aria-label="Star openmlsys/openmlsys-zh on GitHub">Star</a></p>
<!-- OPENMLSYS_LANGUAGE_SWITCH -->
</div>
</div>
<div class = "authors mdl-grid" id = "author" >
<div class = "author-group-title mdl-cell mdl-cell--12-col mdl-cell--top">
<h2>Core Authors</h2>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/1136455?v=4"/>
<h3><a href="https://github.com/luomai">Mai Luo</a></h3>
<p>University of Edinburgh</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/10713581?v=4"/>
<h3><a href="https://github.com/zsdonghao">Hao Dong</a></h3>
<p>Peking University, Peng Cheng Laboratory</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="./_images/jinxuefeng.png"/>
<h3>Xuefeng Jin</h3>
<p>MindSpore Architect</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/90942796?v=4"/>
<h3><a href="https://github.com/ganzhiliang">Zhiliang Gan</a></h3>
<p>MindSpore Architect</p>
</div>
</div>
<div class = "author-group-title mdl-cell mdl-cell--12-col mdl-cell--top">
<h2>Chapter Authors</h2>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/21324004?v=4"/>
<h3><a href="https://github.com/Laicheng0830">Cheng Lai</a></h3>
<p>Peng Cheng Laboratory</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/73918561?v=4"/>
<h3><a href="https://github.com/hanjr92">Jiarong Han</a></h3>
<p>Peng Cheng Laboratory</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/31511840?v=4"/>
<h3><a href="https://github.com/future-xy">Yao Fu</a></h3>
<p>University of Edinburgh</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/39682259?v=4"/>
<h3><a href="https://github.com/eedalong">Xiulong Yuan</a></h3>
<p>Tsinghua University</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/22659010?v=4"/>
<h3><a href="https://github.com/quantumiracle">Zihan Ding</a></h3>
<p>Princeton University</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/25477396?v=4"/>
<h3><a href="https://github.com/Jiankai-Sun">Jiankai Sun</a></h3>
<p>Stanford University</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/34192321?v=4"/>
<h3><a href="https://liaopeiyuan.com/">Peiyuan Liao</a></h3>
<p>Carnegie Mellon University</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/91313652?v=4"/>
<h3><a href="https://github.com/Went-Liang">Wenteng Liang</a></h3>
<p>Beijing University of Posts and Telecommunications</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/49299030?v=4"/>
<h3><a href="https://github.com/JieRen98">Jie Ren</a></h3>
<p>University of Edinburgh</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="./_images/wanghanchen.png"/>
<h3><a href="https://github.com/hansen7">Hanchen Wang</a></h3>
<p>University of Cambridge</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/10811136?v=4"/>
<h3><a href="https://github.com/ds1231h">Pei Mu</a></h3>
<p>University of Edinburgh</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="./_images/zhangqinghua.png"/>
<h3>Qinghua Zhang</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/18185293?v=4"/>
<h3>Zhibo Liang</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/20573763?v=4"/>
<h3>Jianfeng Yu</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/5563661?v=4"/>
<h3>Jinjin Chu</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/68210792?v=4"/>
<h3>Fubi Cai</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="./_images/zhangrenwei.png"/>
<h3>Renwei Zhang</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/2681256?v=4"/>
<h3>Chao Liu</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/5202350?v=4"/>
<h3>Gang Chen</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/61316635?v=4"/>
<h3>Mingqi Li</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/36526001?v=4"/>
<h3>Gangqiang Han</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/55628940?v=4"/>
<h3>Yehui Tang</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="./_images/zhaizhiqiang.png"/>
<h3>Zhiqiang Zhai</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="./_images/wutiancheng.png"/>
<h3>Tiancheng Wu</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/59854698?v=4"/>
<h3>Xiaohui Li</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/20376974?v=4"/>
<h3>Haoyang Li</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/17228522?v=4"/>
<h3>Zhipeng Tan</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="https://avatars.githubusercontent.com/u/105415060?v=4"/>
<h3>Shanni Li</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
<div class="author-item">
<img src="./_images/guozhijian.png"/>
<h3>Zhijian Guo</h3>
<p>Huawei Engineer</p>
</div>
</div>
<div class = "author-group-title mdl-cell mdl-cell--12-col mdl-cell--top">
<h3> Thanks to our <a href="https://github.com/openmlsys/openmlsys-zh/graphs/contributors">contributors</a></h3>
<h3> and everyone who reported valuable <a href="https://github.com/openmlsys/openmlsys-zh/issues">errata and suggestions</a></h3>
<h4><a href="https://github.com/openmlsys/openmlsys-zh">Contribute to this Book</a></h4>
</div>
</div>
<h2 class="toc">Table of Contents</h2>
<script>
var coll = document.getElementsByClassName("collapsible");
var i;
for (i = 0; i < coll.length; i++) {
coll[i].addEventListener("click", function() {
this.classList.toggle("collapsible-active");
var content = this.nextElementSibling;
if (content.style.maxHeight){
content.style.maxHeight = null;
} else {
content.style.maxHeight = content.scrollHeight + "px";
}
});
}
</script>
<script async defer src="https://buttons.github.io/buttons.js"></script>

View File

@@ -1 +0,0 @@
/chivier-disk/hyq-home/Projects/openmlsys-zh/img

View File

@@ -1 +0,0 @@
/chivier-disk/hyq-home/Projects/openmlsys-zh/references

View File

@@ -1 +0,0 @@
/chivier-disk/hyq-home/Projects/openmlsys-zh/static

View File

@@ -201,6 +201,7 @@ a {
<h2 class="mdl-color-text--primary">《机器学习系统:设计和实现》</h2>
<p>做世界上第一本全面讲述机器学习系统知识的开源书籍</p>
<p><a class="github-button" href="https://github.com/openmlsys/openmlsys-zh" data-icon="octicon-star" data-size="large" data-show-count="true" aria-label="Star openmlsys/openmlsys-zh on GitHub">Star</a></p>
<!-- OPENMLSYS_LANGUAGE_SWITCH -->
</div>
</div>
@@ -477,4 +478,3 @@ for (i = 0; i < coll.length; i++) {
</script>
<script async defer src="https://buttons.github.io/buttons.js"></script>

View File

@@ -0,0 +1,87 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from tools.assemble_docs_publish_tree import assemble_publish_tree
class AssembleDocsPublishTreeTests(unittest.TestCase):
def test_assemble_publish_tree_uses_legacy_docs_layout(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
pages_repo = root / "pages"
en_source = root / "en-html"
zh_source = root / "zh-html"
pages_repo.mkdir()
en_source.mkdir()
zh_source.mkdir()
(en_source / "index.html").write_text("english home", encoding="utf-8")
(en_source / "guide.html").write_text("english guide", encoding="utf-8")
(zh_source / "index.html").write_text("chinese home", encoding="utf-8")
(zh_source / "searchindex.js").write_text("zh search", encoding="utf-8")
assemble_publish_tree(
destination_root=pages_repo,
docs_subdir="docs",
en_source=en_source,
zh_source=zh_source,
)
self.assertEqual(
(pages_repo / "docs" / "index.html").read_text(encoding="utf-8"),
"english home",
)
self.assertEqual(
(pages_repo / "docs" / "guide.html").read_text(encoding="utf-8"),
"english guide",
)
self.assertEqual(
(pages_repo / "docs" / "cn" / "index.html").read_text(encoding="utf-8"),
"chinese home",
)
self.assertEqual(
(pages_repo / "docs" / "cn" / "searchindex.js").read_text(encoding="utf-8"),
"zh search",
)
def test_assemble_publish_tree_replaces_stale_docs_content(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
pages_repo = root / "pages"
en_source = root / "en-html"
zh_source = root / "zh-html"
(pages_repo / "docs" / "cn").mkdir(parents=True)
(pages_repo / "docs" / "old.html").write_text("stale en", encoding="utf-8")
(pages_repo / "docs" / "cn" / "old.html").write_text("stale zh", encoding="utf-8")
en_source.mkdir()
zh_source.mkdir()
(en_source / "index.html").write_text("fresh en", encoding="utf-8")
(zh_source / "index.html").write_text("fresh zh", encoding="utf-8")
assemble_publish_tree(
destination_root=pages_repo,
docs_subdir="docs",
en_source=en_source,
zh_source=zh_source,
)
self.assertFalse((pages_repo / "docs" / "old.html").exists())
self.assertFalse((pages_repo / "docs" / "cn" / "old.html").exists())
self.assertEqual(
(pages_repo / "docs" / "index.html").read_text(encoding="utf-8"),
"fresh en",
)
self.assertEqual(
(pages_repo / "docs" / "cn" / "index.html").read_text(encoding="utf-8"),
"fresh zh",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,30 @@
from __future__ import annotations
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
class DarkModeImagesCssTests(unittest.TestCase):
def test_both_theme_css_files_style_dark_mode_body_images_only(self) -> None:
css_paths = [
REPO_ROOT / "theme" / "dark-mode-images.css",
REPO_ROOT / "books" / "zh" / "theme" / "dark-mode-images.css",
]
for css_path in css_paths:
css = css_path.read_text(encoding="utf-8")
self.assertIn(".navy .content main img", css, css_path.as_posix())
self.assertIn(".coal .content main img", css, css_path.as_posix())
self.assertIn(".ayu .content main img", css, css_path.as_posix())
self.assertIn("background-color: #e8e8e8;", css, css_path.as_posix())
self.assertIn(".openmlsys-frontpage img", css, css_path.as_posix())
self.assertIn("background-color: transparent !important;", css, css_path.as_posix())
self.assertIn("padding: 0 !important;", css, css_path.as_posix())
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,55 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from tools.ensure_book_resources import ensure_resource_views
class EnsureBookResourcesTests(unittest.TestCase):
def test_ensure_resource_views_creates_missing_symlinks(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
chapter_dir = root / "en_chapters"
chapter_dir.mkdir()
for directory in ("img", "references", "static"):
(root / directory).mkdir()
(root / "mlsys.bib").write_text("bib", encoding="utf-8")
ensure_resource_views(chapter_dir, root)
for name in ("img", "references", "static", "mlsys.bib"):
path = chapter_dir / name
self.assertTrue(path.is_symlink(), f"{name} should be a symlink")
self.assertEqual(path.resolve(), (root / name).resolve())
def test_ensure_resource_views_keeps_existing_non_symlink_paths(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
chapter_dir = root / "en_chapters"
chapter_dir.mkdir()
for directory in ("img", "references", "static"):
(root / directory).mkdir()
(root / "mlsys.bib").write_text("root bib", encoding="utf-8")
local_bib = chapter_dir / "mlsys.bib"
local_bib.write_text("local bib", encoding="utf-8")
local_static = chapter_dir / "static"
local_static.mkdir()
(local_static / "frontpage.html").write_text("local static", encoding="utf-8")
ensure_resource_views(chapter_dir, root)
self.assertFalse(local_bib.is_symlink())
self.assertEqual(local_bib.read_text(encoding="utf-8"), "local bib")
self.assertFalse(local_static.is_symlink())
self.assertTrue((local_static / "frontpage.html").exists())
self.assertTrue((chapter_dir / "img").is_symlink())
self.assertTrue((chapter_dir / "references").is_symlink())
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,134 @@
from __future__ import annotations
import gzip
import hashlib
import os
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from tools.ensure_mdbook_typst_math import (
ASSET_SHA256,
VERSION,
build_download_url,
ensure_binary,
resolve_asset_name,
resolve_binary_path,
resolve_version_path,
)
class ResolveAssetNameTests(unittest.TestCase):
def test_resolve_asset_name_for_supported_targets(self) -> None:
self.assertEqual(
resolve_asset_name(system="Darwin", machine="arm64"),
"mdbook-typst-math-aarch64-apple-darwin.gz",
)
self.assertEqual(
resolve_asset_name(system="Darwin", machine="x86_64"),
"mdbook-typst-math-x86_64-apple-darwin.gz",
)
self.assertEqual(
resolve_asset_name(system="Linux", machine="aarch64"),
"mdbook-typst-math-aarch64-unknown-linux-gnu.gz",
)
self.assertEqual(
resolve_asset_name(system="Linux", machine="AMD64"),
"mdbook-typst-math-x86_64-unknown-linux-gnu.gz",
)
self.assertEqual(
resolve_asset_name(system="Windows", machine="AMD64"),
"mdbook-typst-math-x86_64-pc-windows-msvc.exe",
)
def test_resolve_asset_name_rejects_unsupported_targets(self) -> None:
with self.assertRaises(ValueError):
resolve_asset_name(system="Linux", machine="riscv64")
class EnsureBinaryTests(unittest.TestCase):
def test_ensure_binary_downloads_and_extracts_gzip_release(self) -> None:
payload = b"linux-binary"
asset_name = "mdbook-typst-math-x86_64-unknown-linux-gnu.gz"
with tempfile.TemporaryDirectory() as tmpdir:
output_dir = Path(tmpdir)
urls: list[str] = []
def fake_downloader(url: str) -> bytes:
urls.append(url)
return gzip.compress(payload)
with patch.dict(ASSET_SHA256, {asset_name: hashlib.sha256(gzip.compress(payload)).hexdigest()}):
binary_path = ensure_binary(
output_dir,
system="Linux",
machine="x86_64",
downloader=fake_downloader,
)
self.assertEqual(binary_path, resolve_binary_path(output_dir, VERSION, asset_name))
self.assertEqual(binary_path.name, "mdbook-typst-math")
self.assertEqual(binary_path.read_bytes(), payload)
self.assertEqual(resolve_version_path(output_dir).read_text(encoding="utf-8"), VERSION)
self.assertEqual(urls, [build_download_url(VERSION, asset_name)])
self.assertTrue(os.access(binary_path, os.X_OK))
def test_ensure_binary_uses_cached_file_without_downloading(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
output_dir = Path(tmpdir)
asset_name = "mdbook-typst-math-x86_64-unknown-linux-gnu.gz"
cached_binary = resolve_binary_path(output_dir, VERSION, asset_name)
output_dir.mkdir(parents=True, exist_ok=True)
cached_binary.write_bytes(b"cached")
cached_binary.chmod(0o755)
resolve_version_path(output_dir).write_text(VERSION, encoding="utf-8")
def fail_downloader(_: str) -> bytes:
raise AssertionError("downloader should not be called for cached binary")
binary_path = ensure_binary(
output_dir,
system="Linux",
machine="x86_64",
downloader=fail_downloader,
)
self.assertEqual(binary_path, cached_binary)
self.assertEqual(binary_path.read_bytes(), b"cached")
def test_ensure_binary_keeps_windows_extension(self) -> None:
payload = b"windows-binary"
asset_name = "mdbook-typst-math-x86_64-pc-windows-msvc.exe"
with tempfile.TemporaryDirectory() as tmpdir:
output_dir = Path(tmpdir)
def fake_downloader(_: str) -> bytes:
return payload
with patch.dict(ASSET_SHA256, {asset_name: hashlib.sha256(payload).hexdigest()}):
binary_path = ensure_binary(
output_dir,
system="Windows",
machine="AMD64",
downloader=fake_downloader,
)
self.assertEqual(binary_path.name, "mdbook-typst-math.exe")
self.assertEqual(binary_path.read_bytes(), payload)
def test_ensure_binary_rejects_checksum_mismatch(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
with self.assertRaises(ValueError):
ensure_binary(
Path(tmpdir),
system="Linux",
machine="x86_64",
downloader=lambda _: gzip.compress(b"bad-binary"),
)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,38 @@
from __future__ import annotations
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
BOOK_PATHS = (
REPO_ROOT / "book.toml",
REPO_ROOT / "books" / "zh" / "book.toml",
)
BUILD_SCRIPTS = (
REPO_ROOT / "build_mdbook.sh",
REPO_ROOT / "build_mdbook_zh.sh",
)
class MdBookTypstMathConfigTests(unittest.TestCase):
def test_books_use_typst_math_without_mathjax(self) -> None:
for path in BOOK_PATHS:
config = path.read_text(encoding="utf-8")
self.assertIn("[preprocessor.typst-math]", config, path.as_posix())
self.assertIn("theme/typst.css", config, path.as_posix())
self.assertNotIn("mathjax-support = true", config, path.as_posix())
def test_build_scripts_bootstrap_prebuilt_typst_math_binary(self) -> None:
for path in BUILD_SCRIPTS:
script = path.read_text(encoding="utf-8")
self.assertIn("ensure_mdbook_typst_math.py", script, path.as_posix())
self.assertIn("MDBOOK_TYPST_MATH_BIN_DIR", script, path.as_posix())
self.assertIn("export PATH=", script, path.as_posix())
self.assertNotIn("cargo install mdbook-typst-math", script, path.as_posix())
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,237 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from tools.prepare_mdbook import build_title_cache, rewrite_markdown, write_summary
REPO_ROOT = Path(__file__).resolve().parents[1]
class PrepareMdBookTests(unittest.TestCase):
def test_write_summary_skips_placeholder_pages(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
source = root / "en_chapters"
source.mkdir()
(source / "index.md").write_text(
"""Machine Learning Systems
========================
```toc
:maxdepth: 2
chapter_preface/index
chapter_introduction/index
```
```toc
:maxdepth: 1
appendix/index
```
""",
encoding="utf-8",
)
chapter_preface = source / "chapter_preface"
chapter_preface.mkdir()
(chapter_preface / "index.md").write_text(
"[TODO: src = zh_chapters/chapter_preface/index.md]\n",
encoding="utf-8",
)
chapter_intro = source / "chapter_introduction"
chapter_intro.mkdir()
(chapter_intro / "index.md").write_text("# Introduction\n", encoding="utf-8")
appendix = source / "appendix"
appendix.mkdir()
(appendix / "index.md").write_text("# Appendix\n", encoding="utf-8")
summary_path = write_summary(
source,
placeholder_prefix="[TODO: src = zh_chapters/",
)
summary = summary_path.read_text(encoding="utf-8")
self.assertEqual(
summary,
"""# Summary
[Machine Learning Systems](index.md)
[Introduction](chapter_introduction/index.md)
[Appendix](appendix/index.md)
""",
)
title_cache = build_title_cache(
source,
placeholder_prefix="[TODO: src = zh_chapters/",
)
rewritten = rewrite_markdown(
(source / "index.md").read_text(encoding="utf-8"),
(source / "index.md").resolve(),
title_cache,
)
self.assertIn("- [Introduction](chapter_introduction/index.md)", rewritten)
self.assertIn("- [Appendix](appendix/index.md)", rewritten)
self.assertNotIn("chapter_preface/index.md", rewritten)
def test_rewrite_markdown_uses_configured_bibliography_title(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
page = root / "chapter.md"
page.write_text(
"""# Introduction
Reference :cite:`smith2024`.
""",
encoding="utf-8",
)
rewritten = rewrite_markdown(
page.read_text(encoding="utf-8"),
page.resolve(),
{page.resolve(): "Introduction"},
bib_db={
"smith2024": {
"author": "Smith, Alice and Doe, Bob",
"title": "Systems Paper",
"year": "2024",
"journal": "ML Systems Journal",
}
},
bibliography_title="References",
)
self.assertIn("## References", rewritten)
self.assertNotIn("## 参考文献", rewritten)
def test_rewrite_markdown_inlines_frontpage_with_language_switch(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
source = root / "en_chapters"
static_dir = source / "static"
static_dir.mkdir(parents=True)
index = source / "index.md"
index.write_text(
"""# Home
```eval_rst
.. raw:: html
:file: frontpage.html
```
""",
encoding="utf-8",
)
(static_dir / "frontpage.html").write_text(
"<p class=\"star-slot\">STAR</p>\n<!-- OPENMLSYS_LANGUAGE_SWITCH -->\n<div class=\"hero\">frontpage</div>\n",
encoding="utf-8",
)
rewritten = rewrite_markdown(
index.read_text(encoding="utf-8"),
index.resolve(),
{index.resolve(): "Home"},
frontpage_switch_label="中文",
frontpage_switch_href="cn/",
)
self.assertIn('class="openmlsys-frontpage-switch-row"', rewritten)
self.assertIn('class="openmlsys-frontpage-switch"', rewritten)
self.assertIn('href="cn/"', rewritten)
self.assertIn(">中文</a>", rewritten)
self.assertLess(
rewritten.index('class="star-slot"'),
rewritten.index('class="openmlsys-frontpage-switch-row"'),
)
def test_rewrite_markdown_prefers_book_local_frontpage_file(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
source = root / "en_chapters"
static_dir = source / "static"
source.mkdir()
static_dir.mkdir()
index = source / "index.md"
index.write_text(
"""# Home
```eval_rst
.. raw:: html
:file: frontpage.html
```
""",
encoding="utf-8",
)
(source / "frontpage.html").write_text(
"<div class=\"hero\">English frontpage</div>\n",
encoding="utf-8",
)
(static_dir / "frontpage.html").write_text(
"<div class=\"hero\">Chinese fallback</div>\n",
encoding="utf-8",
)
rewritten = rewrite_markdown(
index.read_text(encoding="utf-8"),
index.resolve(),
{index.resolve(): "Home"},
)
self.assertIn("English frontpage", rewritten)
self.assertNotIn("Chinese fallback", rewritten)
self.assertIn("background: transparent !important;", rewritten)
self.assertIn("padding: 0 !important;", rewritten)
self.assertIn("border-radius: 6px;", rewritten)
self.assertIn("background: #f6f8fa;", rewritten)
self.assertIn(".content main {", rewritten)
self.assertIn("max-width: min(100%, max(65%, var(--content-max-width)));", rewritten)
self.assertIn(".openmlsys-frontpage {", rewritten)
self.assertIn("width: 100%;", rewritten)
self.assertIn("margin-inline: auto;", rewritten)
def test_regular_page_does_not_render_frontpage_switch(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
page = root / "chapter.md"
page.write_text("# Chapter\n\nRegular body.\n", encoding="utf-8")
rewritten = rewrite_markdown(
page.read_text(encoding="utf-8"),
page.resolve(),
{page.resolve(): "Chapter"},
frontpage_switch_label="中文",
frontpage_switch_href="cn/",
)
self.assertNotIn('openmlsys-frontpage-switch', rewritten)
def test_english_frontpage_author_grid_uses_top_aligned_spacing(self) -> None:
frontpage = (REPO_ROOT / "en_chapters" / "frontpage.html").read_text(encoding="utf-8")
self.assertIn(".authors.mdl-grid {", frontpage)
self.assertIn("align-items: flex-start;", frontpage)
self.assertIn("row-gap: calc(24px + 1.5em);", frontpage)
self.assertIn("height: 120px;", frontpage)
self.assertIn("object-fit: cover;", frontpage)
def test_english_frontpage_footer_titles_are_stacked_not_flex_columns(self) -> None:
frontpage = (REPO_ROOT / "en_chapters" / "frontpage.html").read_text(encoding="utf-8")
self.assertIn(".authors .mdl-cell:not(.author-group-title) {", frontpage)
self.assertIn(".author-group-title {", frontpage)
self.assertIn("display: block;", frontpage)
self.assertIn(".author-group-title h3,", frontpage)
self.assertIn("width: 100%;", frontpage)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,266 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from tools.prepare_mdbook_zh import extract_title, process_equation_labels, rewrite_markdown, write_summary
class PrepareMdBookZhTests(unittest.TestCase):
def test_extract_title_supports_atx_and_setext_headings(self) -> None:
self.assertEqual(extract_title("# 导论\n"), "导论")
self.assertEqual(extract_title("前言文字\n\n## 机器学习应用\n"), "机器学习应用")
self.assertEqual(extract_title("机器学习系统:设计和实现\n=========================\n"), "机器学习系统:设计和实现")
def test_write_summary_generates_nested_navigation(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
source = root / "zh_chapters"
source.mkdir()
(source / "index.md").write_text(
"""机器学习系统:设计和实现
=========================
```eval_rst
.. raw:: html
:file: frontpage.html
```
```toc
:maxdepth: 2
[前言](chapter_preface/index)
# 基础篇
chapter_introduction/index
# 附录
[机器学习基础附录](appendix_machine_learning_introduction/index)
```
""",
encoding="utf-8",
)
chapter_preface = source / "chapter_preface"
chapter_preface.mkdir()
(chapter_preface / "index.md").write_text("# 前言\n", encoding="utf-8")
static_dir = source / "static"
static_dir.mkdir()
(static_dir / "frontpage.html").write_text(
"<div class=\"hero\">frontpage</div>\n",
encoding="utf-8",
)
chapter_intro = source / "chapter_introduction"
chapter_intro.mkdir()
(chapter_intro / "index.md").write_text(
"""# 导论
```toc
:maxdepth: 2
applications
design
```
""",
encoding="utf-8",
)
(chapter_intro / "applications.md").write_text("# 机器学习应用\n", encoding="utf-8")
(chapter_intro / "design.md").write_text("# 设计目标\n", encoding="utf-8")
appendix = source / "appendix_machine_learning_introduction"
appendix.mkdir()
(appendix / "index.md").write_text("# 机器学习基础附录\n", encoding="utf-8")
for name in ("img", "static", "references"):
(root / name).mkdir()
(root / "mlsys.bib").write_text("% bibliography\n", encoding="utf-8")
summary_path = write_summary(source)
summary = summary_path.read_text(encoding="utf-8")
self.assertEqual(
summary,
"""# Summary
[机器学习系统:设计和实现](index.md)
[前言](chapter_preface/index.md)
# 基础篇
- [导论](chapter_introduction/index.md)
- [机器学习应用](chapter_introduction/applications.md)
- [设计目标](chapter_introduction/design.md)
# 附录
- [机器学习基础附录](appendix_machine_learning_introduction/index.md)
""",
)
title_cache = {
(source / "chapter_preface" / "index.md").resolve(): "前言",
(source / "chapter_introduction" / "index.md").resolve(): "导论",
(source / "chapter_introduction" / "applications.md").resolve(): "机器学习应用",
(source / "chapter_introduction" / "design.md").resolve(): "设计目标",
(source / "appendix_machine_learning_introduction" / "index.md").resolve(): "机器学习基础附录",
}
root_index = rewrite_markdown((source / "index.md").read_text(encoding="utf-8"), (source / "index.md").resolve(), title_cache)
self.assertNotIn("```eval_rst", root_index)
self.assertNotIn("```toc", root_index)
self.assertIn("- [前言](chapter_preface/index.md)", root_index)
self.assertIn("- 基础篇", root_index)
self.assertIn(" - [导论](chapter_introduction/index.md)", root_index)
self.assertIn("- 附录", root_index)
self.assertIn(" - [机器学习基础附录](appendix_machine_learning_introduction/index.md)", root_index)
intro_index = rewrite_markdown(
(source / "chapter_introduction" / "index.md").read_text(encoding="utf-8"),
(source / "chapter_introduction" / "index.md").resolve(),
title_cache,
)
self.assertNotIn("```toc", intro_index)
self.assertIn("- [机器学习应用](applications.md)", intro_index)
self.assertIn("- [设计目标](design.md)", intro_index)
def test_write_summary_raises_for_missing_toc_entries(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
source = root / "zh_chapters"
source.mkdir()
(source / "index.md").write_text(
"""# 首页
```toc
:maxdepth: 2
existing
missing
```
""",
encoding="utf-8",
)
(source / "existing.md").write_text("# 现有章节\n", encoding="utf-8")
with self.assertRaises(FileNotFoundError):
write_summary(source)
def test_rewrite_markdown_normalizes_common_d2l_directives(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
source = root / "zh_chapters"
source.mkdir()
page = source / "chapter.md"
page.write_text(
"""# 标题
![配图](../img/example.png)
:width:`800px`
:label:`fig_example`
参见 :numref:`fig_example` 和公式 :eqref:`eq_example`,引用 :cite:`foo2024`。
""",
encoding="utf-8",
)
rewritten = rewrite_markdown(page.read_text(encoding="utf-8"), page.resolve(), {page.resolve(): "标题"})
self.assertNotIn(":width:", rewritten)
self.assertNotIn(":label:", rewritten)
self.assertNotIn(":numref:", rewritten)
self.assertNotIn(":eqref:", rewritten)
self.assertNotIn(":cite:", rewritten)
self.assertIn("`fig_example`", rewritten)
self.assertIn("$\\eqref{eq_example}$", rewritten)
self.assertIn("[foo2024]", rewritten)
def test_rewrite_markdown_inlines_frontpage_html_include(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
source = root / "zh_chapters"
static_dir = source / "static"
static_dir.mkdir(parents=True)
index = source / "index.md"
index.write_text(
"""# 首页
```eval_rst
.. raw:: html
:file: frontpage.html
```
""",
encoding="utf-8",
)
(static_dir / "frontpage.html").write_text(
"""<head>
<style>
.hero { color: red; }
.other { color: blue; }
</style>
</head>
<p class="star-slot">STAR</p>
<!-- OPENMLSYS_LANGUAGE_SWITCH -->
<div class="hero">
<img src="_images/logo.png" />
<img src="./_images/jinxuefeng.png" />
</div>
<script>console.log('frontpage');</script>
""",
encoding="utf-8",
)
rewritten = rewrite_markdown(index.read_text(encoding="utf-8"), index.resolve(), {index.resolve(): "首页"})
self.assertNotIn("```eval_rst", rewritten)
self.assertNotIn("<head>", rewritten)
self.assertIn('class="openmlsys-frontpage"', rewritten)
self.assertIn('<div class="hero">', rewritten)
self.assertIn('<style>', rewritten)
self.assertIn(".hero { color: red; } .other { color: blue; }", rewritten)
self.assertIn("static/image/logo.png", rewritten)
self.assertIn("static/image/jinxuefeng.png", rewritten)
self.assertIn("console.log('frontpage')", rewritten)
self.assertIn('class="openmlsys-frontpage-switch-row"', rewritten)
self.assertIn('class="openmlsys-frontpage-switch"', rewritten)
self.assertIn('href="../"', rewritten)
self.assertIn(">English</a>", rewritten)
self.assertLess(
rewritten.index('class="star-slot"'),
rewritten.index('class="openmlsys-frontpage-switch-row"'),
)
def test_process_equation_labels_single_line(self) -> None:
"""Verify single-line equation gets \\tag and \\label injected."""
md = "# Title\n\n$$a = f(z)$$\n:eqlabel:`sigmoid`\n\nSee :eqref:`sigmoid`.\n"
result, label_map = process_equation_labels(md)
self.assertIn("\\tag{1}\\label{sigmoid}$$", result)
self.assertNotIn(":eqlabel:", result)
self.assertEqual(label_map, {"sigmoid": 1})
def test_process_equation_labels_multiline(self) -> None:
"""Verify multi-line equation (closing $$ on own line) gets \\tag and \\label."""
md = "# Title\n\n$$\na = f(z)\n$$\n:eqlabel:`eq1`\n"
result, label_map = process_equation_labels(md)
lines = result.split("\n")
# \\tag line should appear before the closing $$
tag_idx = next(i for i, l in enumerate(lines) if "\\tag{1}\\label{eq1}" in l)
close_idx = next(i for i, l in enumerate(lines) if l.strip() == "$$" and i > tag_idx)
self.assertLess(tag_idx, close_idx)
self.assertNotIn(":eqlabel:", result)
self.assertEqual(label_map, {"eq1": 1})
def test_process_equation_labels_sequential_numbering(self) -> None:
"""Verify multiple equations get sequential numbers."""
md = "$$a$$\n:eqlabel:`eq1`\n\n$$b$$\n:eqlabel:`eq2`\n"
result, label_map = process_equation_labels(md)
self.assertIn("\\tag{1}\\label{eq1}", result)
self.assertIn("\\tag{2}\\label{eq2}", result)
self.assertEqual(label_map, {"eq1": 1, "eq2": 2})
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,29 @@
from __future__ import annotations
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "update_docs.yml"
CI_WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "main.yml"
class UpdateDocsWorkflowTests(unittest.TestCase):
def test_workflow_deploys_to_openmlsys_github_io(self) -> None:
workflow = WORKFLOW_PATH.read_text(encoding="utf-8")
self.assertIn("DEPLOY_TOKEN", workflow)
self.assertIn("openmlsys.github.io", workflow)
self.assertIn("git push", workflow)
def test_workflows_use_peaceiris_mdbook_action(self) -> None:
for workflow_path in (WORKFLOW_PATH, CI_WORKFLOW_PATH):
workflow = workflow_path.read_text(encoding="utf-8")
self.assertIn("uses: peaceiris/actions-mdbook@v2", workflow, workflow_path.as_posix())
self.assertIn("mdbook-version: 'latest'", workflow, workflow_path.as_posix())
self.assertNotIn("cargo install mdbook --locked", workflow, workflow_path.as_posix())
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,16 @@
/* 暗色模式下仅为正文图片添加浅灰色背景,提高透明背景图片的可读性 */
.navy .content main img,
.coal .content main img,
.ayu .content main img {
background-color: #e8e8e8;
border-radius: 4px;
padding: 8px;
}
/* 首页 frontpage 图片保持透明,不添加正文图像底色。 */
.navy .openmlsys-frontpage img,
.coal .openmlsys-frontpage img,
.ayu .openmlsys-frontpage img {
background-color: transparent !important;
padding: 0 !important;
}

16
theme/typst.css Normal file
View File

@@ -0,0 +1,16 @@
.typst-inline {
display: inline-flex;
vertical-align: -0.2em;
}
.typst-display {
display: flex;
justify-content: center;
margin: 1rem 0;
overflow-x: auto;
}
.typst-doc {
color: var(--fg);
max-width: 100%;
}

View File

@@ -0,0 +1,97 @@
from __future__ import annotations
import argparse
import shutil
from pathlib import Path
def remove_path(path: Path) -> None:
if path.is_symlink() or path.is_file():
path.unlink()
return
if path.is_dir():
shutil.rmtree(path)
def copy_site(source: Path, destination: Path) -> None:
source = source.resolve()
if not source.is_dir():
raise FileNotFoundError(f"Site source does not exist or is not a directory: {source}")
remove_path(destination)
destination.parent.mkdir(parents=True, exist_ok=True)
shutil.copytree(source, destination)
def assemble_publish_tree(
destination_root: Path,
docs_subdir: str = "docs",
en_source: Path | None = None,
zh_source: Path | None = None,
) -> tuple[Path, Path | None]:
if en_source is None and zh_source is None:
raise ValueError("At least one site source must be provided.")
destination_root = destination_root.resolve()
docs_root = (destination_root / docs_subdir).resolve()
remove_path(docs_root)
docs_root.parent.mkdir(parents=True, exist_ok=True)
if en_source is not None:
copy_site(en_source, docs_root)
else:
docs_root.mkdir(parents=True, exist_ok=True)
zh_destination: Path | None = None
if zh_source is not None:
zh_destination = docs_root / "cn"
copy_site(zh_source, zh_destination)
return docs_root, zh_destination
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Assemble the publish tree expected by openmlsys.github.io."
)
parser.add_argument(
"--destination-root",
type=Path,
required=True,
help="Root of the checked-out deployment repository.",
)
parser.add_argument(
"--docs-subdir",
default="docs",
help="Subdirectory inside the destination root that hosts the site.",
)
parser.add_argument(
"--en-source",
type=Path,
help="Built site to publish at docs/.",
)
parser.add_argument(
"--zh-source",
type=Path,
help="Built site to publish at docs/cn/.",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
docs_root, zh_root = assemble_publish_tree(
destination_root=args.destination_root,
docs_subdir=args.docs_subdir,
en_source=args.en_source,
zh_source=args.zh_source,
)
print(f"Assembled root site at {docs_root}")
if zh_root is not None:
print(f"Assembled Chinese site at {zh_root}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,60 @@
from __future__ import annotations
import argparse
import os
from pathlib import Path
RESOURCE_NAMES = ("img", "references", "static", "mlsys.bib")
def ensure_resource_views(
chapter_dir: Path,
repo_root: Path,
resource_names: tuple[str, ...] = RESOURCE_NAMES,
) -> None:
chapter_dir = chapter_dir.resolve()
repo_root = repo_root.resolve()
for name in resource_names:
destination = chapter_dir / name
source = repo_root / name
if not source.exists():
raise FileNotFoundError(f"Resource does not exist: {source}")
if destination.is_symlink():
destination.unlink()
elif destination.exists():
continue
relative_source = os.path.relpath(source, start=chapter_dir)
destination.symlink_to(relative_source)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Ensure chapter directories can see shared book resources."
)
parser.add_argument(
"--chapter-dir",
type=Path,
required=True,
help="Book source directory such as en_chapters or zh_chapters.",
)
parser.add_argument(
"--repo-root",
type=Path,
default=Path(__file__).resolve().parent.parent,
help="Repository root that owns the shared resources.",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
ensure_resource_views(args.chapter_dir, args.repo_root)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,132 @@
from __future__ import annotations
import argparse
import gzip
import hashlib
import os
import platform
import urllib.request
from pathlib import Path
VERSION = "v0.3.0"
RELEASE_BASE_URL = "https://github.com/duskmoon314/mdbook-typst-math/releases/download"
ASSET_SHA256 = {
"mdbook-typst-math-aarch64-apple-darwin.gz": "9c7a94113e16a465edd1324010e2cc432be3c0794320c13d6a44d9523f069384",
"mdbook-typst-math-aarch64-unknown-linux-gnu.gz": "bbcf4574e380663400af74dda76dd6ecafd36aff185d653a2e24e294c45321c3",
"mdbook-typst-math-x86_64-apple-darwin.gz": "8bb36eb558fc438c55162b442975eca588a7654b8069860526e46cc08c2aee6a",
"mdbook-typst-math-x86_64-pc-windows-msvc.exe": "b5d3e07108a7286007d153c66efe434d06ab6caf43fcd22f78b4e6af8a294314",
"mdbook-typst-math-x86_64-unknown-linux-gnu.gz": "3b785a42fb3a93bcd3f80106e6ded5c55bb0bcd4cd0634edf8232d14444b6987",
}
SUPPORTED_ASSETS = {
("darwin", "aarch64"): "mdbook-typst-math-aarch64-apple-darwin.gz",
("darwin", "x86_64"): "mdbook-typst-math-x86_64-apple-darwin.gz",
("linux", "aarch64"): "mdbook-typst-math-aarch64-unknown-linux-gnu.gz",
("linux", "x86_64"): "mdbook-typst-math-x86_64-unknown-linux-gnu.gz",
("windows", "x86_64"): "mdbook-typst-math-x86_64-pc-windows-msvc.exe",
}
def normalize_machine(machine: str) -> str:
normalized = machine.strip().lower()
if normalized in {"arm64", "aarch64"}:
return "aarch64"
if normalized in {"amd64", "x86_64", "x64"}:
return "x86_64"
return normalized
def normalize_system(system: str) -> str:
normalized = system.strip().lower()
if normalized.startswith("mingw") or normalized.startswith("msys") or normalized.startswith("cygwin"):
return "windows"
return normalized
def resolve_asset_name(system: str | None = None, machine: str | None = None) -> str:
resolved_system = normalize_system(system or platform.system())
resolved_machine = normalize_machine(machine or platform.machine())
asset_name = SUPPORTED_ASSETS.get((resolved_system, resolved_machine))
if asset_name is None:
raise ValueError(
f"Unsupported platform for mdbook-typst-math: system={resolved_system!r}, machine={resolved_machine!r}"
)
return asset_name
def build_download_url(version: str, asset_name: str) -> str:
return f"{RELEASE_BASE_URL}/{version}/{asset_name}"
def resolve_binary_path(output_dir: Path, version: str, asset_name: str) -> Path:
binary_name = "mdbook-typst-math.exe" if asset_name.endswith(".exe") else "mdbook-typst-math"
return output_dir / binary_name
def resolve_version_path(output_dir: Path) -> Path:
return output_dir / ".mdbook-typst-math.version"
def download_bytes(url: str) -> bytes:
request = urllib.request.Request(url, headers={"User-Agent": "openmlsys-mdbook-bootstrap/1.0"})
with urllib.request.urlopen(request) as response:
return response.read()
def ensure_binary(
output_dir: Path,
*,
version: str = VERSION,
system: str | None = None,
machine: str | None = None,
downloader=download_bytes,
) -> Path:
asset_name = resolve_asset_name(system=system, machine=machine)
binary_path = resolve_binary_path(output_dir, version, asset_name)
version_path = resolve_version_path(output_dir)
if binary_path.exists() and version_path.exists() and version_path.read_text(encoding="utf-8").strip() == version:
if binary_path.suffix != ".exe":
binary_path.chmod(binary_path.stat().st_mode | 0o111)
return binary_path
expected_sha256 = ASSET_SHA256[asset_name]
download_url = build_download_url(version, asset_name)
archive_bytes = downloader(download_url)
digest = hashlib.sha256(archive_bytes).hexdigest()
if digest != expected_sha256:
raise ValueError(
f"Checksum mismatch for {asset_name}: expected {expected_sha256}, got {digest}"
)
binary_path.parent.mkdir(parents=True, exist_ok=True)
payload = gzip.decompress(archive_bytes) if asset_name.endswith(".gz") else archive_bytes
output_dir.mkdir(parents=True, exist_ok=True)
temporary_path = binary_path.with_suffix(f"{binary_path.suffix}.tmp")
temporary_path.write_bytes(payload)
if binary_path.suffix != ".exe":
temporary_path.chmod(0o755)
os.replace(temporary_path, binary_path)
version_path.write_text(version, encoding="utf-8")
return binary_path
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Download a pinned mdbook-typst-math release binary.")
parser.add_argument(
"--output-dir",
type=Path,
default=Path(".mdbook-bin"),
help="Directory used to cache the downloaded mdbook-typst-math binary.",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
binary_path = ensure_binary(args.output_dir.resolve())
print(binary_path)
return 0
if __name__ == "__main__":
raise SystemExit(main())

614
tools/latex_to_typst.py Normal file
View File

@@ -0,0 +1,614 @@
"""Convert LaTeX math notation to Typst math notation within markdown content.
This module provides a best-effort converter for the LaTeX math subset used in
the OpenMLSys textbook. It is **not** a general-purpose LaTeX→Typst transpiler;
only the commands that actually appear in the zh_chapters are handled.
"""
from __future__ import annotations
import re
# ---------------------------------------------------------------------------
# Brace-matching helper
# ---------------------------------------------------------------------------
def _find_brace_group(s: str, pos: int) -> tuple[str, int] | None:
"""Return ``(content, end_pos)`` for the ``{…}`` group starting at *pos*.
Skips leading whitespace. Returns ``None`` when no opening brace is found
or braces are unbalanced.
"""
while pos < len(s) and s[pos] in " \t":
pos += 1
if pos >= len(s) or s[pos] != "{":
return None
depth = 0
start = pos + 1
for i in range(pos, len(s)):
if s[i] == "{":
depth += 1
elif s[i] == "}":
depth -= 1
if depth == 0:
return (s[start:i], i + 1)
return None
# ---------------------------------------------------------------------------
# Command tables
# ---------------------------------------------------------------------------
# Commands whose Typst name equals the LaTeX name (just drop the backslash).
SIMPLE_COMMANDS: set[str] = {
# Greek
"alpha", "beta", "gamma", "delta", "Delta",
"epsilon", "zeta", "eta", "theta", "Theta",
"iota", "kappa", "lambda", "Lambda",
"mu", "nu", "xi", "Xi",
"pi", "Pi", "rho",
"sigma", "Sigma", "tau",
"upsilon", "Upsilon",
"phi", "Phi", "chi", "psi", "Psi",
"omega", "Omega",
# Operators / relations
"times", "partial", "nabla", "in",
"top", "prime",
"forall", "exists", "approx", "equiv",
"subset", "supset",
# Big operators / functions
"log", "ln", "exp", "sin", "cos", "tan",
"min", "max", "lim", "sum",
"det", "dim", "ker", "inf", "sup",
}
# Commands that map to a *different* Typst identifier.
RENAMED_COMMANDS: dict[str, str] = {
"cdot": "dot.c",
"cdots": "dots.c",
"ldots": "dots",
"dots": "dots",
"to": "->",
"rightarrow": "->",
"leftarrow": "<-",
"Rightarrow": "=>",
"rightsquigarrow": "arrow.r.squiggly",
"leq": "<=",
"geq": ">=",
"prod": "product",
"notag": "",
"quad": "quad",
"qquad": "wide",
"label": "", # consumed by :eqlabel: already
"sim": "tilde.op",
"infty": "infinity",
"neq": "eq.not",
"ast": "ast.op",
"vdots": "dots.v",
"ddots": "dots.down",
"lVert": "||",
"rVert": "||",
"vert": "|",
"lvert": "|",
"rvert": "|",
"mid": "|",
"cap": "inter",
"cup": "union",
"le": "<=",
"ge": ">=",
"odot": "dot.o",
"oplus": "plus.circle",
"otimes": "times.circle",
}
# \cmd{arg} → typst_func(arg)
ONE_ARG_COMMANDS: dict[str, str] = {
"boldsymbol": "bold",
"mathcal": "cal",
"mathbf": "bold",
"mathbb": "bb",
"hat": "hat",
"bar": "overline",
"dot": "dot",
"tilde": "tilde",
"sqrt": "sqrt",
"overline": "overline",
"pmb": "bold",
"textbf": "bold",
"textit": "italic",
"bm": "bold",
}
# \cmd{arg1}{arg2} → typst_func(arg1, arg2)
TWO_ARG_COMMANDS: dict[str, str] = {
"frac": "frac",
"binom": "binom",
}
# Delimiter-sizing commands to strip (the delimiter char after them is kept).
_SIZING_COMMANDS: set[str] = {
"left", "right", "bigg", "Bigg", "big", "Big", "biggl", "biggr",
}
# ---------------------------------------------------------------------------
# Core single-pass converter
# ---------------------------------------------------------------------------
def _last_char(out: list[str]) -> str:
"""Return the last non-empty character in the output buffer, or ``""``."""
for part in reversed(out):
if part:
return part[-1]
return ""
def _emit(out: list[str], text: str) -> None:
"""Append *text* to *out*, adding a space separator when needed.
In Typst math, consecutive letters form a multi-letter identifier which
will error if unknown. Similarly, letter→digit transitions form tokens
like ``W1``. This helper inserts spaces to prevent such merging, matching
LaTeX math semantics where adjacent characters are separate symbols.
"""
if not text:
out.append(text)
return
lc = _last_char(out)
fc = text[0]
if lc and (
# letter→letter (e.g. "ou" → "o u")
(lc.isalpha() and fc.isalpha())
# letter→digit (e.g. "W1" → "W 1")
or (lc.isalpha() and fc.isdigit())
# digit→letter (e.g. "2x" → "2 x")
or (lc.isdigit() and fc.isalpha())
# )→letter/digit (e.g. "bold(X)y" → "bold(X) y")
or (lc == ")" and (fc.isalpha() or fc.isdigit()))
):
out.append(" ")
out.append(text)
def _convert(s: str) -> str:
"""Convert a single LaTeX math expression to Typst math."""
out: list[str] = []
i = 0
n = len(s)
while i < n:
ch = s[i]
# ---- backslash commands ----
if ch == "\\" and i + 1 < n:
nxt = s[i + 1]
# Double backslash: either markdown-escaped bracket or line-break
if nxt == "\\":
# \\{ \\} \\[ \\] \\( \\) → markdown-escaped LaTeX delimiters
if i + 2 < n and s[i + 2] in "{}[]()":
out.append(s[i + 2])
i += 3
continue
out.append(" \\\n")
i += 2
continue
# Escaped characters: \{ \} \[ \] \( \) \, \; \! \ \.
if nxt in "{}[]()":
out.append(nxt)
i += 2
continue
if nxt == ",":
out.append("thin ") # thin space
i += 2
continue
if nxt == ";":
out.append("med ") # medium space
i += 2
continue
if nxt == "!":
out.append("") # negative thin space → ignore
i += 2
continue
if nxt == " ":
out.append(" ")
i += 2
continue
if nxt == "\n":
out.append(" ")
i += 2
continue
# Try to match an alphabetic command name
m = re.match(r"[a-zA-Z]+", s[i + 1:])
if not m:
# Bare backslash before non-alpha → keep the char after
out.append(nxt)
i += 2
continue
cmd = m.group()
after = i + 1 + m.end()
# -- environments --
if cmd == "begin":
g = _find_brace_group(s, after)
if g:
env_name, env_pos = g
end_marker = f"\\end{{{env_name}}}"
end_idx = s.find(end_marker, env_pos)
if end_idx != -1:
body = s[env_pos:end_idx]
i = end_idx + len(end_marker)
_emit(out, _convert_environment(env_name, body))
continue
# Fallthrough: couldn't parse, skip \begin
i = after
continue
if cmd == "end":
# Stray \end (shouldn't happen if \begin matched)
g = _find_brace_group(s, after)
i = g[1] if g else after
continue
# -- special multi-arg commands --
if cmd == "underset":
# \underset{below}{base} → attach(base, b: below)
g1 = _find_brace_group(s, after)
if g1:
below, p1 = g1
g2 = _find_brace_group(s, p1)
if g2:
base, p2 = g2
_emit(out, f"attach({_convert(base)}, b: {_convert(below)})")
i = p2
continue
if cmd == "overset":
# \overset{above}{base} → attach(base, t: above)
g1 = _find_brace_group(s, after)
if g1:
above, p1 = g1
g2 = _find_brace_group(s, p1)
if g2:
base, p2 = g2
_emit(out, f"attach({_convert(base)}, t: {_convert(above)})")
i = p2
continue
if cmd == "operatorname":
# \operatorname{name} → op("name")
g = _find_brace_group(s, after)
if g:
name, pos = g
_emit(out, f'op("{name}")')
i = pos
continue
if cmd == "tag":
# \tag{n} → visual equation number
g = _find_brace_group(s, after)
if g:
content, pos = g
_emit(out, f'quad upright("({content})")')
i = pos
continue
if cmd == "eqref":
# \eqref{name} → show label name as fallback
g = _find_brace_group(s, after)
if g:
content, pos = g
_emit(out, f'upright("({content})")')
i = pos
continue
if cmd in ("mathrm", "text"):
# \mathrm{text} → upright("text") — treat as text, not math
g = _find_brace_group(s, after)
if g:
content, pos = g
stripped = content.strip()
if stripped:
_emit(out, f'upright("{stripped}")')
# else: empty mathrm (spacing hack) → drop
i = pos
continue
# -- two-arg commands --
if cmd in TWO_ARG_COMMANDS:
g1 = _find_brace_group(s, after)
if g1:
c1, p1 = g1
g2 = _find_brace_group(s, p1)
if g2:
c2, p2 = g2
func = TWO_ARG_COMMANDS[cmd]
_emit(out, f"{func}({_convert(c1)}, {_convert(c2)})")
i = p2
continue
# Fallthrough
_emit(out, cmd)
i = after
continue
# -- one-arg commands --
if cmd in ONE_ARG_COMMANDS:
g = _find_brace_group(s, after)
if g:
content, pos = g
func = ONE_ARG_COMMANDS[cmd]
_emit(out, f"{func}({_convert(content)})")
i = pos
continue
# Fallthrough: no brace group → just emit the typst name
_emit(out, ONE_ARG_COMMANDS[cmd])
i = after
continue
# -- \rm (applies upright to the rest of the current scope) --
if cmd == "rm":
raw_rest = s[after:]
leading = len(raw_rest) - len(raw_rest.lstrip())
rest = raw_rest.lstrip()
# Grab one "word"
wm = re.match(r"[A-Za-z0-9]+", rest)
if wm:
word = wm.group()
_emit(out, f"upright({word})")
i = after + leading + len(word)
continue
_emit(out, "upright")
i = after
continue
# -- delimiter sizing --
if cmd in _SIZING_COMMANDS:
# Skip the command; keep whatever delimiter follows.
i = after
continue
# -- simple (same name) --
if cmd in SIMPLE_COMMANDS:
_emit(out, cmd)
# Also add right-side space when next char would merge
if after < n and (s[after].isalnum() or s[after] == "\\"):
out.append(" ")
i = after
continue
# -- renamed --
if cmd in RENAMED_COMMANDS:
repl = RENAMED_COMMANDS[cmd]
if repl:
_emit(out, repl)
if after < n and s[after].isalnum():
out.append(" ")
# If repl is empty the command is silently dropped.
# For \label{...} consume the brace group too.
if cmd == "label":
g = _find_brace_group(s, after)
if g:
i = g[1]
continue
i = after
continue
# -- unknown command → emit name without backslash --
_emit(out, cmd)
if after < n and s[after].isalnum():
out.append(" ")
i = after
continue
# ---- brace groups (not consumed by a command) ----
if ch == "{":
g = _find_brace_group(s, i)
if g:
content, end = g
# Check if preceded by ^ or _ → superscript/subscript grouping
if out and out[-1] and out[-1][-1] in "^_":
out.append(f"({_convert(content)})")
i = end
continue
# Check for {\rm ...} pattern
rm_m = re.match(r"\\rm\s+", content)
if rm_m:
inner = content[rm_m.end():]
_emit(out, f"upright({_convert(inner)})")
i = end
continue
# Otherwise, just emit the converted content (braces act as
# invisible grouping in LaTeX — no Typst equivalent needed
# in most contexts).
_emit(out, _convert(content))
i = end
continue
# Unmatched brace — emit as-is
out.append(ch)
i += 1
continue
# ---- everything else (digits, letters, operators, whitespace) ----
# Use _emit so consecutive raw letters get spaces inserted,
# matching LaTeX math semantics where adjacent letters are
# separate variables (e.g. "out" → "o u t" in Typst).
_emit(out, ch)
i += 1
result = "".join(out)
# Typst math requires a base before ^ or _; add an invisible base
# when the expression starts with a script marker (e.g. $^2$).
if result and result.lstrip() and result.lstrip()[0] in "^_":
result = '""' + result.lstrip()
return result
# ---------------------------------------------------------------------------
# Environment converters
# ---------------------------------------------------------------------------
def _convert_environment(name: str, body: str) -> str:
"""Convert a ``\\begin{name}\\end{name}`` block to Typst."""
if name in ("matrix", "bmatrix", "pmatrix", "vmatrix"):
return _convert_matrix_env(name, body)
if name == "cases":
return _convert_cases_env(body)
if name in ("aligned", "split"):
# Just unwrap — Typst math handles & alignment and \ line-breaks.
converted = _convert(body)
return converted
if name == "figure":
# Not real math; pass through as-is.
return f"\\begin{{{name}}}{body}\\end{{{name}}}"
# Unknown environment — pass through converted content
return _convert(body)
def _convert_matrix_env(name: str, body: str) -> str:
"""Convert matrix/bmatrix/pmatrix/vmatrix to ``mat(…)``."""
delim_map = {
"matrix": "",
"bmatrix": '"["',
"pmatrix": '"("',
"vmatrix": '"|"',
}
# Split rows on \\, columns on &
rows: list[str] = []
for row_text in re.split(r"\\\\", body):
row_text = row_text.strip()
if not row_text:
continue
cells = [_convert(c.strip()) for c in row_text.split("&")]
rows.append(", ".join(cells))
inner = "; ".join(rows)
delim = delim_map.get(name, "")
if delim:
return f"mat(delim: {delim}, {inner})"
return f"mat({inner})"
def _convert_cases_env(body: str) -> str:
"""Convert cases environment to ``cases(…)``."""
branches: list[str] = []
for branch_text in re.split(r"\\\\", body):
branch_text = branch_text.strip()
if not branch_text:
continue
branches.append(_convert(branch_text))
return "cases(" + ", ".join(branches) + ")"
# ---------------------------------------------------------------------------
# Markdown-level math-span detection
# ---------------------------------------------------------------------------
_FENCE_RE = re.compile(r"^(`{3,}|~{3,})", re.MULTILINE)
def _iter_math_spans(content: str):
"""Yield ``(start, end, is_display)`` for every math span.
Skips spans inside fenced code blocks and inline code.
"""
n = len(content)
i = 0
in_fence: str | None = None # fence marker when inside a code block
while i < n:
# Track fenced code blocks
if content[i] == "`" or content[i] == "~":
m = _FENCE_RE.match(content, i)
if m and (i == 0 or content[i - 1] == "\n"):
marker = m.group(1)
if in_fence is None:
in_fence = marker[0] # opening
i = content.index("\n", i) + 1 if "\n" in content[i:] else n
continue
elif marker[0] == in_fence:
in_fence = None # closing
i = m.end()
continue
if in_fence:
i += 1
continue
# Skip inline code
if content[i] == "`":
end_tick = content.find("`", i + 1)
if end_tick != -1:
i = end_tick + 1
continue
# Display math $$...$$
if content[i:i + 2] == "$$":
start = i
close = content.find("$$", i + 2)
if close != -1:
yield (start + 2, close, True)
i = close + 2
continue
# Inline math $...$
if content[i] == "$":
start = i
# Find closing $ — any next $ closes the span (even if followed
# by another $, which starts a NEW span).
j = i + 1
while j < n:
if content[j] == "$":
if j > i + 1: # non-empty
yield (start + 1, j, False)
j += 1
break
if content[j] == "\n" and not content[i + 1:j].strip():
break # empty line → not math
j += 1
i = j
continue
i += 1
_CJK_RE = re.compile(r"[\u4e00-\u9fff\u3400-\u4dbf\uf900-\ufaff]")
def convert_latex_math_to_typst(content: str) -> str:
"""Replace LaTeX math with Typst math throughout *content* (markdown)."""
spans = list(_iter_math_spans(content))
if not spans:
return content
parts: list[str] = []
prev = 0
for start, end, is_display in spans:
delim = "$$" if is_display else "$"
delim_len = len(delim)
delim_start = start - delim_len
latex = content[start:end]
# Spans containing CJK characters are almost certainly mismatched $.
# Strip the $ delimiters and emit the raw text.
if _CJK_RE.search(latex):
parts.append(content[prev:delim_start])
parts.append(latex) # emit without $ delimiters
prev = end + delim_len
continue
parts.append(content[prev:delim_start])
converted = _convert(latex)
# Strip leading/trailing whitespace from inline math so that
# ``$ text$`` (space after opening $) never occurs — CommonMark
# and mdbook-typst-math treat that as non-math.
if not is_display:
converted = converted.strip()
parts.append(f"{delim}{converted}{delim}")
prev = end + delim_len
parts.append(content[prev:])
return "".join(parts)

View File

@@ -0,0 +1,66 @@
from __future__ import annotations
import json
import sys
from pathlib import Path
try:
from tools.prepare_mdbook import build_title_cache, parse_bib, rewrite_markdown
except ModuleNotFoundError:
from prepare_mdbook import build_title_cache, parse_bib, rewrite_markdown
PLACEHOLDER_PREFIX = "[TODO: src = zh_chapters/"
BIBLIOGRAPHY_TITLE = "References"
FRONTPAGE_SWITCH_LABEL = "中文"
FRONTPAGE_SWITCH_HREF = "cn/"
def iter_chapters(items: list[dict]) -> list[dict]:
chapters: list[dict] = []
for item in items:
chapter = item.get("Chapter")
if not chapter:
continue
chapters.append(chapter)
chapters.extend(iter_chapters(chapter.get("sub_items", [])))
return chapters
def main() -> int:
if len(sys.argv) > 1 and sys.argv[1] == "supports":
return 0
context, book = json.load(sys.stdin)
root = Path(context["root"]).resolve()
source_dir = (root / context["config"]["book"]["src"]).resolve()
title_cache = build_title_cache(source_dir, placeholder_prefix=PLACEHOLDER_PREFIX)
bib_path = source_dir.parent / "mlsys.bib"
bib_db = parse_bib(bib_path) if bib_path.exists() else {}
refs_dir = source_dir.parent / "references"
if refs_dir.is_dir():
for extra_bib in sorted(refs_dir.glob("*.bib")):
for key, fields in parse_bib(extra_bib).items():
bib_db.setdefault(key, fields)
for chapter in iter_chapters(book.get("items", [])):
source_path = chapter.get("source_path") or chapter.get("path")
if not source_path:
continue
current_file = (source_dir / source_path).resolve()
chapter["content"] = rewrite_markdown(
chapter["content"],
current_file,
title_cache,
bib_db,
bibliography_title=BIBLIOGRAPHY_TITLE,
frontpage_switch_label=FRONTPAGE_SWITCH_LABEL,
frontpage_switch_href=FRONTPAGE_SWITCH_HREF,
)
json.dump(book, sys.stdout, ensure_ascii=False)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,68 @@
from __future__ import annotations
import json
import sys
from pathlib import Path
try:
from tools.prepare_mdbook import build_title_cache, parse_bib, rewrite_markdown
from tools.latex_to_typst import convert_latex_math_to_typst
except ModuleNotFoundError:
from prepare_mdbook import build_title_cache, parse_bib, rewrite_markdown
from latex_to_typst import convert_latex_math_to_typst
BIBLIOGRAPHY_TITLE = "参考文献"
FRONTPAGE_SWITCH_LABEL = "English"
FRONTPAGE_SWITCH_HREF = "../"
def iter_chapters(items: list[dict]) -> list[dict]:
chapters: list[dict] = []
for item in items:
chapter = item.get("Chapter")
if not chapter:
continue
chapters.append(chapter)
chapters.extend(iter_chapters(chapter.get("sub_items", [])))
return chapters
def main() -> int:
if len(sys.argv) > 1 and sys.argv[1] == "supports":
return 0
context, book = json.load(sys.stdin)
root = Path(context["root"]).resolve()
source_dir = (root / context["config"]["book"]["src"]).resolve()
title_cache = build_title_cache(source_dir)
bib_path = source_dir.parent / "mlsys.bib"
bib_db = parse_bib(bib_path) if bib_path.exists() else {}
refs_dir = source_dir.parent / "references"
if refs_dir.is_dir():
for extra_bib in sorted(refs_dir.glob("*.bib")):
for key, fields in parse_bib(extra_bib).items():
bib_db.setdefault(key, fields)
for chapter in iter_chapters(book.get("items", [])):
source_path = chapter.get("source_path") or chapter.get("path")
if not source_path:
continue
current_file = (source_dir / source_path).resolve()
chapter["content"] = rewrite_markdown(
chapter["content"],
current_file,
title_cache,
bib_db,
bibliography_title=BIBLIOGRAPHY_TITLE,
frontpage_switch_label=FRONTPAGE_SWITCH_LABEL,
frontpage_switch_href=FRONTPAGE_SWITCH_HREF,
)
chapter["content"] = convert_latex_math_to_typst(chapter["content"])
json.dump(book, sys.stdout, ensure_ascii=False)
return 0
if __name__ == "__main__":
raise SystemExit(main())

790
tools/prepare_mdbook.py Normal file
View File

@@ -0,0 +1,790 @@
from __future__ import annotations
import argparse
import os
import re
from dataclasses import dataclass
from pathlib import Path
TOC_FENCE = "toc"
EVAL_RST_FENCE = "eval_rst"
OPTION_LINE_RE = re.compile(r"^:(width|label):`[^`]+`\s*$", re.MULTILINE)
NUMREF_RE = re.compile(r":numref:`([^`]+)`")
EQREF_RE = re.compile(r":eqref:`([^`]+)`")
EQLABEL_LINE_RE = re.compile(r"^:eqlabel:`([^`]+)`\s*$")
CITE_RE = re.compile(r":cite:`([^`]+)`")
BIB_ENTRY_RE = re.compile(r"@(\w+)\{([^,]+),")
LATEX_ESCAPE_RE = re.compile(r"\\([_%#&])")
RAW_HTML_FILE_RE = re.compile(r"^\s*:file:\s*([^\s]+)\s*$")
TOC_LINK_RE = re.compile(r"^\[([^\]]+)\]\(([^)]+)\)\s*$")
TOC_PART_RE = re.compile(r"^#+\s+(.+?)\s*$")
HEAD_TAG_RE = re.compile(r"</?head>", re.IGNORECASE)
STYLE_BLOCK_RE = re.compile(r"<style>(.*?)</style>", re.IGNORECASE | re.DOTALL)
DEFAULT_BIBLIOGRAPHY_TITLE = "References"
FRONTPAGE_SWITCH_PLACEHOLDER = "<!-- OPENMLSYS_LANGUAGE_SWITCH -->"
FRONTPAGE_LAYOUT_CSS = """
<style>
.openmlsys-frontpage {
width: 100%;
margin: 0 auto 3rem;
margin-inline: auto;
}
.openmlsys-frontpage-switch-row {
margin: 12px 0 0;
display: flex;
justify-content: center;
}
.openmlsys-frontpage-switch {
display: inline-flex;
align-items: center;
justify-content: center;
min-width: 82px;
height: 28px;
padding: 0 14px;
border-radius: 6px;
border: 1px solid rgba(31, 35, 40, 0.15);
background: #f6f8fa;
color: #24292f;
font-size: 13px;
font-weight: 600;
text-decoration: none;
box-shadow: 0 1px 0 rgba(31, 35, 40, 0.04);
}
.openmlsys-frontpage-switch:hover {
background: #f3f4f6;
border-color: rgba(31, 35, 40, 0.2);
}
.openmlsys-frontpage .mdl-grid {
display: flex;
flex-wrap: wrap;
gap: 24px;
width: 100%;
box-sizing: border-box;
}
.openmlsys-frontpage .mdl-cell {
box-sizing: border-box;
flex: 1 1 220px;
min-width: 0;
}
.openmlsys-frontpage .mdl-cell--1-col {
flex: 0 0 48px;
}
.openmlsys-frontpage .mdl-cell--3-col {
flex: 0 1 calc(16.666% - 20px);
max-width: calc(16.666% - 20px);
}
.openmlsys-frontpage .authors.mdl-grid {
justify-content: center;
}
.openmlsys-frontpage .mdl-cell--5-col {
flex: 1 1 calc(41.666% - 24px);
max-width: calc(41.666% - 18px);
}
.openmlsys-frontpage .mdl-cell--12-col {
flex: 1 1 100%;
max-width: 100%;
}
.openmlsys-frontpage .mdl-cell--middle {
align-self: center;
}
.openmlsys-frontpage .mdl-color-text--primary {
color: var(--links, #0b6bcb);
}
.openmlsys-frontpage img {
max-width: 100%;
height: auto;
background: transparent !important;
padding: 0 !important;
}
.openmlsys-frontpage + ul,
.openmlsys-frontpage + ul ul {
max-width: 960px;
margin-inline: auto;
}
.content main {
max-width: min(100%, max(65%, var(--content-max-width)));
}
@media (max-width: 1000px) {
.openmlsys-frontpage .mdl-cell,
.openmlsys-frontpage .mdl-cell--1-col,
.openmlsys-frontpage .mdl-cell--3-col,
.openmlsys-frontpage .mdl-cell--5-col {
flex: 1 1 100%;
max-width: 100%;
}
}
</style>
""".strip()
@dataclass(frozen=True)
class TocItem:
kind: str
label: str
target: str | None = None
def is_placeholder_markdown(markdown: str, placeholder_prefix: str | None = None) -> bool:
if not placeholder_prefix:
return False
stripped = markdown.strip()
return stripped.startswith(placeholder_prefix) and stripped.endswith("]")
def extract_title(markdown: str, fallback: str = "Untitled") -> str:
lines = markdown.splitlines()
for index, line in enumerate(lines):
stripped = line.strip()
if not stripped:
continue
if stripped.startswith("#"):
heading = stripped.lstrip("#").strip()
if heading:
return heading
next_index = index + 1
if next_index < len(lines):
underline = lines[next_index].strip()
if underline and set(underline) <= {"=", "-"}:
return stripped
return fallback
def parse_toc_entries(block_lines: list[str]) -> list[TocItem]:
entries: list[TocItem] = []
for line in block_lines:
stripped = line.strip()
if not stripped or stripped.startswith(":"):
continue
part_match = TOC_PART_RE.match(stripped)
if part_match:
entries.append(TocItem(kind="part", label=part_match.group(1).strip()))
continue
link_match = TOC_LINK_RE.match(stripped)
if link_match:
entries.append(
TocItem(
kind="chapter",
label=link_match.group(1).strip(),
target=link_match.group(2).strip(),
)
)
continue
entries.append(TocItem(kind="chapter", label="", target=stripped))
return entries
def parse_toc_blocks(markdown: str) -> list[list[TocItem]]:
blocks: list[list[TocItem]] = []
lines = markdown.splitlines()
index = 0
while index < len(lines):
if lines[index].strip() == f"```{TOC_FENCE}":
index += 1
block_lines: list[str] = []
while index < len(lines) and lines[index].strip() != "```":
block_lines.append(lines[index])
index += 1
entries = parse_toc_entries(block_lines)
blocks.append(entries)
index += 1
return blocks
def resolve_toc_target(current_file: Path, entry: str) -> Path:
target_name = entry if entry.endswith(".md") else f"{entry}.md"
target = (current_file.parent / target_name).resolve()
if not target.exists():
raise FileNotFoundError(f"TOC entry '{entry}' from '{current_file}' does not exist")
return target
def relative_link(from_file: Path, target_file: Path) -> str:
return Path(os.path.relpath(target_file, start=from_file.parent)).as_posix()
def _strip_latex_escapes_outside_math(text: str) -> str:
"""Remove LaTeX text-mode escapes (``\\_``, ``\\#``, etc.) outside math
spans, fenced code blocks, and inline code.
Operates on the full text (not per-line) to correctly handle multi-line
display math ``$$...$$`` blocks.
"""
# 1. Find all protected regions where escapes must NOT be stripped.
protected: list[tuple[int, int]] = [] # (start, end)
n = len(text)
i = 0
in_fence: str | None = None
fence_start = 0
while i < n:
# Fenced code blocks (``` or ~~~) at start of line
if (i == 0 or text[i - 1] == "\n") and text[i] in ("`", "~"):
m = re.match(r"`{3,}|~{3,}", text[i:])
if m:
if in_fence is None:
in_fence = m.group()[0]
fence_start = i
elif m.group()[0] == in_fence:
eol = text.find("\n", m.end() + i)
end = eol + 1 if eol != -1 else n
protected.append((fence_start, end))
in_fence = None
i = end
continue
eol = text.find("\n", i)
i = eol + 1 if eol != -1 else n
continue
if in_fence is not None:
i += 1
continue
# Inline code `...`
if text[i] == "`":
close = text.find("`", i + 1)
if close != -1:
protected.append((i, close + 1))
i = close + 1
continue
# Display math $$...$$
if text[i:i + 2] == "$$":
close = text.find("$$", i + 2)
if close != -1:
protected.append((i, close + 2))
i = close + 2
continue
# Inline math $...$
if text[i] == "$":
j = i + 1
while j < n and text[j] != "$" and text[j] != "\n":
j += 1
if j < n and text[j] == "$" and j > i + 1:
protected.append((i, j + 1))
i = j + 1
continue
i += 1
# Unclosed fence → protect everything from fence_start to end
if in_fence is not None:
protected.append((fence_start, n))
# 2. Apply substitution only to unprotected gaps.
parts: list[str] = []
prev = 0
for start, end in protected:
if start > prev:
parts.append(LATEX_ESCAPE_RE.sub(r"\1", text[prev:start]))
parts.append(text[start:end])
prev = end
if prev < n:
parts.append(LATEX_ESCAPE_RE.sub(r"\1", text[prev:]))
return "".join(parts)
def process_equation_labels(markdown: str) -> tuple[str, dict[str, int]]:
"""Convert :eqlabel: directives to MathJax \\tag + \\label in preceding equations.
Args:
markdown: The markdown content to process.
Returns:
A tuple of (processed_markdown, label_map) where label_map maps
label names to their equation numbers.
"""
lines = markdown.split("\n")
result: list[str] = []
eq_counter = 0
label_map: dict[str, int] = {}
for line in lines:
match = EQLABEL_LINE_RE.match(line.strip())
if not match:
result.append(line)
continue
label_name = match.group(1)
eq_counter += 1
label_map[label_name] = eq_counter
tag = f"\\tag{{{eq_counter}}}\\label{{{label_name}}}"
# Search backward for the closing $$ of the preceding equation
inserted = False
for j in range(len(result) - 1, -1, -1):
stripped = result[j].rstrip()
if not stripped:
continue # skip blank lines
if stripped == "$$":
# Multi-line equation: $$ on its own line
result.insert(j, tag)
inserted = True
break
if stripped.endswith("$$"):
# Single-line or end-of-content $$
result[j] = stripped[:-2] + tag + "$$"
inserted = True
break
break # non-blank, non-$$ line: no equation found
if not inserted:
# Fallback: keep original line if no equation found
result.append(line)
return "\n".join(result), label_map
def normalize_directives(
markdown: str,
label_map: dict[str, int] | None = None,
) -> str:
normalized = OPTION_LINE_RE.sub("", markdown)
normalized = NUMREF_RE.sub(lambda match: f"`{match.group(1)}`", normalized)
if label_map:
normalized = EQREF_RE.sub(
lambda m: f"({label_map[m.group(1)]})" if m.group(1) in label_map else f"$\\eqref{{{m.group(1)}}}$",
normalized,
)
else:
normalized = EQREF_RE.sub(lambda match: f"$\\eqref{{{match.group(1)}}}$", normalized)
normalized = _strip_latex_escapes_outside_math(normalized)
lines = [line.rstrip() for line in normalized.splitlines()]
collapsed: list[str] = []
previous_blank = False
for line in lines:
is_blank = line == ""
if is_blank and previous_blank:
continue
collapsed.append(line)
previous_blank = is_blank
while collapsed and collapsed[-1] == "":
collapsed.pop()
return "\n".join(collapsed) + "\n"
def clean_bibtex(value: str) -> str:
value = re.sub(r"\{\\[`'^\"~=.](\w)\}", r"\1", value)
value = re.sub(r"\\[`'^\"~=.](\w)", r"\1", value)
value = value.replace("{", "").replace("}", "")
return value.strip()
def _parse_bib_fields(body: str) -> dict[str, str]:
fields: dict[str, str] = {}
i = 0
while i < len(body):
while i < len(body) and body[i] in " \t\n\r,":
i += 1
if i >= len(body):
break
start = i
while i < len(body) and body[i] not in "= \t\n\r":
i += 1
name = body[start:i].strip().lower()
while i < len(body) and body[i] != "=":
i += 1
if i >= len(body):
break
i += 1
while i < len(body) and body[i] in " \t\n\r":
i += 1
if i >= len(body):
break
if body[i] == "{":
depth = 1
i += 1
vstart = i
while i < len(body) and depth > 0:
if body[i] == "{":
depth += 1
elif body[i] == "}":
depth -= 1
i += 1
value = body[vstart : i - 1]
elif body[i] == '"':
i += 1
vstart = i
while i < len(body) and body[i] != '"':
i += 1
value = body[vstart:i]
i += 1
else:
vstart = i
while i < len(body) and body[i] not in ", \t\n\r}":
i += 1
value = body[vstart:i]
if name:
fields[name] = value.strip()
return fields
def parse_bib(bib_path: Path) -> dict[str, dict[str, str]]:
text = bib_path.read_text(encoding="utf-8")
entries: dict[str, dict[str, str]] = {}
for match in BIB_ENTRY_RE.finditer(text):
key = match.group(2).strip()
start = match.end()
depth = 1
pos = start
while pos < len(text) and depth > 0:
if text[pos] == "{":
depth += 1
elif text[pos] == "}":
depth -= 1
pos += 1
fields = _parse_bib_fields(text[start : pos - 1])
fields["_type"] = match.group(1).lower()
entries[key] = fields
return entries
def _render_bibliography(
cited_keys: list[str],
bib_db: dict[str, dict[str, str]],
bibliography_title: str,
) -> list[str]:
lines: list[str] = ["---", "", f"## {bibliography_title}", "", "<ol>"]
for key in cited_keys:
entry = bib_db.get(key)
if not entry:
continue
author = clean_bibtex(entry.get("author", ""))
title = clean_bibtex(entry.get("title", ""))
year = entry.get("year", "")
venue = clean_bibtex(entry.get("journal", "") or entry.get("booktitle", ""))
parts: list[str] = []
if author:
parts.append(author)
if title:
parts.append(f"<em>{title}</em>")
if venue:
parts.append(venue)
if year:
parts.append(year)
text = ". ".join(parts) + "." if parts else f"{key}."
lines.append(f'<li id="ref-{key}">{text} <a href="#cite-{key}">↩</a></li>')
lines.append("</ol>")
return lines
def process_citations(
markdown: str,
bib_db: dict[str, dict[str, str]],
bibliography_title: str = DEFAULT_BIBLIOGRAPHY_TITLE,
) -> str:
cited_keys: list[str] = []
def _replace_cite(match: re.Match[str]) -> str:
keys = [k.strip() for k in match.group(1).split(",")]
for key in keys:
if key not in cited_keys and key in bib_db:
cited_keys.append(key)
if not bib_db:
return "[" + ", ".join(keys) + "]"
nums: list[str] = []
for key in keys:
if key not in bib_db:
continue
idx = cited_keys.index(key) + 1
nums.append(f'<sup id="cite-{key}"><a href="#ref-{key}">[{idx}]</a></sup>')
return "".join(nums)
processed = CITE_RE.sub(_replace_cite, markdown)
if cited_keys and bib_db:
bib_lines = _render_bibliography(cited_keys, bib_db, bibliography_title)
processed = processed.rstrip("\n") + "\n\n" + "\n".join(bib_lines) + "\n"
return processed
def resolve_raw_html_file(current_file: Path, filename: str) -> Path:
direct = (current_file.parent / filename).resolve()
if direct.exists():
return direct
static_fallback = (current_file.parent / "static" / filename).resolve()
if static_fallback.exists():
return static_fallback
repo_static = (Path(__file__).resolve().parent.parent / "static" / filename)
if repo_static.exists():
return repo_static
raise FileNotFoundError(f"Raw HTML include '{filename}' from '{current_file}' does not exist")
def rewrite_frontpage_assets(html: str) -> str:
rewritten = html.replace("./_images/", "static/image/")
rewritten = rewritten.replace("_images/", "static/image/")
rewritten = HEAD_TAG_RE.sub("", rewritten)
rewritten = STYLE_BLOCK_RE.sub(_minify_style_block, rewritten)
return rewritten
def _minify_style_block(match: re.Match[str]) -> str:
content = match.group(1)
parts = [line.strip() for line in content.splitlines() if line.strip()]
return f"<style>{' '.join(parts)}</style>"
def render_frontpage_switch(label: str, href: str) -> str:
return (
'<p class="openmlsys-frontpage-switch-row">'
f'<a class="openmlsys-frontpage-switch" href="{href}">{label}</a>'
"</p>"
)
def wrap_frontpage_html(
html: str,
frontpage_switch_label: str | None = None,
frontpage_switch_href: str | None = None,
) -> str:
rendered_html = html.strip()
if frontpage_switch_label and frontpage_switch_href:
switch_html = render_frontpage_switch(frontpage_switch_label, frontpage_switch_href)
if FRONTPAGE_SWITCH_PLACEHOLDER in rendered_html:
rendered_html = rendered_html.replace(FRONTPAGE_SWITCH_PLACEHOLDER, switch_html)
else:
rendered_html = "\n".join([switch_html, rendered_html])
parts = [FRONTPAGE_LAYOUT_CSS, '<div class="openmlsys-frontpage">', rendered_html, "</div>"]
return "\n".join(parts)
def inline_raw_html(
block_lines: list[str],
current_file: Path,
frontpage_switch_label: str | None = None,
frontpage_switch_href: str | None = None,
) -> str | None:
stripped = [line.strip() for line in block_lines if line.strip()]
if not stripped or stripped[0] != ".. raw:: html":
return None
filename: str | None = None
for line in stripped[1:]:
match = RAW_HTML_FILE_RE.match(line)
if match:
filename = match.group(1)
break
if filename is None:
return None
html_path = resolve_raw_html_file(current_file, filename)
html = rewrite_frontpage_assets(html_path.read_text(encoding="utf-8")).strip()
if Path(filename).name == "frontpage.html":
return wrap_frontpage_html(
html,
frontpage_switch_label=frontpage_switch_label,
frontpage_switch_href=frontpage_switch_href,
)
return html
def chapter_label(item: TocItem, target: Path, title_cache: dict[Path, str]) -> str:
return item.label or title_cache[target]
def render_toc_list(entries: list[TocItem], current_file: Path, title_cache: dict[Path, str]) -> list[str]:
rendered: list[str] = []
current_indent = 0
for entry in entries:
if entry.kind == "part":
rendered.append(f"- {entry.label}")
current_indent = 1
continue
if entry.target is None:
continue
target = resolve_toc_target(current_file, entry.target)
if target not in title_cache:
continue
label = chapter_label(entry, target, title_cache)
rendered.append(f"{' ' * current_indent}- [{label}]({relative_link(current_file, target)})")
return rendered
def rewrite_markdown(
markdown: str,
current_file: Path,
title_cache: dict[Path, str],
bib_db: dict[str, dict[str, str]] | None = None,
bibliography_title: str = DEFAULT_BIBLIOGRAPHY_TITLE,
frontpage_switch_label: str | None = None,
frontpage_switch_href: str | None = None,
) -> str:
output: list[str] = []
lines = markdown.splitlines()
index = 0
while index < len(lines):
stripped = lines[index].strip()
if stripped in (f"```{TOC_FENCE}", f"```{EVAL_RST_FENCE}"):
fence = stripped[3:]
index += 1
block_lines: list[str] = []
while index < len(lines) and lines[index].strip() != "```":
block_lines.append(lines[index])
index += 1
if fence == TOC_FENCE:
entries = parse_toc_entries(block_lines)
if entries:
if output and output[-1] != "":
output.append("")
rendered = render_toc_list(entries, current_file, title_cache)
output.extend(rendered)
if rendered and output and output[-1] != "":
output.append("")
elif fence == EVAL_RST_FENCE:
raw_html = inline_raw_html(
block_lines,
current_file,
frontpage_switch_label=frontpage_switch_label,
frontpage_switch_href=frontpage_switch_href,
)
if raw_html:
if output and output[-1] != "":
output.append("")
output.extend(raw_html.splitlines())
if output and output[-1] != "":
output.append("")
index += 1
continue
output.append(lines[index])
index += 1
while output and output[-1] == "":
output.pop()
raw = "\n".join(output) + "\n"
result, label_map = process_equation_labels(raw)
result = normalize_directives(result, label_map=label_map)
result = process_citations(result, bib_db or {}, bibliography_title=bibliography_title)
return result
def build_title_cache(
source_dir: Path,
placeholder_prefix: str | None = None,
) -> dict[Path, str]:
cache: dict[Path, str] = {}
for markdown_file in sorted(source_dir.rglob("*.md")):
if "_build" in markdown_file.parts or markdown_file.name == "SUMMARY.md":
continue
text = markdown_file.read_text(encoding="utf-8")
if is_placeholder_markdown(text, placeholder_prefix):
continue
cache[markdown_file.resolve()] = extract_title(text, fallback=markdown_file.stem)
return cache
def build_summary(source_dir: Path, title_cache: dict[Path, str]) -> str:
root_index = (source_dir / "index.md").resolve()
root_markdown = root_index.read_text(encoding="utf-8")
lines = ["# Summary", "", f"[{title_cache[root_index]}](index.md)"]
seen: set[Path] = {root_index}
def append_entry(target: Path, indent: int, label: str | None = None) -> None:
target = target.resolve()
if target in seen or target not in title_cache:
return
seen.add(target)
rel = target.relative_to(source_dir.resolve()).as_posix()
title = label or title_cache[target]
lines.append(f"{' ' * indent}- [{title}]({rel})")
child_markdown = target.read_text(encoding="utf-8")
for block in parse_toc_blocks(child_markdown):
for entry in block:
if entry.kind != "chapter" or entry.target is None:
continue
append_entry(resolve_toc_target(target, entry.target), indent + 1, entry.label or None)
def append_prefix_chapter(target: Path, label: str | None = None) -> None:
target = target.resolve()
if target in seen or target not in title_cache:
return
seen.add(target)
rel = target.relative_to(source_dir.resolve()).as_posix()
title = label or title_cache[target]
lines.append(f"[{title}]({rel})")
numbered_started = False
for block in parse_toc_blocks(root_markdown):
for entry in block:
if entry.kind == "part":
if lines and lines[-1] != "":
lines.append("")
lines.append(f"# {entry.label}")
lines.append("")
numbered_started = True
continue
if entry.target is None:
continue
target = resolve_toc_target(root_index, entry.target)
if numbered_started:
append_entry(target, 0, entry.label or None)
else:
append_prefix_chapter(target, entry.label or None)
return "\n".join(lines) + "\n"
def write_summary(
source_dir: Path,
summary_path: Path | None = None,
placeholder_prefix: str | None = None,
) -> Path:
source_dir = source_dir.resolve()
summary_path = summary_path.resolve() if summary_path else (source_dir / "SUMMARY.md")
title_cache = build_title_cache(source_dir, placeholder_prefix=placeholder_prefix)
summary_path.write_text(build_summary(source_dir, title_cache), encoding="utf-8")
return summary_path
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Generate mdBook SUMMARY.md for a chapter directory.")
parser.add_argument("--source", type=Path, required=True, help="Source chapter directory")
parser.add_argument("--summary-output", type=Path, required=True, help="Where to write the generated SUMMARY.md")
parser.add_argument(
"--placeholder-prefix",
default=None,
help="If set, files whose entire contents start with this prefix are skipped from mdBook output.",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
summary_path = write_summary(
args.source,
summary_path=args.summary_output,
placeholder_prefix=args.placeholder_prefix,
)
print(f"Wrote mdBook summary to {summary_path}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,75 @@
from __future__ import annotations
import argparse
from pathlib import Path
try:
from tools.prepare_mdbook import (
build_title_cache,
extract_title,
inline_raw_html,
parse_bib,
process_equation_labels,
rewrite_markdown,
write_summary,
)
except ModuleNotFoundError:
from prepare_mdbook import (
build_title_cache,
extract_title,
inline_raw_html,
parse_bib,
process_equation_labels,
rewrite_markdown,
write_summary,
)
FRONTPAGE_SWITCH_LABEL = "English"
FRONTPAGE_SWITCH_HREF = "../"
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Generate mdBook SUMMARY.md for zh_chapters.")
parser.add_argument("--source", type=Path, default=Path("zh_chapters"), help="Source chapter directory")
parser.add_argument(
"--summary-output",
type=Path,
default=Path("zh_chapters/SUMMARY.md"),
help="Where to write the generated SUMMARY.md",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
summary_path = write_summary(args.source, summary_path=args.summary_output)
print(f"Wrote mdBook summary to {summary_path}")
return 0
def rewrite_markdown(
markdown: str,
current_file: Path,
title_cache: dict[Path, str],
bib_db: dict[str, dict[str, str]] | None = None,
bibliography_title: str = "参考文献",
) -> str:
try:
from tools.prepare_mdbook import rewrite_markdown as generic_rewrite_markdown
except ModuleNotFoundError:
from prepare_mdbook import rewrite_markdown as generic_rewrite_markdown
return generic_rewrite_markdown(
markdown,
current_file,
title_cache,
bib_db=bib_db,
bibliography_title=bibliography_title,
frontpage_switch_label=FRONTPAGE_SWITCH_LABEL,
frontpage_switch_href=FRONTPAGE_SWITCH_HREF,
)
if __name__ == "__main__":
raise SystemExit(main())

110
zh_chapters/SUMMARY.md Normal file
View File

@@ -0,0 +1,110 @@
# Summary
[机器学习系统:设计和实现](index.md)
[前言](chapter_preface/index.md)
# 基础篇
- [导论](chapter_introduction/index.md)
- [机器学习应用](chapter_introduction/applications.md)
- [机器学习框架的设计目标](chapter_introduction/design.md)
- [机器学习框架的基本组成原理](chapter_introduction/architecture.md)
- [机器学习系统生态](chapter_introduction/ecosystem.md)
- [图书结构和读者](chapter_introduction/readers.md)
- [编程接口](chapter_programming_interface/index.md)
- [机器学习系统编程模型的演进](chapter_programming_interface/development_history.md)
- [机器学习工作流](chapter_programming_interface/ml_workflow.md)
- [定义深度神经网络](chapter_programming_interface/neural_network_layer.md)
- [C/C++编程接口](chapter_programming_interface/c_python_interaction.md)
- [机器学习框架的编程范式](chapter_programming_interface/ml_programming_paradigm.md)
- [总结](chapter_programming_interface/summary.md)
- [计算图](chapter_computational_graph/index.md)
- [计算图的设计背景和作用](chapter_computational_graph/background_and_functionality.md)
- [计算图的基本构成](chapter_computational_graph/components_of_computational_graph.md)
- [计算图的生成](chapter_computational_graph/generation_of_computational_graph.md)
- [计算图的调度](chapter_computational_graph/schedule_of_computational_graph.md)
- [总结](chapter_computational_graph/summary.md)
# 进阶篇
- [导读](chapter_preface_advanced/index.md)
- [AI编译器和前端技术](chapter_frontend_and_ir/index.md)
- [AI编译器设计原理](chapter_frontend_and_ir/ai_compiler_design_principle.md)
- [AI编译器前端技术概述](chapter_frontend_and_ir/overview_of_frontend.md)
- [中间表示](chapter_frontend_and_ir/intermediate_representation.md)
- [自动微分](chapter_frontend_and_ir/ad.md)
- [类型系统和静态分析](chapter_frontend_and_ir/type_system_and_static_analysis.md)
- [常见前端编译优化方法](chapter_frontend_and_ir/common_frontend_optimization_pass.md)
- [总结](chapter_frontend_and_ir/summary.md)
- [编译器后端和运行时](chapter_backend_and_runtime/index.md)
- [概述](chapter_backend_and_runtime/overview.md)
- [计算图优化](chapter_backend_and_runtime/graph_optimizer.md)
- [算子选择](chapter_backend_and_runtime/kernel_selecter.md)
- [内存分配](chapter_backend_and_runtime/memory_allocator.md)
- [计算调度与执行](chapter_backend_and_runtime/compute_schedule_and_execute.md)
- [算子编译器](chapter_backend_and_runtime/op_compiler.md)
- [总结](chapter_backend_and_runtime/summary.md)
- [硬件加速器](chapter_accelerator/index.md)
- [概述](chapter_accelerator/accelerator_introduction.md)
- [加速器基本组成原理](chapter_accelerator/accelerator_architecture.md)
- [加速器基本编程原理](chapter_accelerator/accelerator_programming.md)
- [加速器实践](chapter_accelerator/accelerator_practise.md)
- [总结](chapter_accelerator/summary.md)
- [数据处理框架](chapter_data_processing/index.md)
- [概述](chapter_data_processing/requirements.md)
- [易用性设计](chapter_data_processing/program_model.md)
- [高效性设计](chapter_data_processing/performance.md)
- [保序性设计](chapter_data_processing/data_order.md)
- [单机数据处理性能的扩展](chapter_data_processing/extension.md)
- [总结](chapter_data_processing/summary.md)
- [模型部署](chapter_model_deployment/index.md)
- [概述](chapter_model_deployment/model_deployment_introduction.md)
- [训练模型到推理模型的转换及优化](chapter_model_deployment/model_converter_and_optimizer.md)
- [模型压缩](chapter_model_deployment/model_compression.md)
- [模型推理](chapter_model_deployment/model_inference.md)
- [模型的安全保护](chapter_model_deployment/model_security.md)
- [总结](chapter_model_deployment/summary.md)
- [分布式训练](chapter_distributed_training/index.md)
- [系统概述](chapter_distributed_training/overview.md)
- [实现方法](chapter_distributed_training/methods.md)
- [机器学习集群架构](chapter_distributed_training/cluster.md)
- [集合通信](chapter_distributed_training/collective.md)
- [参数服务器](chapter_distributed_training/parameter_servers.md)
- [总结](chapter_distributed_training/summary.md)
# 拓展篇
- [导读](chapter_preface_extension/index.md)
- [深度学习推荐系统](chapter_recommender_system/index.md)
- [系统基本组成](chapter_recommender_system/system_architecture.md)
- [多阶段推荐系统](chapter_recommender_system/multi_stage_recommender_system.md)
- [模型更新](chapter_recommender_system/model_update.md)
- [案例分析:支持在线模型更新的大型推荐系统](chapter_recommender_system/case_study.md)
- [小结](chapter_recommender_system/summary.md)
- [联邦学习系统](chapter_federated_learning/index.md)
- [概述](chapter_federated_learning/overview.md)
- [横向联邦学习](chapter_federated_learning/horizontal_fl.md)
- [纵向联邦学习](chapter_federated_learning/vertical_fl.md)
- [隐私加密算法](chapter_federated_learning/privacy_encryption_algorithm.md)
- [展望](chapter_federated_learning/outlook.md)
- [小结](chapter_federated_learning/summary.md)
- [强化学习系统](chapter_reinforcement_learning/index.md)
- [强化学习介绍](chapter_reinforcement_learning/rl_introduction.md)
- [单节点强化学习系统](chapter_reinforcement_learning/single_node_rl.md)
- [多智能体强化学习](chapter_reinforcement_learning/marl.md)
- [多智能体强化学习系统](chapter_reinforcement_learning/marl_sys.md)
- [小结](chapter_reinforcement_learning/summary.md)
- [可解释性AI系统](chapter_explainable_AI/index.md)
- [背景](chapter_explainable_AI/explainable_ai.md)
- [机器人系统](chapter_rl_sys/index.md)
- [机器人系统概述](chapter_rl_sys/rl_sys_intro.md)
- [通用机器人操作系统](chapter_rl_sys/ros.md)
- [案例分析:使用机器人操作系统](chapter_rl_sys/ros_code_ex.md)
- [总结](chapter_rl_sys/summary.md)
# 附录
- [机器学习介绍](appendix_machine_learning_introduction/index.md)
- [神经网络](appendix_machine_learning_introduction/neural_network.md)
- [梯度下降与反向传播](appendix_machine_learning_introduction/gradient_descent.md)
- [经典机器学习方法](appendix_machine_learning_introduction/classic_machine_learning.md)

View File

@@ -20,15 +20,11 @@
上一节介绍了中间表示的基本概念,初步阐述了中间表示的重要作用和发展历程。接下来从组织结构的角度出发,介绍通用编译器的中间表示的类型以及各自特点 :cite:`2020MLIR`,如下表所示。中间表示组织结构的设计,对编译阶段的分析优化、代码生成等有着重要影响。编译器的设计需求不同,采用的中间表示组织结构也有所不同。
::: {#tab:ch04/ch04-categorize}
组织结构 特点 举例
-------------- ---------------------- ----------------------------------
Linear IR 基于线性代码 堆栈机代码、三地址代码
Graphical IR 基于图 抽象语法树、有向无环图、控制流图
Hybrid IR 基于图与线性代码混合 LLVM IR
: 中间表示的分类
:::
| 组织结构 | 特点 | 举例 |
|---|---|---|
| Linear IR | 基于线性代码 | 堆栈机代码、三地址代码 |
| Graphical IR | 基于图 | 抽象语法树、有向无环图、控制流图 |
| Hybrid IR | 基于图与线性代码混合 | LLVM IR |
1\) 线性中间表示

View File

@@ -53,14 +53,10 @@ $$\pmb{Y_{bn}}=\pmb{A}*\pmb{X_{conv}}+\pmb{B}$$
在融合过程中Convolution计算公式和Batchnorm计算公式中被认为是常量的符号在训练时均为参数并不是常量。训练阶段如果进行该融合会导致模型参数的缺失。从该融合Pattern的结果来看融合后网络中减少了一个Batchnorm算子减少了一个Batchnorm算子的参数量其实就是改变了深度神经网络的算法会影响到网络的准确率这是不可接受的。所以Convolution算子与Batchnorm算子的融合一般是在部署阶段特有的一种优化手段其优化效果以MindSpore Lite为例构造了包含一个Convolution和一个Batchnorm的sample网络分别以样例网络和mobilenet-v2网络为例在华为Mate30手机上以两线程运行模型推理取3000轮推理的平均时耗作为模型推理性能的指标对比融合前后该指标的变化。从表 :numref:`ch08-tab-conv_bn_fusion`可以看到对于sample网络和mobilenet-v2网络融合后分别获得了8.5%和11.7%的推理性能提升,这个性能提升非常可观。并且这个性能提升没有带来任何的副作用,也没有对于硬件或算子库的提出额外要求。
::: {#tab:ch08-tab-conv_bn_fusion}
网络 sample mobilenet-v2
------- ------------- -----------------
融合前 0.035 15.415
融合后 0.031 13.606
: Convolution + Batchnorm融合前后推理性能单位ms
:::
| 网络 | sample | mobilenet-v2 |
|---|---|---|
| 融合前 | 0.035 | 15.415 |
| 融合后 | 0.031 | 13.606 |
:label:`ch08-tab-conv_bn_fusion`
### 算子替换

View File

@@ -13,7 +13,6 @@
rl_introduction
single_node_rl
distributed_node_rl
marl
marl_sys
summary

View File

@@ -1 +0,0 @@
/chivier-disk/hyq-home/Projects/openmlsys-zh/img

View File

@@ -8,15 +8,16 @@
```toc
:maxdepth: 2
:numbered:
chapter_preface/index
[前言](chapter_preface/index)
# 基础篇
chapter_introduction/index
chapter_programming_interface/index
chapter_computational_graph/index
chapter_preface_advanced/index
# 进阶篇
[导读](chapter_preface_advanced/index)
chapter_frontend_and_ir/index
chapter_backend_and_runtime/index
chapter_accelerator/index
@@ -24,18 +25,14 @@ chapter_data_processing/index
chapter_model_deployment/index
chapter_distributed_training/index
chapter_preface_extension/index
# 拓展篇
[导读](chapter_preface_extension/index)
chapter_recommender_system/index
chapter_federated_learning/index
chapter_reinforcement_learning/index
chapter_explainable_AI/index
chapter_rl_sys/index
```
```toc
:maxdepth: 1
appendix_machine_learning_introduction/index
# 附录
[机器学习介绍](appendix_machine_learning_introduction/index)
```

File diff suppressed because it is too large Load Diff

1
zh_chapters/mlsys.bib Symbolic link
View File

@@ -0,0 +1 @@
../mlsys.bib

View File

@@ -1 +0,0 @@
/chivier-disk/hyq-home/Projects/openmlsys-zh/references

View File

@@ -1 +0,0 @@
/chivier-disk/hyq-home/Projects/openmlsys-zh/static