mirror of
https://github.com/openmlsys/openmlsys-zh.git
synced 2026-07-08 04:47:28 +08:00
build: migrate docs build and deploy to mdbook (#490)
* build: add mdbook support for zh chapters
Add mdBook configuration rooted at zh_chapters, generate and commit SUMMARY.md, rewrite d2l-specific directives through a Python preprocessor, refresh chapter resource symlinks from the build scripts, and ignore local build-only links and helper directories.
* feat: add raw HTML inline and frontpage layout support for mdbook preprocessor
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add dark mode image background for mdbook dark themes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add resource symlinks and repo root static fallback to mdbook build
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add BibTeX citation support with inline links and bibliography
Parse mlsys.bib to generate author-year inline citations linked to
per-page bibliography sections. Missing bib keys degrade gracefully
to plain text placeholders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: switch citation display to footnote style
Use numbered superscript references [1] [2] inline with an ordered
list bibliography at page bottom. Each entry has a back-link (↩)
to the citation site.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: strip LaTeX escapes outside math mode in mdbook preprocessor
Remove \_, \%, \#, \& escapes from text outside $...$ math spans
while preserving them inside math mode for MathJax compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: set frontpage author grid to 6 columns and widen main content area
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: group mdbook toc by part titles
* fix: enable inline math rendering in mdbook
* build: migrate docs publishing to mdbook
Move the English root site to mdBook, keep the Chinese site as a sub-book, and update CI/deploy to publish .mdbook outputs to docs/ and docs/cn/. Also add regression coverage for placeholder skipping, publish-tree assembly, and shared resource setup.
* ci: use official pages deployment workflow
Switch the docs deployment workflow to the official GitHub Pages actions flow and verify it uses Pages action outputs for the deployment URL.
* feat: add homepage language switch links
Inject a homepage-only language switch into the mdBook frontpage wrapper so the English homepage links to the Chinese homepage and the Chinese homepage links back to the English homepage.
* fix: correct english homepage frontpage
Add an English-specific frontpage template so the default homepage no longer falls back to the Chinese frontpage, and clear homepage image backgrounds in the frontpage wrapper CSS.
* fix: align english homepage author grid
Top-align the English homepage author cards, enlarge the row gap, and normalize avatar sizing so author portraits line up consistently.
* fix: restore dark mode body image backgrounds
Apply light gray backgrounds to body images in dark themes for both English and Chinese mdBook themes while explicitly excluding homepage frontpage images.
* fix: restyle homepage language switch button
Move the homepage language switch below the GitHub star button and restyle it to match the same button family on both the English and Chinese homepages.
* fix: center homepage content container
Align the English and Chinese homepage frontpage wrapper with the main content container so homepage content is centered like normal body content.
* fix: stack english homepage footer copy
Keep the English homepage contributor and errata footer lines in normal block flow so each sentence stays on its own line instead of being laid out as author-grid columns.
* fix: widen centered homepage container
Keep the homepage frontpage wrapper centered while ensuring it uses at least 80% of the available content area, without changing normal body page layout.
* fix: widen homepage main content area
Apply a homepage-only override so mdbook-content > main uses at least 80% of the available content width while keeping normal body pages on the default layout.
* ci: use peaceiris action for mdbook
Replace manual mdBook installation in CI and Pages workflows with peaceiris/actions-mdbook@v2 and keep a regression test to ensure the action stays in use.
* fix: reduce homepage main width floor
Lower the homepage-only mdbook-content > main minimum width from 80% to 65% while leaving normal body pages unchanged.
* build: switch math rendering to mdbook-katex
Use mdbook-katex in pre-render mode for both books, pin mdBook to a compatible version, update build scripts and workflows, and replace the old MathJax regression tests with KaTeX coverage.
* Revert "build: switch math rendering to mdbook-katex"
This reverts commit b9cf38a5d1.
* build: switch math rendering from MathJax to mdbook-typst-math
* ci: deploy docs to openmlsys.github.io repo
* fix: convert pandoc tables to GFM pipe tables for mdbook
* feat: convert :eqlabel:/:eqref: to MathJax \tag/\label/\eqref
- Add process_equation_labels() to inject \tag{n}\label{name} into
preceding $$ equations, replacing :eqlabel: directives
- Change :eqref: conversion from backtick code to $\eqref{name}$
for clickable cross-references
- Add TeX.equationNumbers.autoNumber:"none" to MathJax config to
prevent conflicts with manual \tag numbering
- Add tests for single-line, multi-line, and sequential numbering
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: cache mdbook-typst-math binary in workflows
* feat: add LaTeX-to-Typst math converter with eqref/tag support
* feat: integrate LaTeX-to-Typst conversion into zh preprocessor
* fix: strip LaTeX escapes only outside math spans and code blocks
* fix: load references/*.bib so all citations render correctly
* fix: skip citations with no bib entry instead of rendering raw keys
* ci: remove redundant CI workflow, keep only deploy workflow
* ci: Add CI workflow for testing and building mdBook
* ci: remove concurrency settings from update_docs.yml
Removed concurrency settings from the update_docs workflow.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
78
.github/workflows/main.yml
vendored
78
.github/workflows/main.yml
vendored
@@ -6,8 +6,8 @@ on:
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
build-en:
|
||||
name: Build (English)
|
||||
test:
|
||||
name: Tests
|
||||
runs-on: ubuntu-22.04
|
||||
|
||||
steps:
|
||||
@@ -17,62 +17,30 @@ jobs:
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install pandoc
|
||||
run: |
|
||||
wget -q https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-1-amd64.deb
|
||||
sudo dpkg -i pandoc-2.19.2-1-amd64.deb
|
||||
|
||||
- name: Install d2lbook
|
||||
run: |
|
||||
git clone https://github.com/openmlsys/d2l-book.git
|
||||
cd d2l-book
|
||||
# Fix Python 3.10+ incompatibility: bibtex<2.0.0 depends on oset which
|
||||
# uses collections.MutableSet removed in Python 3.10.
|
||||
sed -i "s/'sphinxcontrib-bibtex<2.0.0'/'sphinxcontrib-bibtex>=2.5.0'/" setup.py
|
||||
python3 -m pip install .
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: python3 -m pip install -r requirements.txt
|
||||
|
||||
- name: Build English HTML
|
||||
run: bash build_html.sh
|
||||
|
||||
build-zh:
|
||||
name: Build (Chinese)
|
||||
runs-on: ubuntu-22.04
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.10
|
||||
uses: actions/setup-python@v5
|
||||
- name: Setup mdBook
|
||||
uses: peaceiris/actions-mdbook@v2
|
||||
with:
|
||||
python-version: '3.10'
|
||||
cache: 'pip'
|
||||
mdbook-version: 'latest'
|
||||
|
||||
- name: Install pandoc
|
||||
- name: Cache mdbook-typst-math binary
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .mdbook-bin
|
||||
key: mdbook-typst-math-v0.3.0-linux-x86_64
|
||||
|
||||
- name: Run mdBook regression tests
|
||||
run: |
|
||||
wget -q https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-1-amd64.deb
|
||||
sudo dpkg -i pandoc-2.19.2-1-amd64.deb
|
||||
python3 -m unittest discover -s tests -p 'test_prepare_mdbook.py'
|
||||
python3 -m unittest discover -s tests -p 'test_prepare_mdbook_zh.py'
|
||||
python3 -m unittest discover -s tests -p 'test_assemble_docs_publish_tree.py'
|
||||
python3 -m unittest discover -s tests -p 'test_ensure_book_resources.py'
|
||||
python3 -m unittest discover -s tests -p 'test_mdbook_typst_math.py'
|
||||
python3 -m unittest discover -s tests -p 'test_ensure_mdbook_typst_math.py'
|
||||
python3 -m unittest discover -s tests -p 'test_update_docs_workflow.py'
|
||||
|
||||
- name: Install d2lbook
|
||||
run: |
|
||||
git clone https://github.com/openmlsys/d2l-book.git
|
||||
cd d2l-book
|
||||
sed -i "s/'sphinxcontrib-bibtex<2.0.0'/'sphinxcontrib-bibtex>=2.5.0'/" setup.py
|
||||
python3 -m pip install .
|
||||
- name: Build English HTML with mdBook
|
||||
run: bash build_mdbook.sh
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: python3 -m pip install -r requirements.txt
|
||||
|
||||
- name: Build Chinese HTML
|
||||
run: bash build_html_zh.sh
|
||||
|
||||
build:
|
||||
name: build
|
||||
needs: [build-en, build-zh]
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- run: echo "All builds passed"
|
||||
- name: Build Chinese HTML with mdBook
|
||||
run: bash build_mdbook_zh.sh
|
||||
|
||||
66
.github/workflows/update_docs.yml
vendored
66
.github/workflows/update_docs.yml
vendored
@@ -16,47 +16,53 @@ jobs:
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install pandoc
|
||||
- name: Cache mdbook-typst-math binary
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: .mdbook-bin
|
||||
key: mdbook-typst-math-v0.3.0-linux-x86_64
|
||||
|
||||
- name: Setup mdBook
|
||||
uses: peaceiris/actions-mdbook@v2
|
||||
with:
|
||||
mdbook-version: 'latest'
|
||||
|
||||
- name: Run mdBook regression tests
|
||||
run: |
|
||||
wget -q https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-1-amd64.deb
|
||||
sudo dpkg -i pandoc-2.19.2-1-amd64.deb
|
||||
python3 -m unittest discover -s tests -p 'test_prepare_mdbook.py'
|
||||
python3 -m unittest discover -s tests -p 'test_prepare_mdbook_zh.py'
|
||||
python3 -m unittest discover -s tests -p 'test_assemble_docs_publish_tree.py'
|
||||
python3 -m unittest discover -s tests -p 'test_ensure_book_resources.py'
|
||||
python3 -m unittest discover -s tests -p 'test_mdbook_typst_math.py'
|
||||
python3 -m unittest discover -s tests -p 'test_ensure_mdbook_typst_math.py'
|
||||
|
||||
- name: Install d2lbook
|
||||
run: |
|
||||
git clone https://github.com/openmlsys/d2l-book.git
|
||||
cd d2l-book
|
||||
# Fix Python 3.10+ incompatibility: bibtex<2.0.0 depends on oset which
|
||||
# uses collections.MutableSet removed in Python 3.10.
|
||||
sed -i "s/'sphinxcontrib-bibtex<2.0.0'/'sphinxcontrib-bibtex>=2.5.0'/" setup.py
|
||||
python3 -m pip install .
|
||||
- name: Build English HTML with mdBook
|
||||
run: bash build_mdbook.sh
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: python3 -m pip install -r requirements.txt sphinx-mathjax-offline
|
||||
- name: Build Chinese HTML with mdBook
|
||||
run: bash build_mdbook_zh.sh
|
||||
|
||||
- name: Build English HTML
|
||||
run: bash build_html.sh
|
||||
|
||||
- name: Build Chinese HTML
|
||||
run: bash build_html_zh.sh
|
||||
|
||||
- name: Deploy to GitHub Pages
|
||||
- name: Deploy to openmlsys.github.io
|
||||
env:
|
||||
DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
|
||||
run: |
|
||||
git clone https://x-access-token:${DEPLOY_TOKEN}@github.com/openmlsys/openmlsys.github.io.git
|
||||
|
||||
# English → root (default language)
|
||||
cp -r en_chapters/_build/html/* openmlsys.github.io/docs/
|
||||
|
||||
# Chinese → /cn/ subdirectory
|
||||
mkdir -p openmlsys.github.io/docs/cn
|
||||
cp -r zh_chapters/_build/html/* openmlsys.github.io/docs/cn/
|
||||
python3 tools/assemble_docs_publish_tree.py \
|
||||
--destination-root openmlsys.github.io \
|
||||
--docs-subdir docs \
|
||||
--en-source .mdbook/book \
|
||||
--zh-source .mdbook-zh/book
|
||||
|
||||
cd openmlsys.github.io
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "github-actions[bot]@users.noreply.github.com"
|
||||
git add .
|
||||
git commit -m "deploy: update docs (en+zh) from openmlsys-zh@${{ github.sha }}" || echo "No changes to commit"
|
||||
git push
|
||||
git add docs
|
||||
|
||||
if git diff --cached --quiet; then
|
||||
echo "No changes to commit"
|
||||
else
|
||||
git commit -m "deploy: update docs from ${GITHUB_REPOSITORY}@${GITHUB_SHA}"
|
||||
git push
|
||||
fi
|
||||
|
||||
17
.gitignore
vendored
17
.gitignore
vendored
@@ -12,3 +12,20 @@ test*.md
|
||||
run.sh
|
||||
.idea
|
||||
env
|
||||
.mdbook/
|
||||
.mdbook-zh/
|
||||
.mdbook-zh-test/
|
||||
.mdbook-bin/
|
||||
task_plan.md
|
||||
findings.md
|
||||
progress.md
|
||||
d2l-book/
|
||||
docs/
|
||||
en_chapters/img
|
||||
en_chapters/references
|
||||
en_chapters/static
|
||||
en_chapters/mlsys.bib
|
||||
zh_chapters/img
|
||||
zh_chapters/references
|
||||
zh_chapters/static
|
||||
zh_chapters/mlsys.bib
|
||||
|
||||
19
book.toml
Normal file
19
book.toml
Normal file
@@ -0,0 +1,19 @@
|
||||
[book]
|
||||
authors = ["OpenMLSys Contributors"]
|
||||
language = "en"
|
||||
src = "en_chapters"
|
||||
title = "Machine Learning Systems: Design and Implementation"
|
||||
|
||||
[build]
|
||||
build-dir = ".mdbook/book"
|
||||
create-missing = false
|
||||
|
||||
[preprocessor.openmlsys]
|
||||
command = "python3 tools/mdbook_preprocessor.py"
|
||||
|
||||
[preprocessor.typst-math]
|
||||
|
||||
[output.html]
|
||||
git-repository-url = "https://github.com/openmlsys/openmlsys-zh"
|
||||
preferred-dark-theme = "navy"
|
||||
additional-css = ["theme/dark-mode-images.css", "theme/typst.css"]
|
||||
19
books/zh/book.toml
Normal file
19
books/zh/book.toml
Normal file
@@ -0,0 +1,19 @@
|
||||
[book]
|
||||
authors = ["OpenMLSys Contributors"]
|
||||
language = "zh-CN"
|
||||
src = "../../zh_chapters"
|
||||
title = "机器学习系统:设计和实现"
|
||||
|
||||
[build]
|
||||
build-dir = "../../.mdbook-zh/book"
|
||||
create-missing = false
|
||||
|
||||
[preprocessor.openmlsys-zh]
|
||||
command = "python3 ../../tools/mdbook_zh_preprocessor.py"
|
||||
|
||||
[preprocessor.typst-math]
|
||||
|
||||
[output.html]
|
||||
git-repository-url = "https://github.com/openmlsys/openmlsys-zh"
|
||||
preferred-dark-theme = "navy"
|
||||
additional-css = ["theme/dark-mode-images.css", "theme/typst.css"]
|
||||
16
books/zh/theme/dark-mode-images.css
Normal file
16
books/zh/theme/dark-mode-images.css
Normal file
@@ -0,0 +1,16 @@
|
||||
/* 暗色模式下仅为正文图片添加浅灰色背景,提高透明背景图片的可读性 */
|
||||
.navy .content main img,
|
||||
.coal .content main img,
|
||||
.ayu .content main img {
|
||||
background-color: #e8e8e8;
|
||||
border-radius: 4px;
|
||||
padding: 8px;
|
||||
}
|
||||
|
||||
/* 首页 frontpage 图片保持透明,不添加正文图像底色。 */
|
||||
.navy .openmlsys-frontpage img,
|
||||
.coal .openmlsys-frontpage img,
|
||||
.ayu .openmlsys-frontpage img {
|
||||
background-color: transparent !important;
|
||||
padding: 0 !important;
|
||||
}
|
||||
16
books/zh/theme/typst.css
Normal file
16
books/zh/theme/typst.css
Normal file
@@ -0,0 +1,16 @@
|
||||
.typst-inline {
|
||||
display: inline-flex;
|
||||
vertical-align: -0.2em;
|
||||
}
|
||||
|
||||
.typst-display {
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
margin: 1rem 0;
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
.typst-doc {
|
||||
color: var(--fg);
|
||||
max-width: 100%;
|
||||
}
|
||||
@@ -10,12 +10,7 @@ set -e
|
||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
|
||||
# ── Create resource symlinks ──────────────────────────────────────────────────
|
||||
for target in img references static mlsys.bib; do
|
||||
link="$ROOT/en_chapters/$target"
|
||||
if [ ! -e "$link" ]; then
|
||||
ln -sf "$ROOT/$target" "$link"
|
||||
fi
|
||||
done
|
||||
python3 "$ROOT/tools/ensure_book_resources.py" --chapter-dir "$ROOT/en_chapters"
|
||||
|
||||
# ── Build ─────────────────────────────────────────────────────────────────────
|
||||
cd "$ROOT/en_chapters"
|
||||
|
||||
@@ -10,12 +10,7 @@ set -e
|
||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
|
||||
# ── Create resource symlinks ──────────────────────────────────────────────────
|
||||
for target in img references static mlsys.bib; do
|
||||
link="$ROOT/zh_chapters/$target"
|
||||
if [ ! -e "$link" ]; then
|
||||
ln -sf "$ROOT/$target" "$link"
|
||||
fi
|
||||
done
|
||||
python3 "$ROOT/tools/ensure_book_resources.py" --chapter-dir "$ROOT/zh_chapters"
|
||||
|
||||
# ── Build ─────────────────────────────────────────────────────────────────────
|
||||
cd "$ROOT/zh_chapters"
|
||||
|
||||
27
build_mdbook.sh
Normal file
27
build_mdbook.sh
Normal file
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PYTHON_BIN="$(command -v python3 || command -v python || true)"
|
||||
|
||||
if [[ -z "${PYTHON_BIN}" ]]; then
|
||||
echo "Python is required to prepare the mdBook staging tree." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v mdbook >/dev/null 2>&1; then
|
||||
echo "mdbook is not installed. Install it first, for example with: cargo install mdbook" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
MDBOOK_TYPST_MATH_BIN_DIR="${ROOT}/.mdbook-bin"
|
||||
"${PYTHON_BIN}" "${ROOT}/tools/ensure_mdbook_typst_math.py" --output-dir "${MDBOOK_TYPST_MATH_BIN_DIR}" >/dev/null
|
||||
export PATH="${MDBOOK_TYPST_MATH_BIN_DIR}:${PATH}"
|
||||
|
||||
"${PYTHON_BIN}" "${ROOT}/tools/ensure_book_resources.py" --chapter-dir "${ROOT}/en_chapters"
|
||||
"${PYTHON_BIN}" "${ROOT}/tools/prepare_mdbook.py" \
|
||||
--source "${ROOT}/en_chapters" \
|
||||
--summary-output "${ROOT}/en_chapters/SUMMARY.md" \
|
||||
--placeholder-prefix "[TODO: src = zh_chapters/"
|
||||
|
||||
mdbook build "${ROOT}"
|
||||
29
build_mdbook_zh.sh
Executable file
29
build_mdbook_zh.sh
Executable file
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PYTHON_BIN="$(command -v python3 || command -v python || true)"
|
||||
|
||||
if [[ -z "${PYTHON_BIN}" ]]; then
|
||||
echo "Python is required to prepare the mdBook staging tree." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v mdbook >/dev/null 2>&1; then
|
||||
echo "mdbook is not installed. Install it first, for example with: cargo install mdbook" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
MDBOOK_TYPST_MATH_BIN_DIR="${ROOT}/.mdbook-bin"
|
||||
"${PYTHON_BIN}" "${ROOT}/tools/ensure_mdbook_typst_math.py" --output-dir "${MDBOOK_TYPST_MATH_BIN_DIR}" >/dev/null
|
||||
export PATH="${MDBOOK_TYPST_MATH_BIN_DIR}:${PATH}"
|
||||
|
||||
# ── Create resource links ─────────────────────────────────────────────────────
|
||||
"${PYTHON_BIN}" "${ROOT}/tools/ensure_book_resources.py" --chapter-dir "${ROOT}/zh_chapters"
|
||||
|
||||
# ── Build ─────────────────────────────────────────────────────────────────────
|
||||
"${PYTHON_BIN}" "${ROOT}/tools/prepare_mdbook_zh.py" \
|
||||
--source "${ROOT}/zh_chapters" \
|
||||
--summary-output "${ROOT}/zh_chapters/SUMMARY.md"
|
||||
|
||||
mdbook build "${ROOT}/books/zh"
|
||||
3
en_chapters/SUMMARY.md
Normal file
3
en_chapters/SUMMARY.md
Normal file
@@ -0,0 +1,3 @@
|
||||
# Summary
|
||||
|
||||
[Machine Learning Systems: Design and Implementation](index.md)
|
||||
507
en_chapters/frontpage.html
Normal file
507
en_chapters/frontpage.html
Normal file
@@ -0,0 +1,507 @@
|
||||
<head>
|
||||
<style>
|
||||
h1, .side-doc-outline {
|
||||
display: none;
|
||||
}
|
||||
.document .page-content {
|
||||
width: 100%;
|
||||
}
|
||||
h2, h3, h4 {
|
||||
letter-spacing: 2px;
|
||||
}
|
||||
.mdl-grid {
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
text-align: center;
|
||||
padding: 150px 0 0 0;
|
||||
}
|
||||
h2.toc {
|
||||
padding: 150px 0 0 0;
|
||||
}
|
||||
|
||||
.header h2 {
|
||||
font-size: 2rem;
|
||||
padding-bottom: 35px;
|
||||
}
|
||||
|
||||
.header p {
|
||||
font-size: 1.25rem;
|
||||
letter-spacing: 0px;
|
||||
line-height: 1.5rempx;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
|
||||
.header .mdl-button {
|
||||
margin: 0px 10px;
|
||||
letter-spacing: 2px;
|
||||
font-size: 16px;
|
||||
margin: 1rem 1.5rem;
|
||||
height: 40px;
|
||||
width: 120px;
|
||||
}
|
||||
.author-item {
|
||||
max-width: 300px;
|
||||
width: 100%;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
}
|
||||
.author-item h3 {
|
||||
padding-top: 25px;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
.author-item img {
|
||||
border-radius: 50%;
|
||||
width: 120px;
|
||||
height: 120px;
|
||||
object-fit: cover;
|
||||
}
|
||||
.authors.mdl-grid {
|
||||
align-items: flex-start;
|
||||
row-gap: calc(24px + 1.5em);
|
||||
}
|
||||
.authors .mdl-cell:not(.author-group-title) {
|
||||
display: flex;
|
||||
align-items: flex-start;
|
||||
justify-content: center;
|
||||
}
|
||||
.author-group-title {
|
||||
display: block;
|
||||
width: 100%;
|
||||
}
|
||||
.author-group-title h3,
|
||||
.author-group-title h4 {
|
||||
width: 100%;
|
||||
margin: 0 0 0.75em;
|
||||
}
|
||||
.author-group-title h4:last-child {
|
||||
margin-bottom: 0;
|
||||
}
|
||||
.authors h4 {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.authors h2 {
|
||||
padding-bottom: 18px;
|
||||
}
|
||||
|
||||
.running-item {
|
||||
max-width: 300px;
|
||||
}
|
||||
|
||||
.running-item img {
|
||||
height: 40px;
|
||||
}
|
||||
|
||||
.running-item p {
|
||||
padding: 20px;
|
||||
}
|
||||
|
||||
.running.mdl-grid {
|
||||
padding: 20px 0 0 0;
|
||||
}
|
||||
|
||||
.logos img {
|
||||
display: inline;
|
||||
max-height: 70px;
|
||||
max-width: 100px;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
}
|
||||
|
||||
.logos {
|
||||
padding-top: 1em;
|
||||
padding-bottom: 2em;
|
||||
}
|
||||
|
||||
.logos .mdl-grid {
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.features .logos .mdl-cell {
|
||||
padding: 10px 0;
|
||||
}
|
||||
|
||||
|
||||
.features .mdl-cell {
|
||||
padding: 0 36px;
|
||||
}
|
||||
.features h2, .features-2 h2 {
|
||||
padding-bottom: 24px;
|
||||
}
|
||||
.features-2 p {
|
||||
padding: 0 200px;
|
||||
}
|
||||
.institute.mdl-grid {
|
||||
padding: 24px;
|
||||
}
|
||||
.institute .mdl-cell {
|
||||
text-align: left;
|
||||
padding: 0;
|
||||
}
|
||||
#logoimg {
|
||||
padding-top: 0em;
|
||||
padding-bottom: 2em;
|
||||
width: 100%;
|
||||
}
|
||||
#map {
|
||||
height: 400px;
|
||||
width: 100%;
|
||||
}
|
||||
#mapimg {
|
||||
width: 100%;
|
||||
}
|
||||
a {
|
||||
font-size: normal;
|
||||
}
|
||||
|
||||
@media (max-width: 1000px) {
|
||||
.mdl-grid, h2.toc {
|
||||
padding: 50px 0 0 0;
|
||||
}
|
||||
.header img, .features-2 img {
|
||||
max-width: 300px;
|
||||
}
|
||||
.header .mdl-cell--1-col {
|
||||
display: none;
|
||||
}
|
||||
.features img {
|
||||
max-width: 350px;
|
||||
}
|
||||
.features-2 p {
|
||||
padding: 0 24px;
|
||||
}
|
||||
.features .mdl-cell {
|
||||
padding: 0 8px;
|
||||
}
|
||||
}
|
||||
|
||||
.collapsible {
|
||||
background-color: #EEE;
|
||||
color: black;
|
||||
cursor: pointer;
|
||||
padding: 6px;
|
||||
width: 100%;
|
||||
border: none;
|
||||
text-align: left;
|
||||
outline: none;
|
||||
font-size: 16px;
|
||||
}
|
||||
.collapsible-active, .collapsible:hover {
|
||||
background-color: #DDD;
|
||||
}
|
||||
.expansion {
|
||||
padding: 3px 6px;
|
||||
max-height: 0;
|
||||
overflow: hidden;
|
||||
transition: max-height 0.2s ease-out;
|
||||
}
|
||||
|
||||
.cover h2 {
|
||||
font-size: 34px;
|
||||
margin-bottom: 0px;
|
||||
padding-bottom: 20px;
|
||||
}
|
||||
|
||||
.cover p.version {
|
||||
font-size: 18px;
|
||||
padding: 0 0 9px 0;
|
||||
|
||||
}
|
||||
|
||||
.cover p {
|
||||
margin: 0;
|
||||
padding: 20px 0 0px 0;
|
||||
font-size: 20px;
|
||||
}
|
||||
|
||||
</style>
|
||||
</head>
|
||||
|
||||
<div class="mdl-grid header">
|
||||
<div class="mdl-cell mdl-cell--3-col">
|
||||
<img src="_images/logo.png" class="img-fluid" alt="">
|
||||
</div>
|
||||
<div class="mdl-cell mdl-cell--1-col">
|
||||
</div>
|
||||
<div class="mdl-cell mdl-cell--5-col mdl-cell--middle cover">
|
||||
<h2 class="mdl-color-text--primary">Machine Learning Systems: Design and Implementation</h2>
|
||||
<p>The first open-source book to comprehensively cover machine learning systems</p>
|
||||
<p><a class="github-button" href="https://github.com/openmlsys/openmlsys-zh" data-icon="octicon-star" data-size="large" data-show-count="true" aria-label="Star openmlsys/openmlsys-zh on GitHub">Star</a></p>
|
||||
<!-- OPENMLSYS_LANGUAGE_SWITCH -->
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<div class = "authors mdl-grid" id = "author" >
|
||||
<div class = "author-group-title mdl-cell mdl-cell--12-col mdl-cell--top">
|
||||
<h2>Core Authors</h2>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/1136455?v=4"/>
|
||||
<h3><a href="https://github.com/luomai">Mai Luo</a></h3>
|
||||
<p>University of Edinburgh</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/10713581?v=4"/>
|
||||
<h3><a href="https://github.com/zsdonghao">Hao Dong</a></h3>
|
||||
<p>Peking University, Peng Cheng Laboratory</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="./_images/jinxuefeng.png"/>
|
||||
<h3>Xuefeng Jin</h3>
|
||||
<p>MindSpore Architect</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/90942796?v=4"/>
|
||||
<h3><a href="https://github.com/ganzhiliang">Zhiliang Gan</a></h3>
|
||||
<p>MindSpore Architect</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class = "author-group-title mdl-cell mdl-cell--12-col mdl-cell--top">
|
||||
<h2>Chapter Authors</h2>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/21324004?v=4"/>
|
||||
<h3><a href="https://github.com/Laicheng0830">Cheng Lai</a></h3>
|
||||
<p>Peng Cheng Laboratory</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/73918561?v=4"/>
|
||||
<h3><a href="https://github.com/hanjr92">Jiarong Han</a></h3>
|
||||
<p>Peng Cheng Laboratory</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/31511840?v=4"/>
|
||||
<h3><a href="https://github.com/future-xy">Yao Fu</a></h3>
|
||||
<p>University of Edinburgh</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/39682259?v=4"/>
|
||||
<h3><a href="https://github.com/eedalong">Xiulong Yuan</a></h3>
|
||||
<p>Tsinghua University</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/22659010?v=4"/>
|
||||
<h3><a href="https://github.com/quantumiracle">Zihan Ding</a></h3>
|
||||
<p>Princeton University</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/25477396?v=4"/>
|
||||
<h3><a href="https://github.com/Jiankai-Sun">Jiankai Sun</a></h3>
|
||||
<p>Stanford University</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/34192321?v=4"/>
|
||||
<h3><a href="https://liaopeiyuan.com/">Peiyuan Liao</a></h3>
|
||||
<p>Carnegie Mellon University</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/91313652?v=4"/>
|
||||
<h3><a href="https://github.com/Went-Liang">Wenteng Liang</a></h3>
|
||||
<p>Beijing University of Posts and Telecommunications</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/49299030?v=4"/>
|
||||
<h3><a href="https://github.com/JieRen98">Jie Ren</a></h3>
|
||||
<p>University of Edinburgh</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="./_images/wanghanchen.png"/>
|
||||
<h3><a href="https://github.com/hansen7">Hanchen Wang</a></h3>
|
||||
<p>University of Cambridge</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/10811136?v=4"/>
|
||||
<h3><a href="https://github.com/ds1231h">Pei Mu</a></h3>
|
||||
<p>University of Edinburgh</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="./_images/zhangqinghua.png"/>
|
||||
<h3>Qinghua Zhang</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/18185293?v=4"/>
|
||||
<h3>Zhibo Liang</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/20573763?v=4"/>
|
||||
<h3>Jianfeng Yu</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/5563661?v=4"/>
|
||||
<h3>Jinjin Chu</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/68210792?v=4"/>
|
||||
<h3>Fubi Cai</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="./_images/zhangrenwei.png"/>
|
||||
<h3>Renwei Zhang</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/2681256?v=4"/>
|
||||
<h3>Chao Liu</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/5202350?v=4"/>
|
||||
<h3>Gang Chen</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/61316635?v=4"/>
|
||||
<h3>Mingqi Li</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/36526001?v=4"/>
|
||||
<h3>Gangqiang Han</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/55628940?v=4"/>
|
||||
<h3>Yehui Tang</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="./_images/zhaizhiqiang.png"/>
|
||||
<h3>Zhiqiang Zhai</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="./_images/wutiancheng.png"/>
|
||||
<h3>Tiancheng Wu</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/59854698?v=4"/>
|
||||
<h3>Xiaohui Li</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/20376974?v=4"/>
|
||||
<h3>Haoyang Li</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/17228522?v=4"/>
|
||||
<h3>Zhipeng Tan</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="https://avatars.githubusercontent.com/u/105415060?v=4"/>
|
||||
<h3>Shanni Li</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class = "mdl-cell mdl-cell--3-col mdl-cell--top">
|
||||
<div class="author-item">
|
||||
<img src="./_images/guozhijian.png"/>
|
||||
<h3>Zhijian Guo</h3>
|
||||
<p>Huawei Engineer</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<div class = "author-group-title mdl-cell mdl-cell--12-col mdl-cell--top">
|
||||
<h3> Thanks to our <a href="https://github.com/openmlsys/openmlsys-zh/graphs/contributors">contributors</a></h3>
|
||||
<h3> and everyone who reported valuable <a href="https://github.com/openmlsys/openmlsys-zh/issues">errata and suggestions</a></h3>
|
||||
<h4><a href="https://github.com/openmlsys/openmlsys-zh">Contribute to this Book</a></h4>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<h2 class="toc">Table of Contents</h2>
|
||||
|
||||
<script>
|
||||
var coll = document.getElementsByClassName("collapsible");
|
||||
var i;
|
||||
for (i = 0; i < coll.length; i++) {
|
||||
coll[i].addEventListener("click", function() {
|
||||
this.classList.toggle("collapsible-active");
|
||||
var content = this.nextElementSibling;
|
||||
if (content.style.maxHeight){
|
||||
content.style.maxHeight = null;
|
||||
} else {
|
||||
content.style.maxHeight = content.scrollHeight + "px";
|
||||
}
|
||||
});
|
||||
}
|
||||
</script>
|
||||
|
||||
<script async defer src="https://buttons.github.io/buttons.js"></script>
|
||||
@@ -1 +0,0 @@
|
||||
/chivier-disk/hyq-home/Projects/openmlsys-zh/img
|
||||
@@ -1 +0,0 @@
|
||||
/chivier-disk/hyq-home/Projects/openmlsys-zh/references
|
||||
@@ -1 +0,0 @@
|
||||
/chivier-disk/hyq-home/Projects/openmlsys-zh/static
|
||||
@@ -201,6 +201,7 @@ a {
|
||||
<h2 class="mdl-color-text--primary">《机器学习系统:设计和实现》</h2>
|
||||
<p>做世界上第一本全面讲述机器学习系统知识的开源书籍</p>
|
||||
<p><a class="github-button" href="https://github.com/openmlsys/openmlsys-zh" data-icon="octicon-star" data-size="large" data-show-count="true" aria-label="Star openmlsys/openmlsys-zh on GitHub">Star</a></p>
|
||||
<!-- OPENMLSYS_LANGUAGE_SWITCH -->
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -477,4 +478,3 @@ for (i = 0; i < coll.length; i++) {
|
||||
</script>
|
||||
|
||||
<script async defer src="https://buttons.github.io/buttons.js"></script>
|
||||
|
||||
|
||||
87
tests/test_assemble_docs_publish_tree.py
Normal file
87
tests/test_assemble_docs_publish_tree.py
Normal file
@@ -0,0 +1,87 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from tools.assemble_docs_publish_tree import assemble_publish_tree
|
||||
|
||||
|
||||
class AssembleDocsPublishTreeTests(unittest.TestCase):
|
||||
def test_assemble_publish_tree_uses_legacy_docs_layout(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
pages_repo = root / "pages"
|
||||
en_source = root / "en-html"
|
||||
zh_source = root / "zh-html"
|
||||
|
||||
pages_repo.mkdir()
|
||||
en_source.mkdir()
|
||||
zh_source.mkdir()
|
||||
|
||||
(en_source / "index.html").write_text("english home", encoding="utf-8")
|
||||
(en_source / "guide.html").write_text("english guide", encoding="utf-8")
|
||||
(zh_source / "index.html").write_text("chinese home", encoding="utf-8")
|
||||
(zh_source / "searchindex.js").write_text("zh search", encoding="utf-8")
|
||||
|
||||
assemble_publish_tree(
|
||||
destination_root=pages_repo,
|
||||
docs_subdir="docs",
|
||||
en_source=en_source,
|
||||
zh_source=zh_source,
|
||||
)
|
||||
|
||||
self.assertEqual(
|
||||
(pages_repo / "docs" / "index.html").read_text(encoding="utf-8"),
|
||||
"english home",
|
||||
)
|
||||
self.assertEqual(
|
||||
(pages_repo / "docs" / "guide.html").read_text(encoding="utf-8"),
|
||||
"english guide",
|
||||
)
|
||||
self.assertEqual(
|
||||
(pages_repo / "docs" / "cn" / "index.html").read_text(encoding="utf-8"),
|
||||
"chinese home",
|
||||
)
|
||||
self.assertEqual(
|
||||
(pages_repo / "docs" / "cn" / "searchindex.js").read_text(encoding="utf-8"),
|
||||
"zh search",
|
||||
)
|
||||
|
||||
def test_assemble_publish_tree_replaces_stale_docs_content(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
pages_repo = root / "pages"
|
||||
en_source = root / "en-html"
|
||||
zh_source = root / "zh-html"
|
||||
|
||||
(pages_repo / "docs" / "cn").mkdir(parents=True)
|
||||
(pages_repo / "docs" / "old.html").write_text("stale en", encoding="utf-8")
|
||||
(pages_repo / "docs" / "cn" / "old.html").write_text("stale zh", encoding="utf-8")
|
||||
|
||||
en_source.mkdir()
|
||||
zh_source.mkdir()
|
||||
(en_source / "index.html").write_text("fresh en", encoding="utf-8")
|
||||
(zh_source / "index.html").write_text("fresh zh", encoding="utf-8")
|
||||
|
||||
assemble_publish_tree(
|
||||
destination_root=pages_repo,
|
||||
docs_subdir="docs",
|
||||
en_source=en_source,
|
||||
zh_source=zh_source,
|
||||
)
|
||||
|
||||
self.assertFalse((pages_repo / "docs" / "old.html").exists())
|
||||
self.assertFalse((pages_repo / "docs" / "cn" / "old.html").exists())
|
||||
self.assertEqual(
|
||||
(pages_repo / "docs" / "index.html").read_text(encoding="utf-8"),
|
||||
"fresh en",
|
||||
)
|
||||
self.assertEqual(
|
||||
(pages_repo / "docs" / "cn" / "index.html").read_text(encoding="utf-8"),
|
||||
"fresh zh",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
30
tests/test_dark_mode_images_css.py
Normal file
30
tests/test_dark_mode_images_css.py
Normal file
@@ -0,0 +1,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
class DarkModeImagesCssTests(unittest.TestCase):
|
||||
def test_both_theme_css_files_style_dark_mode_body_images_only(self) -> None:
|
||||
css_paths = [
|
||||
REPO_ROOT / "theme" / "dark-mode-images.css",
|
||||
REPO_ROOT / "books" / "zh" / "theme" / "dark-mode-images.css",
|
||||
]
|
||||
|
||||
for css_path in css_paths:
|
||||
css = css_path.read_text(encoding="utf-8")
|
||||
|
||||
self.assertIn(".navy .content main img", css, css_path.as_posix())
|
||||
self.assertIn(".coal .content main img", css, css_path.as_posix())
|
||||
self.assertIn(".ayu .content main img", css, css_path.as_posix())
|
||||
self.assertIn("background-color: #e8e8e8;", css, css_path.as_posix())
|
||||
self.assertIn(".openmlsys-frontpage img", css, css_path.as_posix())
|
||||
self.assertIn("background-color: transparent !important;", css, css_path.as_posix())
|
||||
self.assertIn("padding: 0 !important;", css, css_path.as_posix())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
55
tests/test_ensure_book_resources.py
Normal file
55
tests/test_ensure_book_resources.py
Normal file
@@ -0,0 +1,55 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from tools.ensure_book_resources import ensure_resource_views
|
||||
|
||||
|
||||
class EnsureBookResourcesTests(unittest.TestCase):
|
||||
def test_ensure_resource_views_creates_missing_symlinks(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
chapter_dir = root / "en_chapters"
|
||||
chapter_dir.mkdir()
|
||||
|
||||
for directory in ("img", "references", "static"):
|
||||
(root / directory).mkdir()
|
||||
(root / "mlsys.bib").write_text("bib", encoding="utf-8")
|
||||
|
||||
ensure_resource_views(chapter_dir, root)
|
||||
|
||||
for name in ("img", "references", "static", "mlsys.bib"):
|
||||
path = chapter_dir / name
|
||||
self.assertTrue(path.is_symlink(), f"{name} should be a symlink")
|
||||
self.assertEqual(path.resolve(), (root / name).resolve())
|
||||
|
||||
def test_ensure_resource_views_keeps_existing_non_symlink_paths(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
chapter_dir = root / "en_chapters"
|
||||
chapter_dir.mkdir()
|
||||
|
||||
for directory in ("img", "references", "static"):
|
||||
(root / directory).mkdir()
|
||||
(root / "mlsys.bib").write_text("root bib", encoding="utf-8")
|
||||
|
||||
local_bib = chapter_dir / "mlsys.bib"
|
||||
local_bib.write_text("local bib", encoding="utf-8")
|
||||
local_static = chapter_dir / "static"
|
||||
local_static.mkdir()
|
||||
(local_static / "frontpage.html").write_text("local static", encoding="utf-8")
|
||||
|
||||
ensure_resource_views(chapter_dir, root)
|
||||
|
||||
self.assertFalse(local_bib.is_symlink())
|
||||
self.assertEqual(local_bib.read_text(encoding="utf-8"), "local bib")
|
||||
self.assertFalse(local_static.is_symlink())
|
||||
self.assertTrue((local_static / "frontpage.html").exists())
|
||||
self.assertTrue((chapter_dir / "img").is_symlink())
|
||||
self.assertTrue((chapter_dir / "references").is_symlink())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
134
tests/test_ensure_mdbook_typst_math.py
Normal file
134
tests/test_ensure_mdbook_typst_math.py
Normal file
@@ -0,0 +1,134 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import hashlib
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from tools.ensure_mdbook_typst_math import (
|
||||
ASSET_SHA256,
|
||||
VERSION,
|
||||
build_download_url,
|
||||
ensure_binary,
|
||||
resolve_asset_name,
|
||||
resolve_binary_path,
|
||||
resolve_version_path,
|
||||
)
|
||||
|
||||
|
||||
class ResolveAssetNameTests(unittest.TestCase):
|
||||
def test_resolve_asset_name_for_supported_targets(self) -> None:
|
||||
self.assertEqual(
|
||||
resolve_asset_name(system="Darwin", machine="arm64"),
|
||||
"mdbook-typst-math-aarch64-apple-darwin.gz",
|
||||
)
|
||||
self.assertEqual(
|
||||
resolve_asset_name(system="Darwin", machine="x86_64"),
|
||||
"mdbook-typst-math-x86_64-apple-darwin.gz",
|
||||
)
|
||||
self.assertEqual(
|
||||
resolve_asset_name(system="Linux", machine="aarch64"),
|
||||
"mdbook-typst-math-aarch64-unknown-linux-gnu.gz",
|
||||
)
|
||||
self.assertEqual(
|
||||
resolve_asset_name(system="Linux", machine="AMD64"),
|
||||
"mdbook-typst-math-x86_64-unknown-linux-gnu.gz",
|
||||
)
|
||||
self.assertEqual(
|
||||
resolve_asset_name(system="Windows", machine="AMD64"),
|
||||
"mdbook-typst-math-x86_64-pc-windows-msvc.exe",
|
||||
)
|
||||
|
||||
def test_resolve_asset_name_rejects_unsupported_targets(self) -> None:
|
||||
with self.assertRaises(ValueError):
|
||||
resolve_asset_name(system="Linux", machine="riscv64")
|
||||
|
||||
|
||||
class EnsureBinaryTests(unittest.TestCase):
|
||||
def test_ensure_binary_downloads_and_extracts_gzip_release(self) -> None:
|
||||
payload = b"linux-binary"
|
||||
asset_name = "mdbook-typst-math-x86_64-unknown-linux-gnu.gz"
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
output_dir = Path(tmpdir)
|
||||
urls: list[str] = []
|
||||
|
||||
def fake_downloader(url: str) -> bytes:
|
||||
urls.append(url)
|
||||
return gzip.compress(payload)
|
||||
|
||||
with patch.dict(ASSET_SHA256, {asset_name: hashlib.sha256(gzip.compress(payload)).hexdigest()}):
|
||||
binary_path = ensure_binary(
|
||||
output_dir,
|
||||
system="Linux",
|
||||
machine="x86_64",
|
||||
downloader=fake_downloader,
|
||||
)
|
||||
|
||||
self.assertEqual(binary_path, resolve_binary_path(output_dir, VERSION, asset_name))
|
||||
self.assertEqual(binary_path.name, "mdbook-typst-math")
|
||||
self.assertEqual(binary_path.read_bytes(), payload)
|
||||
self.assertEqual(resolve_version_path(output_dir).read_text(encoding="utf-8"), VERSION)
|
||||
self.assertEqual(urls, [build_download_url(VERSION, asset_name)])
|
||||
self.assertTrue(os.access(binary_path, os.X_OK))
|
||||
|
||||
def test_ensure_binary_uses_cached_file_without_downloading(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
output_dir = Path(tmpdir)
|
||||
asset_name = "mdbook-typst-math-x86_64-unknown-linux-gnu.gz"
|
||||
cached_binary = resolve_binary_path(output_dir, VERSION, asset_name)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
cached_binary.write_bytes(b"cached")
|
||||
cached_binary.chmod(0o755)
|
||||
resolve_version_path(output_dir).write_text(VERSION, encoding="utf-8")
|
||||
|
||||
def fail_downloader(_: str) -> bytes:
|
||||
raise AssertionError("downloader should not be called for cached binary")
|
||||
|
||||
binary_path = ensure_binary(
|
||||
output_dir,
|
||||
system="Linux",
|
||||
machine="x86_64",
|
||||
downloader=fail_downloader,
|
||||
)
|
||||
|
||||
self.assertEqual(binary_path, cached_binary)
|
||||
self.assertEqual(binary_path.read_bytes(), b"cached")
|
||||
|
||||
def test_ensure_binary_keeps_windows_extension(self) -> None:
|
||||
payload = b"windows-binary"
|
||||
asset_name = "mdbook-typst-math-x86_64-pc-windows-msvc.exe"
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
output_dir = Path(tmpdir)
|
||||
|
||||
def fake_downloader(_: str) -> bytes:
|
||||
return payload
|
||||
|
||||
with patch.dict(ASSET_SHA256, {asset_name: hashlib.sha256(payload).hexdigest()}):
|
||||
binary_path = ensure_binary(
|
||||
output_dir,
|
||||
system="Windows",
|
||||
machine="AMD64",
|
||||
downloader=fake_downloader,
|
||||
)
|
||||
|
||||
self.assertEqual(binary_path.name, "mdbook-typst-math.exe")
|
||||
self.assertEqual(binary_path.read_bytes(), payload)
|
||||
|
||||
def test_ensure_binary_rejects_checksum_mismatch(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
with self.assertRaises(ValueError):
|
||||
ensure_binary(
|
||||
Path(tmpdir),
|
||||
system="Linux",
|
||||
machine="x86_64",
|
||||
downloader=lambda _: gzip.compress(b"bad-binary"),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
38
tests/test_mdbook_typst_math.py
Normal file
38
tests/test_mdbook_typst_math.py
Normal file
@@ -0,0 +1,38 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
BOOK_PATHS = (
|
||||
REPO_ROOT / "book.toml",
|
||||
REPO_ROOT / "books" / "zh" / "book.toml",
|
||||
)
|
||||
BUILD_SCRIPTS = (
|
||||
REPO_ROOT / "build_mdbook.sh",
|
||||
REPO_ROOT / "build_mdbook_zh.sh",
|
||||
)
|
||||
|
||||
|
||||
class MdBookTypstMathConfigTests(unittest.TestCase):
|
||||
def test_books_use_typst_math_without_mathjax(self) -> None:
|
||||
for path in BOOK_PATHS:
|
||||
config = path.read_text(encoding="utf-8")
|
||||
|
||||
self.assertIn("[preprocessor.typst-math]", config, path.as_posix())
|
||||
self.assertIn("theme/typst.css", config, path.as_posix())
|
||||
self.assertNotIn("mathjax-support = true", config, path.as_posix())
|
||||
|
||||
def test_build_scripts_bootstrap_prebuilt_typst_math_binary(self) -> None:
|
||||
for path in BUILD_SCRIPTS:
|
||||
script = path.read_text(encoding="utf-8")
|
||||
|
||||
self.assertIn("ensure_mdbook_typst_math.py", script, path.as_posix())
|
||||
self.assertIn("MDBOOK_TYPST_MATH_BIN_DIR", script, path.as_posix())
|
||||
self.assertIn("export PATH=", script, path.as_posix())
|
||||
self.assertNotIn("cargo install mdbook-typst-math", script, path.as_posix())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
237
tests/test_prepare_mdbook.py
Normal file
237
tests/test_prepare_mdbook.py
Normal file
@@ -0,0 +1,237 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from tools.prepare_mdbook import build_title_cache, rewrite_markdown, write_summary
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
class PrepareMdBookTests(unittest.TestCase):
|
||||
def test_write_summary_skips_placeholder_pages(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
source = root / "en_chapters"
|
||||
source.mkdir()
|
||||
|
||||
(source / "index.md").write_text(
|
||||
"""Machine Learning Systems
|
||||
========================
|
||||
|
||||
```toc
|
||||
:maxdepth: 2
|
||||
|
||||
chapter_preface/index
|
||||
chapter_introduction/index
|
||||
```
|
||||
|
||||
```toc
|
||||
:maxdepth: 1
|
||||
|
||||
appendix/index
|
||||
```
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
chapter_preface = source / "chapter_preface"
|
||||
chapter_preface.mkdir()
|
||||
(chapter_preface / "index.md").write_text(
|
||||
"[TODO: src = zh_chapters/chapter_preface/index.md]\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
chapter_intro = source / "chapter_introduction"
|
||||
chapter_intro.mkdir()
|
||||
(chapter_intro / "index.md").write_text("# Introduction\n", encoding="utf-8")
|
||||
|
||||
appendix = source / "appendix"
|
||||
appendix.mkdir()
|
||||
(appendix / "index.md").write_text("# Appendix\n", encoding="utf-8")
|
||||
|
||||
summary_path = write_summary(
|
||||
source,
|
||||
placeholder_prefix="[TODO: src = zh_chapters/",
|
||||
)
|
||||
summary = summary_path.read_text(encoding="utf-8")
|
||||
|
||||
self.assertEqual(
|
||||
summary,
|
||||
"""# Summary
|
||||
|
||||
[Machine Learning Systems](index.md)
|
||||
[Introduction](chapter_introduction/index.md)
|
||||
[Appendix](appendix/index.md)
|
||||
""",
|
||||
)
|
||||
|
||||
title_cache = build_title_cache(
|
||||
source,
|
||||
placeholder_prefix="[TODO: src = zh_chapters/",
|
||||
)
|
||||
rewritten = rewrite_markdown(
|
||||
(source / "index.md").read_text(encoding="utf-8"),
|
||||
(source / "index.md").resolve(),
|
||||
title_cache,
|
||||
)
|
||||
|
||||
self.assertIn("- [Introduction](chapter_introduction/index.md)", rewritten)
|
||||
self.assertIn("- [Appendix](appendix/index.md)", rewritten)
|
||||
self.assertNotIn("chapter_preface/index.md", rewritten)
|
||||
|
||||
def test_rewrite_markdown_uses_configured_bibliography_title(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
page = root / "chapter.md"
|
||||
page.write_text(
|
||||
"""# Introduction
|
||||
|
||||
Reference :cite:`smith2024`.
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
rewritten = rewrite_markdown(
|
||||
page.read_text(encoding="utf-8"),
|
||||
page.resolve(),
|
||||
{page.resolve(): "Introduction"},
|
||||
bib_db={
|
||||
"smith2024": {
|
||||
"author": "Smith, Alice and Doe, Bob",
|
||||
"title": "Systems Paper",
|
||||
"year": "2024",
|
||||
"journal": "ML Systems Journal",
|
||||
}
|
||||
},
|
||||
bibliography_title="References",
|
||||
)
|
||||
|
||||
self.assertIn("## References", rewritten)
|
||||
self.assertNotIn("## 参考文献", rewritten)
|
||||
|
||||
def test_rewrite_markdown_inlines_frontpage_with_language_switch(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
source = root / "en_chapters"
|
||||
static_dir = source / "static"
|
||||
static_dir.mkdir(parents=True)
|
||||
|
||||
index = source / "index.md"
|
||||
index.write_text(
|
||||
"""# Home
|
||||
|
||||
```eval_rst
|
||||
.. raw:: html
|
||||
:file: frontpage.html
|
||||
```
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(static_dir / "frontpage.html").write_text(
|
||||
"<p class=\"star-slot\">STAR</p>\n<!-- OPENMLSYS_LANGUAGE_SWITCH -->\n<div class=\"hero\">frontpage</div>\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
rewritten = rewrite_markdown(
|
||||
index.read_text(encoding="utf-8"),
|
||||
index.resolve(),
|
||||
{index.resolve(): "Home"},
|
||||
frontpage_switch_label="中文",
|
||||
frontpage_switch_href="cn/",
|
||||
)
|
||||
|
||||
self.assertIn('class="openmlsys-frontpage-switch-row"', rewritten)
|
||||
self.assertIn('class="openmlsys-frontpage-switch"', rewritten)
|
||||
self.assertIn('href="cn/"', rewritten)
|
||||
self.assertIn(">中文</a>", rewritten)
|
||||
self.assertLess(
|
||||
rewritten.index('class="star-slot"'),
|
||||
rewritten.index('class="openmlsys-frontpage-switch-row"'),
|
||||
)
|
||||
|
||||
def test_rewrite_markdown_prefers_book_local_frontpage_file(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
source = root / "en_chapters"
|
||||
static_dir = source / "static"
|
||||
source.mkdir()
|
||||
static_dir.mkdir()
|
||||
|
||||
index = source / "index.md"
|
||||
index.write_text(
|
||||
"""# Home
|
||||
|
||||
```eval_rst
|
||||
.. raw:: html
|
||||
:file: frontpage.html
|
||||
```
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(source / "frontpage.html").write_text(
|
||||
"<div class=\"hero\">English frontpage</div>\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(static_dir / "frontpage.html").write_text(
|
||||
"<div class=\"hero\">Chinese fallback</div>\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
rewritten = rewrite_markdown(
|
||||
index.read_text(encoding="utf-8"),
|
||||
index.resolve(),
|
||||
{index.resolve(): "Home"},
|
||||
)
|
||||
|
||||
self.assertIn("English frontpage", rewritten)
|
||||
self.assertNotIn("Chinese fallback", rewritten)
|
||||
self.assertIn("background: transparent !important;", rewritten)
|
||||
self.assertIn("padding: 0 !important;", rewritten)
|
||||
self.assertIn("border-radius: 6px;", rewritten)
|
||||
self.assertIn("background: #f6f8fa;", rewritten)
|
||||
self.assertIn(".content main {", rewritten)
|
||||
self.assertIn("max-width: min(100%, max(65%, var(--content-max-width)));", rewritten)
|
||||
self.assertIn(".openmlsys-frontpage {", rewritten)
|
||||
self.assertIn("width: 100%;", rewritten)
|
||||
self.assertIn("margin-inline: auto;", rewritten)
|
||||
|
||||
def test_regular_page_does_not_render_frontpage_switch(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
page = root / "chapter.md"
|
||||
page.write_text("# Chapter\n\nRegular body.\n", encoding="utf-8")
|
||||
|
||||
rewritten = rewrite_markdown(
|
||||
page.read_text(encoding="utf-8"),
|
||||
page.resolve(),
|
||||
{page.resolve(): "Chapter"},
|
||||
frontpage_switch_label="中文",
|
||||
frontpage_switch_href="cn/",
|
||||
)
|
||||
|
||||
self.assertNotIn('openmlsys-frontpage-switch', rewritten)
|
||||
|
||||
def test_english_frontpage_author_grid_uses_top_aligned_spacing(self) -> None:
|
||||
frontpage = (REPO_ROOT / "en_chapters" / "frontpage.html").read_text(encoding="utf-8")
|
||||
|
||||
self.assertIn(".authors.mdl-grid {", frontpage)
|
||||
self.assertIn("align-items: flex-start;", frontpage)
|
||||
self.assertIn("row-gap: calc(24px + 1.5em);", frontpage)
|
||||
self.assertIn("height: 120px;", frontpage)
|
||||
self.assertIn("object-fit: cover;", frontpage)
|
||||
|
||||
def test_english_frontpage_footer_titles_are_stacked_not_flex_columns(self) -> None:
|
||||
frontpage = (REPO_ROOT / "en_chapters" / "frontpage.html").read_text(encoding="utf-8")
|
||||
|
||||
self.assertIn(".authors .mdl-cell:not(.author-group-title) {", frontpage)
|
||||
self.assertIn(".author-group-title {", frontpage)
|
||||
self.assertIn("display: block;", frontpage)
|
||||
self.assertIn(".author-group-title h3,", frontpage)
|
||||
self.assertIn("width: 100%;", frontpage)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
266
tests/test_prepare_mdbook_zh.py
Normal file
266
tests/test_prepare_mdbook_zh.py
Normal file
@@ -0,0 +1,266 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from tools.prepare_mdbook_zh import extract_title, process_equation_labels, rewrite_markdown, write_summary
|
||||
|
||||
|
||||
class PrepareMdBookZhTests(unittest.TestCase):
|
||||
def test_extract_title_supports_atx_and_setext_headings(self) -> None:
|
||||
self.assertEqual(extract_title("# 导论\n"), "导论")
|
||||
self.assertEqual(extract_title("前言文字\n\n## 机器学习应用\n"), "机器学习应用")
|
||||
self.assertEqual(extract_title("机器学习系统:设计和实现\n=========================\n"), "机器学习系统:设计和实现")
|
||||
|
||||
def test_write_summary_generates_nested_navigation(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
source = root / "zh_chapters"
|
||||
source.mkdir()
|
||||
|
||||
(source / "index.md").write_text(
|
||||
"""机器学习系统:设计和实现
|
||||
=========================
|
||||
|
||||
```eval_rst
|
||||
.. raw:: html
|
||||
:file: frontpage.html
|
||||
```
|
||||
|
||||
```toc
|
||||
:maxdepth: 2
|
||||
|
||||
[前言](chapter_preface/index)
|
||||
|
||||
# 基础篇
|
||||
chapter_introduction/index
|
||||
|
||||
# 附录
|
||||
[机器学习基础附录](appendix_machine_learning_introduction/index)
|
||||
```
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
chapter_preface = source / "chapter_preface"
|
||||
chapter_preface.mkdir()
|
||||
(chapter_preface / "index.md").write_text("# 前言\n", encoding="utf-8")
|
||||
static_dir = source / "static"
|
||||
static_dir.mkdir()
|
||||
(static_dir / "frontpage.html").write_text(
|
||||
"<div class=\"hero\">frontpage</div>\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
chapter_intro = source / "chapter_introduction"
|
||||
chapter_intro.mkdir()
|
||||
(chapter_intro / "index.md").write_text(
|
||||
"""# 导论
|
||||
|
||||
```toc
|
||||
:maxdepth: 2
|
||||
|
||||
applications
|
||||
design
|
||||
```
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(chapter_intro / "applications.md").write_text("# 机器学习应用\n", encoding="utf-8")
|
||||
(chapter_intro / "design.md").write_text("# 设计目标\n", encoding="utf-8")
|
||||
|
||||
appendix = source / "appendix_machine_learning_introduction"
|
||||
appendix.mkdir()
|
||||
(appendix / "index.md").write_text("# 机器学习基础附录\n", encoding="utf-8")
|
||||
|
||||
for name in ("img", "static", "references"):
|
||||
(root / name).mkdir()
|
||||
(root / "mlsys.bib").write_text("% bibliography\n", encoding="utf-8")
|
||||
|
||||
summary_path = write_summary(source)
|
||||
summary = summary_path.read_text(encoding="utf-8")
|
||||
self.assertEqual(
|
||||
summary,
|
||||
"""# Summary
|
||||
|
||||
[机器学习系统:设计和实现](index.md)
|
||||
[前言](chapter_preface/index.md)
|
||||
|
||||
# 基础篇
|
||||
|
||||
- [导论](chapter_introduction/index.md)
|
||||
- [机器学习应用](chapter_introduction/applications.md)
|
||||
- [设计目标](chapter_introduction/design.md)
|
||||
|
||||
# 附录
|
||||
|
||||
- [机器学习基础附录](appendix_machine_learning_introduction/index.md)
|
||||
""",
|
||||
)
|
||||
|
||||
title_cache = {
|
||||
(source / "chapter_preface" / "index.md").resolve(): "前言",
|
||||
(source / "chapter_introduction" / "index.md").resolve(): "导论",
|
||||
(source / "chapter_introduction" / "applications.md").resolve(): "机器学习应用",
|
||||
(source / "chapter_introduction" / "design.md").resolve(): "设计目标",
|
||||
(source / "appendix_machine_learning_introduction" / "index.md").resolve(): "机器学习基础附录",
|
||||
}
|
||||
root_index = rewrite_markdown((source / "index.md").read_text(encoding="utf-8"), (source / "index.md").resolve(), title_cache)
|
||||
self.assertNotIn("```eval_rst", root_index)
|
||||
self.assertNotIn("```toc", root_index)
|
||||
self.assertIn("- [前言](chapter_preface/index.md)", root_index)
|
||||
self.assertIn("- 基础篇", root_index)
|
||||
self.assertIn(" - [导论](chapter_introduction/index.md)", root_index)
|
||||
self.assertIn("- 附录", root_index)
|
||||
self.assertIn(" - [机器学习基础附录](appendix_machine_learning_introduction/index.md)", root_index)
|
||||
|
||||
intro_index = rewrite_markdown(
|
||||
(source / "chapter_introduction" / "index.md").read_text(encoding="utf-8"),
|
||||
(source / "chapter_introduction" / "index.md").resolve(),
|
||||
title_cache,
|
||||
)
|
||||
self.assertNotIn("```toc", intro_index)
|
||||
self.assertIn("- [机器学习应用](applications.md)", intro_index)
|
||||
self.assertIn("- [设计目标](design.md)", intro_index)
|
||||
|
||||
def test_write_summary_raises_for_missing_toc_entries(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
source = root / "zh_chapters"
|
||||
source.mkdir()
|
||||
|
||||
(source / "index.md").write_text(
|
||||
"""# 首页
|
||||
|
||||
```toc
|
||||
:maxdepth: 2
|
||||
|
||||
existing
|
||||
missing
|
||||
```
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(source / "existing.md").write_text("# 现有章节\n", encoding="utf-8")
|
||||
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
write_summary(source)
|
||||
|
||||
def test_rewrite_markdown_normalizes_common_d2l_directives(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
source = root / "zh_chapters"
|
||||
source.mkdir()
|
||||
|
||||
page = source / "chapter.md"
|
||||
page.write_text(
|
||||
"""# 标题
|
||||
|
||||

|
||||
:width:`800px`
|
||||
:label:`fig_example`
|
||||
|
||||
参见 :numref:`fig_example` 和公式 :eqref:`eq_example`,引用 :cite:`foo2024`。
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
rewritten = rewrite_markdown(page.read_text(encoding="utf-8"), page.resolve(), {page.resolve(): "标题"})
|
||||
self.assertNotIn(":width:", rewritten)
|
||||
self.assertNotIn(":label:", rewritten)
|
||||
self.assertNotIn(":numref:", rewritten)
|
||||
self.assertNotIn(":eqref:", rewritten)
|
||||
self.assertNotIn(":cite:", rewritten)
|
||||
self.assertIn("`fig_example`", rewritten)
|
||||
self.assertIn("$\\eqref{eq_example}$", rewritten)
|
||||
self.assertIn("[foo2024]", rewritten)
|
||||
|
||||
def test_rewrite_markdown_inlines_frontpage_html_include(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
source = root / "zh_chapters"
|
||||
static_dir = source / "static"
|
||||
static_dir.mkdir(parents=True)
|
||||
|
||||
index = source / "index.md"
|
||||
index.write_text(
|
||||
"""# 首页
|
||||
|
||||
```eval_rst
|
||||
.. raw:: html
|
||||
:file: frontpage.html
|
||||
```
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(static_dir / "frontpage.html").write_text(
|
||||
"""<head>
|
||||
<style>
|
||||
.hero { color: red; }
|
||||
|
||||
.other { color: blue; }
|
||||
</style>
|
||||
</head>
|
||||
<p class="star-slot">STAR</p>
|
||||
<!-- OPENMLSYS_LANGUAGE_SWITCH -->
|
||||
<div class="hero">
|
||||
<img src="_images/logo.png" />
|
||||
<img src="./_images/jinxuefeng.png" />
|
||||
</div>
|
||||
<script>console.log('frontpage');</script>
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
rewritten = rewrite_markdown(index.read_text(encoding="utf-8"), index.resolve(), {index.resolve(): "首页"})
|
||||
self.assertNotIn("```eval_rst", rewritten)
|
||||
self.assertNotIn("<head>", rewritten)
|
||||
self.assertIn('class="openmlsys-frontpage"', rewritten)
|
||||
self.assertIn('<div class="hero">', rewritten)
|
||||
self.assertIn('<style>', rewritten)
|
||||
self.assertIn(".hero { color: red; } .other { color: blue; }", rewritten)
|
||||
self.assertIn("static/image/logo.png", rewritten)
|
||||
self.assertIn("static/image/jinxuefeng.png", rewritten)
|
||||
self.assertIn("console.log('frontpage')", rewritten)
|
||||
self.assertIn('class="openmlsys-frontpage-switch-row"', rewritten)
|
||||
self.assertIn('class="openmlsys-frontpage-switch"', rewritten)
|
||||
self.assertIn('href="../"', rewritten)
|
||||
self.assertIn(">English</a>", rewritten)
|
||||
self.assertLess(
|
||||
rewritten.index('class="star-slot"'),
|
||||
rewritten.index('class="openmlsys-frontpage-switch-row"'),
|
||||
)
|
||||
|
||||
|
||||
def test_process_equation_labels_single_line(self) -> None:
|
||||
"""Verify single-line equation gets \\tag and \\label injected."""
|
||||
md = "# Title\n\n$$a = f(z)$$\n:eqlabel:`sigmoid`\n\nSee :eqref:`sigmoid`.\n"
|
||||
result, label_map = process_equation_labels(md)
|
||||
self.assertIn("\\tag{1}\\label{sigmoid}$$", result)
|
||||
self.assertNotIn(":eqlabel:", result)
|
||||
self.assertEqual(label_map, {"sigmoid": 1})
|
||||
|
||||
def test_process_equation_labels_multiline(self) -> None:
|
||||
"""Verify multi-line equation (closing $$ on own line) gets \\tag and \\label."""
|
||||
md = "# Title\n\n$$\na = f(z)\n$$\n:eqlabel:`eq1`\n"
|
||||
result, label_map = process_equation_labels(md)
|
||||
lines = result.split("\n")
|
||||
# \\tag line should appear before the closing $$
|
||||
tag_idx = next(i for i, l in enumerate(lines) if "\\tag{1}\\label{eq1}" in l)
|
||||
close_idx = next(i for i, l in enumerate(lines) if l.strip() == "$$" and i > tag_idx)
|
||||
self.assertLess(tag_idx, close_idx)
|
||||
self.assertNotIn(":eqlabel:", result)
|
||||
self.assertEqual(label_map, {"eq1": 1})
|
||||
|
||||
def test_process_equation_labels_sequential_numbering(self) -> None:
|
||||
"""Verify multiple equations get sequential numbers."""
|
||||
md = "$$a$$\n:eqlabel:`eq1`\n\n$$b$$\n:eqlabel:`eq2`\n"
|
||||
result, label_map = process_equation_labels(md)
|
||||
self.assertIn("\\tag{1}\\label{eq1}", result)
|
||||
self.assertIn("\\tag{2}\\label{eq2}", result)
|
||||
self.assertEqual(label_map, {"eq1": 1, "eq2": 2})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
29
tests/test_update_docs_workflow.py
Normal file
29
tests/test_update_docs_workflow.py
Normal file
@@ -0,0 +1,29 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "update_docs.yml"
|
||||
CI_WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "main.yml"
|
||||
|
||||
|
||||
class UpdateDocsWorkflowTests(unittest.TestCase):
|
||||
def test_workflow_deploys_to_openmlsys_github_io(self) -> None:
|
||||
workflow = WORKFLOW_PATH.read_text(encoding="utf-8")
|
||||
|
||||
self.assertIn("DEPLOY_TOKEN", workflow)
|
||||
self.assertIn("openmlsys.github.io", workflow)
|
||||
self.assertIn("git push", workflow)
|
||||
|
||||
def test_workflows_use_peaceiris_mdbook_action(self) -> None:
|
||||
for workflow_path in (WORKFLOW_PATH, CI_WORKFLOW_PATH):
|
||||
workflow = workflow_path.read_text(encoding="utf-8")
|
||||
self.assertIn("uses: peaceiris/actions-mdbook@v2", workflow, workflow_path.as_posix())
|
||||
self.assertIn("mdbook-version: 'latest'", workflow, workflow_path.as_posix())
|
||||
self.assertNotIn("cargo install mdbook --locked", workflow, workflow_path.as_posix())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
16
theme/dark-mode-images.css
Normal file
16
theme/dark-mode-images.css
Normal file
@@ -0,0 +1,16 @@
|
||||
/* 暗色模式下仅为正文图片添加浅灰色背景,提高透明背景图片的可读性 */
|
||||
.navy .content main img,
|
||||
.coal .content main img,
|
||||
.ayu .content main img {
|
||||
background-color: #e8e8e8;
|
||||
border-radius: 4px;
|
||||
padding: 8px;
|
||||
}
|
||||
|
||||
/* 首页 frontpage 图片保持透明,不添加正文图像底色。 */
|
||||
.navy .openmlsys-frontpage img,
|
||||
.coal .openmlsys-frontpage img,
|
||||
.ayu .openmlsys-frontpage img {
|
||||
background-color: transparent !important;
|
||||
padding: 0 !important;
|
||||
}
|
||||
16
theme/typst.css
Normal file
16
theme/typst.css
Normal file
@@ -0,0 +1,16 @@
|
||||
.typst-inline {
|
||||
display: inline-flex;
|
||||
vertical-align: -0.2em;
|
||||
}
|
||||
|
||||
.typst-display {
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
margin: 1rem 0;
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
.typst-doc {
|
||||
color: var(--fg);
|
||||
max-width: 100%;
|
||||
}
|
||||
97
tools/assemble_docs_publish_tree.py
Normal file
97
tools/assemble_docs_publish_tree.py
Normal file
@@ -0,0 +1,97 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def remove_path(path: Path) -> None:
|
||||
if path.is_symlink() or path.is_file():
|
||||
path.unlink()
|
||||
return
|
||||
if path.is_dir():
|
||||
shutil.rmtree(path)
|
||||
|
||||
|
||||
def copy_site(source: Path, destination: Path) -> None:
|
||||
source = source.resolve()
|
||||
if not source.is_dir():
|
||||
raise FileNotFoundError(f"Site source does not exist or is not a directory: {source}")
|
||||
|
||||
remove_path(destination)
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copytree(source, destination)
|
||||
|
||||
|
||||
def assemble_publish_tree(
|
||||
destination_root: Path,
|
||||
docs_subdir: str = "docs",
|
||||
en_source: Path | None = None,
|
||||
zh_source: Path | None = None,
|
||||
) -> tuple[Path, Path | None]:
|
||||
if en_source is None and zh_source is None:
|
||||
raise ValueError("At least one site source must be provided.")
|
||||
|
||||
destination_root = destination_root.resolve()
|
||||
docs_root = (destination_root / docs_subdir).resolve()
|
||||
|
||||
remove_path(docs_root)
|
||||
docs_root.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if en_source is not None:
|
||||
copy_site(en_source, docs_root)
|
||||
else:
|
||||
docs_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
zh_destination: Path | None = None
|
||||
if zh_source is not None:
|
||||
zh_destination = docs_root / "cn"
|
||||
copy_site(zh_source, zh_destination)
|
||||
|
||||
return docs_root, zh_destination
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Assemble the publish tree expected by openmlsys.github.io."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--destination-root",
|
||||
type=Path,
|
||||
required=True,
|
||||
help="Root of the checked-out deployment repository.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--docs-subdir",
|
||||
default="docs",
|
||||
help="Subdirectory inside the destination root that hosts the site.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--en-source",
|
||||
type=Path,
|
||||
help="Built site to publish at docs/.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--zh-source",
|
||||
type=Path,
|
||||
help="Built site to publish at docs/cn/.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
docs_root, zh_root = assemble_publish_tree(
|
||||
destination_root=args.destination_root,
|
||||
docs_subdir=args.docs_subdir,
|
||||
en_source=args.en_source,
|
||||
zh_source=args.zh_source,
|
||||
)
|
||||
print(f"Assembled root site at {docs_root}")
|
||||
if zh_root is not None:
|
||||
print(f"Assembled Chinese site at {zh_root}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
60
tools/ensure_book_resources.py
Normal file
60
tools/ensure_book_resources.py
Normal file
@@ -0,0 +1,60 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
RESOURCE_NAMES = ("img", "references", "static", "mlsys.bib")
|
||||
|
||||
|
||||
def ensure_resource_views(
|
||||
chapter_dir: Path,
|
||||
repo_root: Path,
|
||||
resource_names: tuple[str, ...] = RESOURCE_NAMES,
|
||||
) -> None:
|
||||
chapter_dir = chapter_dir.resolve()
|
||||
repo_root = repo_root.resolve()
|
||||
|
||||
for name in resource_names:
|
||||
destination = chapter_dir / name
|
||||
source = repo_root / name
|
||||
if not source.exists():
|
||||
raise FileNotFoundError(f"Resource does not exist: {source}")
|
||||
|
||||
if destination.is_symlink():
|
||||
destination.unlink()
|
||||
elif destination.exists():
|
||||
continue
|
||||
|
||||
relative_source = os.path.relpath(source, start=chapter_dir)
|
||||
destination.symlink_to(relative_source)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Ensure chapter directories can see shared book resources."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--chapter-dir",
|
||||
type=Path,
|
||||
required=True,
|
||||
help="Book source directory such as en_chapters or zh_chapters.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--repo-root",
|
||||
type=Path,
|
||||
default=Path(__file__).resolve().parent.parent,
|
||||
help="Repository root that owns the shared resources.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
ensure_resource_views(args.chapter_dir, args.repo_root)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
132
tools/ensure_mdbook_typst_math.py
Normal file
132
tools/ensure_mdbook_typst_math.py
Normal file
@@ -0,0 +1,132 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import os
|
||||
import platform
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
VERSION = "v0.3.0"
|
||||
RELEASE_BASE_URL = "https://github.com/duskmoon314/mdbook-typst-math/releases/download"
|
||||
ASSET_SHA256 = {
|
||||
"mdbook-typst-math-aarch64-apple-darwin.gz": "9c7a94113e16a465edd1324010e2cc432be3c0794320c13d6a44d9523f069384",
|
||||
"mdbook-typst-math-aarch64-unknown-linux-gnu.gz": "bbcf4574e380663400af74dda76dd6ecafd36aff185d653a2e24e294c45321c3",
|
||||
"mdbook-typst-math-x86_64-apple-darwin.gz": "8bb36eb558fc438c55162b442975eca588a7654b8069860526e46cc08c2aee6a",
|
||||
"mdbook-typst-math-x86_64-pc-windows-msvc.exe": "b5d3e07108a7286007d153c66efe434d06ab6caf43fcd22f78b4e6af8a294314",
|
||||
"mdbook-typst-math-x86_64-unknown-linux-gnu.gz": "3b785a42fb3a93bcd3f80106e6ded5c55bb0bcd4cd0634edf8232d14444b6987",
|
||||
}
|
||||
SUPPORTED_ASSETS = {
|
||||
("darwin", "aarch64"): "mdbook-typst-math-aarch64-apple-darwin.gz",
|
||||
("darwin", "x86_64"): "mdbook-typst-math-x86_64-apple-darwin.gz",
|
||||
("linux", "aarch64"): "mdbook-typst-math-aarch64-unknown-linux-gnu.gz",
|
||||
("linux", "x86_64"): "mdbook-typst-math-x86_64-unknown-linux-gnu.gz",
|
||||
("windows", "x86_64"): "mdbook-typst-math-x86_64-pc-windows-msvc.exe",
|
||||
}
|
||||
|
||||
|
||||
def normalize_machine(machine: str) -> str:
|
||||
normalized = machine.strip().lower()
|
||||
if normalized in {"arm64", "aarch64"}:
|
||||
return "aarch64"
|
||||
if normalized in {"amd64", "x86_64", "x64"}:
|
||||
return "x86_64"
|
||||
return normalized
|
||||
|
||||
|
||||
def normalize_system(system: str) -> str:
|
||||
normalized = system.strip().lower()
|
||||
if normalized.startswith("mingw") or normalized.startswith("msys") or normalized.startswith("cygwin"):
|
||||
return "windows"
|
||||
return normalized
|
||||
|
||||
|
||||
def resolve_asset_name(system: str | None = None, machine: str | None = None) -> str:
|
||||
resolved_system = normalize_system(system or platform.system())
|
||||
resolved_machine = normalize_machine(machine or platform.machine())
|
||||
asset_name = SUPPORTED_ASSETS.get((resolved_system, resolved_machine))
|
||||
if asset_name is None:
|
||||
raise ValueError(
|
||||
f"Unsupported platform for mdbook-typst-math: system={resolved_system!r}, machine={resolved_machine!r}"
|
||||
)
|
||||
return asset_name
|
||||
|
||||
|
||||
def build_download_url(version: str, asset_name: str) -> str:
|
||||
return f"{RELEASE_BASE_URL}/{version}/{asset_name}"
|
||||
|
||||
|
||||
def resolve_binary_path(output_dir: Path, version: str, asset_name: str) -> Path:
|
||||
binary_name = "mdbook-typst-math.exe" if asset_name.endswith(".exe") else "mdbook-typst-math"
|
||||
return output_dir / binary_name
|
||||
|
||||
|
||||
def resolve_version_path(output_dir: Path) -> Path:
|
||||
return output_dir / ".mdbook-typst-math.version"
|
||||
|
||||
|
||||
def download_bytes(url: str) -> bytes:
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "openmlsys-mdbook-bootstrap/1.0"})
|
||||
with urllib.request.urlopen(request) as response:
|
||||
return response.read()
|
||||
|
||||
|
||||
def ensure_binary(
|
||||
output_dir: Path,
|
||||
*,
|
||||
version: str = VERSION,
|
||||
system: str | None = None,
|
||||
machine: str | None = None,
|
||||
downloader=download_bytes,
|
||||
) -> Path:
|
||||
asset_name = resolve_asset_name(system=system, machine=machine)
|
||||
binary_path = resolve_binary_path(output_dir, version, asset_name)
|
||||
version_path = resolve_version_path(output_dir)
|
||||
if binary_path.exists() and version_path.exists() and version_path.read_text(encoding="utf-8").strip() == version:
|
||||
if binary_path.suffix != ".exe":
|
||||
binary_path.chmod(binary_path.stat().st_mode | 0o111)
|
||||
return binary_path
|
||||
|
||||
expected_sha256 = ASSET_SHA256[asset_name]
|
||||
download_url = build_download_url(version, asset_name)
|
||||
archive_bytes = downloader(download_url)
|
||||
digest = hashlib.sha256(archive_bytes).hexdigest()
|
||||
if digest != expected_sha256:
|
||||
raise ValueError(
|
||||
f"Checksum mismatch for {asset_name}: expected {expected_sha256}, got {digest}"
|
||||
)
|
||||
|
||||
binary_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = gzip.decompress(archive_bytes) if asset_name.endswith(".gz") else archive_bytes
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
temporary_path = binary_path.with_suffix(f"{binary_path.suffix}.tmp")
|
||||
temporary_path.write_bytes(payload)
|
||||
if binary_path.suffix != ".exe":
|
||||
temporary_path.chmod(0o755)
|
||||
os.replace(temporary_path, binary_path)
|
||||
version_path.write_text(version, encoding="utf-8")
|
||||
return binary_path
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Download a pinned mdbook-typst-math release binary.")
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=Path,
|
||||
default=Path(".mdbook-bin"),
|
||||
help="Directory used to cache the downloaded mdbook-typst-math binary.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
binary_path = ensure_binary(args.output_dir.resolve())
|
||||
print(binary_path)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
614
tools/latex_to_typst.py
Normal file
614
tools/latex_to_typst.py
Normal file
@@ -0,0 +1,614 @@
|
||||
"""Convert LaTeX math notation to Typst math notation within markdown content.
|
||||
|
||||
This module provides a best-effort converter for the LaTeX math subset used in
|
||||
the OpenMLSys textbook. It is **not** a general-purpose LaTeX→Typst transpiler;
|
||||
only the commands that actually appear in the zh_chapters are handled.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Brace-matching helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _find_brace_group(s: str, pos: int) -> tuple[str, int] | None:
|
||||
"""Return ``(content, end_pos)`` for the ``{…}`` group starting at *pos*.
|
||||
|
||||
Skips leading whitespace. Returns ``None`` when no opening brace is found
|
||||
or braces are unbalanced.
|
||||
"""
|
||||
while pos < len(s) and s[pos] in " \t":
|
||||
pos += 1
|
||||
if pos >= len(s) or s[pos] != "{":
|
||||
return None
|
||||
depth = 0
|
||||
start = pos + 1
|
||||
for i in range(pos, len(s)):
|
||||
if s[i] == "{":
|
||||
depth += 1
|
||||
elif s[i] == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return (s[start:i], i + 1)
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Command tables
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Commands whose Typst name equals the LaTeX name (just drop the backslash).
|
||||
SIMPLE_COMMANDS: set[str] = {
|
||||
# Greek
|
||||
"alpha", "beta", "gamma", "delta", "Delta",
|
||||
"epsilon", "zeta", "eta", "theta", "Theta",
|
||||
"iota", "kappa", "lambda", "Lambda",
|
||||
"mu", "nu", "xi", "Xi",
|
||||
"pi", "Pi", "rho",
|
||||
"sigma", "Sigma", "tau",
|
||||
"upsilon", "Upsilon",
|
||||
"phi", "Phi", "chi", "psi", "Psi",
|
||||
"omega", "Omega",
|
||||
# Operators / relations
|
||||
"times", "partial", "nabla", "in",
|
||||
"top", "prime",
|
||||
"forall", "exists", "approx", "equiv",
|
||||
"subset", "supset",
|
||||
# Big operators / functions
|
||||
"log", "ln", "exp", "sin", "cos", "tan",
|
||||
"min", "max", "lim", "sum",
|
||||
"det", "dim", "ker", "inf", "sup",
|
||||
}
|
||||
|
||||
# Commands that map to a *different* Typst identifier.
|
||||
RENAMED_COMMANDS: dict[str, str] = {
|
||||
"cdot": "dot.c",
|
||||
"cdots": "dots.c",
|
||||
"ldots": "dots",
|
||||
"dots": "dots",
|
||||
"to": "->",
|
||||
"rightarrow": "->",
|
||||
"leftarrow": "<-",
|
||||
"Rightarrow": "=>",
|
||||
"rightsquigarrow": "arrow.r.squiggly",
|
||||
"leq": "<=",
|
||||
"geq": ">=",
|
||||
"prod": "product",
|
||||
"notag": "",
|
||||
"quad": "quad",
|
||||
"qquad": "wide",
|
||||
"label": "", # consumed by :eqlabel: already
|
||||
"sim": "tilde.op",
|
||||
"infty": "infinity",
|
||||
"neq": "eq.not",
|
||||
"ast": "ast.op",
|
||||
"vdots": "dots.v",
|
||||
"ddots": "dots.down",
|
||||
"lVert": "||",
|
||||
"rVert": "||",
|
||||
"vert": "|",
|
||||
"lvert": "|",
|
||||
"rvert": "|",
|
||||
"mid": "|",
|
||||
"cap": "inter",
|
||||
"cup": "union",
|
||||
"le": "<=",
|
||||
"ge": ">=",
|
||||
"odot": "dot.o",
|
||||
"oplus": "plus.circle",
|
||||
"otimes": "times.circle",
|
||||
}
|
||||
|
||||
# \cmd{arg} → typst_func(arg)
|
||||
ONE_ARG_COMMANDS: dict[str, str] = {
|
||||
"boldsymbol": "bold",
|
||||
"mathcal": "cal",
|
||||
"mathbf": "bold",
|
||||
"mathbb": "bb",
|
||||
"hat": "hat",
|
||||
"bar": "overline",
|
||||
"dot": "dot",
|
||||
"tilde": "tilde",
|
||||
"sqrt": "sqrt",
|
||||
"overline": "overline",
|
||||
"pmb": "bold",
|
||||
"textbf": "bold",
|
||||
"textit": "italic",
|
||||
"bm": "bold",
|
||||
}
|
||||
|
||||
# \cmd{arg1}{arg2} → typst_func(arg1, arg2)
|
||||
TWO_ARG_COMMANDS: dict[str, str] = {
|
||||
"frac": "frac",
|
||||
"binom": "binom",
|
||||
}
|
||||
|
||||
# Delimiter-sizing commands to strip (the delimiter char after them is kept).
|
||||
_SIZING_COMMANDS: set[str] = {
|
||||
"left", "right", "bigg", "Bigg", "big", "Big", "biggl", "biggr",
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Core single-pass converter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _last_char(out: list[str]) -> str:
|
||||
"""Return the last non-empty character in the output buffer, or ``""``."""
|
||||
for part in reversed(out):
|
||||
if part:
|
||||
return part[-1]
|
||||
return ""
|
||||
|
||||
|
||||
def _emit(out: list[str], text: str) -> None:
|
||||
"""Append *text* to *out*, adding a space separator when needed.
|
||||
|
||||
In Typst math, consecutive letters form a multi-letter identifier which
|
||||
will error if unknown. Similarly, letter→digit transitions form tokens
|
||||
like ``W1``. This helper inserts spaces to prevent such merging, matching
|
||||
LaTeX math semantics where adjacent characters are separate symbols.
|
||||
"""
|
||||
if not text:
|
||||
out.append(text)
|
||||
return
|
||||
lc = _last_char(out)
|
||||
fc = text[0]
|
||||
if lc and (
|
||||
# letter→letter (e.g. "ou" → "o u")
|
||||
(lc.isalpha() and fc.isalpha())
|
||||
# letter→digit (e.g. "W1" → "W 1")
|
||||
or (lc.isalpha() and fc.isdigit())
|
||||
# digit→letter (e.g. "2x" → "2 x")
|
||||
or (lc.isdigit() and fc.isalpha())
|
||||
# )→letter/digit (e.g. "bold(X)y" → "bold(X) y")
|
||||
or (lc == ")" and (fc.isalpha() or fc.isdigit()))
|
||||
):
|
||||
out.append(" ")
|
||||
out.append(text)
|
||||
|
||||
|
||||
def _convert(s: str) -> str:
|
||||
"""Convert a single LaTeX math expression to Typst math."""
|
||||
out: list[str] = []
|
||||
i = 0
|
||||
n = len(s)
|
||||
|
||||
while i < n:
|
||||
ch = s[i]
|
||||
|
||||
# ---- backslash commands ----
|
||||
if ch == "\\" and i + 1 < n:
|
||||
nxt = s[i + 1]
|
||||
|
||||
# Double backslash: either markdown-escaped bracket or line-break
|
||||
if nxt == "\\":
|
||||
# \\{ \\} \\[ \\] \\( \\) → markdown-escaped LaTeX delimiters
|
||||
if i + 2 < n and s[i + 2] in "{}[]()":
|
||||
out.append(s[i + 2])
|
||||
i += 3
|
||||
continue
|
||||
out.append(" \\\n")
|
||||
i += 2
|
||||
continue
|
||||
|
||||
# Escaped characters: \{ \} \[ \] \( \) \, \; \! \ \.
|
||||
if nxt in "{}[]()":
|
||||
out.append(nxt)
|
||||
i += 2
|
||||
continue
|
||||
if nxt == ",":
|
||||
out.append("thin ") # thin space
|
||||
i += 2
|
||||
continue
|
||||
if nxt == ";":
|
||||
out.append("med ") # medium space
|
||||
i += 2
|
||||
continue
|
||||
if nxt == "!":
|
||||
out.append("") # negative thin space → ignore
|
||||
i += 2
|
||||
continue
|
||||
if nxt == " ":
|
||||
out.append(" ")
|
||||
i += 2
|
||||
continue
|
||||
if nxt == "\n":
|
||||
out.append(" ")
|
||||
i += 2
|
||||
continue
|
||||
|
||||
# Try to match an alphabetic command name
|
||||
m = re.match(r"[a-zA-Z]+", s[i + 1:])
|
||||
if not m:
|
||||
# Bare backslash before non-alpha → keep the char after
|
||||
out.append(nxt)
|
||||
i += 2
|
||||
continue
|
||||
|
||||
cmd = m.group()
|
||||
after = i + 1 + m.end()
|
||||
|
||||
# -- environments --
|
||||
if cmd == "begin":
|
||||
g = _find_brace_group(s, after)
|
||||
if g:
|
||||
env_name, env_pos = g
|
||||
end_marker = f"\\end{{{env_name}}}"
|
||||
end_idx = s.find(end_marker, env_pos)
|
||||
if end_idx != -1:
|
||||
body = s[env_pos:end_idx]
|
||||
i = end_idx + len(end_marker)
|
||||
_emit(out, _convert_environment(env_name, body))
|
||||
continue
|
||||
# Fallthrough: couldn't parse, skip \begin
|
||||
i = after
|
||||
continue
|
||||
|
||||
if cmd == "end":
|
||||
# Stray \end (shouldn't happen if \begin matched)
|
||||
g = _find_brace_group(s, after)
|
||||
i = g[1] if g else after
|
||||
continue
|
||||
|
||||
# -- special multi-arg commands --
|
||||
if cmd == "underset":
|
||||
# \underset{below}{base} → attach(base, b: below)
|
||||
g1 = _find_brace_group(s, after)
|
||||
if g1:
|
||||
below, p1 = g1
|
||||
g2 = _find_brace_group(s, p1)
|
||||
if g2:
|
||||
base, p2 = g2
|
||||
_emit(out, f"attach({_convert(base)}, b: {_convert(below)})")
|
||||
i = p2
|
||||
continue
|
||||
|
||||
if cmd == "overset":
|
||||
# \overset{above}{base} → attach(base, t: above)
|
||||
g1 = _find_brace_group(s, after)
|
||||
if g1:
|
||||
above, p1 = g1
|
||||
g2 = _find_brace_group(s, p1)
|
||||
if g2:
|
||||
base, p2 = g2
|
||||
_emit(out, f"attach({_convert(base)}, t: {_convert(above)})")
|
||||
i = p2
|
||||
continue
|
||||
|
||||
if cmd == "operatorname":
|
||||
# \operatorname{name} → op("name")
|
||||
g = _find_brace_group(s, after)
|
||||
if g:
|
||||
name, pos = g
|
||||
_emit(out, f'op("{name}")')
|
||||
i = pos
|
||||
continue
|
||||
|
||||
if cmd == "tag":
|
||||
# \tag{n} → visual equation number
|
||||
g = _find_brace_group(s, after)
|
||||
if g:
|
||||
content, pos = g
|
||||
_emit(out, f'quad upright("({content})")')
|
||||
i = pos
|
||||
continue
|
||||
|
||||
if cmd == "eqref":
|
||||
# \eqref{name} → show label name as fallback
|
||||
g = _find_brace_group(s, after)
|
||||
if g:
|
||||
content, pos = g
|
||||
_emit(out, f'upright("({content})")')
|
||||
i = pos
|
||||
continue
|
||||
|
||||
if cmd in ("mathrm", "text"):
|
||||
# \mathrm{text} → upright("text") — treat as text, not math
|
||||
g = _find_brace_group(s, after)
|
||||
if g:
|
||||
content, pos = g
|
||||
stripped = content.strip()
|
||||
if stripped:
|
||||
_emit(out, f'upright("{stripped}")')
|
||||
# else: empty mathrm (spacing hack) → drop
|
||||
i = pos
|
||||
continue
|
||||
|
||||
# -- two-arg commands --
|
||||
if cmd in TWO_ARG_COMMANDS:
|
||||
g1 = _find_brace_group(s, after)
|
||||
if g1:
|
||||
c1, p1 = g1
|
||||
g2 = _find_brace_group(s, p1)
|
||||
if g2:
|
||||
c2, p2 = g2
|
||||
func = TWO_ARG_COMMANDS[cmd]
|
||||
_emit(out, f"{func}({_convert(c1)}, {_convert(c2)})")
|
||||
i = p2
|
||||
continue
|
||||
# Fallthrough
|
||||
_emit(out, cmd)
|
||||
i = after
|
||||
continue
|
||||
|
||||
# -- one-arg commands --
|
||||
if cmd in ONE_ARG_COMMANDS:
|
||||
g = _find_brace_group(s, after)
|
||||
if g:
|
||||
content, pos = g
|
||||
func = ONE_ARG_COMMANDS[cmd]
|
||||
_emit(out, f"{func}({_convert(content)})")
|
||||
i = pos
|
||||
continue
|
||||
# Fallthrough: no brace group → just emit the typst name
|
||||
_emit(out, ONE_ARG_COMMANDS[cmd])
|
||||
i = after
|
||||
continue
|
||||
|
||||
# -- \rm (applies upright to the rest of the current scope) --
|
||||
if cmd == "rm":
|
||||
raw_rest = s[after:]
|
||||
leading = len(raw_rest) - len(raw_rest.lstrip())
|
||||
rest = raw_rest.lstrip()
|
||||
# Grab one "word"
|
||||
wm = re.match(r"[A-Za-z0-9]+", rest)
|
||||
if wm:
|
||||
word = wm.group()
|
||||
_emit(out, f"upright({word})")
|
||||
i = after + leading + len(word)
|
||||
continue
|
||||
_emit(out, "upright")
|
||||
i = after
|
||||
continue
|
||||
|
||||
# -- delimiter sizing --
|
||||
if cmd in _SIZING_COMMANDS:
|
||||
# Skip the command; keep whatever delimiter follows.
|
||||
i = after
|
||||
continue
|
||||
|
||||
# -- simple (same name) --
|
||||
if cmd in SIMPLE_COMMANDS:
|
||||
_emit(out, cmd)
|
||||
# Also add right-side space when next char would merge
|
||||
if after < n and (s[after].isalnum() or s[after] == "\\"):
|
||||
out.append(" ")
|
||||
i = after
|
||||
continue
|
||||
|
||||
# -- renamed --
|
||||
if cmd in RENAMED_COMMANDS:
|
||||
repl = RENAMED_COMMANDS[cmd]
|
||||
if repl:
|
||||
_emit(out, repl)
|
||||
if after < n and s[after].isalnum():
|
||||
out.append(" ")
|
||||
# If repl is empty the command is silently dropped.
|
||||
# For \label{...} consume the brace group too.
|
||||
if cmd == "label":
|
||||
g = _find_brace_group(s, after)
|
||||
if g:
|
||||
i = g[1]
|
||||
continue
|
||||
i = after
|
||||
continue
|
||||
|
||||
# -- unknown command → emit name without backslash --
|
||||
_emit(out, cmd)
|
||||
if after < n and s[after].isalnum():
|
||||
out.append(" ")
|
||||
i = after
|
||||
continue
|
||||
|
||||
# ---- brace groups (not consumed by a command) ----
|
||||
if ch == "{":
|
||||
g = _find_brace_group(s, i)
|
||||
if g:
|
||||
content, end = g
|
||||
# Check if preceded by ^ or _ → superscript/subscript grouping
|
||||
if out and out[-1] and out[-1][-1] in "^_":
|
||||
out.append(f"({_convert(content)})")
|
||||
i = end
|
||||
continue
|
||||
# Check for {\rm ...} pattern
|
||||
rm_m = re.match(r"\\rm\s+", content)
|
||||
if rm_m:
|
||||
inner = content[rm_m.end():]
|
||||
_emit(out, f"upright({_convert(inner)})")
|
||||
i = end
|
||||
continue
|
||||
# Otherwise, just emit the converted content (braces act as
|
||||
# invisible grouping in LaTeX — no Typst equivalent needed
|
||||
# in most contexts).
|
||||
_emit(out, _convert(content))
|
||||
i = end
|
||||
continue
|
||||
# Unmatched brace — emit as-is
|
||||
out.append(ch)
|
||||
i += 1
|
||||
continue
|
||||
|
||||
# ---- everything else (digits, letters, operators, whitespace) ----
|
||||
# Use _emit so consecutive raw letters get spaces inserted,
|
||||
# matching LaTeX math semantics where adjacent letters are
|
||||
# separate variables (e.g. "out" → "o u t" in Typst).
|
||||
_emit(out, ch)
|
||||
i += 1
|
||||
|
||||
result = "".join(out)
|
||||
# Typst math requires a base before ^ or _; add an invisible base
|
||||
# when the expression starts with a script marker (e.g. $^2$).
|
||||
if result and result.lstrip() and result.lstrip()[0] in "^_":
|
||||
result = '""' + result.lstrip()
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Environment converters
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _convert_environment(name: str, body: str) -> str:
|
||||
"""Convert a ``\\begin{name}…\\end{name}`` block to Typst."""
|
||||
if name in ("matrix", "bmatrix", "pmatrix", "vmatrix"):
|
||||
return _convert_matrix_env(name, body)
|
||||
if name == "cases":
|
||||
return _convert_cases_env(body)
|
||||
if name in ("aligned", "split"):
|
||||
# Just unwrap — Typst math handles & alignment and \ line-breaks.
|
||||
converted = _convert(body)
|
||||
return converted
|
||||
if name == "figure":
|
||||
# Not real math; pass through as-is.
|
||||
return f"\\begin{{{name}}}{body}\\end{{{name}}}"
|
||||
# Unknown environment — pass through converted content
|
||||
return _convert(body)
|
||||
|
||||
|
||||
def _convert_matrix_env(name: str, body: str) -> str:
|
||||
"""Convert matrix/bmatrix/pmatrix/vmatrix to ``mat(…)``."""
|
||||
delim_map = {
|
||||
"matrix": "",
|
||||
"bmatrix": '"["',
|
||||
"pmatrix": '"("',
|
||||
"vmatrix": '"|"',
|
||||
}
|
||||
# Split rows on \\, columns on &
|
||||
rows: list[str] = []
|
||||
for row_text in re.split(r"\\\\", body):
|
||||
row_text = row_text.strip()
|
||||
if not row_text:
|
||||
continue
|
||||
cells = [_convert(c.strip()) for c in row_text.split("&")]
|
||||
rows.append(", ".join(cells))
|
||||
|
||||
inner = "; ".join(rows)
|
||||
delim = delim_map.get(name, "")
|
||||
if delim:
|
||||
return f"mat(delim: {delim}, {inner})"
|
||||
return f"mat({inner})"
|
||||
|
||||
|
||||
def _convert_cases_env(body: str) -> str:
|
||||
"""Convert cases environment to ``cases(…)``."""
|
||||
branches: list[str] = []
|
||||
for branch_text in re.split(r"\\\\", body):
|
||||
branch_text = branch_text.strip()
|
||||
if not branch_text:
|
||||
continue
|
||||
branches.append(_convert(branch_text))
|
||||
return "cases(" + ", ".join(branches) + ")"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Markdown-level math-span detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_FENCE_RE = re.compile(r"^(`{3,}|~{3,})", re.MULTILINE)
|
||||
|
||||
|
||||
def _iter_math_spans(content: str):
|
||||
"""Yield ``(start, end, is_display)`` for every math span.
|
||||
|
||||
Skips spans inside fenced code blocks and inline code.
|
||||
"""
|
||||
n = len(content)
|
||||
i = 0
|
||||
in_fence: str | None = None # fence marker when inside a code block
|
||||
|
||||
while i < n:
|
||||
# Track fenced code blocks
|
||||
if content[i] == "`" or content[i] == "~":
|
||||
m = _FENCE_RE.match(content, i)
|
||||
if m and (i == 0 or content[i - 1] == "\n"):
|
||||
marker = m.group(1)
|
||||
if in_fence is None:
|
||||
in_fence = marker[0] # opening
|
||||
i = content.index("\n", i) + 1 if "\n" in content[i:] else n
|
||||
continue
|
||||
elif marker[0] == in_fence:
|
||||
in_fence = None # closing
|
||||
i = m.end()
|
||||
continue
|
||||
|
||||
if in_fence:
|
||||
i += 1
|
||||
continue
|
||||
|
||||
# Skip inline code
|
||||
if content[i] == "`":
|
||||
end_tick = content.find("`", i + 1)
|
||||
if end_tick != -1:
|
||||
i = end_tick + 1
|
||||
continue
|
||||
|
||||
# Display math $$...$$
|
||||
if content[i:i + 2] == "$$":
|
||||
start = i
|
||||
close = content.find("$$", i + 2)
|
||||
if close != -1:
|
||||
yield (start + 2, close, True)
|
||||
i = close + 2
|
||||
continue
|
||||
|
||||
# Inline math $...$
|
||||
if content[i] == "$":
|
||||
start = i
|
||||
# Find closing $ — any next $ closes the span (even if followed
|
||||
# by another $, which starts a NEW span).
|
||||
j = i + 1
|
||||
while j < n:
|
||||
if content[j] == "$":
|
||||
if j > i + 1: # non-empty
|
||||
yield (start + 1, j, False)
|
||||
j += 1
|
||||
break
|
||||
if content[j] == "\n" and not content[i + 1:j].strip():
|
||||
break # empty line → not math
|
||||
j += 1
|
||||
i = j
|
||||
continue
|
||||
|
||||
i += 1
|
||||
|
||||
|
||||
_CJK_RE = re.compile(r"[\u4e00-\u9fff\u3400-\u4dbf\uf900-\ufaff]")
|
||||
|
||||
|
||||
def convert_latex_math_to_typst(content: str) -> str:
|
||||
"""Replace LaTeX math with Typst math throughout *content* (markdown)."""
|
||||
spans = list(_iter_math_spans(content))
|
||||
if not spans:
|
||||
return content
|
||||
|
||||
parts: list[str] = []
|
||||
prev = 0
|
||||
for start, end, is_display in spans:
|
||||
delim = "$$" if is_display else "$"
|
||||
delim_len = len(delim)
|
||||
delim_start = start - delim_len
|
||||
|
||||
latex = content[start:end]
|
||||
|
||||
# Spans containing CJK characters are almost certainly mismatched $.
|
||||
# Strip the $ delimiters and emit the raw text.
|
||||
if _CJK_RE.search(latex):
|
||||
parts.append(content[prev:delim_start])
|
||||
parts.append(latex) # emit without $ delimiters
|
||||
prev = end + delim_len
|
||||
continue
|
||||
|
||||
parts.append(content[prev:delim_start])
|
||||
converted = _convert(latex)
|
||||
# Strip leading/trailing whitespace from inline math so that
|
||||
# ``$ text$`` (space after opening $) never occurs — CommonMark
|
||||
# and mdbook-typst-math treat that as non-math.
|
||||
if not is_display:
|
||||
converted = converted.strip()
|
||||
parts.append(f"{delim}{converted}{delim}")
|
||||
|
||||
prev = end + delim_len
|
||||
|
||||
parts.append(content[prev:])
|
||||
return "".join(parts)
|
||||
66
tools/mdbook_preprocessor.py
Normal file
66
tools/mdbook_preprocessor.py
Normal file
@@ -0,0 +1,66 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from tools.prepare_mdbook import build_title_cache, parse_bib, rewrite_markdown
|
||||
except ModuleNotFoundError:
|
||||
from prepare_mdbook import build_title_cache, parse_bib, rewrite_markdown
|
||||
|
||||
|
||||
PLACEHOLDER_PREFIX = "[TODO: src = zh_chapters/"
|
||||
BIBLIOGRAPHY_TITLE = "References"
|
||||
FRONTPAGE_SWITCH_LABEL = "中文"
|
||||
FRONTPAGE_SWITCH_HREF = "cn/"
|
||||
|
||||
|
||||
def iter_chapters(items: list[dict]) -> list[dict]:
|
||||
chapters: list[dict] = []
|
||||
for item in items:
|
||||
chapter = item.get("Chapter")
|
||||
if not chapter:
|
||||
continue
|
||||
chapters.append(chapter)
|
||||
chapters.extend(iter_chapters(chapter.get("sub_items", [])))
|
||||
return chapters
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) > 1 and sys.argv[1] == "supports":
|
||||
return 0
|
||||
|
||||
context, book = json.load(sys.stdin)
|
||||
root = Path(context["root"]).resolve()
|
||||
source_dir = (root / context["config"]["book"]["src"]).resolve()
|
||||
title_cache = build_title_cache(source_dir, placeholder_prefix=PLACEHOLDER_PREFIX)
|
||||
bib_path = source_dir.parent / "mlsys.bib"
|
||||
bib_db = parse_bib(bib_path) if bib_path.exists() else {}
|
||||
refs_dir = source_dir.parent / "references"
|
||||
if refs_dir.is_dir():
|
||||
for extra_bib in sorted(refs_dir.glob("*.bib")):
|
||||
for key, fields in parse_bib(extra_bib).items():
|
||||
bib_db.setdefault(key, fields)
|
||||
|
||||
for chapter in iter_chapters(book.get("items", [])):
|
||||
source_path = chapter.get("source_path") or chapter.get("path")
|
||||
if not source_path:
|
||||
continue
|
||||
current_file = (source_dir / source_path).resolve()
|
||||
chapter["content"] = rewrite_markdown(
|
||||
chapter["content"],
|
||||
current_file,
|
||||
title_cache,
|
||||
bib_db,
|
||||
bibliography_title=BIBLIOGRAPHY_TITLE,
|
||||
frontpage_switch_label=FRONTPAGE_SWITCH_LABEL,
|
||||
frontpage_switch_href=FRONTPAGE_SWITCH_HREF,
|
||||
)
|
||||
|
||||
json.dump(book, sys.stdout, ensure_ascii=False)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
68
tools/mdbook_zh_preprocessor.py
Normal file
68
tools/mdbook_zh_preprocessor.py
Normal file
@@ -0,0 +1,68 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from tools.prepare_mdbook import build_title_cache, parse_bib, rewrite_markdown
|
||||
from tools.latex_to_typst import convert_latex_math_to_typst
|
||||
except ModuleNotFoundError:
|
||||
from prepare_mdbook import build_title_cache, parse_bib, rewrite_markdown
|
||||
from latex_to_typst import convert_latex_math_to_typst
|
||||
|
||||
|
||||
BIBLIOGRAPHY_TITLE = "参考文献"
|
||||
FRONTPAGE_SWITCH_LABEL = "English"
|
||||
FRONTPAGE_SWITCH_HREF = "../"
|
||||
|
||||
|
||||
def iter_chapters(items: list[dict]) -> list[dict]:
|
||||
chapters: list[dict] = []
|
||||
for item in items:
|
||||
chapter = item.get("Chapter")
|
||||
if not chapter:
|
||||
continue
|
||||
chapters.append(chapter)
|
||||
chapters.extend(iter_chapters(chapter.get("sub_items", [])))
|
||||
return chapters
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) > 1 and sys.argv[1] == "supports":
|
||||
return 0
|
||||
|
||||
context, book = json.load(sys.stdin)
|
||||
root = Path(context["root"]).resolve()
|
||||
source_dir = (root / context["config"]["book"]["src"]).resolve()
|
||||
title_cache = build_title_cache(source_dir)
|
||||
bib_path = source_dir.parent / "mlsys.bib"
|
||||
bib_db = parse_bib(bib_path) if bib_path.exists() else {}
|
||||
refs_dir = source_dir.parent / "references"
|
||||
if refs_dir.is_dir():
|
||||
for extra_bib in sorted(refs_dir.glob("*.bib")):
|
||||
for key, fields in parse_bib(extra_bib).items():
|
||||
bib_db.setdefault(key, fields)
|
||||
|
||||
for chapter in iter_chapters(book.get("items", [])):
|
||||
source_path = chapter.get("source_path") or chapter.get("path")
|
||||
if not source_path:
|
||||
continue
|
||||
current_file = (source_dir / source_path).resolve()
|
||||
chapter["content"] = rewrite_markdown(
|
||||
chapter["content"],
|
||||
current_file,
|
||||
title_cache,
|
||||
bib_db,
|
||||
bibliography_title=BIBLIOGRAPHY_TITLE,
|
||||
frontpage_switch_label=FRONTPAGE_SWITCH_LABEL,
|
||||
frontpage_switch_href=FRONTPAGE_SWITCH_HREF,
|
||||
)
|
||||
chapter["content"] = convert_latex_math_to_typst(chapter["content"])
|
||||
|
||||
json.dump(book, sys.stdout, ensure_ascii=False)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
790
tools/prepare_mdbook.py
Normal file
790
tools/prepare_mdbook.py
Normal file
@@ -0,0 +1,790 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
TOC_FENCE = "toc"
|
||||
EVAL_RST_FENCE = "eval_rst"
|
||||
OPTION_LINE_RE = re.compile(r"^:(width|label):`[^`]+`\s*$", re.MULTILINE)
|
||||
NUMREF_RE = re.compile(r":numref:`([^`]+)`")
|
||||
EQREF_RE = re.compile(r":eqref:`([^`]+)`")
|
||||
EQLABEL_LINE_RE = re.compile(r"^:eqlabel:`([^`]+)`\s*$")
|
||||
CITE_RE = re.compile(r":cite:`([^`]+)`")
|
||||
BIB_ENTRY_RE = re.compile(r"@(\w+)\{([^,]+),")
|
||||
LATEX_ESCAPE_RE = re.compile(r"\\([_%#&])")
|
||||
RAW_HTML_FILE_RE = re.compile(r"^\s*:file:\s*([^\s]+)\s*$")
|
||||
TOC_LINK_RE = re.compile(r"^\[([^\]]+)\]\(([^)]+)\)\s*$")
|
||||
TOC_PART_RE = re.compile(r"^#+\s+(.+?)\s*$")
|
||||
HEAD_TAG_RE = re.compile(r"</?head>", re.IGNORECASE)
|
||||
STYLE_BLOCK_RE = re.compile(r"<style>(.*?)</style>", re.IGNORECASE | re.DOTALL)
|
||||
DEFAULT_BIBLIOGRAPHY_TITLE = "References"
|
||||
FRONTPAGE_SWITCH_PLACEHOLDER = "<!-- OPENMLSYS_LANGUAGE_SWITCH -->"
|
||||
FRONTPAGE_LAYOUT_CSS = """
|
||||
<style>
|
||||
.openmlsys-frontpage {
|
||||
width: 100%;
|
||||
margin: 0 auto 3rem;
|
||||
margin-inline: auto;
|
||||
}
|
||||
.openmlsys-frontpage-switch-row {
|
||||
margin: 12px 0 0;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
}
|
||||
.openmlsys-frontpage-switch {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
min-width: 82px;
|
||||
height: 28px;
|
||||
padding: 0 14px;
|
||||
border-radius: 6px;
|
||||
border: 1px solid rgba(31, 35, 40, 0.15);
|
||||
background: #f6f8fa;
|
||||
color: #24292f;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
text-decoration: none;
|
||||
box-shadow: 0 1px 0 rgba(31, 35, 40, 0.04);
|
||||
}
|
||||
.openmlsys-frontpage-switch:hover {
|
||||
background: #f3f4f6;
|
||||
border-color: rgba(31, 35, 40, 0.2);
|
||||
}
|
||||
.openmlsys-frontpage .mdl-grid {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 24px;
|
||||
width: 100%;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
.openmlsys-frontpage .mdl-cell {
|
||||
box-sizing: border-box;
|
||||
flex: 1 1 220px;
|
||||
min-width: 0;
|
||||
}
|
||||
.openmlsys-frontpage .mdl-cell--1-col {
|
||||
flex: 0 0 48px;
|
||||
}
|
||||
.openmlsys-frontpage .mdl-cell--3-col {
|
||||
flex: 0 1 calc(16.666% - 20px);
|
||||
max-width: calc(16.666% - 20px);
|
||||
}
|
||||
.openmlsys-frontpage .authors.mdl-grid {
|
||||
justify-content: center;
|
||||
}
|
||||
.openmlsys-frontpage .mdl-cell--5-col {
|
||||
flex: 1 1 calc(41.666% - 24px);
|
||||
max-width: calc(41.666% - 18px);
|
||||
}
|
||||
.openmlsys-frontpage .mdl-cell--12-col {
|
||||
flex: 1 1 100%;
|
||||
max-width: 100%;
|
||||
}
|
||||
.openmlsys-frontpage .mdl-cell--middle {
|
||||
align-self: center;
|
||||
}
|
||||
.openmlsys-frontpage .mdl-color-text--primary {
|
||||
color: var(--links, #0b6bcb);
|
||||
}
|
||||
.openmlsys-frontpage img {
|
||||
max-width: 100%;
|
||||
height: auto;
|
||||
background: transparent !important;
|
||||
padding: 0 !important;
|
||||
}
|
||||
.openmlsys-frontpage + ul,
|
||||
.openmlsys-frontpage + ul ul {
|
||||
max-width: 960px;
|
||||
margin-inline: auto;
|
||||
}
|
||||
.content main {
|
||||
max-width: min(100%, max(65%, var(--content-max-width)));
|
||||
}
|
||||
@media (max-width: 1000px) {
|
||||
.openmlsys-frontpage .mdl-cell,
|
||||
.openmlsys-frontpage .mdl-cell--1-col,
|
||||
.openmlsys-frontpage .mdl-cell--3-col,
|
||||
.openmlsys-frontpage .mdl-cell--5-col {
|
||||
flex: 1 1 100%;
|
||||
max-width: 100%;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
""".strip()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TocItem:
|
||||
kind: str
|
||||
label: str
|
||||
target: str | None = None
|
||||
|
||||
|
||||
def is_placeholder_markdown(markdown: str, placeholder_prefix: str | None = None) -> bool:
|
||||
if not placeholder_prefix:
|
||||
return False
|
||||
|
||||
stripped = markdown.strip()
|
||||
return stripped.startswith(placeholder_prefix) and stripped.endswith("]")
|
||||
|
||||
|
||||
def extract_title(markdown: str, fallback: str = "Untitled") -> str:
|
||||
lines = markdown.splitlines()
|
||||
|
||||
for index, line in enumerate(lines):
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
if stripped.startswith("#"):
|
||||
heading = stripped.lstrip("#").strip()
|
||||
if heading:
|
||||
return heading
|
||||
|
||||
next_index = index + 1
|
||||
if next_index < len(lines):
|
||||
underline = lines[next_index].strip()
|
||||
if underline and set(underline) <= {"=", "-"}:
|
||||
return stripped
|
||||
|
||||
return fallback
|
||||
|
||||
|
||||
def parse_toc_entries(block_lines: list[str]) -> list[TocItem]:
|
||||
entries: list[TocItem] = []
|
||||
for line in block_lines:
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith(":"):
|
||||
continue
|
||||
part_match = TOC_PART_RE.match(stripped)
|
||||
if part_match:
|
||||
entries.append(TocItem(kind="part", label=part_match.group(1).strip()))
|
||||
continue
|
||||
link_match = TOC_LINK_RE.match(stripped)
|
||||
if link_match:
|
||||
entries.append(
|
||||
TocItem(
|
||||
kind="chapter",
|
||||
label=link_match.group(1).strip(),
|
||||
target=link_match.group(2).strip(),
|
||||
)
|
||||
)
|
||||
continue
|
||||
entries.append(TocItem(kind="chapter", label="", target=stripped))
|
||||
return entries
|
||||
|
||||
|
||||
def parse_toc_blocks(markdown: str) -> list[list[TocItem]]:
|
||||
blocks: list[list[TocItem]] = []
|
||||
lines = markdown.splitlines()
|
||||
index = 0
|
||||
|
||||
while index < len(lines):
|
||||
if lines[index].strip() == f"```{TOC_FENCE}":
|
||||
index += 1
|
||||
block_lines: list[str] = []
|
||||
while index < len(lines) and lines[index].strip() != "```":
|
||||
block_lines.append(lines[index])
|
||||
index += 1
|
||||
entries = parse_toc_entries(block_lines)
|
||||
blocks.append(entries)
|
||||
index += 1
|
||||
|
||||
return blocks
|
||||
|
||||
|
||||
def resolve_toc_target(current_file: Path, entry: str) -> Path:
|
||||
target_name = entry if entry.endswith(".md") else f"{entry}.md"
|
||||
target = (current_file.parent / target_name).resolve()
|
||||
if not target.exists():
|
||||
raise FileNotFoundError(f"TOC entry '{entry}' from '{current_file}' does not exist")
|
||||
return target
|
||||
|
||||
|
||||
def relative_link(from_file: Path, target_file: Path) -> str:
|
||||
return Path(os.path.relpath(target_file, start=from_file.parent)).as_posix()
|
||||
|
||||
|
||||
def _strip_latex_escapes_outside_math(text: str) -> str:
|
||||
"""Remove LaTeX text-mode escapes (``\\_``, ``\\#``, etc.) outside math
|
||||
spans, fenced code blocks, and inline code.
|
||||
|
||||
Operates on the full text (not per-line) to correctly handle multi-line
|
||||
display math ``$$...$$`` blocks.
|
||||
"""
|
||||
# 1. Find all protected regions where escapes must NOT be stripped.
|
||||
protected: list[tuple[int, int]] = [] # (start, end)
|
||||
n = len(text)
|
||||
i = 0
|
||||
in_fence: str | None = None
|
||||
fence_start = 0
|
||||
|
||||
while i < n:
|
||||
# Fenced code blocks (``` or ~~~) at start of line
|
||||
if (i == 0 or text[i - 1] == "\n") and text[i] in ("`", "~"):
|
||||
m = re.match(r"`{3,}|~{3,}", text[i:])
|
||||
if m:
|
||||
if in_fence is None:
|
||||
in_fence = m.group()[0]
|
||||
fence_start = i
|
||||
elif m.group()[0] == in_fence:
|
||||
eol = text.find("\n", m.end() + i)
|
||||
end = eol + 1 if eol != -1 else n
|
||||
protected.append((fence_start, end))
|
||||
in_fence = None
|
||||
i = end
|
||||
continue
|
||||
eol = text.find("\n", i)
|
||||
i = eol + 1 if eol != -1 else n
|
||||
continue
|
||||
|
||||
if in_fence is not None:
|
||||
i += 1
|
||||
continue
|
||||
|
||||
# Inline code `...`
|
||||
if text[i] == "`":
|
||||
close = text.find("`", i + 1)
|
||||
if close != -1:
|
||||
protected.append((i, close + 1))
|
||||
i = close + 1
|
||||
continue
|
||||
|
||||
# Display math $$...$$
|
||||
if text[i:i + 2] == "$$":
|
||||
close = text.find("$$", i + 2)
|
||||
if close != -1:
|
||||
protected.append((i, close + 2))
|
||||
i = close + 2
|
||||
continue
|
||||
|
||||
# Inline math $...$
|
||||
if text[i] == "$":
|
||||
j = i + 1
|
||||
while j < n and text[j] != "$" and text[j] != "\n":
|
||||
j += 1
|
||||
if j < n and text[j] == "$" and j > i + 1:
|
||||
protected.append((i, j + 1))
|
||||
i = j + 1
|
||||
continue
|
||||
|
||||
i += 1
|
||||
|
||||
# Unclosed fence → protect everything from fence_start to end
|
||||
if in_fence is not None:
|
||||
protected.append((fence_start, n))
|
||||
|
||||
# 2. Apply substitution only to unprotected gaps.
|
||||
parts: list[str] = []
|
||||
prev = 0
|
||||
for start, end in protected:
|
||||
if start > prev:
|
||||
parts.append(LATEX_ESCAPE_RE.sub(r"\1", text[prev:start]))
|
||||
parts.append(text[start:end])
|
||||
prev = end
|
||||
if prev < n:
|
||||
parts.append(LATEX_ESCAPE_RE.sub(r"\1", text[prev:]))
|
||||
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def process_equation_labels(markdown: str) -> tuple[str, dict[str, int]]:
|
||||
"""Convert :eqlabel: directives to MathJax \\tag + \\label in preceding equations.
|
||||
|
||||
Args:
|
||||
markdown: The markdown content to process.
|
||||
|
||||
Returns:
|
||||
A tuple of (processed_markdown, label_map) where label_map maps
|
||||
label names to their equation numbers.
|
||||
"""
|
||||
lines = markdown.split("\n")
|
||||
result: list[str] = []
|
||||
eq_counter = 0
|
||||
label_map: dict[str, int] = {}
|
||||
|
||||
for line in lines:
|
||||
match = EQLABEL_LINE_RE.match(line.strip())
|
||||
if not match:
|
||||
result.append(line)
|
||||
continue
|
||||
|
||||
label_name = match.group(1)
|
||||
eq_counter += 1
|
||||
label_map[label_name] = eq_counter
|
||||
tag = f"\\tag{{{eq_counter}}}\\label{{{label_name}}}"
|
||||
|
||||
# Search backward for the closing $$ of the preceding equation
|
||||
inserted = False
|
||||
for j in range(len(result) - 1, -1, -1):
|
||||
stripped = result[j].rstrip()
|
||||
if not stripped:
|
||||
continue # skip blank lines
|
||||
if stripped == "$$":
|
||||
# Multi-line equation: $$ on its own line
|
||||
result.insert(j, tag)
|
||||
inserted = True
|
||||
break
|
||||
if stripped.endswith("$$"):
|
||||
# Single-line or end-of-content $$
|
||||
result[j] = stripped[:-2] + tag + "$$"
|
||||
inserted = True
|
||||
break
|
||||
break # non-blank, non-$$ line: no equation found
|
||||
|
||||
if not inserted:
|
||||
# Fallback: keep original line if no equation found
|
||||
result.append(line)
|
||||
|
||||
return "\n".join(result), label_map
|
||||
|
||||
|
||||
def normalize_directives(
|
||||
markdown: str,
|
||||
label_map: dict[str, int] | None = None,
|
||||
) -> str:
|
||||
normalized = OPTION_LINE_RE.sub("", markdown)
|
||||
normalized = NUMREF_RE.sub(lambda match: f"`{match.group(1)}`", normalized)
|
||||
if label_map:
|
||||
normalized = EQREF_RE.sub(
|
||||
lambda m: f"({label_map[m.group(1)]})" if m.group(1) in label_map else f"$\\eqref{{{m.group(1)}}}$",
|
||||
normalized,
|
||||
)
|
||||
else:
|
||||
normalized = EQREF_RE.sub(lambda match: f"$\\eqref{{{match.group(1)}}}$", normalized)
|
||||
|
||||
normalized = _strip_latex_escapes_outside_math(normalized)
|
||||
|
||||
lines = [line.rstrip() for line in normalized.splitlines()]
|
||||
collapsed: list[str] = []
|
||||
previous_blank = False
|
||||
for line in lines:
|
||||
is_blank = line == ""
|
||||
if is_blank and previous_blank:
|
||||
continue
|
||||
collapsed.append(line)
|
||||
previous_blank = is_blank
|
||||
|
||||
while collapsed and collapsed[-1] == "":
|
||||
collapsed.pop()
|
||||
|
||||
return "\n".join(collapsed) + "\n"
|
||||
|
||||
|
||||
def clean_bibtex(value: str) -> str:
|
||||
value = re.sub(r"\{\\[`'^\"~=.](\w)\}", r"\1", value)
|
||||
value = re.sub(r"\\[`'^\"~=.](\w)", r"\1", value)
|
||||
value = value.replace("{", "").replace("}", "")
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _parse_bib_fields(body: str) -> dict[str, str]:
|
||||
fields: dict[str, str] = {}
|
||||
i = 0
|
||||
while i < len(body):
|
||||
while i < len(body) and body[i] in " \t\n\r,":
|
||||
i += 1
|
||||
if i >= len(body):
|
||||
break
|
||||
start = i
|
||||
while i < len(body) and body[i] not in "= \t\n\r":
|
||||
i += 1
|
||||
name = body[start:i].strip().lower()
|
||||
while i < len(body) and body[i] != "=":
|
||||
i += 1
|
||||
if i >= len(body):
|
||||
break
|
||||
i += 1
|
||||
while i < len(body) and body[i] in " \t\n\r":
|
||||
i += 1
|
||||
if i >= len(body):
|
||||
break
|
||||
if body[i] == "{":
|
||||
depth = 1
|
||||
i += 1
|
||||
vstart = i
|
||||
while i < len(body) and depth > 0:
|
||||
if body[i] == "{":
|
||||
depth += 1
|
||||
elif body[i] == "}":
|
||||
depth -= 1
|
||||
i += 1
|
||||
value = body[vstart : i - 1]
|
||||
elif body[i] == '"':
|
||||
i += 1
|
||||
vstart = i
|
||||
while i < len(body) and body[i] != '"':
|
||||
i += 1
|
||||
value = body[vstart:i]
|
||||
i += 1
|
||||
else:
|
||||
vstart = i
|
||||
while i < len(body) and body[i] not in ", \t\n\r}":
|
||||
i += 1
|
||||
value = body[vstart:i]
|
||||
if name:
|
||||
fields[name] = value.strip()
|
||||
return fields
|
||||
|
||||
|
||||
def parse_bib(bib_path: Path) -> dict[str, dict[str, str]]:
|
||||
text = bib_path.read_text(encoding="utf-8")
|
||||
entries: dict[str, dict[str, str]] = {}
|
||||
for match in BIB_ENTRY_RE.finditer(text):
|
||||
key = match.group(2).strip()
|
||||
start = match.end()
|
||||
depth = 1
|
||||
pos = start
|
||||
while pos < len(text) and depth > 0:
|
||||
if text[pos] == "{":
|
||||
depth += 1
|
||||
elif text[pos] == "}":
|
||||
depth -= 1
|
||||
pos += 1
|
||||
fields = _parse_bib_fields(text[start : pos - 1])
|
||||
fields["_type"] = match.group(1).lower()
|
||||
entries[key] = fields
|
||||
return entries
|
||||
|
||||
|
||||
def _render_bibliography(
|
||||
cited_keys: list[str],
|
||||
bib_db: dict[str, dict[str, str]],
|
||||
bibliography_title: str,
|
||||
) -> list[str]:
|
||||
lines: list[str] = ["---", "", f"## {bibliography_title}", "", "<ol>"]
|
||||
for key in cited_keys:
|
||||
entry = bib_db.get(key)
|
||||
if not entry:
|
||||
continue
|
||||
author = clean_bibtex(entry.get("author", ""))
|
||||
title = clean_bibtex(entry.get("title", ""))
|
||||
year = entry.get("year", "")
|
||||
venue = clean_bibtex(entry.get("journal", "") or entry.get("booktitle", ""))
|
||||
parts: list[str] = []
|
||||
if author:
|
||||
parts.append(author)
|
||||
if title:
|
||||
parts.append(f"<em>{title}</em>")
|
||||
if venue:
|
||||
parts.append(venue)
|
||||
if year:
|
||||
parts.append(year)
|
||||
text = ". ".join(parts) + "." if parts else f"{key}."
|
||||
lines.append(f'<li id="ref-{key}">{text} <a href="#cite-{key}">↩</a></li>')
|
||||
lines.append("</ol>")
|
||||
return lines
|
||||
|
||||
|
||||
def process_citations(
|
||||
markdown: str,
|
||||
bib_db: dict[str, dict[str, str]],
|
||||
bibliography_title: str = DEFAULT_BIBLIOGRAPHY_TITLE,
|
||||
) -> str:
|
||||
cited_keys: list[str] = []
|
||||
|
||||
def _replace_cite(match: re.Match[str]) -> str:
|
||||
keys = [k.strip() for k in match.group(1).split(",")]
|
||||
for key in keys:
|
||||
if key not in cited_keys and key in bib_db:
|
||||
cited_keys.append(key)
|
||||
if not bib_db:
|
||||
return "[" + ", ".join(keys) + "]"
|
||||
nums: list[str] = []
|
||||
for key in keys:
|
||||
if key not in bib_db:
|
||||
continue
|
||||
idx = cited_keys.index(key) + 1
|
||||
nums.append(f'<sup id="cite-{key}"><a href="#ref-{key}">[{idx}]</a></sup>')
|
||||
return "".join(nums)
|
||||
|
||||
processed = CITE_RE.sub(_replace_cite, markdown)
|
||||
if cited_keys and bib_db:
|
||||
bib_lines = _render_bibliography(cited_keys, bib_db, bibliography_title)
|
||||
processed = processed.rstrip("\n") + "\n\n" + "\n".join(bib_lines) + "\n"
|
||||
return processed
|
||||
|
||||
|
||||
def resolve_raw_html_file(current_file: Path, filename: str) -> Path:
|
||||
direct = (current_file.parent / filename).resolve()
|
||||
if direct.exists():
|
||||
return direct
|
||||
|
||||
static_fallback = (current_file.parent / "static" / filename).resolve()
|
||||
if static_fallback.exists():
|
||||
return static_fallback
|
||||
|
||||
repo_static = (Path(__file__).resolve().parent.parent / "static" / filename)
|
||||
if repo_static.exists():
|
||||
return repo_static
|
||||
|
||||
raise FileNotFoundError(f"Raw HTML include '{filename}' from '{current_file}' does not exist")
|
||||
|
||||
|
||||
def rewrite_frontpage_assets(html: str) -> str:
|
||||
rewritten = html.replace("./_images/", "static/image/")
|
||||
rewritten = rewritten.replace("_images/", "static/image/")
|
||||
rewritten = HEAD_TAG_RE.sub("", rewritten)
|
||||
rewritten = STYLE_BLOCK_RE.sub(_minify_style_block, rewritten)
|
||||
return rewritten
|
||||
|
||||
|
||||
def _minify_style_block(match: re.Match[str]) -> str:
|
||||
content = match.group(1)
|
||||
parts = [line.strip() for line in content.splitlines() if line.strip()]
|
||||
return f"<style>{' '.join(parts)}</style>"
|
||||
|
||||
|
||||
def render_frontpage_switch(label: str, href: str) -> str:
|
||||
return (
|
||||
'<p class="openmlsys-frontpage-switch-row">'
|
||||
f'<a class="openmlsys-frontpage-switch" href="{href}">{label}</a>'
|
||||
"</p>"
|
||||
)
|
||||
|
||||
|
||||
def wrap_frontpage_html(
|
||||
html: str,
|
||||
frontpage_switch_label: str | None = None,
|
||||
frontpage_switch_href: str | None = None,
|
||||
) -> str:
|
||||
rendered_html = html.strip()
|
||||
if frontpage_switch_label and frontpage_switch_href:
|
||||
switch_html = render_frontpage_switch(frontpage_switch_label, frontpage_switch_href)
|
||||
if FRONTPAGE_SWITCH_PLACEHOLDER in rendered_html:
|
||||
rendered_html = rendered_html.replace(FRONTPAGE_SWITCH_PLACEHOLDER, switch_html)
|
||||
else:
|
||||
rendered_html = "\n".join([switch_html, rendered_html])
|
||||
|
||||
parts = [FRONTPAGE_LAYOUT_CSS, '<div class="openmlsys-frontpage">', rendered_html, "</div>"]
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def inline_raw_html(
|
||||
block_lines: list[str],
|
||||
current_file: Path,
|
||||
frontpage_switch_label: str | None = None,
|
||||
frontpage_switch_href: str | None = None,
|
||||
) -> str | None:
|
||||
stripped = [line.strip() for line in block_lines if line.strip()]
|
||||
if not stripped or stripped[0] != ".. raw:: html":
|
||||
return None
|
||||
|
||||
filename: str | None = None
|
||||
for line in stripped[1:]:
|
||||
match = RAW_HTML_FILE_RE.match(line)
|
||||
if match:
|
||||
filename = match.group(1)
|
||||
break
|
||||
|
||||
if filename is None:
|
||||
return None
|
||||
|
||||
html_path = resolve_raw_html_file(current_file, filename)
|
||||
html = rewrite_frontpage_assets(html_path.read_text(encoding="utf-8")).strip()
|
||||
if Path(filename).name == "frontpage.html":
|
||||
return wrap_frontpage_html(
|
||||
html,
|
||||
frontpage_switch_label=frontpage_switch_label,
|
||||
frontpage_switch_href=frontpage_switch_href,
|
||||
)
|
||||
return html
|
||||
|
||||
|
||||
def chapter_label(item: TocItem, target: Path, title_cache: dict[Path, str]) -> str:
|
||||
return item.label or title_cache[target]
|
||||
|
||||
|
||||
def render_toc_list(entries: list[TocItem], current_file: Path, title_cache: dict[Path, str]) -> list[str]:
|
||||
rendered: list[str] = []
|
||||
current_indent = 0
|
||||
for entry in entries:
|
||||
if entry.kind == "part":
|
||||
rendered.append(f"- {entry.label}")
|
||||
current_indent = 1
|
||||
continue
|
||||
|
||||
if entry.target is None:
|
||||
continue
|
||||
|
||||
target = resolve_toc_target(current_file, entry.target)
|
||||
if target not in title_cache:
|
||||
continue
|
||||
|
||||
label = chapter_label(entry, target, title_cache)
|
||||
rendered.append(f"{' ' * current_indent}- [{label}]({relative_link(current_file, target)})")
|
||||
return rendered
|
||||
|
||||
|
||||
def rewrite_markdown(
|
||||
markdown: str,
|
||||
current_file: Path,
|
||||
title_cache: dict[Path, str],
|
||||
bib_db: dict[str, dict[str, str]] | None = None,
|
||||
bibliography_title: str = DEFAULT_BIBLIOGRAPHY_TITLE,
|
||||
frontpage_switch_label: str | None = None,
|
||||
frontpage_switch_href: str | None = None,
|
||||
) -> str:
|
||||
output: list[str] = []
|
||||
lines = markdown.splitlines()
|
||||
index = 0
|
||||
|
||||
while index < len(lines):
|
||||
stripped = lines[index].strip()
|
||||
if stripped in (f"```{TOC_FENCE}", f"```{EVAL_RST_FENCE}"):
|
||||
fence = stripped[3:]
|
||||
index += 1
|
||||
block_lines: list[str] = []
|
||||
while index < len(lines) and lines[index].strip() != "```":
|
||||
block_lines.append(lines[index])
|
||||
index += 1
|
||||
|
||||
if fence == TOC_FENCE:
|
||||
entries = parse_toc_entries(block_lines)
|
||||
if entries:
|
||||
if output and output[-1] != "":
|
||||
output.append("")
|
||||
rendered = render_toc_list(entries, current_file, title_cache)
|
||||
output.extend(rendered)
|
||||
if rendered and output and output[-1] != "":
|
||||
output.append("")
|
||||
elif fence == EVAL_RST_FENCE:
|
||||
raw_html = inline_raw_html(
|
||||
block_lines,
|
||||
current_file,
|
||||
frontpage_switch_label=frontpage_switch_label,
|
||||
frontpage_switch_href=frontpage_switch_href,
|
||||
)
|
||||
if raw_html:
|
||||
if output and output[-1] != "":
|
||||
output.append("")
|
||||
output.extend(raw_html.splitlines())
|
||||
if output and output[-1] != "":
|
||||
output.append("")
|
||||
index += 1
|
||||
continue
|
||||
|
||||
output.append(lines[index])
|
||||
index += 1
|
||||
|
||||
while output and output[-1] == "":
|
||||
output.pop()
|
||||
|
||||
raw = "\n".join(output) + "\n"
|
||||
result, label_map = process_equation_labels(raw)
|
||||
result = normalize_directives(result, label_map=label_map)
|
||||
result = process_citations(result, bib_db or {}, bibliography_title=bibliography_title)
|
||||
return result
|
||||
|
||||
|
||||
def build_title_cache(
|
||||
source_dir: Path,
|
||||
placeholder_prefix: str | None = None,
|
||||
) -> dict[Path, str]:
|
||||
cache: dict[Path, str] = {}
|
||||
for markdown_file in sorted(source_dir.rglob("*.md")):
|
||||
if "_build" in markdown_file.parts or markdown_file.name == "SUMMARY.md":
|
||||
continue
|
||||
text = markdown_file.read_text(encoding="utf-8")
|
||||
if is_placeholder_markdown(text, placeholder_prefix):
|
||||
continue
|
||||
cache[markdown_file.resolve()] = extract_title(text, fallback=markdown_file.stem)
|
||||
return cache
|
||||
|
||||
|
||||
def build_summary(source_dir: Path, title_cache: dict[Path, str]) -> str:
|
||||
root_index = (source_dir / "index.md").resolve()
|
||||
root_markdown = root_index.read_text(encoding="utf-8")
|
||||
|
||||
lines = ["# Summary", "", f"[{title_cache[root_index]}](index.md)"]
|
||||
seen: set[Path] = {root_index}
|
||||
|
||||
def append_entry(target: Path, indent: int, label: str | None = None) -> None:
|
||||
target = target.resolve()
|
||||
if target in seen or target not in title_cache:
|
||||
return
|
||||
seen.add(target)
|
||||
rel = target.relative_to(source_dir.resolve()).as_posix()
|
||||
title = label or title_cache[target]
|
||||
lines.append(f"{' ' * indent}- [{title}]({rel})")
|
||||
|
||||
child_markdown = target.read_text(encoding="utf-8")
|
||||
for block in parse_toc_blocks(child_markdown):
|
||||
for entry in block:
|
||||
if entry.kind != "chapter" or entry.target is None:
|
||||
continue
|
||||
append_entry(resolve_toc_target(target, entry.target), indent + 1, entry.label or None)
|
||||
|
||||
def append_prefix_chapter(target: Path, label: str | None = None) -> None:
|
||||
target = target.resolve()
|
||||
if target in seen or target not in title_cache:
|
||||
return
|
||||
seen.add(target)
|
||||
rel = target.relative_to(source_dir.resolve()).as_posix()
|
||||
title = label or title_cache[target]
|
||||
lines.append(f"[{title}]({rel})")
|
||||
|
||||
numbered_started = False
|
||||
for block in parse_toc_blocks(root_markdown):
|
||||
for entry in block:
|
||||
if entry.kind == "part":
|
||||
if lines and lines[-1] != "":
|
||||
lines.append("")
|
||||
lines.append(f"# {entry.label}")
|
||||
lines.append("")
|
||||
numbered_started = True
|
||||
continue
|
||||
|
||||
if entry.target is None:
|
||||
continue
|
||||
|
||||
target = resolve_toc_target(root_index, entry.target)
|
||||
if numbered_started:
|
||||
append_entry(target, 0, entry.label or None)
|
||||
else:
|
||||
append_prefix_chapter(target, entry.label or None)
|
||||
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def write_summary(
|
||||
source_dir: Path,
|
||||
summary_path: Path | None = None,
|
||||
placeholder_prefix: str | None = None,
|
||||
) -> Path:
|
||||
source_dir = source_dir.resolve()
|
||||
summary_path = summary_path.resolve() if summary_path else (source_dir / "SUMMARY.md")
|
||||
title_cache = build_title_cache(source_dir, placeholder_prefix=placeholder_prefix)
|
||||
summary_path.write_text(build_summary(source_dir, title_cache), encoding="utf-8")
|
||||
return summary_path
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Generate mdBook SUMMARY.md for a chapter directory.")
|
||||
parser.add_argument("--source", type=Path, required=True, help="Source chapter directory")
|
||||
parser.add_argument("--summary-output", type=Path, required=True, help="Where to write the generated SUMMARY.md")
|
||||
parser.add_argument(
|
||||
"--placeholder-prefix",
|
||||
default=None,
|
||||
help="If set, files whose entire contents start with this prefix are skipped from mdBook output.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
summary_path = write_summary(
|
||||
args.source,
|
||||
summary_path=args.summary_output,
|
||||
placeholder_prefix=args.placeholder_prefix,
|
||||
)
|
||||
print(f"Wrote mdBook summary to {summary_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
75
tools/prepare_mdbook_zh.py
Normal file
75
tools/prepare_mdbook_zh.py
Normal file
@@ -0,0 +1,75 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from tools.prepare_mdbook import (
|
||||
build_title_cache,
|
||||
extract_title,
|
||||
inline_raw_html,
|
||||
parse_bib,
|
||||
process_equation_labels,
|
||||
rewrite_markdown,
|
||||
write_summary,
|
||||
)
|
||||
except ModuleNotFoundError:
|
||||
from prepare_mdbook import (
|
||||
build_title_cache,
|
||||
extract_title,
|
||||
inline_raw_html,
|
||||
parse_bib,
|
||||
process_equation_labels,
|
||||
rewrite_markdown,
|
||||
write_summary,
|
||||
)
|
||||
|
||||
|
||||
FRONTPAGE_SWITCH_LABEL = "English"
|
||||
FRONTPAGE_SWITCH_HREF = "../"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Generate mdBook SUMMARY.md for zh_chapters.")
|
||||
parser.add_argument("--source", type=Path, default=Path("zh_chapters"), help="Source chapter directory")
|
||||
parser.add_argument(
|
||||
"--summary-output",
|
||||
type=Path,
|
||||
default=Path("zh_chapters/SUMMARY.md"),
|
||||
help="Where to write the generated SUMMARY.md",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
summary_path = write_summary(args.source, summary_path=args.summary_output)
|
||||
print(f"Wrote mdBook summary to {summary_path}")
|
||||
return 0
|
||||
|
||||
|
||||
def rewrite_markdown(
|
||||
markdown: str,
|
||||
current_file: Path,
|
||||
title_cache: dict[Path, str],
|
||||
bib_db: dict[str, dict[str, str]] | None = None,
|
||||
bibliography_title: str = "参考文献",
|
||||
) -> str:
|
||||
try:
|
||||
from tools.prepare_mdbook import rewrite_markdown as generic_rewrite_markdown
|
||||
except ModuleNotFoundError:
|
||||
from prepare_mdbook import rewrite_markdown as generic_rewrite_markdown
|
||||
|
||||
return generic_rewrite_markdown(
|
||||
markdown,
|
||||
current_file,
|
||||
title_cache,
|
||||
bib_db=bib_db,
|
||||
bibliography_title=bibliography_title,
|
||||
frontpage_switch_label=FRONTPAGE_SWITCH_LABEL,
|
||||
frontpage_switch_href=FRONTPAGE_SWITCH_HREF,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
110
zh_chapters/SUMMARY.md
Normal file
110
zh_chapters/SUMMARY.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Summary
|
||||
|
||||
[机器学习系统:设计和实现](index.md)
|
||||
[前言](chapter_preface/index.md)
|
||||
|
||||
# 基础篇
|
||||
|
||||
- [导论](chapter_introduction/index.md)
|
||||
- [机器学习应用](chapter_introduction/applications.md)
|
||||
- [机器学习框架的设计目标](chapter_introduction/design.md)
|
||||
- [机器学习框架的基本组成原理](chapter_introduction/architecture.md)
|
||||
- [机器学习系统生态](chapter_introduction/ecosystem.md)
|
||||
- [图书结构和读者](chapter_introduction/readers.md)
|
||||
- [编程接口](chapter_programming_interface/index.md)
|
||||
- [机器学习系统编程模型的演进](chapter_programming_interface/development_history.md)
|
||||
- [机器学习工作流](chapter_programming_interface/ml_workflow.md)
|
||||
- [定义深度神经网络](chapter_programming_interface/neural_network_layer.md)
|
||||
- [C/C++编程接口](chapter_programming_interface/c_python_interaction.md)
|
||||
- [机器学习框架的编程范式](chapter_programming_interface/ml_programming_paradigm.md)
|
||||
- [总结](chapter_programming_interface/summary.md)
|
||||
- [计算图](chapter_computational_graph/index.md)
|
||||
- [计算图的设计背景和作用](chapter_computational_graph/background_and_functionality.md)
|
||||
- [计算图的基本构成](chapter_computational_graph/components_of_computational_graph.md)
|
||||
- [计算图的生成](chapter_computational_graph/generation_of_computational_graph.md)
|
||||
- [计算图的调度](chapter_computational_graph/schedule_of_computational_graph.md)
|
||||
- [总结](chapter_computational_graph/summary.md)
|
||||
|
||||
# 进阶篇
|
||||
|
||||
- [导读](chapter_preface_advanced/index.md)
|
||||
- [AI编译器和前端技术](chapter_frontend_and_ir/index.md)
|
||||
- [AI编译器设计原理](chapter_frontend_and_ir/ai_compiler_design_principle.md)
|
||||
- [AI编译器前端技术概述](chapter_frontend_and_ir/overview_of_frontend.md)
|
||||
- [中间表示](chapter_frontend_and_ir/intermediate_representation.md)
|
||||
- [自动微分](chapter_frontend_and_ir/ad.md)
|
||||
- [类型系统和静态分析](chapter_frontend_and_ir/type_system_and_static_analysis.md)
|
||||
- [常见前端编译优化方法](chapter_frontend_and_ir/common_frontend_optimization_pass.md)
|
||||
- [总结](chapter_frontend_and_ir/summary.md)
|
||||
- [编译器后端和运行时](chapter_backend_and_runtime/index.md)
|
||||
- [概述](chapter_backend_and_runtime/overview.md)
|
||||
- [计算图优化](chapter_backend_and_runtime/graph_optimizer.md)
|
||||
- [算子选择](chapter_backend_and_runtime/kernel_selecter.md)
|
||||
- [内存分配](chapter_backend_and_runtime/memory_allocator.md)
|
||||
- [计算调度与执行](chapter_backend_and_runtime/compute_schedule_and_execute.md)
|
||||
- [算子编译器](chapter_backend_and_runtime/op_compiler.md)
|
||||
- [总结](chapter_backend_and_runtime/summary.md)
|
||||
- [硬件加速器](chapter_accelerator/index.md)
|
||||
- [概述](chapter_accelerator/accelerator_introduction.md)
|
||||
- [加速器基本组成原理](chapter_accelerator/accelerator_architecture.md)
|
||||
- [加速器基本编程原理](chapter_accelerator/accelerator_programming.md)
|
||||
- [加速器实践](chapter_accelerator/accelerator_practise.md)
|
||||
- [总结](chapter_accelerator/summary.md)
|
||||
- [数据处理框架](chapter_data_processing/index.md)
|
||||
- [概述](chapter_data_processing/requirements.md)
|
||||
- [易用性设计](chapter_data_processing/program_model.md)
|
||||
- [高效性设计](chapter_data_processing/performance.md)
|
||||
- [保序性设计](chapter_data_processing/data_order.md)
|
||||
- [单机数据处理性能的扩展](chapter_data_processing/extension.md)
|
||||
- [总结](chapter_data_processing/summary.md)
|
||||
- [模型部署](chapter_model_deployment/index.md)
|
||||
- [概述](chapter_model_deployment/model_deployment_introduction.md)
|
||||
- [训练模型到推理模型的转换及优化](chapter_model_deployment/model_converter_and_optimizer.md)
|
||||
- [模型压缩](chapter_model_deployment/model_compression.md)
|
||||
- [模型推理](chapter_model_deployment/model_inference.md)
|
||||
- [模型的安全保护](chapter_model_deployment/model_security.md)
|
||||
- [总结](chapter_model_deployment/summary.md)
|
||||
- [分布式训练](chapter_distributed_training/index.md)
|
||||
- [系统概述](chapter_distributed_training/overview.md)
|
||||
- [实现方法](chapter_distributed_training/methods.md)
|
||||
- [机器学习集群架构](chapter_distributed_training/cluster.md)
|
||||
- [集合通信](chapter_distributed_training/collective.md)
|
||||
- [参数服务器](chapter_distributed_training/parameter_servers.md)
|
||||
- [总结](chapter_distributed_training/summary.md)
|
||||
|
||||
# 拓展篇
|
||||
|
||||
- [导读](chapter_preface_extension/index.md)
|
||||
- [深度学习推荐系统](chapter_recommender_system/index.md)
|
||||
- [系统基本组成](chapter_recommender_system/system_architecture.md)
|
||||
- [多阶段推荐系统](chapter_recommender_system/multi_stage_recommender_system.md)
|
||||
- [模型更新](chapter_recommender_system/model_update.md)
|
||||
- [案例分析:支持在线模型更新的大型推荐系统](chapter_recommender_system/case_study.md)
|
||||
- [小结](chapter_recommender_system/summary.md)
|
||||
- [联邦学习系统](chapter_federated_learning/index.md)
|
||||
- [概述](chapter_federated_learning/overview.md)
|
||||
- [横向联邦学习](chapter_federated_learning/horizontal_fl.md)
|
||||
- [纵向联邦学习](chapter_federated_learning/vertical_fl.md)
|
||||
- [隐私加密算法](chapter_federated_learning/privacy_encryption_algorithm.md)
|
||||
- [展望](chapter_federated_learning/outlook.md)
|
||||
- [小结](chapter_federated_learning/summary.md)
|
||||
- [强化学习系统](chapter_reinforcement_learning/index.md)
|
||||
- [强化学习介绍](chapter_reinforcement_learning/rl_introduction.md)
|
||||
- [单节点强化学习系统](chapter_reinforcement_learning/single_node_rl.md)
|
||||
- [多智能体强化学习](chapter_reinforcement_learning/marl.md)
|
||||
- [多智能体强化学习系统](chapter_reinforcement_learning/marl_sys.md)
|
||||
- [小结](chapter_reinforcement_learning/summary.md)
|
||||
- [可解释性AI系统](chapter_explainable_AI/index.md)
|
||||
- [背景](chapter_explainable_AI/explainable_ai.md)
|
||||
- [机器人系统](chapter_rl_sys/index.md)
|
||||
- [机器人系统概述](chapter_rl_sys/rl_sys_intro.md)
|
||||
- [通用机器人操作系统](chapter_rl_sys/ros.md)
|
||||
- [案例分析:使用机器人操作系统](chapter_rl_sys/ros_code_ex.md)
|
||||
- [总结](chapter_rl_sys/summary.md)
|
||||
|
||||
# 附录
|
||||
|
||||
- [机器学习介绍](appendix_machine_learning_introduction/index.md)
|
||||
- [神经网络](appendix_machine_learning_introduction/neural_network.md)
|
||||
- [梯度下降与反向传播](appendix_machine_learning_introduction/gradient_descent.md)
|
||||
- [经典机器学习方法](appendix_machine_learning_introduction/classic_machine_learning.md)
|
||||
@@ -20,15 +20,11 @@
|
||||
|
||||
上一节介绍了中间表示的基本概念,初步阐述了中间表示的重要作用和发展历程。接下来从组织结构的角度出发,介绍通用编译器的中间表示的类型以及各自特点 :cite:`2020MLIR`,如下表所示。中间表示组织结构的设计,对编译阶段的分析优化、代码生成等有着重要影响。编译器的设计需求不同,采用的中间表示组织结构也有所不同。
|
||||
|
||||
::: {#tab:ch04/ch04-categorize}
|
||||
组织结构 特点 举例
|
||||
-------------- ---------------------- ----------------------------------
|
||||
Linear IR 基于线性代码 堆栈机代码、三地址代码
|
||||
Graphical IR 基于图 抽象语法树、有向无环图、控制流图
|
||||
Hybrid IR 基于图与线性代码混合 LLVM IR
|
||||
|
||||
: 中间表示的分类
|
||||
:::
|
||||
| 组织结构 | 特点 | 举例 |
|
||||
|---|---|---|
|
||||
| Linear IR | 基于线性代码 | 堆栈机代码、三地址代码 |
|
||||
| Graphical IR | 基于图 | 抽象语法树、有向无环图、控制流图 |
|
||||
| Hybrid IR | 基于图与线性代码混合 | LLVM IR |
|
||||
|
||||
1\) 线性中间表示
|
||||
|
||||
|
||||
@@ -53,14 +53,10 @@ $$\pmb{Y_{bn}}=\pmb{A}*\pmb{X_{conv}}+\pmb{B}$$
|
||||
|
||||
在融合过程中,Convolution计算公式和Batchnorm计算公式中被认为是常量的符号在训练时均为参数,并不是常量。训练阶段如果进行该融合会导致模型参数的缺失。从该融合Pattern的结果来看,融合后网络中减少了一个Batchnorm算子,减少了一个Batchnorm算子的参数量,其实就是改变了深度神经网络的算法,会影响到网络的准确率,这是不可接受的。所以Convolution算子与Batchnorm算子的融合一般是在部署阶段特有的一种优化手段,其优化效果以MindSpore Lite为例,构造了包含一个Convolution和一个Batchnorm的sample网络,分别以样例网络和mobilenet-v2网络为例,在华为Mate30手机上,以两线程运行模型推理,取3000轮推理的平均时耗作为模型推理性能的指标,对比融合前后该指标的变化。从表 :numref:`ch08-tab-conv_bn_fusion`可以看到,对于sample网络和mobilenet-v2网络,融合后分别获得了8.5%和11.7%的推理性能提升,这个性能提升非常可观。并且这个性能提升没有带来任何的副作用,也没有对于硬件或算子库的提出额外要求。
|
||||
|
||||
::: {#tab:ch08-tab-conv_bn_fusion}
|
||||
网络 sample mobilenet-v2
|
||||
------- ------------- -----------------
|
||||
融合前 0.035 15.415
|
||||
融合后 0.031 13.606
|
||||
|
||||
: Convolution + Batchnorm融合前后推理性能(单位:ms)
|
||||
:::
|
||||
| 网络 | sample | mobilenet-v2 |
|
||||
|---|---|---|
|
||||
| 融合前 | 0.035 | 15.415 |
|
||||
| 融合后 | 0.031 | 13.606 |
|
||||
:label:`ch08-tab-conv_bn_fusion`
|
||||
|
||||
### 算子替换
|
||||
|
||||
@@ -13,7 +13,6 @@
|
||||
|
||||
rl_introduction
|
||||
single_node_rl
|
||||
distributed_node_rl
|
||||
marl
|
||||
marl_sys
|
||||
summary
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
/chivier-disk/hyq-home/Projects/openmlsys-zh/img
|
||||
@@ -8,15 +8,16 @@
|
||||
|
||||
```toc
|
||||
:maxdepth: 2
|
||||
:numbered:
|
||||
|
||||
chapter_preface/index
|
||||
[前言](chapter_preface/index)
|
||||
|
||||
# 基础篇
|
||||
chapter_introduction/index
|
||||
chapter_programming_interface/index
|
||||
chapter_computational_graph/index
|
||||
|
||||
chapter_preface_advanced/index
|
||||
|
||||
# 进阶篇
|
||||
[导读](chapter_preface_advanced/index)
|
||||
chapter_frontend_and_ir/index
|
||||
chapter_backend_and_runtime/index
|
||||
chapter_accelerator/index
|
||||
@@ -24,18 +25,14 @@ chapter_data_processing/index
|
||||
chapter_model_deployment/index
|
||||
chapter_distributed_training/index
|
||||
|
||||
chapter_preface_extension/index
|
||||
|
||||
# 拓展篇
|
||||
[导读](chapter_preface_extension/index)
|
||||
chapter_recommender_system/index
|
||||
chapter_federated_learning/index
|
||||
chapter_reinforcement_learning/index
|
||||
chapter_explainable_AI/index
|
||||
chapter_rl_sys/index
|
||||
|
||||
```
|
||||
|
||||
```toc
|
||||
:maxdepth: 1
|
||||
|
||||
appendix_machine_learning_introduction/index
|
||||
# 附录
|
||||
[机器学习介绍](appendix_machine_learning_introduction/index)
|
||||
```
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
1
zh_chapters/mlsys.bib
Symbolic link
1
zh_chapters/mlsys.bib
Symbolic link
@@ -0,0 +1 @@
|
||||
../mlsys.bib
|
||||
@@ -1 +0,0 @@
|
||||
/chivier-disk/hyq-home/Projects/openmlsys-zh/references
|
||||
@@ -1 +0,0 @@
|
||||
/chivier-disk/hyq-home/Projects/openmlsys-zh/static
|
||||
Reference in New Issue
Block a user