Files
bpf-developer-tutorial/37-uprobe-rust/index.html

334 lines
26 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE HTML>
<html lang="en" class="light" dir="ltr">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>使用 uprobe 追踪 Rust 应用程序 - bpf-developer-tutorial</title>
<!-- Custom HTML head -->
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff">
<link rel="icon" href="../favicon.svg">
<link rel="shortcut icon" href="../favicon.png">
<link rel="stylesheet" href="../css/variables.css">
<link rel="stylesheet" href="../css/general.css">
<link rel="stylesheet" href="../css/chrome.css">
<link rel="stylesheet" href="../css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="../FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="../fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" href="../highlight.css">
<link rel="stylesheet" href="../tomorrow-night.css">
<link rel="stylesheet" href="../ayu-highlight.css">
<!-- Custom theme stylesheets -->
</head>
<body class="sidebar-visible no-js">
<div id="body-container">
<!-- Provide site root to javascript -->
<script>
var path_to_root = "../";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";
</script>
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script>
try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script>
var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('light')
html.classList.add(theme);
var body = document.querySelector('body');
body.classList.remove('no-js')
body.classList.add('js');
</script>
<input type="checkbox" id="sidebar-toggle-anchor" class="hidden">
<!-- Hide / unhide sidebar before it is displayed -->
<script>
var body = document.querySelector('body');
var sidebar = null;
var sidebar_toggle = document.getElementById("sidebar-toggle-anchor");
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
}
sidebar_toggle.checked = sidebar === 'visible';
body.classList.remove('sidebar-visible');
body.classList.add("sidebar-" + sidebar);
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded affix "><a href="../https://github.com/eunomia-bpf/bpf-developer-tutorial.html">https://github.com/eunomia-bpf/bpf-developer-tutorial</a></li><li class="chapter-item expanded affix "><li class="part-title">入门文档</li><li class="chapter-item expanded "><a href="../0-introduce/index.html"><strong aria-hidden="true">1.</strong> lesson 0-introduce</a></li><li class="chapter-item expanded "><a href="../1-helloworld/index.html"><strong aria-hidden="true">2.</strong> lesson 1-helloworld</a></li><li class="chapter-item expanded "><a href="../2-kprobe-unlink/index.html"><strong aria-hidden="true">3.</strong> lesson 2-kprobe-unlink</a></li><li class="chapter-item expanded "><a href="../3-fentry-unlink/index.html"><strong aria-hidden="true">4.</strong> lesson 3-fentry-unlink</a></li><li class="chapter-item expanded "><a href="../4-opensnoop/index.html"><strong aria-hidden="true">5.</strong> lesson 4-opensnoop</a></li><li class="chapter-item expanded "><a href="../5-uprobe-bashreadline/index.html"><strong aria-hidden="true">6.</strong> lesson 5-uprobe-bashreadline</a></li><li class="chapter-item expanded "><a href="../6-sigsnoop/index.html"><strong aria-hidden="true">7.</strong> lesson 6-sigsnoop</a></li><li class="chapter-item expanded "><a href="../7-execsnoop/index.html"><strong aria-hidden="true">8.</strong> lesson 7-execsnoop</a></li><li class="chapter-item expanded "><a href="../8-exitsnoop/index.html"><strong aria-hidden="true">9.</strong> lesson 8-execsnoop</a></li><li class="chapter-item expanded "><a href="../9-runqlat/index.html"><strong aria-hidden="true">10.</strong> lesson 9-runqlat</a></li><li class="chapter-item expanded "><a href="../10-hardirqs/index.html"><strong aria-hidden="true">11.</strong> lesson 10-hardirqs</a></li><li class="chapter-item expanded affix "><li class="part-title">进阶文档和示例</li><li class="chapter-item expanded "><a href="../11-bootstrap/index.html"><strong aria-hidden="true">12.</strong> lesson 11-bootstrap</a></li><li class="chapter-item expanded "><a href="../12-profile/index.html"><strong aria-hidden="true">13.</strong> lesson 12-profile</a></li><li class="chapter-item expanded "><a href="../13-tcpconnlat/index.html"><strong aria-hidden="true">14.</strong> lesson 13-tcpconnlat</a></li><li class="chapter-item expanded "><a href="../14-tcpstates/index.html"><strong aria-hidden="true">15.</strong> lesson 14-tcpstates</a></li><li class="chapter-item expanded "><a href="../15-javagc/index.html"><strong aria-hidden="true">16.</strong> lesson 15-javagc</a></li><li class="chapter-item expanded "><a href="../16-memleak/index.html"><strong aria-hidden="true">17.</strong> lesson 16-memleak</a></li><li class="chapter-item expanded "><a href="../17-biopattern/index.html"><strong aria-hidden="true">18.</strong> lesson 17-biopattern</a></li><li class="chapter-item expanded "><a href="../18-further-reading/index.html"><strong aria-hidden="true">19.</strong> lesson 18-further-reading</a></li><li class="chapter-item expanded "><a href="../19-lsm-connect/index.html"><strong aria-hidden="true">20.</strong> lesson 19-lsm-connect</a></li><li class="chapter-item expanded "><a href="../20-tc/index.html"><strong aria-hidden="true">21.</strong> lesson 20-tc</a></li><li class="chapter-item expanded "><a href="../21-xdp/index.html"><strong aria-hidden="true">22.</strong> lesson 21-xdp</a></li><li class="chapter-item expanded affix "><li class="part-title">高级主题</li><li class="chapter-item expanded "><a href="../22-android/index.html"><strong aria-hidden="true">23.</strong> 在 Android 上使用 eBPF 程序</a></li><li class="chapter-item expanded "><a href="../30-sslsniff/index.html"><strong aria-hidden="true">24.</strong> 使用 uprobe 捕获多种库的 SSL/TLS 明文数据</a></li><li class="chapter-item expanded "><a href="../23-http/index.html"><strong aria-hidden="true">25.</strong> 使用 eBPF socket filter 或 syscall trace 追踪 HTTP 请求和其他七层协议</a></li><li class="chapter-item expanded "><a href="../29-sockops/index.html"><strong aria-hidden="true">26.</strong> 使用 sockops 加速网络请求转发</a></li><li class="chapter-item expanded "><a href="../24-hide/index.html"><strong aria-hidden="true">27.</strong> 使用 eBPF 隐藏进程或文件信息</a></li><li class="chapter-item expanded "><a href="../25-signal/index.html"><strong aria-hidden="true">28.</strong> 使用 bpf_send_signal 发送信号终止进程</a></li><li class="chapter-item expanded "><a href="../26-sudo/index.html"><strong aria-hidden="true">29.</strong> 使用 eBPF 添加 sudo 用户</a></li><li class="chapter-item expanded "><a href="../27-replace/index.html"><strong aria-hidden="true">30.</strong> 使用 eBPF 替换任意程序读取或写入的文本</a></li><li class="chapter-item expanded "><a href="../28-detach/index.html"><strong aria-hidden="true">31.</strong> BPF 的生命周期:使用 Detached 模式在用户态应用退出后持续运行 eBPF 程序</a></li><li class="chapter-item expanded "><a href="../18-further-reading/ebpf-security.zh.html"><strong aria-hidden="true">32.</strong> eBPF 运行时的安全性与面临的挑战</a></li><li class="chapter-item expanded "><a href="../34-syscall/index.html"><strong aria-hidden="true">33.</strong> 使用 eBPF 修改系统调用参数</a></li><li class="chapter-item expanded "><a href="../35-user-ringbuf/index.html"><strong aria-hidden="true">34.</strong> eBPF开发实践使用 user ring buffer 向内核异步发送信息</a></li><li class="chapter-item expanded "><a href="../36-userspace-ebpf/index.html"><strong aria-hidden="true">35.</strong> 用户空间 eBPF 运行时:深度解析与应用实践</a></li><li class="chapter-item expanded "><a href="../37-uprobe-rust/index.html" class="active"><strong aria-hidden="true">36.</strong> 使用 uprobe 追踪 Rust 应用程序</a></li><li class="chapter-item expanded "><a href="../38-btf-uprobe/index.html"><strong aria-hidden="true">37.</strong> 借助 eBPF 和 BTF让用户态也能一次编译、到处运行</a></li><li class="chapter-item expanded affix "><li class="part-title">bcc 和 bpftrace 教程与文档</li><li class="chapter-item expanded "><a href="../bcc-documents/kernel-versions.html"><strong aria-hidden="true">38.</strong> BPF Features by Linux Kernel Version</a></li><li class="chapter-item expanded "><a href="../bcc-documents/kernel_config.html"><strong aria-hidden="true">39.</strong> Kernel Configuration for BPF Features</a></li><li class="chapter-item expanded "><a href="../bcc-documents/reference_guide.html"><strong aria-hidden="true">40.</strong> bcc Reference Guide</a></li><li class="chapter-item expanded "><a href="../bcc-documents/special_filtering.html"><strong aria-hidden="true">41.</strong> Special Filtering</a></li><li class="chapter-item expanded "><a href="../bcc-documents/tutorial.html"><strong aria-hidden="true">42.</strong> bcc Tutorial</a></li><li class="chapter-item expanded "><a href="../bcc-documents/tutorial_bcc_python_developer.html"><strong aria-hidden="true">43.</strong> bcc Python Developer Tutorial</a></li><li class="chapter-item expanded "><a href="../bpftrace-tutorial/index.html"><strong aria-hidden="true">44.</strong> bpftrace Tutorial</a></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
</div>
</nav>
<!-- Track and set sidebar scroll position -->
<script>
var sidebarScrollbox = document.querySelector('#sidebar .sidebar-scrollbox');
sidebarScrollbox.addEventListener('click', function(e) {
if (e.target.tagName === 'A') {
sessionStorage.setItem('sidebar-scroll', sidebarScrollbox.scrollTop);
}
}, { passive: true });
var sidebarScrollTop = sessionStorage.getItem('sidebar-scroll');
sessionStorage.removeItem('sidebar-scroll');
if (sidebarScrollTop) {
// preserve sidebar scroll position when navigating via links within sidebar
sidebarScrollbox.scrollTop = sidebarScrollTop;
} else {
// scroll sidebar to current active section when navigating via "next/previous chapter" buttons
var activeSection = document.querySelector('#sidebar .active');
if (activeSection) {
activeSection.scrollIntoView({ block: 'center' });
}
}
</script>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky">
<div class="left-buttons">
<label id="sidebar-toggle" class="icon-button" for="sidebar-toggle-anchor" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</label>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">bpf-developer-tutorial</h1>
<div class="right-buttons">
<a href="../print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="ebpf-实践使用-uprobe-追踪用户态-rust-应用"><a class="header" href="#ebpf-实践使用-uprobe-追踪用户态-rust-应用">eBPF 实践:使用 Uprobe 追踪用户态 Rust 应用</a></h1>
<p>eBPF即扩展的Berkeley包过滤器Extended Berkeley Packet Filter是Linux内核中的一种革命性技术它允许开发者在内核态中运行自定义的“微程序”从而在不修改内核代码的情况下改变系统行为或收集系统细粒度的性能数据。</p>
<p>本文讨论如何使用 Uprobe 和 eBPF 追踪用户态 Rust 应用,包括如何获取符号名称并 attach、获取函数参数、获取返回值等。本文是 eBPF 开发者教程的一部分,更详细的内容可以在这里找到:<a href="https://eunomia.dev/tutorials/">https://eunomia.dev/tutorials/</a> 源代码在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">GitHub 仓库</a> 中开源。</p>
<h2 id="uprobe"><a class="header" href="#uprobe">Uprobe</a></h2>
<p>Uprobe是一种用户空间探针uprobe探针允许在用户空间程序中动态插桩插桩位置包括函数入口、特定偏移处以及函数返回处。当我们定义uprobe时内核会在附加的指令上创建快速断点指令x86机器上为int3指令当程序执行到该指令时内核将触发事件程序陷入到内核态并以回调函数的方式调用探针函数执行完探针函数再返回到用户态继续执行后序的指令。</p>
<p>uprobe 适用于在用户态去解析一些内核态探针无法解析的流量,例如 http2 流量https 流量,同时也可以分析程序运行时、业务逻辑等。关于 Uprobe 的更多信息,可以参考:</p>
<ul>
<li><a href="../30-sslsniff/README.html">eBPF 实践教程:使用 uprobe 捕获多种库的 SSL/TLS 明文数据</a></li>
<li><a href="../31-goroutine/README.html">eBPF 实践教程:使用 uprobe 捕获 Golang 的协程切换</a></li>
<li><a href="../32-http2/README.html">eBPF 实践教程:使用 uprobe 捕获用户态 http2 流量</a></li>
</ul>
<p>Uprobe 在内核态 eBPF 运行时,也可能产生比较大的性能开销,这时候也可以考虑使用用户态 eBPF 运行时,例如 <a href="https://github.com/eunomia-bpf/bpftime">bpftime</a>。bpftime 是一个基于 LLVM JIT/AOT 的用户态 eBPF 运行时,它可以在用户态运行 eBPF Uprobe 程序,和内核态的 eBPF 兼容由于避免了内核态和用户态之间的上下文切换bpftime 的 Uprobe 开销比内核少约 10 倍,并且也更容易扩展。</p>
<h2 id="rust"><a class="header" href="#rust">Rust</a></h2>
<p>Rust 是一种开源的系统编程语言注重安全、速度和并行性。它于2010年由Graydon Hoare在Mozilla研究中心开发并于2015年发布了第一个稳定版本。Rust 语言的设计哲学旨在提供C++的性能优势同时大幅减少内存安全漏洞。Rust在系统编程领域逐渐受到欢迎特别是在需要高性能、安全性和可靠性的应用场景例如操作系统、文件系统、游戏引擎、网络服务等领域。许多大型技术公司包括Mozilla、Google、Microsoft和Amazon等都在使用或支持Rust语言。</p>
<p>可以参考 <a href="https://www.rust-lang.org/">Rust 官方网站</a> 了解更多 Rust 语言的信息,并安装 Rust 的工具链。</p>
<h2 id="最简单的例子symbol-name-mangling"><a class="header" href="#最简单的例子symbol-name-mangling">最简单的例子Symbol name mangling</a></h2>
<p>我们先来看一个简单的例子,使用 Uprobe 追踪 Rust 程序的 <code>main</code> 函数,代码如下:</p>
<pre><pre class="playground"><code class="language-rust">pub fn hello() -&gt; i32 {
println!("Hello, world!");
0
}
fn main() {
hello();
}</code></pre></pre>
<p>构建和尝试获取符号:</p>
<pre><code class="language-console">$ cd helloworld
$ cargo build
$ nm helloworld/target/release/helloworld | grep hello
0000000000008940 t _ZN10helloworld4main17h2dce92cb81426b91E
</code></pre>
<p>我们会发现,对应的符号被转换为了 <code>_ZN10helloworld4main17h2dce92cb81426b91E</code>,这是因为 rustc 使用 <a href="https://en.wikipedia.org/wiki/Name_mangling">Symbol name mangling</a> 来为代码生成过程中使用的符号编码一个唯一的名称。编码后的名称会被链接器用于将名称与所指向的内容关联起来。可以使用 -C symbol-mangling-version 选项来控制符号名称的处理方法。</p>
<p>我们可以使用 <a href="https://crates.io/crates/rustfilt"><code>rustfilt</code></a> 工具来解析和获取对应的符号:</p>
<pre><code class="language-console">$ cargo install rustfilt
$ nm helloworld/target/release/helloworld &gt; name.txt
$ rustfilt _ZN10helloworld4main17h2dce92cb81426b91E
helloworld::main
$ rustfilt -i name.txt | grep hello
0000000000008b60 t helloworld::main
</code></pre>
<p>接下来我们可以尝试使用 bpftrace 跟踪对应的函数:</p>
<pre><code class="language-console">$ sudo bpftrace -e 'uprobe:helloworld/target/release/helloworld:_ZN10helloworld4main17h2dce92cb81426b91E { printf("Function hello-world called\n"); }'
Attaching 1 probe...
Function hello-world called
</code></pre>
<h2 id="一个奇怪的现象多次调用获取参数"><a class="header" href="#一个奇怪的现象多次调用获取参数">一个奇怪的现象:多次调用、获取参数</a></h2>
<p>对于一个更复杂的例子,包含多次调用和获取参数:</p>
<pre><pre class="playground"><code class="language-rust">use std::env;
pub fn hello(i: i32, len: usize) -&gt; i32 {
println!("Hello, world! {} in {}", i, len);
i + len as i32
}
fn main() {
let args: Vec&lt;String&gt; = env::args().collect();
// Skip the first argument, which is the path to the binary, and iterate over the rest
for arg in args.iter().skip(1) {
match arg.parse::&lt;i32&gt;() {
Ok(i) =&gt; {
let ret = hello(i, args.len());
println!("return value: {}", ret);
}
Err(_) =&gt; {
eprintln!("Error: Argument '{}' is not a valid integer", arg);
}
}
}
}</code></pre></pre>
<p>我们再次进行类似的操作,会发现一个奇怪的现象:</p>
<pre><code class="language-console">$ sudo bpftrace -e 'uprobe:args/target/release/helloworld:_ZN10helloworld4main17h2dce92cb81426b91E { printf("Function hello-world called\n"); }'
Attaching 1 probe...
Function hello-world called
</code></pre>
<p>这时候我们希望 hello 函数运行多次,但 bpftrace 中只输出了一次调用:</p>
<pre><code class="language-console">$ args/target/release/helloworld 1 2 3 4
Hello, world! 1 in 5
return value: 6
Hello, world! 2 in 5
return value: 7
Hello, world! 3 in 5
return value: 8
Hello, world! 4 in 5
return value: 9
</code></pre>
<p>而且看起来 bpftrace 并不能正确获取参数:</p>
<pre><code class="language-console">$ sudo bpftrace -e 'uprobe:args/target/release/helloworld:_ZN10helloworld4main17h2dce92cb81426b91E { printf("Function hello-world called %d\n"
, arg0); }'
Attaching 1 probe...
Function hello-world called 63642464
</code></pre>
<p>Uretprobe 捕捉到了第一次调用的返回值:</p>
<pre><code class="language-console">$ sudo bpftrace -e 'uretprobe:args/tar
get/release/helloworld:_ZN10helloworld4main17h2dce92
cb81426b91E { printf("Function hello-world called %d
\n", retval); }'
Attaching 1 probe...
Function hello-world called 6
</code></pre>
<p>这可能是由于 Rust 没有稳定的 ABI。 Rust正如它迄今为止所存在的那样保留了以任何它想要的方式对这些结构成员进行排序的权利。 因此,被调用者的编译版本可能会完全按照上面的方式对成员进行排序,而调用库的编程的编译版本可能会认为它实际上是这样布局的:</p>
<p>TODO: 进一步分析(未完待续)</p>
<h2 id="参考资料"><a class="header" href="#参考资料">参考资料</a></h2>
<ul>
<li><a href="https://doc.rust-lang.org/rustc/symbol-mangling/index.html">https://doc.rust-lang.org/rustc/symbol-mangling/index.html</a></li>
</ul>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<!-- Mobile navigation buttons -->
<a rel="prev" href="../36-userspace-ebpf/index.html" class="mobile-nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="../38-btf-uprobe/index.html" class="mobile-nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
<a rel="prev" href="../36-userspace-ebpf/index.html" class="nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="../38-btf-uprobe/index.html" class="nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
</nav>
</div>
<script>
window.playground_copyable = true;
</script>
<script src="../elasticlunr.min.js"></script>
<script src="../mark.min.js"></script>
<script src="../searcher.js"></script>
<script src="../clipboard.min.js"></script>
<script src="../highlight.js"></script>
<script src="../book.js"></script>
<!-- Custom JS scripts -->
</div>
</body>
</html>