Files
bpf-developer-tutorial/14-tcpstates/index.html

432 lines
24 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE HTML>
<html lang="en" class="sidebar-visible no-js light">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>eBPF入门实践教程使用 libbpf-bootstrap 开发程序统计 TCP 连接延时 - bpf-developer-tutorial</title>
<!-- Custom HTML head -->
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff" />
<link rel="icon" href="../favicon.svg">
<link rel="shortcut icon" href="../favicon.png">
<link rel="stylesheet" href="../css/variables.css">
<link rel="stylesheet" href="../css/general.css">
<link rel="stylesheet" href="../css/chrome.css">
<link rel="stylesheet" href="../css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="../FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="../fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" href="../highlight.css">
<link rel="stylesheet" href="../tomorrow-night.css">
<link rel="stylesheet" href="../ayu-highlight.css">
<!-- Custom theme stylesheets -->
</head>
<body>
<div id="body-container">
<!-- Provide site root to javascript -->
<script>
var path_to_root = "../";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";
</script>
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script>
try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script>
var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('no-js')
html.classList.remove('light')
html.classList.add(theme);
html.classList.add('js');
</script>
<!-- Hide / unhide sidebar before it is displayed -->
<script>
var html = document.querySelector('html');
var sidebar = null;
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
}
html.classList.remove('sidebar-visible');
html.classList.add("sidebar-" + sidebar);
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded affix "><li class="part-title">eBPF 实践教程:基于 libbpf 和 CO-RE</li><li class="chapter-item expanded "><a href="../0-introduce/index.html"><strong aria-hidden="true">1.</strong> eBPF 入门开发实践教程一:介绍 eBPF 的基本概念、常见的开发工具</a></li><li class="chapter-item expanded "><a href="../1-helloworld/index.html"><strong aria-hidden="true">2.</strong> eBPF 入门开发实践教程二Hello World基本框架和开发流程</a></li><li class="chapter-item expanded "><a href="../2-kprobe-unlink/index.html"><strong aria-hidden="true">3.</strong> eBPF 入门开发实践教程二:在 eBPF 中使用 kprobe 监测捕获 unlink 系统调用</a></li><li class="chapter-item expanded "><a href="../3-fentry-unlink/index.html"><strong aria-hidden="true">4.</strong> eBPF 入门开发实践教程三:在 eBPF 中使用 fentry 监测捕获 unlink 系统调用</a></li><li class="chapter-item expanded "><a href="../4-opensnoop/index.html"><strong aria-hidden="true">5.</strong> eBPF 入门开发实践教程四:在 eBPF 中捕获进程打开文件的系统调用集合,使用全局变量过滤进程 pid</a></li><li class="chapter-item expanded "><a href="../5-uprobe-bashreadline/index.html"><strong aria-hidden="true">6.</strong> eBPF 入门开发实践教程五:在 eBPF 中使用 uprobe 捕获 bash 的 readline 函数调用</a></li><li class="chapter-item expanded "><a href="../6-sigsnoop/index.html"><strong aria-hidden="true">7.</strong> eBPF 入门开发实践教程六:捕获进程发送信号的系统调用集合,使用 hash map 保存状态</a></li><li class="chapter-item expanded "><a href="../7-execsnoop/index.html"><strong aria-hidden="true">8.</strong> eBPF 入门实践教程七:捕获进程执行/退出时间,通过 perf event array 向用户态打印输出</a></li><li class="chapter-item expanded "><a href="../8-exitsnoop/index.html"><strong aria-hidden="true">9.</strong> eBPF 入门开发实践教程八:在 eBPF 中使用 exitsnoop 监控进程退出事件,使用 ring buffer 向用户态打印输出</a></li><li class="chapter-item expanded "><a href="../9-runqlat/index.html"><strong aria-hidden="true">10.</strong> eBPF 入门开发实践教程九:一个 Linux 内核 BPF 程序,通过柱状图来总结调度程序运行队列延迟,显示任务等待运行在 CPU 上的时间长度</a></li><li class="chapter-item expanded "><a href="../10-hardirqs/index.html"><strong aria-hidden="true">11.</strong> eBPF 入门开发实践教程十:在 eBPF 中使用 hardirqs 或 softirqs 捕获中断事件</a></li><li class="chapter-item expanded "><a href="../11-bootstrap/index.html"><strong aria-hidden="true">12.</strong> eBPF 入门开发实践教程十一:在 eBPF 中使用 bootstrap 开发用户态程序并跟踪 exec() 和 exit() 系统调用</a></li><li class="chapter-item expanded "><a href="../13-tcpconnlat/index.html"><strong aria-hidden="true">13.</strong> eBPF入门实践教程使用 libbpf-bootstrap 开发程序统计 TCP 连接延时</a></li><li class="chapter-item expanded "><a href="../13-tcpconnlat/tcpconnlat.html"><strong aria-hidden="true">14.</strong> eBPF 入门实践教程:编写 eBPF 程序 tcpconnlat 测量 tcp 连接延时</a></li><li class="chapter-item expanded "><a href="../14-tcpstates/index.html" class="active"><strong aria-hidden="true">15.</strong> eBPF入门实践教程使用 libbpf-bootstrap 开发程序统计 TCP 连接延时</a></li><li class="chapter-item expanded "><a href="../15-tcprtt/index.html"><strong aria-hidden="true">16.</strong> eBPF 入门实践教程:编写 eBPF 程序 Tcprtt 测量 TCP 连接的往返时间</a></li><li class="chapter-item expanded "><a href="../16-memleak/index.html"><strong aria-hidden="true">17.</strong> eBPF 入门实践教程:编写 eBPF 程序 Memleak 监控内存泄漏</a></li><li class="chapter-item expanded "><a href="../17-biopattern/index.html"><strong aria-hidden="true">18.</strong> eBPF 入门实践教程:编写 eBPF 程序 Biopattern: 统计随机/顺序磁盘 I/O</a></li><li class="chapter-item expanded "><a href="../18-further-reading/index.html"><strong aria-hidden="true">19.</strong> 更多的参考资料</a></li><li class="chapter-item expanded "><a href="../19-lsm-connect/index.html"><strong aria-hidden="true">20.</strong> eBPF 入门实践教程:使用 LSM 进行安全检测防御</a></li><li class="chapter-item expanded "><a href="../20-tc/index.html"><strong aria-hidden="true">21.</strong> eBPF 入门实践教程:使用 eBPF 进行 tc 流量控制</a></li><li class="chapter-item expanded affix "><li class="part-title">bcc 开发者教程</li><li class="chapter-item expanded "><a href="../bcc-documents/kernel-versions.html"><strong aria-hidden="true">22.</strong> BPF Features by Linux Kernel Version</a></li><li class="chapter-item expanded "><a href="../bcc-documents/kernel_config.html"><strong aria-hidden="true">23.</strong> Kernel Configuration for BPF Features</a></li><li class="chapter-item expanded "><a href="../bcc-documents/reference_guide.html"><strong aria-hidden="true">24.</strong> bcc Reference Guide</a></li><li class="chapter-item expanded "><a href="../bcc-documents/special_filtering.html"><strong aria-hidden="true">25.</strong> Special Filtering</a></li><li class="chapter-item expanded "><a href="../bcc-documents/tutorial.html"><strong aria-hidden="true">26.</strong> bcc Tutorial</a></li><li class="chapter-item expanded "><a href="../bcc-documents/tutorial_bcc_python_developer.html"><strong aria-hidden="true">27.</strong> bcc Python Developer Tutorial</a></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle"></div>
</nav>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky bordered">
<div class="left-buttons">
<button id="sidebar-toggle" class="icon-button" type="button" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</button>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">bpf-developer-tutorial</h1>
<div class="right-buttons">
<a href="../print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="ebpf入门实践教程使用-libbpf-bootstrap-开发程序统计-tcp-连接延时"><a class="header" href="#ebpf入门实践教程使用-libbpf-bootstrap-开发程序统计-tcp-连接延时">eBPF入门实践教程使用 libbpf-bootstrap 开发程序统计 TCP 连接延时</a></h1>
<h2 id="内核态代码"><a class="header" href="#内核态代码">内核态代码</a></h2>
<pre><code class="language-c">// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
/* Copyright (c) 2021 Hengqi Chen */
#include &lt;vmlinux.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#include &lt;bpf/bpf_core_read.h&gt;
#include &quot;tcpstates.h&quot;
#define MAX_ENTRIES 10240
#define AF_INET 2
#define AF_INET6 10
const volatile bool filter_by_sport = false;
const volatile bool filter_by_dport = false;
const volatile short target_family = 0;
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_ENTRIES);
__type(key, __u16);
__type(value, __u16);
} sports SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_ENTRIES);
__type(key, __u16);
__type(value, __u16);
} dports SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_ENTRIES);
__type(key, struct sock *);
__type(value, __u64);
} timestamps SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
__uint(key_size, sizeof(__u32));
__uint(value_size, sizeof(__u32));
} events SEC(&quot;.maps&quot;);
SEC(&quot;tracepoint/sock/inet_sock_set_state&quot;)
int handle_set_state(struct trace_event_raw_inet_sock_set_state *ctx)
{
struct sock *sk = (struct sock *)ctx-&gt;skaddr;
__u16 family = ctx-&gt;family;
__u16 sport = ctx-&gt;sport;
__u16 dport = ctx-&gt;dport;
__u64 *tsp, delta_us, ts;
struct event event = {};
if (ctx-&gt;protocol != IPPROTO_TCP)
return 0;
if (target_family &amp;&amp; target_family != family)
return 0;
if (filter_by_sport &amp;&amp; !bpf_map_lookup_elem(&amp;sports, &amp;sport))
return 0;
if (filter_by_dport &amp;&amp; !bpf_map_lookup_elem(&amp;dports, &amp;dport))
return 0;
tsp = bpf_map_lookup_elem(&amp;timestamps, &amp;sk);
ts = bpf_ktime_get_ns();
if (!tsp)
delta_us = 0;
else
delta_us = (ts - *tsp) / 1000;
event.skaddr = (__u64)sk;
event.ts_us = ts / 1000;
event.delta_us = delta_us;
event.pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
event.oldstate = ctx-&gt;oldstate;
event.newstate = ctx-&gt;newstate;
event.family = family;
event.sport = sport;
event.dport = dport;
bpf_get_current_comm(&amp;event.task, sizeof(event.task));
if (family == AF_INET) {
bpf_probe_read_kernel(&amp;event.saddr, sizeof(event.saddr), &amp;sk-&gt;__sk_common.skc_rcv_saddr);
bpf_probe_read_kernel(&amp;event.daddr, sizeof(event.daddr), &amp;sk-&gt;__sk_common.skc_daddr);
} else { /* family == AF_INET6 */
bpf_probe_read_kernel(&amp;event.saddr, sizeof(event.saddr), &amp;sk-&gt;__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
bpf_probe_read_kernel(&amp;event.daddr, sizeof(event.daddr), &amp;sk-&gt;__sk_common.skc_v6_daddr.in6_u.u6_addr32);
}
bpf_perf_event_output(ctx, &amp;events, BPF_F_CURRENT_CPU, &amp;event, sizeof(event));
if (ctx-&gt;newstate == TCP_CLOSE)
bpf_map_delete_elem(&amp;timestamps, &amp;sk);
else
bpf_map_update_elem(&amp;timestamps, &amp;sk, &amp;ts, BPF_ANY);
return 0;
}
char LICENSE[] SEC(&quot;license&quot;) = &quot;Dual BSD/GPL&quot;;
</code></pre>
<p><code>tcpstates</code> 是一个追踪当前系统上的TCP套接字的TCP状态的程序主要通过跟踪内核跟踪点 <code>inet_sock_set_state</code> 来实现。统计数据通过 <code>perf_event</code>向用户态传输。</p>
<pre><code class="language-c">SEC(&quot;tracepoint/sock/inet_sock_set_state&quot;)
int handle_set_state(struct trace_event_raw_inet_sock_set_state *ctx)
</code></pre>
<p>在套接字改变状态处附加一个eBPF跟踪函数。</p>
<pre><code class="language-c"> if (ctx-&gt;protocol != IPPROTO_TCP)
return 0;
if (target_family &amp;&amp; target_family != family)
return 0;
if (filter_by_sport &amp;&amp; !bpf_map_lookup_elem(&amp;sports, &amp;sport))
return 0;
if (filter_by_dport &amp;&amp; !bpf_map_lookup_elem(&amp;dports, &amp;dport))
return 0;
</code></pre>
<p>跟踪函数被调用后,先判断当前改变状态的套接字是否满足我们需要的过滤条件,如果不满足则不进行记录。</p>
<pre><code class="language-c"> tsp = bpf_map_lookup_elem(&amp;timestamps, &amp;sk);
ts = bpf_ktime_get_ns();
if (!tsp)
delta_us = 0;
else
delta_us = (ts - *tsp) / 1000;
event.skaddr = (__u64)sk;
event.ts_us = ts / 1000;
event.delta_us = delta_us;
event.pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
event.oldstate = ctx-&gt;oldstate;
event.newstate = ctx-&gt;newstate;
event.family = family;
event.sport = sport;
event.dport = dport;
bpf_get_current_comm(&amp;event.task, sizeof(event.task));
if (family == AF_INET) {
bpf_probe_read_kernel(&amp;event.saddr, sizeof(event.saddr), &amp;sk-&gt;__sk_common.skc_rcv_saddr);
bpf_probe_read_kernel(&amp;event.daddr, sizeof(event.daddr), &amp;sk-&gt;__sk_common.skc_daddr);
} else { /* family == AF_INET6 */
bpf_probe_read_kernel(&amp;event.saddr, sizeof(event.saddr), &amp;sk-&gt;__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
bpf_probe_read_kernel(&amp;event.daddr, sizeof(event.daddr), &amp;sk-&gt;__sk_common.skc_v6_daddr.in6_u.u6_addr32);
}
</code></pre>
<p>使用状态改变相关填充event结构体。</p>
<ul>
<li>此处使用了<code>libbpf</code> 的 CO-RE 支持。</li>
</ul>
<pre><code class="language-c"> bpf_perf_event_output(ctx, &amp;events, BPF_F_CURRENT_CPU, &amp;event, sizeof(event));
</code></pre>
<p>将事件结构体发送至用户态程序。</p>
<pre><code class="language-c"> if (ctx-&gt;newstate == TCP_CLOSE)
bpf_map_delete_elem(&amp;timestamps, &amp;sk);
else
bpf_map_update_elem(&amp;timestamps, &amp;sk, &amp;ts, BPF_ANY);
</code></pre>
<p>根据这个TCP链接的新状态决定是更新下时间戳记录还是不再记录它的时间戳。</p>
<h2 id="用户态程序"><a class="header" href="#用户态程序">用户态程序</a></h2>
<pre><code class="language-c"> while (!exiting) {
err = perf_buffer__poll(pb, PERF_POLL_TIMEOUT_MS);
if (err &lt; 0 &amp;&amp; err != -EINTR) {
warn(&quot;error polling perf buffer: %s\n&quot;, strerror(-err));
goto cleanup;
}
/* reset err to return 0 if exiting */
err = 0;
}
</code></pre>
<p>不停轮询内核程序所发过来的 <code>perf event</code></p>
<pre><code class="language-c">static void handle_event(void* ctx, int cpu, void* data, __u32 data_sz) {
char ts[32], saddr[26], daddr[26];
struct event* e = data;
struct tm* tm;
int family;
time_t t;
if (emit_timestamp) {
time(&amp;t);
tm = localtime(&amp;t);
strftime(ts, sizeof(ts), &quot;%H:%M:%S&quot;, tm);
printf(&quot;%8s &quot;, ts);
}
inet_ntop(e-&gt;family, &amp;e-&gt;saddr, saddr, sizeof(saddr));
inet_ntop(e-&gt;family, &amp;e-&gt;daddr, daddr, sizeof(daddr));
if (wide_output) {
family = e-&gt;family == AF_INET ? 4 : 6;
printf(
&quot;%-16llx %-7d %-16s %-2d %-26s %-5d %-26s %-5d %-11s -&gt; %-11s &quot;
&quot;%.3f\n&quot;,
e-&gt;skaddr, e-&gt;pid, e-&gt;task, family, saddr, e-&gt;sport, daddr,
e-&gt;dport, tcp_states[e-&gt;oldstate], tcp_states[e-&gt;newstate],
(double)e-&gt;delta_us / 1000);
} else {
printf(
&quot;%-16llx %-7d %-10.10s %-15s %-5d %-15s %-5d %-11s -&gt; %-11s %.3f\n&quot;,
e-&gt;skaddr, e-&gt;pid, e-&gt;task, saddr, e-&gt;sport, daddr, e-&gt;dport,
tcp_states[e-&gt;oldstate], tcp_states[e-&gt;newstate],
(double)e-&gt;delta_us / 1000);
}
}
static void handle_lost_events(void* ctx, int cpu, __u64 lost_cnt) {
warn(&quot;lost %llu events on CPU #%d\n&quot;, lost_cnt, cpu);
}
</code></pre>
<p>收到事件后所调用对应的处理函数并进行输出打印。</p>
<h2 id="编译运行"><a class="header" href="#编译运行">编译运行</a></h2>
<pre><code class="language-console">$ make
...
BPF .output/tcpstates.bpf.o
GEN-SKEL .output/tcpstates.skel.h
CC .output/tcpstates.o
BINARY tcpstates
$ sudo ./tcpstates
SKADDR PID COMM LADDR LPORT RADDR RPORT OLDSTATE -&gt; NEWSTATE MS
ffff9bf61bb62bc0 164978 node 192.168.88.15 0 52.178.17.2 443 CLOSE -&gt; SYN_SENT 0.000
ffff9bf61bb62bc0 0 swapper/0 192.168.88.15 41596 52.178.17.2 443 SYN_SENT -&gt; ESTABLISHED 225.794
ffff9bf61bb62bc0 0 swapper/0 192.168.88.15 41596 52.178.17.2 443 ESTABLISHED -&gt; CLOSE_WAIT 901.454
ffff9bf61bb62bc0 164978 node 192.168.88.15 41596 52.178.17.2 443 CLOSE_WAIT -&gt; LAST_ACK 0.793
ffff9bf61bb62bc0 164978 node 192.168.88.15 41596 52.178.17.2 443 LAST_ACK -&gt; LAST_ACK 0.086
ffff9bf61bb62bc0 228759 kworker/u6 192.168.88.15 41596 52.178.17.2 443 LAST_ACK -&gt; CLOSE 0.193
ffff9bf6d8ee88c0 229832 redis-serv 0.0.0.0 6379 0.0.0.0 0 CLOSE -&gt; LISTEN 0.000
ffff9bf6d8ee88c0 229832 redis-serv 0.0.0.0 6379 0.0.0.0 0 LISTEN -&gt; CLOSE 1.763
ffff9bf7109d6900 88750 node 127.0.0.1 39755 127.0.0.1 50966 ESTABLISHED -&gt; FIN_WAIT1 0.000
</code></pre>
<h2 id="总结"><a class="header" href="#总结">总结</a></h2>
<p>这里的代码修改自 <a href="https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcpstates.bpf.c">https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcpstates.bpf.c</a></p>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<!-- Mobile navigation buttons -->
<a rel="prev" href="../13-tcpconnlat/tcpconnlat.html" class="mobile-nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next" href="../15-tcprtt/index.html" class="mobile-nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
<a rel="prev" href="../13-tcpconnlat/tcpconnlat.html" class="nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next" href="../15-tcprtt/index.html" class="nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
</nav>
</div>
<script>
window.playground_copyable = true;
</script>
<script src="../elasticlunr.min.js"></script>
<script src="../mark.min.js"></script>
<script src="../searcher.js"></script>
<script src="../clipboard.min.js"></script>
<script src="../highlight.js"></script>
<script src="../book.js"></script>
<!-- Custom JS scripts -->
</div>
</body>
</html>