From 5514e9f8ffe7f193acffeb28e036934e078bb331 Mon Sep 17 00:00:00 2001 From: LinuxDev9002 Date: Sat, 18 Jul 2026 23:09:38 -0700 Subject: [PATCH] Teach the egress pacer component by component --- src/53-egress-pacer/README.md | 120 +++++++++++++++---------------- src/53-egress-pacer/README.zh.md | 120 +++++++++++++++---------------- 2 files changed, 118 insertions(+), 122 deletions(-) diff --git a/src/53-egress-pacer/README.md b/src/53-egress-pacer/README.md index b9ddb65..baf8762 100644 --- a/src/53-egress-pacer/README.md +++ b/src/53-egress-pacer/README.md @@ -1,25 +1,24 @@ # eBPF Tutorial by Example: Build an Egress Pacer with a BPF Qdisc -Most traffic-control BPF programs inspect packets and return a verdict. The kernel's queueing discipline still decides where those packets wait and when they leave. Linux 6.16 added another option. A BPF `struct_ops` program can implement the qdisc itself. +Have you ever wanted to slow down traffic from one test interface and see exactly when each packet becomes ready to leave? Most TC BPF examples cannot show that part. They inspect an skb and return an action, but a separate queueing discipline still owns the packet queue and the send time. -This tutorial uses that interface to build `egress_pacer`, a deliberately small FIFO pacer for a disposable veth, TAP, or IFB interface. Give it a rate and a queue limit, and it holds each skb until the packet's departure time arrives. When the run ends, it reports how many packets entered the queue, left it, or were dropped. +Linux 6.16 lets BPF go one step further. A `struct_ops` program can implement the qdisc itself. In this tutorial, we will build `egress_pacer`, a small FIFO pacer for a disposable veth, TAP, or IFB interface. You give it a rate and a queue limit. It holds each skb until its departure time arrives, then reports how many packets were queued, sent, or dropped. -The example is a lab tool, not a general-purpose scheduler for a production NIC. Its value is that every important qdisc responsibility stays visible: skb ownership, queue accounting, pacing time, watchdog wakeups, and reset-time cleanup. The complete project lives in the [`53-egress-pacer` source directory](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer). +We keep the design intentionally small. There are no classes, fairness rules, ECN, or burst controls. That makes it easier to follow the parts a qdisc must get right: skb ownership, queue accounting, pacing time, watchdog wakeups, and cleanup. Treat it as a lab tool, not as a production scheduler for a physical NIC. -## From classifier to queue owner +> The complete source code: -An ordinary TC classifier runs while another qdisc owns the queue. [Lesson 20](../20-tc/README.md) follows that familiar model. `egress_pacer` goes one layer deeper. Its `struct_ops` object implements `struct Qdisc_ops`, so the BPF program becomes the root qdisc and takes responsibility for every skb it accepts. +## What Changes When BPF Becomes the Qdisc? -Follow one packet. User space first loads the object with a rate and packet limit, registers `bpf_pacer`, and creates it at `TC_H_ROOT`. The qdisc's `enqueue` callback moves the skb into a BPF-owned node and assigns a departure time. Later, `dequeue` either returns the packet or schedules the qdisc watchdog for the moment it becomes eligible. Removing the qdisc invokes `reset`, which releases anything still waiting. +In an ordinary TC example, the BPF program sees a packet while another qdisc owns the queue. [Lesson 20](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/20-tc) uses that familiar model. `egress_pacer` goes one layer deeper. Its `struct_ops` object implements `struct Qdisc_ops`, so the BPF program becomes the root qdisc and takes responsibility for every skb it accepts. -That single trip through the queue is the thread for the rest of the tutorial. +Follow one packet through the example. User space loads the BPF object with a rate and queue limit, registers `bpf_pacer`, and attaches it at `TC_H_ROOT`. The `enqueue` callback moves the skb into a BPF-owned node and assigns a departure time. The `dequeue` callback either returns the packet or asks the qdisc watchdog to wake up when the packet is ready. When user space removes the qdisc, `reset` releases anything still waiting. -## Complete source code +## Building the Pacer -The shared header defines the statistics contract used by BPF and user space. +We will build the pacer one component at a time. The first component is the small statistics contract shared by BPF and user space. -
-egress_pacer.h: shared statistics +### Sharing Statistics with User Space ```c @@ -40,13 +39,14 @@ struct pacer_stats { ``` -
+The BPF program updates these six counters while packets move through the queue. `enqueued` and `dequeued` cover the normal path. `policy_dropped` counts packets rejected because the queue is full or a node cannot be allocated. `cleanup_dropped` counts packets still waiting when the qdisc is reset. The last two fields record the number of bytes sent and the largest queue depth seen during the run. + +The loader reads the same structure from `skel->bss->stats` after it removes the qdisc. Keeping this definition in one header prevents the BPF program and the loader from disagreeing about the layout. + +### Adding the Missing BPF Declarations The repository's generated headers predate a few BPF qdisc definitions. This local compatibility header supplies those declarations without changing the vendored global headers. -
-bpf_experimental.h: local compatibility declarations - ```c /* SPDX-License-Identifier: GPL-2.0 */ @@ -84,12 +84,13 @@ bpf_list_pop_front(struct bpf_list_head *head) __ksym; ``` -
+This file does not implement pacing. It exposes the graph-object kfuncs used by the real qdisc: `bpf_obj_new` and `bpf_obj_drop` manage packet nodes, while `bpf_list_push_back` and `bpf_list_pop_front` manage the FIFO. The `__contains` BTF tag tells the verifier which node type belongs to the list head. -The kernel-mode program implements the qdisc operations, packet queue, pacing clock, and counters. +Keeping these declarations local matters because they are compatibility glue for this example, not a replacement for the repository's generated libbpf headers. Once those headers provide the same definitions, this file can disappear without changing the pacing algorithm. -
-egress_pacer.bpf.c: BPF qdisc implementation +### Implementing the Qdisc Callbacks + +Now we can look at the kernel-mode program. It implements the qdisc operations, packet queue, pacing clock, and counters. ```c @@ -293,13 +294,47 @@ struct Qdisc_ops pacer = { ``` -
+#### Taking Ownership of an skb + +The `init` callback first copies `queue_limit` into `sch->limit`. From then on, `egress_pacer_enqueue` must either take ownership of each skb or reject it. It reads `qdisc_skb_cb(skb)->pkt_len`, checks the packet limit, and allocates a `packet_node` with `bpf_obj_new`. A full queue and an allocation failure use the same drop path, so both increase `policy_dropped`. + +For an accepted packet, `bpf_kptr_xchg` moves the skb into the node's `__kptr` field. The node now owns the packet and stores its qdisc length and `eligible_ns` timestamp. While holding `queue_lock`, the program adds the node to the FIFO and updates qlen, backlog, `enqueued`, and `max_qlen` together. + +The departure time comes from the serialization time of this packet at the requested rate: + +```text +interval_ns = packet_len * 8 * 1,000,000 / rate_kbps +eligible_ns = max(next_departure_ns, now) +next_departure_ns = eligible_ns + interval_ns +``` + +If the queue was idle, `next_departure_ns` is already in the past, so the first packet can leave immediately. Every later packet advances the clock by its own length. This creates one aggregate FIFO schedule. It is not a token bucket and does not have a separate burst budget. + +#### Waiting Without Busy Looping + +`egress_pacer_dequeue` pops the first node and compares its `eligible_ns` with `bpf_ktime_get_ns()`. If the time has arrived, `bpf_kptr_xchg` moves the skb out of the node. The callback reduces qlen and backlog, frees the empty node, updates the qdisc byte statistics, and increases `dequeued` and `bytes_dequeued`. + +If the packet is early, the callback puts the node back at the front and calls `bpf_qdisc_watchdog_schedule(sch, expire, 0)`. Returning `NULL` tells the qdisc core that nothing is ready. The core wakes the qdisc at `eligible_ns`, so the BPF program never spins while it waits. + +#### Releasing Packets During Reset + +Removal completes the ownership story. `egress_pacer_reset` uses `bpf_for` to drain the current qlen. It releases every retained skb with `bpf_kfree_skb` and drops each node object. These packets increase `cleanup_dropped`, while admission failures during normal enqueue increase `policy_dropped`. Reset also clears the pacing clock, qlen, and backlog. + +Together, the BPF callbacks update the six counters we saw in the shared header: + +| Field | Meaning | +| --- | --- | +| `enqueued` | Packets successfully added to the FIFO | +| `dequeued` | Packets returned for transmission | +| `policy_dropped` | Packets rejected at the queue limit or after node allocation failed | +| `cleanup_dropped` | Packets drained by qdisc reset | +| `bytes_dequeued` | Total qdisc packet bytes returned by dequeue | +| `max_qlen` | Peak packet occupancy of the FIFO | + +### Loading and Removing the Qdisc The user-space loader validates the command line, registers the BPF qdisc, installs it on one interface, waits for the requested duration, and removes it. -
-egress_pacer.c: loader and qdisc lifecycle - ```c // SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) @@ -601,44 +636,7 @@ cleanup: ``` -
- -## Owning the skb and assigning a departure time - -The qdisc's `init` callback copies `queue_limit` into `sch->limit`. From that point on, `egress_pacer_enqueue` must either take ownership of each skb or reject it. The callback reads `qdisc_skb_cb(skb)->pkt_len`, checks the packet limit, and allocates a `packet_node` with `bpf_obj_new`. A full queue and an allocation failure share the same drop path, so both increase `policy_dropped`. - -For an accepted packet, `bpf_kptr_xchg` moves the skb into the node's `__kptr` field. The node then carries the packet, its qdisc length, and an `eligible_ns` timestamp. The spin lock protects the list, the pacing clock, qlen, and backlog while the node enters the FIFO. Successful insertion increases `enqueued`, and `max_qlen` remembers the largest queue seen during the run. - -The timestamp comes from the packet's serialization time at the requested rate: - -```text -interval_ns = packet_len * 8 * 1,000,000 / rate_kbps -eligible_ns = max(next_departure_ns, now) -next_departure_ns = eligible_ns + interval_ns -``` - -After an idle period, `next_departure_ns` is already in the past, so the first packet may leave immediately. Each later packet advances the clock by its own length. This produces one aggregate FIFO schedule. It is not a token bucket and has no separate burst budget. - -## Letting the watchdog wait - -`egress_pacer_dequeue` pops the first node and compares `eligible_ns` with `bpf_ktime_get_ns()`. If the time has arrived, another `bpf_kptr_xchg` moves the skb out of the node. The callback reduces qlen and backlog, frees the empty node, updates the qdisc byte statistics, and increases `dequeued` and `bytes_dequeued`. - -If the packet is early, the callback puts the node back at the front and calls `bpf_qdisc_watchdog_schedule(sch, expire, 0)`. Returning `NULL` tells the qdisc core that nothing is ready. The core wakes the qdisc at `eligible_ns`, so the BPF program never spins while waiting. - -Removal completes the ownership story. `egress_pacer_reset` uses `bpf_for` to drain the current qlen, releases every retained skb with `bpf_kfree_skb`, and drops each node object. Those packets increase `cleanup_dropped`, while admission failures during normal enqueue increase `policy_dropped`. Reset also clears the pacing clock, qlen, and backlog. - -The final BSS `pacer_stats` counters therefore answer six separate questions: - -| Field | Meaning | -| --- | --- | -| `enqueued` | Packets successfully added to the FIFO | -| `dequeued` | Packets returned for transmission | -| `policy_dropped` | Packets rejected at the queue limit or after node allocation failed | -| `cleanup_dropped` | Packets drained by qdisc reset | -| `bytes_dequeued` | Total qdisc packet bytes returned by dequeue | -| `max_qlen` | Peak packet occupancy of the FIFO | - -## Keeping the experiment inside one interface +#### Keeping the Experiment on One Interface Before `egress_pacer_bpf__load()`, the CLI writes the chosen rate and queue limit into `skel->rodata->rate_kbps` and `skel->rodata->queue_limit`. Loading invokes the verifier, attaching the skeleton registers the `struct_ops` implementation, and `bpf_tc_hook_create` creates `bpf_pacer` at `BPF_TC_QDISC` and `TC_H_ROOT` on the selected ifindex. diff --git a/src/53-egress-pacer/README.zh.md b/src/53-egress-pacer/README.zh.md index a6f4936..bd3e43e 100644 --- a/src/53-egress-pacer/README.zh.md +++ b/src/53-egress-pacer/README.zh.md @@ -1,25 +1,24 @@ # eBPF 教程：用 BPF Qdisc 实现出口限速 -常见的流量控制 BPF 程序只负责检查报文并给出处理结果，真正的排队和发送时机仍由内核 qdisc 决定。Linux 6.16 开始允许 BPF `struct_ops` 直接实现 qdisc，这意味着 BPF 程序也可以接管队列本身。 +你有没有遇到过这样的情况：只想限制一个测试接口的出口速率，同时看清每个报文究竟什么时候可以发出去？常见的 TC BPF 示例看不到这一段。它们可以检查 skb 并返回处理动作，但报文怎样排队、什么时候发送，仍由另一个 qdisc 决定。 -本课用这个接口实现 `egress_pacer`。它是一个刻意保持简单的 FIFO 限速器，适合运行在一次性的 veth、TAP 或 IFB 接口上。给它一个速率和队列上限，它就会保存 skb，等到发送时间后再交还给网络栈，并在结束时报告入队、出队和丢包统计。 +Linux 6.16 让 BPF 可以再向前一步：`struct_ops` 程序能够直接实现 qdisc。本课会用它构建 `egress_pacer`，一个运行在临时 veth、TAP 或 IFB 接口上的小型 FIFO 限速器。你只要给出速率和队列上限，它就会保存 skb，等到发送时间后再交还给网络栈，最后报告入队、发送和丢包统计。 -这不是面向生产网卡的通用调度器。它的作用是把 qdisc 最关键的责任完整展示出来，包括 skb 所有权、队列统计、发送时间、watchdog 唤醒和 reset 清理。完整项目位于 [`53-egress-pacer` 源码目录](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer)。 +这个例子故意没有加入 class、公平调度、ECN 和 burst 控制。这样我们才能把 qdisc 必须做对的事情看清楚：skb 所有权、队列统计、发送时间、watchdog 唤醒和清理。请把它当作实验工具，不要直接用作物理网卡上的生产调度器。 -## 从检查报文到管理队列 +> 完整源码： -普通 TC classifier 运行时，队列仍由另一个 qdisc 管理。[第 20 课](../20-tc/README.zh.md) 展示的就是这种模式。`egress_pacer` 则再向下走了一层。它通过 `struct_ops` 实现 `struct Qdisc_ops`，把自己注册为 root qdisc，并对接收的每个 skb 负责。 +## 当 BPF 自己成为 Qdisc -我们可以跟着一个报文看完整流程。用户态先写入速率和报文上限，再加载并注册 `bpf_pacer`，最后把它创建在 `TC_H_ROOT`。报文到达后，`enqueue` 回调把 skb 移进 BPF 管理的节点，并计算允许发送的时间。`dequeue` 回调随后决定立即返回报文，还是让 qdisc watchdog 到点后再来取。移除 qdisc 时，`reset` 负责释放仍在等待的报文。 +在普通的 TC 示例中，BPF 程序可以看到报文，但队列仍由另一个 qdisc 管理。[第 20 课](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/20-tc) 用的就是这种模式。`egress_pacer` 再向下走一层。它通过 `struct_ops` 实现 `struct Qdisc_ops`，把自己注册为 root qdisc，并对接收的每个 skb 负责。 -后面的代码分析都会沿着这条路径展开。 +跟着一个报文走一遍就容易理解了。用户态先写入速率和队列上限，加载并注册 `bpf_pacer`，再把它挂到 `TC_H_ROOT`。`enqueue` 回调把 skb 移进 BPF 管理的节点，并计算发送时间。`dequeue` 回调要么返回报文，要么请 qdisc watchdog 等到报文可以发送时再来。用户态移除 qdisc 后，`reset` 会释放仍在队列里的报文。 -## 完整源码 +## 一步步构建 Pacer -共享头文件定义了 BPF 程序与用户态共同使用的统计结构。 +我们按组件来构建这个限速器。先从 BPF 程序和用户态共同使用的统计结构开始。 -
-egress_pacer.h：共享统计结构 +### 让 BPF 和用户态共享统计信息 ```c @@ -40,12 +39,13 @@ struct pacer_stats { ``` -
+BPF 程序会在报文经过队列时更新这六个计数器。`enqueued` 和 `dequeued` 记录正常的入队与出队；`policy_dropped` 记录队列已满或节点分配失败造成的拒绝；`cleanup_dropped` 记录 reset 时仍留在队列里的报文。最后两个字段分别保存已经发送的字节数和本次运行观察到的最大队长。 -仓库中较早生成的头文件缺少部分 BPF qdisc 定义。下面这个本地兼容头文件补齐了所需声明，不会修改仓库全局集成的头文件。 +loader 移除 qdisc 后，会从 `skel->bss->stats` 读取同一个结构。把定义放在共享头文件中，可以避免 BPF 程序和用户态对字段布局产生不同理解。 -
-bpf_experimental.h：本地兼容声明 +### 补齐缺少的 BPF 声明 + +仓库中较早生成的头文件缺少部分 BPF qdisc 定义。下面这个本地兼容头文件补齐所需声明，不会修改仓库全局集成的头文件。 ```c @@ -84,12 +84,13 @@ bpf_list_pop_front(struct bpf_list_head *head) __ksym; ``` -
+这个文件本身不做限速。它只是把真正的 qdisc 会用到的 graph-object kfunc 暴露出来：`bpf_obj_new` 和 `bpf_obj_drop` 管理报文节点的生命周期，`bpf_list_push_back` 和 `bpf_list_pop_front` 管理 FIFO。`__contains` 生成的 BTF tag 则告诉 verifier，list head 中保存的是哪一种节点。 -内核态程序实现 qdisc 操作、报文队列、限速时钟和统计计数。 +这些声明放在教程目录内，是因为它们只是当前仓库的兼容胶水，不应该取代仓库统一生成的 libbpf 头文件。以后生成的头文件补齐同样的定义时，删除这个文件也不会影响限速算法。 -
-egress_pacer.bpf.c：BPF qdisc 实现 +### 实现 Qdisc 回调 + +接下来进入内核态程序。它实现 qdisc 回调、报文队列、限速时钟和统计计数。 ```c @@ -293,13 +294,47 @@ struct Qdisc_ops pacer = { ``` -
+#### 接管一个 skb + +`init` 回调先把 `queue_limit` 写入 `sch->limit`。从这一步开始，`egress_pacer_enqueue` 必须明确处理每个 skb：要么接管它，要么拒绝它。程序读取 `qdisc_skb_cb(skb)->pkt_len`，检查队列上限，再用 `bpf_obj_new` 分配 `packet_node`。队列已满和节点分配失败会进入同一条 drop 路径，因此都会增加 `policy_dropped`。 + +报文可以入队时，`bpf_kptr_xchg` 会把 skb 转移到节点的 `__kptr` 字段。此后节点拥有这个 skb，同时保存报文长度和 `eligible_ns` 时间戳。程序持有 `queue_lock` 时，把节点加入 FIFO，并一起更新 qlen、backlog、`enqueued` 和 `max_qlen`。 + +发送时间来自这个报文在目标速率下的序列化时间： + +```text +interval_ns = packet_len * 8 * 1,000,000 / rate_kbps +eligible_ns = max(next_departure_ns, now) +next_departure_ns = eligible_ns + interval_ns +``` + +如果队列刚才是空的，`next_departure_ns` 已经落在当前时间之前，所以第一个报文可以立即离开。后续报文再按各自长度推进时钟。这样得到的是一个聚合 FIFO 发送计划，不是 token bucket，也没有单独的突发额度。 + +#### 等待发送，但不忙等 + +`egress_pacer_dequeue` 先取出队首节点，再比较它的 `eligible_ns` 和 `bpf_ktime_get_ns()`。发送时间已经到达时，`bpf_kptr_xchg` 会把 skb 从节点中取出。回调随后减少 qlen 和 backlog，释放空节点，更新 qdisc 字节统计，并增加 `dequeued` 和 `bytes_dequeued`。 + +如果时间还没到，回调会把节点放回队首，调用 `bpf_qdisc_watchdog_schedule(sch, expire, 0)`，然后返回 `NULL`。这会告诉 qdisc 核心当前没有报文可发。核心会在 `eligible_ns` 到来时重新唤醒队列，BPF 程序不需要自己循环等待。 + +#### 在 Reset 中释放报文 + +移除 qdisc 时，skb 所有权也必须有明确去处。`egress_pacer_reset` 使用 `bpf_for` 排空当前 qlen，通过 `bpf_kfree_skb` 释放每个残留 skb，再销毁节点。这些报文计入 `cleanup_dropped`，正常 enqueue 期间的准入失败则计入 `policy_dropped`。reset 最后还会清零限速时钟、qlen 和 backlog。 + +到这里，BPF 回调已经用到了共享头文件中的全部六项统计： + +| 字段 | 含义 | +| --- | --- | +| `enqueued` | 成功进入 FIFO 的报文数 | +| `dequeued` | 已返回给发送路径的报文数 | +| `policy_dropped` | 队列达到上限或节点分配失败时拒绝的报文数 | +| `cleanup_dropped` | qdisc reset 时排空的报文数 | +| `bytes_dequeued` | dequeue 返回的 qdisc 报文总字节数 | +| `max_qlen` | FIFO 的报文占用峰值 | + +### 加载并移除 Qdisc 用户态加载器负责检查参数、注册 BPF qdisc、把它安装到一个接口，等待指定时间后再将其移除。 -
-egress_pacer.c：加载器与 qdisc 生命周期 - ```c // SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) @@ -601,44 +636,7 @@ cleanup: ``` -
- -## 接管 skb，并算出发送时间 - -qdisc 的 `init` 回调先把 `queue_limit` 写入 `sch->limit`。从这一步开始，`egress_pacer_enqueue` 必须明确处理每个 skb，要么接管它，要么拒绝它。回调读取 `qdisc_skb_cb(skb)->pkt_len`，检查报文数是否达到上限，再用 `bpf_obj_new` 分配 `packet_node`。队列已满和节点分配失败共用同一条丢弃路径，因此都会增加 `policy_dropped`。 - -报文可以入队时，`bpf_kptr_xchg` 会把 skb 转移到节点的 `__kptr` 字段。这个节点随后同时保存报文、qdisc 记录的长度和 `eligible_ns` 时间戳。节点进入 FIFO 时，spin lock 保护链表、限速时钟、qlen 和 backlog。入队成功后，程序增加 `enqueued`，并用 `max_qlen` 记录本次运行中的队列峰值。 - -发送时间来自报文在目标速率下的序列化时间： - -```text -interval_ns = packet_len * 8 * 1,000,000 / rate_kbps -eligible_ns = max(next_departure_ns, now) -next_departure_ns = eligible_ns + interval_ns -``` - -队列空闲一段时间后，`next_departure_ns` 已经落在当前时间之前，所以第一个报文可以立即离开。后续报文再按各自长度推进时钟。这样得到的是一个聚合 FIFO 发送计划，不是 token bucket，也没有额外的突发额度。 - -## 让 watchdog 等待，而不是让 BPF 忙等 - -`egress_pacer_dequeue` 先取出队首节点，再比较 `eligible_ns` 和 `bpf_ktime_get_ns()`。到达发送时间后，另一次 `bpf_kptr_xchg` 把 skb 从节点中取出。回调随后减少 qlen 和 backlog，释放空节点，更新 qdisc 字节统计，并增加 `dequeued` 和 `bytes_dequeued`。 - -如果时间还没到，回调会把节点放回队首，调用 `bpf_qdisc_watchdog_schedule(sch, expire, 0)`，然后返回 `NULL`。qdisc 核心会在 `eligible_ns` 到来时重新唤醒队列，BPF 程序不需要自己循环等待。 - -移除 qdisc 时，skb 所有权也必须有明确去处。`egress_pacer_reset` 使用 `bpf_for` 排空当前 qlen，通过 `bpf_kfree_skb` 释放每个残留 skb，再销毁节点。这里释放的报文计入 `cleanup_dropped`，而正常入队期间的准入失败计入 `policy_dropped`。reset 最后还会清零限速时钟、qlen 和 backlog。 - -最终的 BSS `pacer_stats` 统计分别回答六个问题： - -| 字段 | 含义 | -| --- | --- | -| `enqueued` | 成功进入 FIFO 的报文数 | -| `dequeued` | 已返回给发送路径的报文数 | -| `policy_dropped` | 队列达到上限或节点分配失败时拒绝的报文数 | -| `cleanup_dropped` | qdisc reset 时排空的报文数 | -| `bytes_dequeued` | dequeue 返回的 qdisc 报文总字节数 | -| `max_qlen` | FIFO 的报文占用峰值 | - -## 把实验限制在一个接口内 +#### 把实验限制在一个接口内 调用 `egress_pacer_bpf__load()` 之前，命令行程序会把速率和队列上限写入 `skel->rodata->rate_kbps` 与 `skel->rodata->queue_limit`。加载阶段由内核验证器检查程序，挂载 skeleton 会注册 `struct_ops` 实现，随后 `bpf_tc_hook_create` 在目标 ifindex 的 `BPF_TC_QDISC`、`TC_H_ROOT` 位置创建 `bpf_pacer`。