mirror of
https://github.com/eunomia-bpf/bpf-developer-tutorial.git
synced 2026-10-29 03:34:44 +08:00
Use positive framing in the egress tutorial
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# eBPF Tutorial by Example: Build an Egress Pacer with a BPF Qdisc
|
||||
|
||||
Want to throttle a virtual link in your test environment? Say, squeeze a veth down to 64 Kbit/s and watch how many packets queue up before they start dropping. Ordinary TC BPF examples cannot do this. They inspect packets and return actions, but a separate qdisc still decides when packets actually leave.
|
||||
Want to throttle a virtual link in your test environment? Say, squeeze a veth down to 64 Kbit/s and watch how many packets queue up before they start dropping. Ordinary TC BPF handles inspection and actions, while the existing qdisc retains departure-time control.
|
||||
|
||||
Linux 6.16 hands that layer to BPF as well. You write a set of `struct_ops` callbacks and the kernel treats them as a real root qdisc. This tutorial builds `egress_pacer`, a small FIFO pacer that runs on a veth, TAP, or IFB. Give it a rate and a queue limit. It holds packets, releases them on schedule, and reports enqueue, dequeue, and drop statistics at the end.
|
||||
|
||||
@@ -12,7 +12,7 @@ This example uses one aggregate FIFO to demonstrate skb ownership, queue account
|
||||
|
||||
An ordinary TC BPF program leaves queue ownership with the existing qdisc, so it can only inspect packets and decide whether to pass or drop them. [Lesson 20](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/20-tc) shows this familiar path. `egress_pacer` implements `struct Qdisc_ops` through `struct_ops` and becomes the root qdisc, owning both the enqueue and dequeue stages. Once the kernel hands it an skb, this BPF program decides both queueing and release time.
|
||||
|
||||
Follow one packet and the design becomes easier to see. User space sets the rate and queue limit, loads `bpf_pacer`, and attaches it at `TC_H_ROOT`. On arrival, `enqueue` puts the skb into a BPF-managed node and computes the earliest time it may leave. When `dequeue` runs, it returns an eligible skb or sets the watchdog so the kernel comes back later. User space eventually removes the qdisc, and `reset` disposes of packets that are still waiting. Every piece of code below belongs to this path.
|
||||
Follow one packet and the design becomes easier to see. User space sets the rate and queue limit, loads `bpf_pacer`, and attaches it at `TC_H_ROOT`. On arrival, `enqueue` puts the skb into a BPF-managed node and computes the earliest time it may leave. When `dequeue` runs, it returns an eligible skb or sets the watchdog so the kernel comes back later. User space eventually removes the qdisc, and `reset` drains the remaining queue. Every piece of code below belongs to this path.
|
||||
|
||||
## Start the Pacer with Shared Data
|
||||
|
||||
@@ -37,7 +37,7 @@ struct pacer_stats {
|
||||
#endif /* __EGRESS_PACER_H */
|
||||
```
|
||||
|
||||
These six fields cover the possible outcomes for a packet. `enqueued` and `dequeued` count successful entry and exit. A full queue or failed node allocation increases `policy_dropped`. A packet that is still waiting when the qdisc is removed increases `cleanup_dropped`. `bytes_dequeued` adds up the bytes returned to the transmit path, while `max_qlen` remembers the deepest point reached by the queue.
|
||||
These six fields cover the possible outcomes for a packet. `enqueued` and `dequeued` count successful entry and exit. A full queue or failed node allocation increases `policy_dropped`. Packets remaining in the queue at reset time increase `cleanup_dropped`. `bytes_dequeued` adds up the bytes returned to the transmit path, while `max_qlen` remembers the deepest point reached by the queue.
|
||||
|
||||
The counters live in BSS. After removing the qdisc, the loader reads them through `skel->bss->stats`. Including one header on both sides keeps the layout consistent.
|
||||
|
||||
@@ -86,7 +86,7 @@ Keeping the compatibility layer inside this lesson also separates it from the sc
|
||||
|
||||
### Implement the Complete Qdisc Lifecycle in BPF
|
||||
|
||||
Now we can read the complete kernel-side program. One file contains the qdisc callbacks, FIFO, pacing clock, and counter updates. The main path has only three stages: `enqueue` takes an skb, `dequeue` returns it at the right time, and `reset` handles anything that never left.
|
||||
Now we can read the complete kernel-side program. One file contains the qdisc callbacks, FIFO, pacing clock, and counter updates. The main path has only three stages: `enqueue` takes an skb, `dequeue` returns it at the right time, and `reset` drains the remaining queue.
|
||||
|
||||
```c
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
@@ -292,11 +292,11 @@ The full source is long, but ownership makes it manageable. We will follow the s
|
||||
|
||||
#### `enqueue`: Own the skb and Schedule Its Departure
|
||||
|
||||
During initialization, `init` copies the selected `queue_limit` into `sch->limit`. Every later call to `egress_pacer_enqueue` must give the incoming skb a definite outcome. The callback reads the qdisc-visible length from `qdisc_skb_cb(skb)->pkt_len`, checks for space, and creates a `packet_node` with `bpf_obj_new`. A full queue and a failed allocation both mean that the program cannot accept ownership, so they share one drop path and increment `policy_dropped`.
|
||||
During initialization, `init` copies the selected `queue_limit` into `sch->limit`. Every later call to `egress_pacer_enqueue` must give the incoming skb a definite outcome. The callback reads the qdisc-visible length from `qdisc_skb_cb(skb)->pkt_len`, checks for space, and creates a `packet_node` with `bpf_obj_new`. A full queue and a failed allocation both take the policy-drop path and increment `policy_dropped`.
|
||||
|
||||
The actual ownership transfer happens through `bpf_kptr_xchg`. After it succeeds, the node's `__kptr` field holds the skb and the node is responsible for it. The node also stores the packet length and its `eligible_ns` timestamp. Under `queue_lock`, the program pushes that node to the FIFO tail and updates qlen, backlog, `enqueued`, and `max_qlen` in the same critical section. The list and its accounting therefore move together.
|
||||
|
||||
The delay is not a fixed interval. It comes from the time this packet would occupy the link at the selected rate:
|
||||
The departure interval scales with packet length at the selected rate:
|
||||
|
||||
```text
|
||||
interval_ns = packet_len * 8 * 1,000,000 / rate_kbps
|
||||
@@ -306,7 +306,7 @@ next_departure_ns = eligible_ns + interval_ns
|
||||
|
||||
After an idle period, `next_departure_ns` is behind the current time, so the next packet can leave at once. While the FIFO stays busy, every following packet advances the clock by its own length. The result is one aggregate FIFO schedule where each packet's departure interval depends on its own length.
|
||||
|
||||
#### `dequeue`: Hand Early Packets to the Watchdog
|
||||
#### `dequeue`: Watchdog Schedules the Next Dequeue
|
||||
|
||||
`egress_pacer_dequeue` only needs to examine the front node. It pops that node and compares `eligible_ns` with the current value from `bpf_ktime_get_ns()`. If the packet is ready, a second `bpf_kptr_xchg` returns the skb to the qdisc. The callback then reduces qlen and backlog, drops the empty node, updates the qdisc's byte counters, and increments `dequeued` and `bytes_dequeued`.
|
||||
|
||||
@@ -642,7 +642,7 @@ After installation, the loader prints `READY` and checks the duration and `exiti
|
||||
|
||||
## Build, Run, and Watch the Queue
|
||||
|
||||
The lesson's [`Makefile`](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/Makefile) builds the BPF object, skeleton, and user-space loader. This step does not need root:
|
||||
The lesson's [`Makefile`](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/Makefile) builds the BPF object, skeleton, and user-space loader. A regular user can complete the build:
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
@@ -693,19 +693,13 @@ Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKET
|
||||
| `--duration` | 1 to 86400 seconds | 10 |
|
||||
| `--verbose` | Flag | Off |
|
||||
|
||||
`READY` means installation succeeded. After the duration ends and removal succeeds, the loader prints `SUMMARY`. The ellipses below show the output format, not measured values:
|
||||
`READY` means installation succeeded. After the duration ends and removal succeeds, the loader prints `SUMMARY`. The ellipses below are placeholders for run-specific values:
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
If another root qdisc is present, the loader fails before changing it and explains the refusal:
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## Runtime Environment and Extension Points
|
||||
|
||||
The BPF qdisc support used here starts with Linux 6.16. The [BPF qdisc merge commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb) contains the core change. The kernel needs BTF and the BPF JIT enabled, with at least `CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_NET_SCHED=y`, and `CONFIG_NET_SCH_BPF=y`.
|
||||
@@ -718,7 +712,7 @@ This lesson is suitable for validating functional correctness on controlled inte
|
||||
|
||||
## What to Take Away
|
||||
|
||||
This lesson uses one aggregate FIFO to walk through the full BPF qdisc lifecycle. `enqueue` takes ownership of an skb and schedules its departure from the packet length. The watchdog wakes `dequeue` at the right time. `reset` releases skbs still held when the qdisc is removed. These ownership and timing patterns form the reusable core of a qdisc. A fuller scheduler can add fairness, dynamic policy, and persistent recovery.
|
||||
This lesson uses one aggregate FIFO to walk through the full BPF qdisc lifecycle. `enqueue` takes ownership of an skb and schedules its departure from the packet length. The watchdog wakes `dequeue` at the right time. `reset` releases skbs still held when the qdisc is removed. These ownership and timing patterns form the reusable core of a qdisc.
|
||||
|
||||
> To keep learning eBPF by building complete examples, explore the [tutorial repository](https://github.com/eunomia-bpf/bpf-developer-tutorial).
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# eBPF 教程:用 BPF Qdisc 实现出口限速
|
||||
|
||||
想给测试环境里的一条虚拟网线限个速?比方说,把 veth 压到 64 Kbit/s,看看丢包之前队列能塞多少报文。常见的 TC BPF 示例做不到这一点:它们可以检查报文、决定放行还是丢弃,但报文什么时候发出去,仍由另一个 qdisc 来安排。
|
||||
想给测试环境里的一条虚拟网线限个速?比方说,把 veth 压到 64 Kbit/s,看看丢包之前队列能塞多少报文。常见的 TC BPF 示例处理检查和动作,发送时间仍由原有的 qdisc 安排。
|
||||
|
||||
Linux 6.16 把这一层也交给了 BPF。你可以用 `struct_ops` 写一个真正的 qdisc,自己管 skb 队列,自己算发送时间。本课会用它构建 `egress_pacer`:一个运行在 veth、TAP 或 IFB 上的小型 FIFO 限速器。给出速率和队列上限,它就把报文存起来,到点再放,最后报告入队、发送和丢包统计。
|
||||
|
||||
@@ -12,7 +12,7 @@ Linux 6.16 把这一层也交给了 BPF。你可以用 `struct_ops` 写一个真
|
||||
|
||||
普通 TC BPF 程序把队列所有权留给原有的 qdisc,因此只能检查报文和决定放行或丢弃。[第 20 课](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/20-tc) 展示的就是这条常用路径。`egress_pacer` 用 `struct_ops` 实现 `struct Qdisc_ops`,自己成为 root qdisc,同时拥有入队和出队两个阶段。内核把 skb 交过来之后,排队和发送时间都由这段 BPF 程序决定。
|
||||
|
||||
先跟着一个报文走一遍。用户态设置速率和队列上限,加载 `bpf_pacer`,再把它挂到 `TC_H_ROOT`。报文到达时,`enqueue` 把 skb 装进 BPF 管理的节点,并算出它最早可以离开的时间。轮到 `dequeue` 时,时间到了就交还 skb;时间没到就设置 watchdog,等内核稍后再来。最后,用户态移除 qdisc,`reset` 收走仍在队列里的报文。后面的代码都围绕这条路径展开。
|
||||
先跟着一个报文走一遍。用户态设置速率和队列上限,加载 `bpf_pacer`,再把它挂到 `TC_H_ROOT`。报文到达时,`enqueue` 把 skb 装进 BPF 管理的节点,并算出它最早可以离开的时间。轮到 `dequeue` 时,到点的报文交还 skb;早到的报文设置 watchdog,等内核稍后再来。最后,用户态移除 qdisc,`reset` 收走仍在队列里的报文。后面的代码都围绕这条路径展开。
|
||||
|
||||
## 从共享数据开始搭起 Pacer
|
||||
|
||||
@@ -37,7 +37,7 @@ struct pacer_stats {
|
||||
#endif /* __EGRESS_PACER_H */
|
||||
```
|
||||
|
||||
这六个字段把报文的一生分成几种结果。成功进入和离开 FIFO 的报文分别计入 `enqueued`、`dequeued`。队列已满或节点分配失败时,`policy_dropped` 增加;移除 qdisc 时还没发出的报文则进入 `cleanup_dropped`。`bytes_dequeued` 累计实际交还给发送路径的字节数,`max_qlen` 记住队列最深的一刻。
|
||||
这六个字段把报文的一生分成几种结果。成功进入和离开 FIFO 的报文分别计入 `enqueued`、`dequeued`。队列已满或节点分配失败时,`policy_dropped` 增加;reset 时仍在队列中的报文则进入 `cleanup_dropped`。`bytes_dequeued` 累计实际交还给发送路径的字节数,`max_qlen` 记住队列最深的一刻。
|
||||
|
||||
这些计数器位于 BSS。qdisc 被移除后,加载器直接从 `skel->bss->stats` 读取它们。两端包含同一个头文件,布局始终一致。
|
||||
|
||||
@@ -86,7 +86,7 @@ bpf_list_pop_front(struct bpf_list_head *head) __ksym;
|
||||
|
||||
### 在 BPF 中实现整个 Qdisc 生命周期
|
||||
|
||||
现在来看内核态的完整程序。一个文件里放着 qdisc 回调、FIFO、限速时钟和统计更新。代码较长,但主线只有三步:`enqueue` 接管 skb,`dequeue` 等到合适时间再交还 skb,`reset` 处理没来得及发送的 skb。
|
||||
现在来看内核态的完整程序。一个文件里放着 qdisc 回调、FIFO、限速时钟和统计更新。代码较长,但主线只有三步:`enqueue` 接管 skb,`dequeue` 在合适时间交还 skb,`reset` 排空剩余队列。
|
||||
|
||||
```c
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
@@ -290,11 +290,11 @@ struct Qdisc_ops pacer = {
|
||||
|
||||
#### `enqueue`:接管 skb,并排好发送时间
|
||||
|
||||
初始化时,`init` 把用户选择的 `queue_limit` 放进 `sch->limit`。此后每次调用 `egress_pacer_enqueue`,程序都必须给传入的 skb 一个明确结果。它先从 `qdisc_skb_cb(skb)->pkt_len` 取得 qdisc 看到的报文长度,再检查队列是否还有位置,最后用 `bpf_obj_new` 创建 `packet_node`。队列已满或节点创建失败都表示无法接管这个 skb,因此走同一条丢弃路径,并增加 `policy_dropped`。
|
||||
初始化时,`init` 把用户选择的 `queue_limit` 放进 `sch->limit`。此后每次调用 `egress_pacer_enqueue`,程序都必须给传入的 skb 一个明确结果。它先从 `qdisc_skb_cb(skb)->pkt_len` 取得 qdisc 看到的报文长度,再检查队列是否还有位置,最后用 `bpf_obj_new` 创建 `packet_node`。队列已满或节点创建失败都走同一条 policy-drop 路径,并增加 `policy_dropped`。
|
||||
|
||||
真正的所有权转移发生在 `bpf_kptr_xchg`。调用成功后,skb 被放进节点的 `__kptr` 字段,节点从此负责保管它。节点还记录报文长度和 `eligible_ns`。程序随后拿住 `queue_lock`,把节点推到 FIFO 尾部,并在同一个临界区内更新 qlen、backlog、`enqueued` 和 `max_qlen`,避免链表内容与计数彼此脱节。
|
||||
|
||||
`eligible_ns` 不是固定间隔。程序按报文长度计算它在目标速率下需要占用多少发送时间:
|
||||
发送间隔随报文长度按目标速率伸缩:
|
||||
|
||||
```text
|
||||
interval_ns = packet_len * 8 * 1,000,000 / rate_kbps
|
||||
@@ -304,7 +304,7 @@ next_departure_ns = eligible_ns + interval_ns
|
||||
|
||||
队列空闲一段时间后,`next_departure_ns` 会早于当前时间,因此新来的第一个报文可以立刻离开。只要 FIFO 中持续有数据,后面的报文就按自己的长度继续推动这只时钟。这是一个聚合 FIFO 时间表,每个报文的发送间隔由自身长度决定。
|
||||
|
||||
#### `dequeue`:时间未到就交给 watchdog
|
||||
#### `dequeue`:watchdog 安排下一次出队
|
||||
|
||||
`egress_pacer_dequeue` 每次只看 FIFO 最前面的节点。它取出节点,把 `eligible_ns` 与 `bpf_ktime_get_ns()` 返回的当前时间比较。报文已经可以发送时,第二次 `bpf_kptr_xchg` 把 skb 从节点交回 qdisc。程序接着扣减 qlen 和 backlog,释放已经空掉的节点,更新 qdisc 自身的字节统计,再增加 `dequeued` 和 `bytes_dequeued`。
|
||||
|
||||
@@ -638,7 +638,7 @@ cleanup:
|
||||
|
||||
## 编译、运行,再观察队列行为
|
||||
|
||||
本课的 [`Makefile`](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/Makefile) 会生成 BPF 对象、skeleton 和用户态加载器。编译本身不需要 root:
|
||||
本课的 [`Makefile`](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/Makefile) 会生成 BPF 对象、skeleton 和用户态加载器。普通用户即可完成编译:
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
@@ -689,19 +689,13 @@ Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKET
|
||||
| `--duration` | 1 到 86400 秒 | 10 |
|
||||
| `--verbose` | 开关 | 关闭 |
|
||||
|
||||
看到 `READY` 就表示 qdisc 已经安装。运行结束并成功移除后,程序打印 `SUMMARY`。下面只展示输出格式,其中的省略号不是测量结果:
|
||||
看到 `READY` 就表示 qdisc 已经安装。运行结束并成功移除后,程序打印 `SUMMARY`。下面只展示输出格式,省略号是每次运行的实际值占位符:
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
如果 root qdisc 已被占用,程序会在改变接口之前失败,并明确说明拒绝替换:
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## 运行环境与扩展方向
|
||||
|
||||
这个功能从 Linux 6.16 开始可用。[BPF qdisc 合入 commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb) 包含了核心支持。内核需要启用 BTF 和 BPF JIT,并至少包含 `CONFIG_BPF=y`、`CONFIG_BPF_SYSCALL=y`、`CONFIG_BPF_JIT=y`、`CONFIG_DEBUG_INFO_BTF=y`、`CONFIG_NET_SCHED=y` 与 `CONFIG_NET_SCH_BPF=y`。
|
||||
@@ -714,7 +708,7 @@ BPF qdisc TC hook 需要 libbpf 1.6.0 或更新版本。仓库已经在 `src/thi
|
||||
|
||||
## 小结
|
||||
|
||||
本课用一条聚合 FIFO 走完了 BPF qdisc 的完整生命周期。`enqueue` 接管 skb 并按报文长度安排发送时间;watchdog 让 `dequeue` 在适当时刻被唤醒;`reset` 释放移除时仍在队列里的 skb。这三段所有权和时序处理构成可复用的 qdisc 模式,更完整的调度器可以在此基础上加入公平性、动态策略和持久化恢复。
|
||||
本课用一条聚合 FIFO 走完了 BPF qdisc 的完整生命周期。`enqueue` 接管 skb 并按报文长度安排发送时间;watchdog 让 `dequeue` 在适当时刻被唤醒;`reset` 释放移除时仍在队列里的 skb。这三段所有权和时序处理构成可复用的 qdisc 模式。
|
||||
|
||||
> 想继续动手学习 eBPF,可以在[教程仓库](https://github.com/eunomia-bpf/bpf-developer-tutorial)中找到更多完整示例。
|
||||
|
||||
|
||||
Reference in New Issue
Block a user