mirror of
https://github.com/eunomia-bpf/bpf-developer-tutorial.git
synced 2026-10-29 19:58:08 +08:00
Polish and review the BPF egress pacer
This commit is contained in:
LinuxDev9002
@@ -51,6 +51,10 @@ clean:
|
||||
$(Q)rm -rf $(OUTPUT) $(APPS)
|
||||
|
||||
test: all
|
||||
@if [ "$$(id -u)" -ne 0 ]; then \
|
||||
echo "test requires root; run: sudo python3 tests/test_egress_pacer.py ./egress_pacer" >&2; \
|
||||
exit 1; \
|
||||
fi
|
||||
python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
|
||||
$(OUTPUT) $(OUTPUT)/libbpf $(BPFTOOL_OUTPUT):
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
# eBPF Tutorial by Example: Pace Egress Traffic with a BPF Qdisc
|
||||
|
||||
You have a lab, a test bench, or a service running behind a dedicated veth or TAP interface. Something sends a burst, the queue grows unbounded, latency spikes, and you have no visibility into the drop. You want a short-lived egress pacer that installs instantly, reports queue and byte counters, and removes itself on exit or signal.
|
||||
You have a lab, a test bench, or a service running behind a dedicated veth or TAP interface. Something sends a burst, the queue grows unbounded, latency spikes, and you have no visibility into the drop. You want a duration-bounded egress pacer that reports queue and byte counters, then removes itself on normal exit or when its SIGINT or SIGTERM handler runs. An uncatchable signal still requires the recovery path below.
|
||||
|
||||
This tutorial builds a single-purpose BPF qdisc that does exactly that. It uses the Linux 6.16 `struct_ops`-based BPF qdisc infrastructure to register a bounded FIFO, pace packets at an aggregate rate you choose, and expose enqueue/dequeue/drop statistics in user space.
|
||||
|
||||
This is not production-ready for a general-purpose NIC. It is intentionally designed for a controlled interface (veth, TAP, IFB) where you make an explicit lifecycle decision. The loader refuses to replace any existing root qdisc.
|
||||
|
||||
[Browse the complete source for this lesson](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer).
|
||||
|
||||
## Quick start
|
||||
|
||||
Build without root:
|
||||
@@ -16,13 +18,19 @@ make clean
|
||||
make -j2
|
||||
```
|
||||
|
||||
Run on a dedicated interface:
|
||||
For a self-contained first run, let the integration fixture create and remove a disposable veth pair. This step needs root:
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
To run the loader manually on a dedicated interface that you already created and control, inspect it first with `tc qdisc show dev veth-service`. The loader refuses installation if the kernel reports an existing root qdisc:
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
```
|
||||
|
||||
The tool prints a `READY` line when the qdisc is active, then a `SUMMARY` line with final counters when it exits:
|
||||
The tool prints a `READY` line when the qdisc is active, then a `SUMMARY` line with final counters when it exits. The following block shows the output shape. Its ellipses are placeholders, not captured values:
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
@@ -57,10 +65,10 @@ The repository includes a deterministic integration fixture. This output was cap
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8
|
||||
TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, and normal/signal cleanup succeeded
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded
|
||||
```
|
||||
|
||||
This is a functional smoke test, not benchmark-quality performance evidence. 40 raw 1024-byte EtherType `0x88B5` frames are sent in a burst through a 64 Kbit/s pacer with an 8-packet queue. 31 are dropped at the queue limit; 9 are paced and delivered. Do not generalize these numbers beyond this fixture.
|
||||
This is a functional smoke test, not benchmark-quality performance evidence. 40 raw 1024-byte EtherType `0x88B5` frames are sent in a burst through a 64 Kbit/s pacer with an 8-packet queue. 31 are counted as `policy_dropped` when the queue fills in this run, and 9 are paced and delivered. Do not generalize these numbers beyond this fixture.
|
||||
|
||||
Run the fixture yourself on a suitable disposable Linux 6.16+ system:
|
||||
|
||||
@@ -68,15 +76,19 @@ Run the fixture yourself on a suitable disposable Linux 6.16+ system:
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
The fixture creates and deletes `epac_tx`/`epac_rx` veth interfaces, installs a conflicting `pfifo` first to prove refusal, then sends the burst. It asserts bounded drops, byte/packet accounting, pacing, normal cleanup, SIGTERM cleanup, and deletes the veth pair in a `finally` block.
|
||||
The fixture creates and deletes `epac_tx`/`epac_rx` veth interfaces. It installs a conflicting `pfifo` first to prove refusal, then sends the burst and checks bounded drops, byte/packet accounting, pacing, and normal cleanup.
|
||||
|
||||
A second burst leaves packets queued before SIGTERM. The test requires a nonzero `cleanup_dropped` count and verifies `enqueued = dequeued + cleanup_dropped`. Finally, it proves that SIGKILL leaves `bpf_pacer` installed on the tested guest, removes the qdisc explicitly with `tc`, verifies recovery, and deletes the veth pair in a `finally` block.
|
||||
|
||||
## How it works
|
||||
|
||||
The tool has two parts: a BPF object that implements `struct Qdisc_ops`, and a user-space loader that manages the qdisc lifecycle.
|
||||
|
||||
Unlike the [ordinary TC classifier/action example](../20-tc/README.md), where a classifier returns an action while the qdisc remains a separate kernel component, this struct_ops program registers and installs `bpf_pacer` as the root qdisc itself. Its `init` callback copies the configured `queue_limit` into `sch->limit` before packets arrive.
|
||||
|
||||
### BPF qdisc: enqueue
|
||||
|
||||
When a packet arrives at the qdisc, the enqueue callback checks the queue length against `sch->limit`. If the queue is full, the packet is dropped and `policy_dropped` increments. Otherwise:
|
||||
When a packet arrives at the qdisc, the enqueue callback checks the queue length against `sch->limit`. If the queue is full, the packet is dropped and `policy_dropped` increments. A `bpf_obj_new` allocation failure follows the same drop-and-count path. Otherwise:
|
||||
|
||||
1. Allocate a `packet_node` with `bpf_obj_new`.
|
||||
2. Transfer the skb into the node via `bpf_kptr_xchg`.
|
||||
@@ -182,7 +194,7 @@ struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
}
|
||||
```
|
||||
|
||||
The watchdog (`bpf_qdisc_watchdog_schedule`) tells the kernel to call dequeue again at the eligible time. This is the pacing mechanism: the kernel does not poll; it sleeps until the next packet is due.
|
||||
The watchdog (`bpf_qdisc_watchdog_schedule`) asks the qdisc core to retry dequeue when the head packet becomes eligible. This provides pacing without busy-polling in the BPF program.
|
||||
|
||||
### BPF qdisc: reset
|
||||
|
||||
@@ -200,7 +212,7 @@ The loader (`egress_pacer.c`) follows this sequence:
|
||||
6. Print `READY`, then sleep in 100 ms increments until the duration expires or a signal arrives.
|
||||
7. Destroy the qdisc via `bpf_tc_hook_destroy`, read final BSS counters, print `SUMMARY`, and destroy the skeleton.
|
||||
|
||||
Signal handlers set a flag that breaks the sleep loop. SIGINT and SIGTERM are tested cleanup paths.
|
||||
SIGINT and SIGTERM share a handler that sets a flag and breaks the sleep loop. The fixture exercises duration-based normal exit and SIGTERM cleanup. It does not exercise SIGINT separately.
|
||||
|
||||
### Statistics
|
||||
|
||||
@@ -210,7 +222,7 @@ The `pacer_stats` struct in BSS holds:
|
||||
|-------|---------|
|
||||
| `enqueued` | packets successfully queued |
|
||||
| `dequeued` | packets transmitted |
|
||||
| `policy_dropped` | packets dropped at the queue limit |
|
||||
| `policy_dropped` | packets dropped at the queue limit or after node allocation failure |
|
||||
| `cleanup_dropped` | packets drained during reset |
|
||||
| `bytes_dequeued` | total bytes of dequeued packets |
|
||||
| `max_qlen` | peak queue occupancy |
|
||||
@@ -220,9 +232,11 @@ The loader reads these from `skel->bss->stats` after destroying the qdisc.
|
||||
## Requirements
|
||||
|
||||
- Linux kernel 6.16 or later. BPF qdisc support entered in [this commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb).
|
||||
- libbpf 1.6.0 or later for BPF qdisc TC hook support ([libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)). This repository vendors bpftool v7.7.0 and libbpf v1.7.0.
|
||||
- libbpf 1.6.0 or later for BPF qdisc TC hook support ([libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)). The repository vendors libbpf and bpftool under `src/third_party`.
|
||||
- Architecture: x86_64 (tested).
|
||||
- Root privileges for BPF load, qdisc creation, and raw test sockets.
|
||||
- The tool needs root privileges to load BPF and create or destroy a qdisc.
|
||||
- The fixture needs root privileges, Python 3, raw packet sockets, and `iproute2` (`ip` and `tc`).
|
||||
- The explicit recovery path needs `tc`.
|
||||
- BTF and BPF JIT enabled.
|
||||
- Kernel configs: `CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_NET_SCHED=y`, `CONFIG_NET_SCH_BPF=y`.
|
||||
- A controlled network interface. A veth pair is sufficient.
|
||||
@@ -230,13 +244,13 @@ The loader reads these from `skel->bss->stats` after destroying the qdisc.
|
||||
## Limitations
|
||||
|
||||
- One aggregate FIFO. No fairness, per-flow isolation, per-cgroup policy, classes, priorities, ECN, burst budget, or congestion-control integration.
|
||||
- Queue limit is in packets; rate accounting uses `qdisc_skb_cb(skb)->pkt_len`.
|
||||
- Queue limit is in packets. Rate accounting uses `qdisc_skb_cb(skb)->pkt_len`.
|
||||
- The first packet after an idle period departs immediately.
|
||||
- Rate and queue limit are fixed for one invocation.
|
||||
- One root qdisc instance per loader process due to global queue and BSS state.
|
||||
- Refuses every existing root qdisc without inspecting, preserving, stacking, or restoring it.
|
||||
- Duration-bounded CLI. Not a daemon, controller, metrics exporter, or persistent policy manager.
|
||||
- Normal exit, SIGINT, and SIGTERM are tested cleanup paths. SIGKILL, host crash, or abnormal process failure will not clean up the qdisc.
|
||||
- Duration-based normal exit and SIGTERM are tested cleanup paths. SIGINT uses the same handler but was not exercised separately. On the tested guest, SIGKILL left the qdisc installed. Cleanup is not guaranteed after an uncatchable signal or abnormal process failure, so inspect the interface and use the recovery command below. Host-crash and reboot behavior were not tested.
|
||||
- KVM test validates behavior but is not a throughput or precision benchmark. It does not account for driver/hardware offload, GSO/TSO, or real NIC timing.
|
||||
|
||||
## Recovery
|
||||
@@ -258,7 +272,7 @@ This changes traffic scheduling on the interface. Only do this on an interface y
|
||||
## References
|
||||
|
||||
- [Linux BPF qdisc merge commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)
|
||||
- [Qdisc watchdog follow-up commit](https://github.com/torvalds/linux/commit/7a2dafd33c4506e4cb4e63a422530c1cd5a35481)
|
||||
- [Qdisc watchdog follow-up commit](https://github.com/torvalds/linux/commit/7a2dafda950b78611dc441c83d105dfdc7082681)
|
||||
- [libbpf qdisc TC hook support](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)
|
||||
- [Upstream BPF FIFO qdisc selftest](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c)
|
||||
- [Upstream BPF qdisc test runner](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c)
|
||||
@@ -266,7 +280,7 @@ This changes traffic scheduling on the interface. Only do this on an interface y
|
||||
|
||||
## More
|
||||
|
||||
This example is part of the eBPF Developer Tutorial. For more examples and the full tutorial, visit:
|
||||
You can now install a short-lived bounded pacer on a controlled interface, observe its queue and drop accounting, refuse unsafe root replacement, and recover the interface after abnormal termination. This example is part of the eBPF Developer Tutorial. For more examples and the full tutorial, visit:
|
||||
|
||||
- <https://github.com/eunomia-bpf/bpf-developer-tutorial>
|
||||
- <https://eunomia.dev/tutorials/>
|
||||
|
||||
@@ -1,14 +1,16 @@
|
||||
# eBPF 实践教程: 使用 BPF Qdisc 对出口流量进行限速
|
||||
# eBPF 实践教程：使用 BPF Qdisc 对出口流量进行限速
|
||||
|
||||
你有一个实验环境、测试台或通过专用 veth/TAP 接口运行的服务。某个组件突然发出流量突发，队列无限制地增长，延迟飙升，而你对丢包情况一无所知。你需要一个即装即用的短期出口限速器，它能报告队列和字节统计，并在退出或收到信号时自动移除。
|
||||
你有一个实验环境、测试台或通过专用 veth/TAP 接口运行的服务。某个组件突然发出流量突发，队列无限制地增长，延迟飙升，而你对丢包情况一无所知。你需要一个按指定时长运行的出口限速器，它能报告队列和字节统计，并在正常退出或处理 SIGINT、SIGTERM 时移除自身。不可捕获的信号仍需使用下文的恢复流程。
|
||||
|
||||
本教程构建的工具正是做这件事。它利用 Linux 6.16 的 `struct_ops` BPF qdisc 基础设施注册一个有界 FIFO，以你指定的总速率对报文进行整形，并将入队/出队/丢弃统计暴露给用户态。
|
||||
本教程构建的工具正是做这件事。它利用 Linux 6.16 的 `struct_ops` BPF qdisc 基础设施注册一个有界 FIFO，以你指定的总速率限制出口流量。这里的限速通过按报文序列化时间进行整形（pacing）实现，并将入队/出队/丢弃统计暴露给用户态。
|
||||
|
||||
这不是生产级工具，不适用于通用物理网卡。它的设计目标是受控接口（veth、TAP、IFB），由操作者做出明确的生命周期决定。加载器拒绝替换任何已存在的 root qdisc。
|
||||
|
||||
[查看本课的完整源码](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer)。
|
||||
|
||||
## 快速开始
|
||||
|
||||
无需 root 即可编译:
|
||||
无需 root 即可编译：
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
@@ -16,20 +18,26 @@ make clean
|
||||
make -j2
|
||||
```
|
||||
|
||||
在专用接口上运行:
|
||||
首次运行时，建议让集成测试自动创建并删除一次性 veth 对。此步骤需要 root 权限：
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
如果已经创建并控制一个专用接口，也可以手动运行加载器。请先通过 `tc qdisc show dev veth-service` 检查接口；如果内核报告已有 root qdisc，加载器会拒绝安装：
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
```
|
||||
|
||||
qdisc 激活后输出 `READY`，退出时输出 `SUMMARY` 统计:
|
||||
当 qdisc 激活后，工具输出 `READY`；退出时则输出 `SUMMARY` 统计。下面只展示输出格式，其中的省略号是占位符，并非实测值：
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
完整用法:
|
||||
完整用法：
|
||||
|
||||
```text
|
||||
Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKETS] [--duration SEC] [--verbose]
|
||||
@@ -43,7 +51,7 @@ Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKET
|
||||
| `--duration` | 1 到 86400 秒 | 10 |
|
||||
| `--verbose` | 开关 | 关闭 |
|
||||
|
||||
如果接口已有 root qdisc，工具会拒绝并输出:
|
||||
如果接口已有 root qdisc，工具会拒绝并输出：
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
@@ -51,39 +59,43 @@ refusing to replace the existing root qdisc on IFACE
|
||||
|
||||
## 验证过的测试输出
|
||||
|
||||
仓库包含一个确定性集成测试。以下输出在 KVM 客户机（内核 `7.0.0-rc2+`）中于宿主机编译后采集:
|
||||
仓库包含一个确定性集成测试。以下输出在 KVM 客户机（内核 `7.0.0-rc2+`）中于宿主机编译后采集：
|
||||
|
||||
```console
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8
|
||||
TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, and normal/signal cleanup succeeded
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded
|
||||
```
|
||||
|
||||
这是功能性冒烟测试，不是基准性能数据。测试向 64 Kbit/s、队列容量 8 的 pacer 突发 40 个 1024 字节的 EtherType `0x88B5` 原始帧。31 个在队列满时被丢弃，9 个被整形后交付。不要将这些数字推广到其他场景。
|
||||
这是功能性冒烟测试，不是基准性能数据。测试向 64 Kbit/s、队列容量 8 的 pacer 突发 40 个 1024 字节的 EtherType `0x88B5` 原始帧。本次运行中，队列填满后有 31 个计入 `policy_dropped`，另有 9 个经过整形后交付。不要将这些数字推广到其他场景。
|
||||
|
||||
在一个可随意使用的 Linux 6.16+ 系统上自行运行:
|
||||
在一个可随意使用的 Linux 6.16+ 系统上自行运行：
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
该测试会创建并删除 `epac_tx`/`epac_rx` veth 对，先安装一个 `pfifo` 以验证冲突拒绝行为，然后发送突发流量。它断言有界丢包、字节和报文计数正确、整形有效、正常退出清理正常、SIGTERM 清理正常，最后在 `finally` 块中删除 veth 对。
|
||||
该测试会创建并删除 `epac_tx`/`epac_rx` veth 对。它先安装一个 `pfifo` 以验证冲突拒绝行为，再发送突发流量。第一轮检查有界丢包、字节和报文计数、整形效果及正常退出清理。
|
||||
|
||||
第二轮在发送突发后保留队列积压，再发送 SIGTERM。测试要求 `cleanup_dropped` 非零，并验证 `enqueued = dequeued + cleanup_dropped`。最后一轮确认 SIGKILL 会在测试客户机上留下 `bpf_pacer`，随后通过 `tc` 显式删除 qdisc、验证恢复结果，并在 `finally` 块中删除 veth 对。
|
||||
|
||||
## 工作原理
|
||||
|
||||
本工具由两部分组成: 实现 `struct Qdisc_ops` 的 BPF 对象，以及管理 qdisc 生命周期的用户态加载器。
|
||||
本工具由两部分组成：实现 `struct Qdisc_ops` 的 BPF 对象，以及管理 qdisc 生命周期的用户态加载器。
|
||||
|
||||
### BPF qdisc: 入队
|
||||
与[普通 TC classifier/action 示例](../20-tc/README.zh.md)不同，后者由 classifier 在独立的内核 qdisc 上返回处理动作。本例的 struct_ops 程序会注册 `bpf_pacer`，并将它自身安装为 root qdisc。报文到达前，`init` 回调会把配置的 `queue_limit` 写入 `sch->limit`。
|
||||
|
||||
当报文到达 qdisc 时，入队回调检查队列长度是否超过 `sch->limit`。如果队列已满，报文被丢弃并累加 `policy_dropped`。否则:
|
||||
### BPF qdisc：入队
|
||||
|
||||
当报文到达 qdisc 时，入队回调检查队列长度是否超过 `sch->limit`。如果队列已满，报文被丢弃并累加 `policy_dropped`。`bpf_obj_new` 分配失败时也会进入同一丢弃和计数路径。否则：
|
||||
|
||||
1. 通过 `bpf_obj_new` 分配一个 `packet_node`。
|
||||
2. 通过 `bpf_kptr_xchg` 将 skb 转移到节点中。
|
||||
3. 计算报文的允许发送时间戳: `next_departure_ns + (packet_len * 8 * 1,000,000 / rate_kbps)` 纳秒。
|
||||
3. 计算报文的允许发送时间戳 `eligible_ns = max(next_departure_ns, now)`，再将 `next_departure_ns` 推进一个序列化间隔，即 `(packet_len * 8 * 1,000,000 / rate_kbps)` 纳秒。
|
||||
4. 将节点推入 BPF 链表，更新 qlen 和 backlog。
|
||||
|
||||
BPF 源码（`egress_pacer.bpf.c`）:
|
||||
以下代码摘自 BPF 源码（`egress_pacer.bpf.c`）：
|
||||
|
||||
```c
|
||||
SEC("struct_ops/egress_pacer_enqueue")
|
||||
@@ -133,9 +145,9 @@ drop:
|
||||
|
||||
间隔公式意味着空闲后的第一个报文可以立即发出（因为 `next_departure_ns` 已经过去，`eligible_ns = now`）。后续报文按配置速率和序列化报文长度间隔发送。
|
||||
|
||||
### BPF qdisc: 出队
|
||||
### BPF qdisc：出队
|
||||
|
||||
出队回调弹出链表头部。如果当前时间早于报文的 `eligible_ns`，报文被放回链表头部，并调度 qdisc watchdog:
|
||||
出队回调弹出链表头部。如果当前时间早于报文的 `eligible_ns`，报文被放回链表头部，并调度 qdisc watchdog：
|
||||
|
||||
```c
|
||||
SEC("struct_ops/egress_pacer_dequeue")
|
||||
@@ -182,15 +194,15 @@ struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
}
|
||||
```
|
||||
|
||||
watchdog（`bpf_qdisc_watchdog_schedule`）告诉内核在 eligible 时间再次调用 dequeue。这就是整形机制: 内核不轮询，而是休眠到下一个报文到期。
|
||||
内核通过 qdisc watchdog（`bpf_qdisc_watchdog_schedule`）在队首报文满足发送条件时重试 dequeue。BPF 程序因此无需忙轮询即可完成整形。
|
||||
|
||||
### BPF qdisc: reset
|
||||
### BPF qdisc：reset
|
||||
|
||||
qdisc 被移除时，`reset` 回调排空所有残留的 skb，释放 `packet_node` 对象，将 qlen/backlog 归零。在此阶段释放的报文计入 `cleanup_dropped`，区别于正常运行期间的策略丢弃。
|
||||
移除 qdisc 时，`reset` 回调排空所有残留的 skb，释放 `packet_node` 对象，将 qlen/backlog 归零。在此阶段释放的报文计入 `cleanup_dropped`，区别于正常运行期间的策略丢弃。
|
||||
|
||||
### 用户态生命周期
|
||||
|
||||
加载器（`egress_pacer.c`）按以下流程执行:
|
||||
加载器（`egress_pacer.c`）按以下流程执行：
|
||||
|
||||
1. 解析参数，将接口名转换为 ifindex。
|
||||
2. 打开 BPF skeleton，设置 `rodata->rate_kbps` 和 `rodata->queue_limit`。
|
||||
@@ -200,17 +212,17 @@ qdisc 被移除时，`reset` 回调排空所有残留的 skb，释放 `packet_no
|
||||
6. 打印 `READY`，以 100 ms 间隔休眠直到持续时间到期或信号到达。
|
||||
7. 通过 `bpf_tc_hook_destroy` 销毁 qdisc，读取最终 BSS 计数器，打印 `SUMMARY`，销毁 skeleton。
|
||||
|
||||
信号处理器设置标志位来中断休眠循环。SIGINT 和 SIGTERM 都是经过测试的清理路径。
|
||||
程序为 SIGINT 和 SIGTERM 注册了同一个信号处理器。处理器设置标志位来中断休眠循环。集成测试覆盖按时长正常退出和 SIGTERM 清理，但没有单独执行 SIGINT 路径。
|
||||
|
||||
### 统计字段
|
||||
|
||||
BSS 中的 `pacer_stats` 结构体包含:
|
||||
内核 BSS 中的 `pacer_stats` 结构体包含：
|
||||
|
||||
| 字段 | 含义 |
|
||||
|------|------|
|
||||
| `enqueued` | 成功入队的报文数 |
|
||||
| `dequeued` | 已发送的报文数 |
|
||||
| `policy_dropped` | 队列满时丢弃的报文数 |
|
||||
| `policy_dropped` | 队列满或节点分配失败时丢弃的报文数 |
|
||||
| `cleanup_dropped` | reset 阶段排空的报文数 |
|
||||
| `bytes_dequeued` | 已发送报文的总字节数 |
|
||||
| `max_qlen` | 队列峰值占用 |
|
||||
@@ -220,11 +232,13 @@ BSS 中的 `pacer_stats` 结构体包含:
|
||||
## 运行要求
|
||||
|
||||
- Linux 内核 6.16 及以上。BPF qdisc 支持通过[此提交](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)合入。
|
||||
- libbpf 1.6.0 及以上用于 BPF qdisc TC hook 支持（[libbpf 提交](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)）。本仓库集成了 bpftool v7.7.0 和 libbpf v1.7.0。
|
||||
- 架构: x86_64（已测试）。
|
||||
- 需要 root 权限进行 BPF 加载、qdisc 创建和原始测试套接字操作。
|
||||
- libbpf 1.6.0 及以上用于 BPF qdisc TC hook 支持（[libbpf 提交](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)）。仓库在 `src/third_party` 下集成 libbpf 和 bpftool。
|
||||
- 架构：x86_64（已测试）。
|
||||
- 工具需要 root 权限加载 BPF，以及创建或销毁 qdisc。
|
||||
- 集成测试需要 root 权限、Python 3、原始报文套接字和 `iproute2`（`ip` 与 `tc`）。
|
||||
- 显式恢复路径需要 `tc`。
|
||||
- 需要开启 BTF 和 BPF JIT。
|
||||
- 内核配置: `CONFIG_BPF=y`、`CONFIG_BPF_SYSCALL=y`、`CONFIG_BPF_JIT=y`、`CONFIG_DEBUG_INFO_BTF=y`、`CONFIG_NET_SCHED=y`、`CONFIG_NET_SCH_BPF=y`。
|
||||
- 内核配置：`CONFIG_BPF=y`、`CONFIG_BPF_SYSCALL=y`、`CONFIG_BPF_JIT=y`、`CONFIG_DEBUG_INFO_BTF=y`、`CONFIG_NET_SCHED=y`、`CONFIG_NET_SCH_BPF=y`。
|
||||
- 需要受控网络接口，veth 对即可满足。
|
||||
|
||||
## 限制
|
||||
@@ -236,18 +250,18 @@ BSS 中的 `pacer_stats` 结构体包含:
|
||||
- 由于全局队列和 BSS 状态，每个加载器进程只能运行一个 root qdisc 实例。
|
||||
- 拒绝所有已存在的 root qdisc，不检查、不保留、不堆叠、不恢复。
|
||||
- 有限时间的 CLI 工具，不是守护进程、控制器、指标导出器或持久策略管理器。
|
||||
- 正常退出、SIGINT 和 SIGTERM 是经过测试的清理路径。SIGKILL、宿主机崩溃或进程异常终止不保证清理 qdisc。
|
||||
- 程序按时长正常退出和处理 SIGTERM 时的清理路径已经过测试。SIGINT 使用同一处理器，但没有单独执行。在测试客户机上，SIGKILL 会留下已安装的 qdisc。不可捕获的信号或进程异常终止后不保证完成清理，因此需要检查接口并执行下文的恢复命令。宿主机崩溃和重启行为未经过测试。
|
||||
- KVM 测试验证行为正确性，不是吞吐量或精度基准测试，不考虑驱动/硬件卸载、GSO/TSO 或真实网卡时序。
|
||||
|
||||
## 恢复
|
||||
|
||||
异常退出后，验证 qdisc 状态:
|
||||
异常退出后，验证 qdisc 状态：
|
||||
|
||||
```bash
|
||||
tc qdisc show dev IFACE
|
||||
```
|
||||
|
||||
如果 `bpf_pacer` 仍存在于受控接口上，移除它:
|
||||
如果 `bpf_pacer` 仍存在于受控接口上，移除它：
|
||||
|
||||
```bash
|
||||
sudo tc qdisc del dev IFACE root
|
||||
@@ -258,7 +272,7 @@ sudo tc qdisc del dev IFACE root
|
||||
## 参考资料
|
||||
|
||||
- [Linux BPF qdisc 合入提交](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)
|
||||
- [Qdisc watchdog 后续提交](https://github.com/torvalds/linux/commit/7a2dafd33c4506e4cb4e63a422530c1cd5a35481)
|
||||
- [Qdisc watchdog 后续提交](https://github.com/torvalds/linux/commit/7a2dafda950b78611dc441c83d105dfdc7082681)
|
||||
- [libbpf qdisc TC hook 支持](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)
|
||||
- [上游 BPF FIFO qdisc 自测程序](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c)
|
||||
- [上游 BPF qdisc 测试运行器](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c)
|
||||
@@ -266,7 +280,7 @@ sudo tc qdisc del dev IFACE root
|
||||
|
||||
## 更多内容
|
||||
|
||||
本示例是 eBPF 开发者教程的一部分。更多示例和完整教程请访问:
|
||||
现在，你可以在受控接口上安装短期有界 pacer，观察队列和丢包计数，拒绝不安全的 root qdisc 替换，并在异常退出后恢复接口。本示例是 eBPF 开发者教程的一部分。更多示例和完整教程请访问：
|
||||
|
||||
- <https://github.com/eunomia-bpf/bpf-developer-tutorial>
|
||||
- <https://eunomia.dev/tutorials/>
|
||||
|
||||
@@ -68,7 +68,7 @@ GPU:
|
||||
- [lesson 42-xdp-loadbalancer](42-xdp-loadbalancer/README.zh.md) eBPF 开发者教程： 简单的 XDP 负载均衡器
|
||||
- [lesson 46-xdp-test](46-xdp-test/README.zh.md) eBPF 实例教程：构建高性能 XDP 数据包生成器
|
||||
- [lesson 50-tcx](50-tcx/README.zh.md) eBPF 入门实践教程第五十篇：使用 TCX Link 实现可组合的流量控制
|
||||
- [lesson 53-egress-pacer](53-egress-pacer/README.zh.md) 使用 BPF Qdisc 对出口流量进行限速
|
||||
- [lesson 53-egress-pacer](53-egress-pacer/README.zh.md) eBPF 实践教程：使用 BPF Qdisc 对出口流量进行限速
|
||||
|
||||
|
||||
追踪:
|
||||
|
||||
@@ -57,7 +57,7 @@
|
||||
| [eBPF 教程：进程级能源监控与功耗分析](48-energy/README.zh.md) | 5.8 | 必需特性 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_EVENTS=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_TRACEPOINTS=y | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 教程：无需内核补丁修复故障的 HID 设备](49-hid/README.zh.md) | 6.3 | 教程文档 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_HID_BPF=y, CONFIG_UHID=y\|m | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 入门实践教程第五十篇：使用 TCX Link 实现可组合的流量控制](50-tcx/README.zh.md) | 6.6 | 教程文档 | x86_64, arm64 | 可选 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_NET_XGRESS=y | 网络接口 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实践教程: 使用 BPF Qdisc 对出口流量进行限速](53-egress-pacer/README.zh.md) | 6.16 | 教程文档 | x86_64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_NET_SCHED=y, CONFIG_NET_SCH_BPF=y | 网络接口 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实践教程：使用 BPF Qdisc 对出口流量进行限速](53-egress-pacer/README.zh.md) | 6.16 | 教程文档 | x86_64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_NET_SCHED=y, CONFIG_NET_SCH_BPF=y | 网络接口 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实例教程：基于 cgroup 的策略控制](cgroup/README.zh.md) | 5.8 | 必需特性 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_CGROUPS=y, CONFIG_CGROUP_BPF=y, CONFIG_CGROUP_DEVICE=y, CONFIG_DEBUG_INFO_BTF=y | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实例教程：BPF Arena 零拷贝共享内存](features/bpf_arena/README.zh.md) | 6.9 | 教程文档 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 教程：BPF 迭代器用于内核数据导出](features/bpf_iters/README.zh.md) | 5.8 | 教程文档 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_DEBUG_INFO_BTF=y | 无 | 需要 | CI 构建测试 |
|
||||
|
||||
Reference in New Issue
Block a user