mirror of
https://github.com/eunomia-bpf/bpf-developer-tutorial.git
synced 2026-10-29 03:34:44 +08:00
Document the BPF egress pacer
This commit is contained in:
LinuxDev9002
@@ -77,6 +77,7 @@ Networking:
|
||||
- [lesson 42-xdp-loadbalancer](src/42-xdp-loadbalancer/README.md) XDP Load Balancer
|
||||
- [lesson 46-xdp-test](src/46-xdp-test/README.md) Building a High-Performance XDP Packet Generator
|
||||
- [lesson 50-tcx](src/50-tcx/README.md) Composable Traffic Control with TCX Links
|
||||
- [lesson 53-egress-pacer](src/53-egress-pacer/README.md) Pace Egress Traffic with a BPF Qdisc
|
||||
|
||||
|
||||
Tracing:
|
||||
|
||||
@@ -76,6 +76,7 @@ GPU:
|
||||
- [lesson 42-xdp-loadbalancer](src/42-xdp-loadbalancer/README.zh.md) eBPF 开发者教程： 简单的 XDP 负载均衡器
|
||||
- [lesson 46-xdp-test](src/46-xdp-test/README.zh.md) eBPF 实例教程：构建高性能 XDP 数据包生成器
|
||||
- [lesson 50-tcx](src/50-tcx/README.zh.md) eBPF 入门实践教程第五十篇：使用 TCX Link 实现可组合的流量控制
|
||||
- [lesson 53-egress-pacer](src/53-egress-pacer/README.zh.md) 使用 BPF Qdisc 对出口流量进行限速
|
||||
|
||||
|
||||
追踪:
|
||||
|
||||
272
src/53-egress-pacer/README.md
Normal file
272
src/53-egress-pacer/README.md
Normal file
@@ -0,0 +1,272 @@
|
||||
# eBPF Tutorial by Example: Pace Egress Traffic with a BPF Qdisc
|
||||
|
||||
You have a lab, a test bench, or a service running behind a dedicated veth or TAP interface. Something sends a burst, the queue grows unbounded, latency spikes, and you have no visibility into the drop. You want a short-lived egress pacer that installs instantly, reports queue and byte counters, and removes itself on exit or signal.
|
||||
|
||||
This tutorial builds a single-purpose BPF qdisc that does exactly that. It uses the Linux 6.16 `struct_ops`-based BPF qdisc infrastructure to register a bounded FIFO, pace packets at an aggregate rate you choose, and expose enqueue/dequeue/drop statistics in user space.
|
||||
|
||||
This is not production-ready for a general-purpose NIC. It is intentionally designed for a controlled interface (veth, TAP, IFB) where you make an explicit lifecycle decision. The loader refuses to replace any existing root qdisc.
|
||||
|
||||
## Quick start
|
||||
|
||||
Build without root:
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
make clean
|
||||
make -j2
|
||||
```
|
||||
|
||||
Run on a dedicated interface:
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
```
|
||||
|
||||
The tool prints a `READY` line when the qdisc is active, then a `SUMMARY` line with final counters when it exits:
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```text
|
||||
Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKETS] [--duration SEC] [--verbose]
|
||||
```
|
||||
|
||||
| Option | Range | Default |
|
||||
|--------|-------|---------|
|
||||
| `--interface` | required | none |
|
||||
| `--rate-kbps` | 8 to 100000000 | 1024 |
|
||||
| `--queue-limit` | 1 to 65535 packets | 256 |
|
||||
| `--duration` | 1 to 86400 seconds | 10 |
|
||||
| `--verbose` | flag | off |
|
||||
|
||||
If the interface already has a root qdisc, the tool exits with:
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## Verified test output
|
||||
|
||||
The repository includes a deterministic integration fixture. This output was captured in a KVM guest running kernel `7.0.0-rc2+` after a clean host build:
|
||||
|
||||
```console
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8
|
||||
TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, and normal/signal cleanup succeeded
|
||||
```
|
||||
|
||||
This is a functional smoke test, not benchmark-quality performance evidence. 40 raw 1024-byte EtherType `0x88B5` frames are sent in a burst through a 64 Kbit/s pacer with an 8-packet queue. 31 are dropped at the queue limit; 9 are paced and delivered. Do not generalize these numbers beyond this fixture.
|
||||
|
||||
Run the fixture yourself on a suitable disposable Linux 6.16+ system:
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
The fixture creates and deletes `epac_tx`/`epac_rx` veth interfaces, installs a conflicting `pfifo` first to prove refusal, then sends the burst. It asserts bounded drops, byte/packet accounting, pacing, normal cleanup, SIGTERM cleanup, and deletes the veth pair in a `finally` block.
|
||||
|
||||
## How it works
|
||||
|
||||
The tool has two parts: a BPF object that implements `struct Qdisc_ops`, and a user-space loader that manages the qdisc lifecycle.
|
||||
|
||||
### BPF qdisc: enqueue
|
||||
|
||||
When a packet arrives at the qdisc, the enqueue callback checks the queue length against `sch->limit`. If the queue is full, the packet is dropped and `policy_dropped` increments. Otherwise:
|
||||
|
||||
1. Allocate a `packet_node` with `bpf_obj_new`.
|
||||
2. Transfer the skb into the node via `bpf_kptr_xchg`.
|
||||
3. Compute the packet's eligible departure timestamp: `eligible_ns = max(next_departure_ns, now)`. Then advance `next_departure_ns` by the packet's serialization interval `(packet_len * 8 * 1,000,000 / rate_kbps)` nanoseconds.
|
||||
4. Push the node onto a BPF linked list and update qlen/backlog.
|
||||
|
||||
From the BPF source (`egress_pacer.bpf.c`):
|
||||
|
||||
```c
|
||||
SEC("struct_ops/egress_pacer_enqueue")
|
||||
int BPF_PROG(egress_pacer_enqueue, struct sk_buff *skb, struct Qdisc *sch,
|
||||
struct bpf_sk_buff_ptr *to_free)
|
||||
{
|
||||
struct packet_node *packet;
|
||||
__u64 eligible_ns, interval_ns, now;
|
||||
__u32 packet_len;
|
||||
|
||||
packet_len = qdisc_packet_len(skb);
|
||||
if (sch->q.qlen >= sch->limit)
|
||||
goto drop;
|
||||
|
||||
packet = bpf_obj_new(typeof(*packet));
|
||||
if (!packet)
|
||||
goto drop;
|
||||
|
||||
now = bpf_ktime_get_ns();
|
||||
interval_ns = (__u64)packet_len * 8 * 1000000ULL / rate_kbps;
|
||||
if (!interval_ns)
|
||||
interval_ns = 1;
|
||||
packet->packet_len = packet_len;
|
||||
skb = bpf_kptr_xchg(&packet->skb, skb);
|
||||
if (skb)
|
||||
bpf_qdisc_skb_drop(skb, to_free);
|
||||
|
||||
bpf_spin_lock(&queue_lock);
|
||||
eligible_ns = next_departure_ns > now ? next_departure_ns : now;
|
||||
next_departure_ns = eligible_ns + interval_ns;
|
||||
packet->eligible_ns = eligible_ns;
|
||||
bpf_list_push_back(&packet_queue, &packet->node);
|
||||
sch->q.qlen++;
|
||||
sch->qstats.backlog += packet_len;
|
||||
__sync_fetch_and_add(&stats.enqueued, 1);
|
||||
if (sch->q.qlen > stats.max_qlen)
|
||||
stats.max_qlen = sch->q.qlen;
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
return NET_XMIT_SUCCESS;
|
||||
|
||||
drop:
|
||||
bpf_qdisc_skb_drop(skb, to_free);
|
||||
__sync_fetch_and_add(&stats.policy_dropped, 1);
|
||||
return NET_XMIT_DROP;
|
||||
}
|
||||
```
|
||||
|
||||
The interval formula means the first packet after an idle period departs immediately (because `next_departure_ns` will be in the past, so `eligible_ns = now`). Subsequent packets are spaced by serialized packet length at the configured rate.
|
||||
|
||||
### BPF qdisc: dequeue
|
||||
|
||||
The dequeue callback pops the head of the list. If the current time is before the packet's `eligible_ns`, the packet goes back to the front of the list and the qdisc watchdog is scheduled:
|
||||
|
||||
```c
|
||||
SEC("struct_ops/egress_pacer_dequeue")
|
||||
struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
{
|
||||
struct bpf_list_node *node;
|
||||
struct packet_node *packet;
|
||||
struct sk_buff *skb = NULL;
|
||||
__u64 expire, now;
|
||||
__u32 packet_len;
|
||||
|
||||
bpf_spin_lock(&queue_lock);
|
||||
node = bpf_list_pop_front(&packet_queue);
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
if (!node)
|
||||
return NULL;
|
||||
|
||||
packet = container_of(node, struct packet_node, node);
|
||||
now = bpf_ktime_get_ns();
|
||||
if (now < packet->eligible_ns) {
|
||||
expire = packet->eligible_ns;
|
||||
bpf_spin_lock(&queue_lock);
|
||||
bpf_list_push_front(&packet_queue, &packet->node);
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
bpf_qdisc_watchdog_schedule(sch, expire, 0);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
packet_len = packet->packet_len;
|
||||
skb = bpf_kptr_xchg(&packet->skb, skb);
|
||||
bpf_spin_lock(&queue_lock);
|
||||
sch->q.qlen--;
|
||||
sch->qstats.backlog -= packet_len;
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
bpf_obj_drop(packet);
|
||||
|
||||
if (!skb)
|
||||
return NULL;
|
||||
|
||||
bpf_qdisc_bstats_update(sch, skb);
|
||||
__sync_fetch_and_add(&stats.dequeued, 1);
|
||||
__sync_fetch_and_add(&stats.bytes_dequeued, packet_len);
|
||||
return skb;
|
||||
}
|
||||
```
|
||||
|
||||
The watchdog (`bpf_qdisc_watchdog_schedule`) tells the kernel to call dequeue again at the eligible time. This is the pacing mechanism: the kernel does not poll; it sleeps until the next packet is due.
|
||||
|
||||
### BPF qdisc: reset
|
||||
|
||||
On qdisc removal, the `reset` callback drains all retained skbs, frees the `packet_node` objects, and zeroes qlen/backlog. Packets freed here are counted in `cleanup_dropped`, distinct from the policy drops during normal operation.
|
||||
|
||||
### User-space lifecycle
|
||||
|
||||
The loader (`egress_pacer.c`) follows this sequence:
|
||||
|
||||
1. Parse arguments and resolve the interface name to an ifindex.
|
||||
2. Open the BPF skeleton and set `rodata->rate_kbps` and `rodata->queue_limit`.
|
||||
3. Load the BPF object (verifier runs here).
|
||||
4. Attach the struct_ops map, registering `bpf_pacer` as an available qdisc type.
|
||||
5. Create the qdisc at the interface root via `bpf_tc_hook_create` with `BPF_TC_QDISC` and `TC_H_ROOT`. If the interface already has a root qdisc, the call returns `-EEXIST` and the tool prints the refusal and exits.
|
||||
6. Print `READY`, then sleep in 100 ms increments until the duration expires or a signal arrives.
|
||||
7. Destroy the qdisc via `bpf_tc_hook_destroy`, read final BSS counters, print `SUMMARY`, and destroy the skeleton.
|
||||
|
||||
Signal handlers set a flag that breaks the sleep loop. SIGINT and SIGTERM are tested cleanup paths.
|
||||
|
||||
### Statistics
|
||||
|
||||
The `pacer_stats` struct in BSS holds:
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| `enqueued` | packets successfully queued |
|
||||
| `dequeued` | packets transmitted |
|
||||
| `policy_dropped` | packets dropped at the queue limit |
|
||||
| `cleanup_dropped` | packets drained during reset |
|
||||
| `bytes_dequeued` | total bytes of dequeued packets |
|
||||
| `max_qlen` | peak queue occupancy |
|
||||
|
||||
The loader reads these from `skel->bss->stats` after destroying the qdisc.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Linux kernel 6.16 or later. BPF qdisc support entered in [this commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb).
|
||||
- libbpf 1.6.0 or later for BPF qdisc TC hook support ([libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)). This repository vendors bpftool v7.7.0 and libbpf v1.7.0.
|
||||
- Architecture: x86_64 (tested).
|
||||
- Root privileges for BPF load, qdisc creation, and raw test sockets.
|
||||
- BTF and BPF JIT enabled.
|
||||
- Kernel configs: `CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_NET_SCHED=y`, `CONFIG_NET_SCH_BPF=y`.
|
||||
- A controlled network interface. A veth pair is sufficient.
|
||||
|
||||
## Limitations
|
||||
|
||||
- One aggregate FIFO. No fairness, per-flow isolation, per-cgroup policy, classes, priorities, ECN, burst budget, or congestion-control integration.
|
||||
- Queue limit is in packets; rate accounting uses `qdisc_skb_cb(skb)->pkt_len`.
|
||||
- The first packet after an idle period departs immediately.
|
||||
- Rate and queue limit are fixed for one invocation.
|
||||
- One root qdisc instance per loader process due to global queue and BSS state.
|
||||
- Refuses every existing root qdisc without inspecting, preserving, stacking, or restoring it.
|
||||
- Duration-bounded CLI. Not a daemon, controller, metrics exporter, or persistent policy manager.
|
||||
- Normal exit, SIGINT, and SIGTERM are tested cleanup paths. SIGKILL, host crash, or abnormal process failure will not clean up the qdisc.
|
||||
- KVM test validates behavior but is not a throughput or precision benchmark. It does not account for driver/hardware offload, GSO/TSO, or real NIC timing.
|
||||
|
||||
## Recovery
|
||||
|
||||
After an abnormal exit, verify with:
|
||||
|
||||
```bash
|
||||
tc qdisc show dev IFACE
|
||||
```
|
||||
|
||||
If `bpf_pacer` is still present on a controlled interface, remove it with:
|
||||
|
||||
```bash
|
||||
sudo tc qdisc del dev IFACE root
|
||||
```
|
||||
|
||||
This changes traffic scheduling on the interface. Only do this on an interface you control.
|
||||
|
||||
## References
|
||||
|
||||
- [Linux BPF qdisc merge commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)
|
||||
- [Qdisc watchdog follow-up commit](https://github.com/torvalds/linux/commit/7a2dafd33c4506e4cb4e63a422530c1cd5a35481)
|
||||
- [libbpf qdisc TC hook support](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)
|
||||
- [Upstream BPF FIFO qdisc selftest](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c)
|
||||
- [Upstream BPF qdisc test runner](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c)
|
||||
- [libbpf 1.7.0 release](https://github.com/libbpf/libbpf/releases/tag/v1.7.0)
|
||||
|
||||
## More
|
||||
|
||||
This example is part of the eBPF Developer Tutorial. For more examples and the full tutorial, visit:
|
||||
|
||||
- <https://github.com/eunomia-bpf/bpf-developer-tutorial>
|
||||
- <https://eunomia.dev/tutorials/>
|
||||
272
src/53-egress-pacer/README.zh.md
Normal file
272
src/53-egress-pacer/README.zh.md
Normal file
@@ -0,0 +1,272 @@
|
||||
# eBPF 实践教程: 使用 BPF Qdisc 对出口流量进行限速
|
||||
|
||||
你有一个实验环境、测试台或通过专用 veth/TAP 接口运行的服务。某个组件突然发出流量突发，队列无限制地增长，延迟飙升，而你对丢包情况一无所知。你需要一个即装即用的短期出口限速器，它能报告队列和字节统计，并在退出或收到信号时自动移除。
|
||||
|
||||
本教程构建的工具正是做这件事。它利用 Linux 6.16 的 `struct_ops` BPF qdisc 基础设施注册一个有界 FIFO，以你指定的总速率对报文进行整形，并将入队/出队/丢弃统计暴露给用户态。
|
||||
|
||||
这不是生产级工具，不适用于通用物理网卡。它的设计目标是受控接口（veth、TAP、IFB），由操作者做出明确的生命周期决定。加载器拒绝替换任何已存在的 root qdisc。
|
||||
|
||||
## 快速开始
|
||||
|
||||
无需 root 即可编译:
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
make clean
|
||||
make -j2
|
||||
```
|
||||
|
||||
在专用接口上运行:
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
```
|
||||
|
||||
qdisc 激活后输出 `READY`，退出时输出 `SUMMARY` 统计:
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
完整用法:
|
||||
|
||||
```text
|
||||
Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKETS] [--duration SEC] [--verbose]
|
||||
```
|
||||
|
||||
| 选项 | 取值范围 | 默认值 |
|
||||
|------|----------|--------|
|
||||
| `--interface` | 必选 | 无 |
|
||||
| `--rate-kbps` | 8 到 100000000 | 1024 |
|
||||
| `--queue-limit` | 1 到 65535 个报文 | 256 |
|
||||
| `--duration` | 1 到 86400 秒 | 10 |
|
||||
| `--verbose` | 开关 | 关闭 |
|
||||
|
||||
如果接口已有 root qdisc，工具会拒绝并输出:
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## 验证过的测试输出
|
||||
|
||||
仓库包含一个确定性集成测试。以下输出在 KVM 客户机（内核 `7.0.0-rc2+`）中于宿主机编译后采集:
|
||||
|
||||
```console
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8
|
||||
TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, and normal/signal cleanup succeeded
|
||||
```
|
||||
|
||||
这是功能性冒烟测试，不是基准性能数据。测试向 64 Kbit/s、队列容量 8 的 pacer 突发 40 个 1024 字节的 EtherType `0x88B5` 原始帧。31 个在队列满时被丢弃，9 个被整形后交付。不要将这些数字推广到其他场景。
|
||||
|
||||
在一个可随意使用的 Linux 6.16+ 系统上自行运行:
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
该测试会创建并删除 `epac_tx`/`epac_rx` veth 对，先安装一个 `pfifo` 以验证冲突拒绝行为，然后发送突发流量。它断言有界丢包、字节和报文计数正确、整形有效、正常退出清理正常、SIGTERM 清理正常，最后在 `finally` 块中删除 veth 对。
|
||||
|
||||
## 工作原理
|
||||
|
||||
本工具由两部分组成: 实现 `struct Qdisc_ops` 的 BPF 对象，以及管理 qdisc 生命周期的用户态加载器。
|
||||
|
||||
### BPF qdisc: 入队
|
||||
|
||||
当报文到达 qdisc 时，入队回调检查队列长度是否超过 `sch->limit`。如果队列已满，报文被丢弃并累加 `policy_dropped`。否则:
|
||||
|
||||
1. 通过 `bpf_obj_new` 分配一个 `packet_node`。
|
||||
2. 通过 `bpf_kptr_xchg` 将 skb 转移到节点中。
|
||||
3. 计算报文的允许发送时间戳: `next_departure_ns + (packet_len * 8 * 1,000,000 / rate_kbps)` 纳秒。
|
||||
4. 将节点推入 BPF 链表，更新 qlen 和 backlog。
|
||||
|
||||
BPF 源码（`egress_pacer.bpf.c`）:
|
||||
|
||||
```c
|
||||
SEC("struct_ops/egress_pacer_enqueue")
|
||||
int BPF_PROG(egress_pacer_enqueue, struct sk_buff *skb, struct Qdisc *sch,
|
||||
struct bpf_sk_buff_ptr *to_free)
|
||||
{
|
||||
struct packet_node *packet;
|
||||
__u64 eligible_ns, interval_ns, now;
|
||||
__u32 packet_len;
|
||||
|
||||
packet_len = qdisc_packet_len(skb);
|
||||
if (sch->q.qlen >= sch->limit)
|
||||
goto drop;
|
||||
|
||||
packet = bpf_obj_new(typeof(*packet));
|
||||
if (!packet)
|
||||
goto drop;
|
||||
|
||||
now = bpf_ktime_get_ns();
|
||||
interval_ns = (__u64)packet_len * 8 * 1000000ULL / rate_kbps;
|
||||
if (!interval_ns)
|
||||
interval_ns = 1;
|
||||
packet->packet_len = packet_len;
|
||||
skb = bpf_kptr_xchg(&packet->skb, skb);
|
||||
if (skb)
|
||||
bpf_qdisc_skb_drop(skb, to_free);
|
||||
|
||||
bpf_spin_lock(&queue_lock);
|
||||
eligible_ns = next_departure_ns > now ? next_departure_ns : now;
|
||||
next_departure_ns = eligible_ns + interval_ns;
|
||||
packet->eligible_ns = eligible_ns;
|
||||
bpf_list_push_back(&packet_queue, &packet->node);
|
||||
sch->q.qlen++;
|
||||
sch->qstats.backlog += packet_len;
|
||||
__sync_fetch_and_add(&stats.enqueued, 1);
|
||||
if (sch->q.qlen > stats.max_qlen)
|
||||
stats.max_qlen = sch->q.qlen;
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
return NET_XMIT_SUCCESS;
|
||||
|
||||
drop:
|
||||
bpf_qdisc_skb_drop(skb, to_free);
|
||||
__sync_fetch_and_add(&stats.policy_dropped, 1);
|
||||
return NET_XMIT_DROP;
|
||||
}
|
||||
```
|
||||
|
||||
间隔公式意味着空闲后的第一个报文可以立即发出（因为 `next_departure_ns` 已经过去，`eligible_ns = now`）。后续报文按配置速率和序列化报文长度间隔发送。
|
||||
|
||||
### BPF qdisc: 出队
|
||||
|
||||
出队回调弹出链表头部。如果当前时间早于报文的 `eligible_ns`，报文被放回链表头部，并调度 qdisc watchdog:
|
||||
|
||||
```c
|
||||
SEC("struct_ops/egress_pacer_dequeue")
|
||||
struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
{
|
||||
struct bpf_list_node *node;
|
||||
struct packet_node *packet;
|
||||
struct sk_buff *skb = NULL;
|
||||
__u64 expire, now;
|
||||
__u32 packet_len;
|
||||
|
||||
bpf_spin_lock(&queue_lock);
|
||||
node = bpf_list_pop_front(&packet_queue);
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
if (!node)
|
||||
return NULL;
|
||||
|
||||
packet = container_of(node, struct packet_node, node);
|
||||
now = bpf_ktime_get_ns();
|
||||
if (now < packet->eligible_ns) {
|
||||
expire = packet->eligible_ns;
|
||||
bpf_spin_lock(&queue_lock);
|
||||
bpf_list_push_front(&packet_queue, &packet->node);
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
bpf_qdisc_watchdog_schedule(sch, expire, 0);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
packet_len = packet->packet_len;
|
||||
skb = bpf_kptr_xchg(&packet->skb, skb);
|
||||
bpf_spin_lock(&queue_lock);
|
||||
sch->q.qlen--;
|
||||
sch->qstats.backlog -= packet_len;
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
bpf_obj_drop(packet);
|
||||
|
||||
if (!skb)
|
||||
return NULL;
|
||||
|
||||
bpf_qdisc_bstats_update(sch, skb);
|
||||
__sync_fetch_and_add(&stats.dequeued, 1);
|
||||
__sync_fetch_and_add(&stats.bytes_dequeued, packet_len);
|
||||
return skb;
|
||||
}
|
||||
```
|
||||
|
||||
watchdog（`bpf_qdisc_watchdog_schedule`）告诉内核在 eligible 时间再次调用 dequeue。这就是整形机制: 内核不轮询，而是休眠到下一个报文到期。
|
||||
|
||||
### BPF qdisc: reset
|
||||
|
||||
qdisc 被移除时，`reset` 回调排空所有残留的 skb，释放 `packet_node` 对象，将 qlen/backlog 归零。在此阶段释放的报文计入 `cleanup_dropped`，区别于正常运行期间的策略丢弃。
|
||||
|
||||
### 用户态生命周期
|
||||
|
||||
加载器（`egress_pacer.c`）按以下流程执行:
|
||||
|
||||
1. 解析参数，将接口名转换为 ifindex。
|
||||
2. 打开 BPF skeleton，设置 `rodata->rate_kbps` 和 `rodata->queue_limit`。
|
||||
3. 加载 BPF 对象（验证器在此运行）。
|
||||
4. attach struct_ops map，将 `bpf_pacer` 注册为可用的 qdisc 类型。
|
||||
5. 通过 `bpf_tc_hook_create`（带 `BPF_TC_QDISC` 和 `TC_H_ROOT`）在接口 root 创建 qdisc。若接口已有 root qdisc，返回 `-EEXIST`，工具输出拒绝信息并退出。
|
||||
6. 打印 `READY`，以 100 ms 间隔休眠直到持续时间到期或信号到达。
|
||||
7. 通过 `bpf_tc_hook_destroy` 销毁 qdisc，读取最终 BSS 计数器，打印 `SUMMARY`，销毁 skeleton。
|
||||
|
||||
信号处理器设置标志位来中断休眠循环。SIGINT 和 SIGTERM 都是经过测试的清理路径。
|
||||
|
||||
### 统计字段
|
||||
|
||||
BSS 中的 `pacer_stats` 结构体包含:
|
||||
|
||||
| 字段 | 含义 |
|
||||
|------|------|
|
||||
| `enqueued` | 成功入队的报文数 |
|
||||
| `dequeued` | 已发送的报文数 |
|
||||
| `policy_dropped` | 队列满时丢弃的报文数 |
|
||||
| `cleanup_dropped` | reset 阶段排空的报文数 |
|
||||
| `bytes_dequeued` | 已发送报文的总字节数 |
|
||||
| `max_qlen` | 队列峰值占用 |
|
||||
|
||||
加载器在销毁 qdisc 后从 `skel->bss->stats` 读取这些字段。
|
||||
|
||||
## 运行要求
|
||||
|
||||
- Linux 内核 6.16 及以上。BPF qdisc 支持通过[此提交](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)合入。
|
||||
- libbpf 1.6.0 及以上用于 BPF qdisc TC hook 支持（[libbpf 提交](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)）。本仓库集成了 bpftool v7.7.0 和 libbpf v1.7.0。
|
||||
- 架构: x86_64（已测试）。
|
||||
- 需要 root 权限进行 BPF 加载、qdisc 创建和原始测试套接字操作。
|
||||
- 需要开启 BTF 和 BPF JIT。
|
||||
- 内核配置: `CONFIG_BPF=y`、`CONFIG_BPF_SYSCALL=y`、`CONFIG_BPF_JIT=y`、`CONFIG_DEBUG_INFO_BTF=y`、`CONFIG_NET_SCHED=y`、`CONFIG_NET_SCH_BPF=y`。
|
||||
- 需要受控网络接口，veth 对即可满足。
|
||||
|
||||
## 限制
|
||||
|
||||
- 单一聚合 FIFO。没有公平调度、每流隔离、每 cgroup 策略、分类、优先级、ECN、突发预算或拥塞控制集成。
|
||||
- 队列限制以报文数计量，速率计算使用 `qdisc_skb_cb(skb)->pkt_len`。
|
||||
- 空闲后的第一个报文可以立即发出。
|
||||
- 速率和队列限制在一次调用中固定不变。
|
||||
- 由于全局队列和 BSS 状态，每个加载器进程只能运行一个 root qdisc 实例。
|
||||
- 拒绝所有已存在的 root qdisc，不检查、不保留、不堆叠、不恢复。
|
||||
- 有限时间的 CLI 工具，不是守护进程、控制器、指标导出器或持久策略管理器。
|
||||
- 正常退出、SIGINT 和 SIGTERM 是经过测试的清理路径。SIGKILL、宿主机崩溃或进程异常终止不保证清理 qdisc。
|
||||
- KVM 测试验证行为正确性，不是吞吐量或精度基准测试，不考虑驱动/硬件卸载、GSO/TSO 或真实网卡时序。
|
||||
|
||||
## 恢复
|
||||
|
||||
异常退出后，验证 qdisc 状态:
|
||||
|
||||
```bash
|
||||
tc qdisc show dev IFACE
|
||||
```
|
||||
|
||||
如果 `bpf_pacer` 仍存在于受控接口上，移除它:
|
||||
|
||||
```bash
|
||||
sudo tc qdisc del dev IFACE root
|
||||
```
|
||||
|
||||
此操作会改变接口的流量调度。只在你控制的接口上执行。
|
||||
|
||||
## 参考资料
|
||||
|
||||
- [Linux BPF qdisc 合入提交](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)
|
||||
- [Qdisc watchdog 后续提交](https://github.com/torvalds/linux/commit/7a2dafd33c4506e4cb4e63a422530c1cd5a35481)
|
||||
- [libbpf qdisc TC hook 支持](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)
|
||||
- [上游 BPF FIFO qdisc 自测程序](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c)
|
||||
- [上游 BPF qdisc 测试运行器](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c)
|
||||
- [libbpf 1.7.0 发布](https://github.com/libbpf/libbpf/releases/tag/v1.7.0)
|
||||
|
||||
## 更多内容
|
||||
|
||||
本示例是 eBPF 开发者教程的一部分。更多示例和完整教程请访问:
|
||||
|
||||
- <https://github.com/eunomia-bpf/bpf-developer-tutorial>
|
||||
- <https://eunomia.dev/tutorials/>
|
||||
@@ -68,6 +68,7 @@ Networking:
|
||||
- [lesson 42-xdp-loadbalancer](42-xdp-loadbalancer/README.md) XDP Load Balancer
|
||||
- [lesson 46-xdp-test](46-xdp-test/README.md) Building a High-Performance XDP Packet Generator
|
||||
- [lesson 50-tcx](50-tcx/README.md) Composable Traffic Control with TCX Links
|
||||
- [lesson 53-egress-pacer](53-egress-pacer/README.md) Pace Egress Traffic with a BPF Qdisc
|
||||
|
||||
|
||||
Tracing:
|
||||
|
||||
@@ -68,6 +68,7 @@ GPU:
|
||||
- [lesson 42-xdp-loadbalancer](42-xdp-loadbalancer/README.zh.md) eBPF 开发者教程： 简单的 XDP 负载均衡器
|
||||
- [lesson 46-xdp-test](46-xdp-test/README.zh.md) eBPF 实例教程：构建高性能 XDP 数据包生成器
|
||||
- [lesson 50-tcx](50-tcx/README.zh.md) eBPF 入门实践教程第五十篇：使用 TCX Link 实现可组合的流量控制
|
||||
- [lesson 53-egress-pacer](53-egress-pacer/README.zh.md) 使用 BPF Qdisc 对出口流量进行限速
|
||||
|
||||
|
||||
追踪:
|
||||
|
||||
@@ -57,6 +57,7 @@ This table is generated from each tutorial's `.config` metadata. `Minimum kernel
|
||||
| [eBPF Tutorial: Energy Monitoring for Process-Level Power Analysis](48-energy/README.md) | 5.8 | Required feature | x86_64, arm64 | Required | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_EVENTS=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_TRACEPOINTS=y | None | Required | CI build |
|
||||
| [eBPF Tutorial: Fixing Broken HID Devices Without Kernel Patches](49-hid/README.md) | 6.3 | Tutorial docs | x86_64, arm64 | Required | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_HID_BPF=y, CONFIG_UHID=y\|m | None | Required | CI build |
|
||||
| [eBPF Tutorial by Example 50: Composable Traffic Control with TCX Links](50-tcx/README.md) | 6.6 | Tutorial docs | x86_64, arm64 | Optional | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_NET_XGRESS=y | Network interface | Required | CI build |
|
||||
| [eBPF Tutorial by Example: Pace Egress Traffic with a BPF Qdisc](53-egress-pacer/README.md) | 6.16 | Tutorial docs | x86_64 | Required | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_NET_SCHED=y, CONFIG_NET_SCH_BPF=y | Network interface | Required | CI build |
|
||||
| [eBPF Tutorial: cgroup-based Policy Control](cgroup/README.md) | 5.8 | Required feature | x86_64, arm64 | Required | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_CGROUPS=y, CONFIG_CGROUP_BPF=y, CONFIG_CGROUP_DEVICE=y, CONFIG_DEBUG_INFO_BTF=y | None | Required | CI build |
|
||||
| [eBPF Tutorial by Example: BPF Arena for Zero-Copy Shared Memory](features/bpf_arena/README.md) | 6.9 | Tutorial docs | x86_64, arm64 | Required | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y | None | Required | CI build |
|
||||
| [eBPF Tutorial: BPF Iterators for Kernel Data Export](features/bpf_iters/README.md) | 5.8 | Tutorial docs | x86_64, arm64 | Required | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_DEBUG_INFO_BTF=y | None | Required | CI build |
|
||||
|
||||
@@ -57,6 +57,7 @@
|
||||
| [eBPF 教程：进程级能源监控与功耗分析](48-energy/README.zh.md) | 5.8 | 必需特性 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_EVENTS=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_TRACEPOINTS=y | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 教程：无需内核补丁修复故障的 HID 设备](49-hid/README.zh.md) | 6.3 | 教程文档 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_HID_BPF=y, CONFIG_UHID=y\|m | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 入门实践教程第五十篇：使用 TCX Link 实现可组合的流量控制](50-tcx/README.zh.md) | 6.6 | 教程文档 | x86_64, arm64 | 可选 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_NET_XGRESS=y | 网络接口 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实践教程: 使用 BPF Qdisc 对出口流量进行限速](53-egress-pacer/README.zh.md) | 6.16 | 教程文档 | x86_64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_NET_SCHED=y, CONFIG_NET_SCH_BPF=y | 网络接口 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实例教程：基于 cgroup 的策略控制](cgroup/README.zh.md) | 5.8 | 必需特性 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_CGROUPS=y, CONFIG_CGROUP_BPF=y, CONFIG_CGROUP_DEVICE=y, CONFIG_DEBUG_INFO_BTF=y | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实例教程：BPF Arena 零拷贝共享内存](features/bpf_arena/README.zh.md) | 6.9 | 教程文档 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 教程：BPF 迭代器用于内核数据导出](features/bpf_iters/README.zh.md) | 5.8 | 教程文档 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_DEBUG_INFO_BTF=y | 无 | 需要 | CI 构建测试 |
|
||||
|
||||
Reference in New Issue
Block a user