mirror of
https://github.com/eunomia-bpf/bpf-developer-tutorial.git
synced 2026-10-29 03:34:44 +08:00
Rewrite the egress pacer tutorial
This commit is contained in:
LinuxDev9002
@@ -76,7 +76,7 @@ GPU:
|
||||
- [lesson 42-xdp-loadbalancer](src/42-xdp-loadbalancer/README.zh.md) eBPF 开发者教程： 简单的 XDP 负载均衡器
|
||||
- [lesson 46-xdp-test](src/46-xdp-test/README.zh.md) eBPF 实例教程：构建高性能 XDP 数据包生成器
|
||||
- [lesson 50-tcx](src/50-tcx/README.zh.md) eBPF 入门实践教程第五十篇：使用 TCX Link 实现可组合的流量控制
|
||||
- [lesson 53-egress-pacer](src/53-egress-pacer/README.zh.md) eBPF 实践教程：使用 BPF Qdisc 对出口流量进行限速
|
||||
- [lesson 53-egress-pacer](src/53-egress-pacer/README.zh.md) eBPF 教程：用 BPF Qdisc 控制出口流量
|
||||
|
||||
|
||||
追踪:
|
||||
|
||||
@@ -1,103 +1,151 @@
|
||||
# eBPF Tutorial by Example: Pace Egress Traffic with a BPF Qdisc
|
||||
|
||||
You have a lab, a test bench, or a service running behind a dedicated veth or TAP interface. Something sends a burst, the queue grows unbounded, latency spikes, and you have no visibility into the drop. You want a duration-bounded egress pacer that reports queue and byte counters, then removes itself on normal exit or when its SIGINT or SIGTERM handler runs. An uncatchable signal still requires the recovery path below.
|
||||
A lab service sends a sudden burst through a veth interface. The queue grows, latency jumps, and packets disappear without useful accounting. You want to slow that burst to a known rate, keep the queue bounded, and see whether packets left the interface or were dropped.
|
||||
|
||||
This tutorial builds a single-purpose BPF qdisc that does exactly that. It uses the Linux 6.16 `struct_ops`-based BPF qdisc infrastructure to register a bounded FIFO, pace packets at an aggregate rate you choose, and expose enqueue/dequeue/drop statistics in user space.
|
||||
This tutorial builds `egress_pacer`, a small BPF qdisc for that job. It registers a FIFO qdisc with Linux 6.16 BPF `struct_ops`, assigns each packet an eligible departure time, and reports enqueue, dequeue, byte, and drop counters when the run ends. The loader also refuses to overwrite an existing root qdisc.
|
||||
|
||||
This is not production-ready for a general-purpose NIC. It is intentionally designed for a controlled interface (veth, TAP, IFB) where you make an explicit lifecycle decision. The loader refuses to replace any existing root qdisc.
|
||||
The example targets a controlled veth, TAP, or IFB interface. It is not a general-purpose scheduler for a production NIC. By the end, you will understand how a BPF qdisc owns an skb, how its watchdog avoids busy waiting, and how user space installs and removes the qdisc safely enough for a bounded lab run.
|
||||
|
||||
[Browse the complete source for this lesson](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer).
|
||||
The complete lesson is available in the [`53-egress-pacer` source directory](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer).
|
||||
|
||||
## Quick start
|
||||
## Why put the pacer in a qdisc?
|
||||
|
||||
Build without root:
|
||||
Linux already sends outgoing packets through a queueing discipline, or qdisc. An ordinary TC classifier can inspect a packet and return an action, but a separate kernel qdisc still owns the queue. The [TC classifier tutorial](../20-tc/README.md) follows that model.
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
make clean
|
||||
make -j2
|
||||
```
|
||||
This example takes a different path. Its BPF `struct_ops` object implements `struct Qdisc_ops` and registers `bpf_pacer` as the root qdisc itself. The BPF program decides whether a packet enters the queue, when the head packet may leave, and what happens to queued skbs during reset.
|
||||
|
||||
For a self-contained first run, let the integration fixture create and remove a disposable veth pair. This step needs root:
|
||||
That control comes with a lifecycle cost. Replacing a root qdisc can change live traffic, so the loader treats any existing root qdisc as a conflict. It runs for a fixed duration, removes the qdisc after normal completion or a handled SIGINT or SIGTERM, and leaves an explicit recovery command for abnormal termination. An uncatchable SIGKILL can leave the qdisc installed.
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
## Follow one packet through the tool
|
||||
|
||||
To run the loader manually on a dedicated interface that you already created and control, inspect it first with `tc qdisc show dev veth-service`. The loader refuses installation if the kernel reports an existing root qdisc:
|
||||
The kernel and user-space parts cooperate in this order:
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
```
|
||||
1. The loader resolves a controlled interface, writes the rate and packet limit into BPF read-only data, and loads the object.
|
||||
2. Attaching the `struct_ops` map registers `bpf_pacer`. The loader then creates it at `TC_H_ROOT` with the libbpf qdisc TC hook.
|
||||
3. The qdisc `init` callback copies `queue_limit` into `sch->limit` before traffic arrives.
|
||||
4. `egress_pacer_enqueue` either drops a packet at the limit or moves its skb into a BPF-owned queue node. It converts packet length and the configured rate into an eligible departure time.
|
||||
5. `egress_pacer_dequeue` returns an eligible skb. If the head packet is early, the callback puts it back and asks the qdisc watchdog to retry at the right time.
|
||||
6. When user space removes the qdisc, `egress_pacer_reset` drains any retained skbs. The loader then reads the final BSS counters and prints `SUMMARY`.
|
||||
|
||||
The tool prints a `READY` line when the qdisc is active, then a `SUMMARY` line with final counters when it exits. The following block shows the output shape. Its ellipses are placeholders, not captured values:
|
||||
The design is deliberately small. One loader owns one root qdisc, one FIFO, and one global set of counters.
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
## Complete source code
|
||||
|
||||
Usage:
|
||||
|
||||
```text
|
||||
Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKETS] [--duration SEC] [--verbose]
|
||||
```
|
||||
|
||||
| Option | Range | Default |
|
||||
|--------|-------|---------|
|
||||
| `--interface` | required | none |
|
||||
| `--rate-kbps` | 8 to 100000000 | 1024 |
|
||||
| `--queue-limit` | 1 to 65535 packets | 256 |
|
||||
| `--duration` | 1 to 86400 seconds | 10 |
|
||||
| `--verbose` | flag | off |
|
||||
|
||||
If the interface already has a root qdisc, the tool exits with:
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## Verified test output
|
||||
|
||||
The repository includes a deterministic integration fixture. This output was captured in a KVM guest running kernel `7.0.0-rc2+` after a clean host build:
|
||||
|
||||
```console
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8
|
||||
TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded
|
||||
```
|
||||
|
||||
This is a functional smoke test, not benchmark-quality performance evidence. 40 raw 1024-byte EtherType `0x88B5` frames are sent in a burst through a 64 Kbit/s pacer with an 8-packet queue. 31 are counted as `policy_dropped` when the queue fills in this run, and 9 are paced and delivered. Do not generalize these numbers beyond this fixture.
|
||||
|
||||
Run the fixture yourself on a suitable disposable Linux 6.16+ system:
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
The fixture creates and deletes `epac_tx`/`epac_rx` veth interfaces. It installs a conflicting `pfifo` first to prove refusal, then sends the burst and checks bounded drops, byte/packet accounting, pacing, and normal cleanup.
|
||||
|
||||
A second burst leaves packets queued before SIGTERM. The test requires a nonzero `cleanup_dropped` count and verifies `enqueued = dequeued + cleanup_dropped`. Finally, it proves that SIGKILL leaves `bpf_pacer` installed on the tested guest, removes the qdisc explicitly with `tc`, verifies recovery, and deletes the veth pair in a `finally` block.
|
||||
|
||||
## How it works
|
||||
|
||||
The tool has two parts: a BPF object that implements `struct Qdisc_ops`, and a user-space loader that manages the qdisc lifecycle.
|
||||
|
||||
Unlike the [ordinary TC classifier/action example](../20-tc/README.md), where a classifier returns an action while the qdisc remains a separate kernel component, this struct_ops program registers and installs `bpf_pacer` as the root qdisc itself. Its `init` callback copies the configured `queue_limit` into `sch->limit` before packets arrive.
|
||||
|
||||
### BPF qdisc: enqueue
|
||||
|
||||
When a packet arrives at the qdisc, the enqueue callback checks the queue length against `sch->limit`. If the queue is full, the packet is dropped and `policy_dropped` increments. A `bpf_obj_new` allocation failure follows the same drop-and-count path. Otherwise:
|
||||
|
||||
1. Allocate a `packet_node` with `bpf_obj_new`.
|
||||
2. Transfer the skb into the node via `bpf_kptr_xchg`.
|
||||
3. Compute the packet's eligible departure timestamp: `eligible_ns = max(next_departure_ns, now)`. Then advance `next_departure_ns` by the packet's serialization interval `(packet_len * 8 * 1,000,000 / rate_kbps)` nanoseconds.
|
||||
4. Push the node onto a BPF linked list and update qlen/backlog.
|
||||
|
||||
From the BPF source (`egress_pacer.bpf.c`):
|
||||
The shared header defines the statistics contract used by BPF and user space.
|
||||
|
||||
<!-- BEGIN FULL SOURCE: src/53-egress-pacer/egress_pacer.h -->
|
||||
```c
|
||||
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
|
||||
#ifndef __EGRESS_PACER_H
|
||||
#define __EGRESS_PACER_H
|
||||
|
||||
struct pacer_stats {
|
||||
unsigned long long enqueued;
|
||||
unsigned long long dequeued;
|
||||
unsigned long long policy_dropped;
|
||||
unsigned long long cleanup_dropped;
|
||||
unsigned long long bytes_dequeued;
|
||||
unsigned long long max_qlen;
|
||||
};
|
||||
|
||||
#endif /* __EGRESS_PACER_H */
|
||||
```
|
||||
<!-- END FULL SOURCE -->
|
||||
|
||||
The repository's generated headers predate a few BPF qdisc definitions. This local compatibility header supplies those declarations without changing the vendored global headers.
|
||||
|
||||
<!-- BEGIN FULL SOURCE: src/53-egress-pacer/bpf_experimental.h -->
|
||||
```c
|
||||
/* SPDX-License-Identifier: GPL-2.0 */
|
||||
#ifndef __EGRESS_PACER_BPF_EXPERIMENTAL_H
|
||||
#define __EGRESS_PACER_BPF_EXPERIMENTAL_H
|
||||
|
||||
#include <bpf/bpf_core_read.h>
|
||||
|
||||
#define __contains(name, node) \
|
||||
__attribute__((btf_decl_tag("contains:" #name ":" #node)))
|
||||
|
||||
extern void *bpf_obj_new_impl(__u64 local_type_id, void *meta) __ksym;
|
||||
#define bpf_obj_new(type) \
|
||||
((type *)bpf_obj_new_impl(bpf_core_type_id_local(type), NULL))
|
||||
|
||||
extern void bpf_obj_drop_impl(void *kptr, void *meta) __ksym;
|
||||
#define bpf_obj_drop(kptr) bpf_obj_drop_impl(kptr, NULL)
|
||||
|
||||
extern int bpf_list_push_front_impl(struct bpf_list_head *head,
|
||||
struct bpf_list_node *node,
|
||||
void *meta, __u64 off) __ksym;
|
||||
#define bpf_list_push_front(head, node) \
|
||||
bpf_list_push_front_impl(head, node, NULL, 0)
|
||||
|
||||
extern int bpf_list_push_back_impl(struct bpf_list_head *head,
|
||||
struct bpf_list_node *node,
|
||||
void *meta, __u64 off) __ksym;
|
||||
#define bpf_list_push_back(head, node) \
|
||||
bpf_list_push_back_impl(head, node, NULL, 0)
|
||||
|
||||
extern struct bpf_list_node *
|
||||
bpf_list_pop_front(struct bpf_list_head *head) __ksym;
|
||||
|
||||
#endif /* __EGRESS_PACER_BPF_EXPERIMENTAL_H */
|
||||
```
|
||||
<!-- END FULL SOURCE -->
|
||||
|
||||
The kernel-mode program implements the qdisc operations, packet queue, pacing clock, and counters.
|
||||
|
||||
<!-- BEGIN FULL SOURCE: src/53-egress-pacer/egress_pacer.bpf.c -->
|
||||
```c
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
#include "vmlinux.h"
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include <bpf/bpf_tracing.h>
|
||||
#include "bpf_experimental.h"
|
||||
#include "egress_pacer.h"
|
||||
|
||||
char LICENSE[] SEC("license") = "GPL";
|
||||
|
||||
#define NET_XMIT_SUCCESS 0x00
|
||||
#define NET_XMIT_DROP 0x01
|
||||
#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8)))
|
||||
|
||||
struct bpf_sk_buff_ptr {
|
||||
struct sk_buff *skb;
|
||||
};
|
||||
|
||||
extern void bpf_qdisc_skb_drop(struct sk_buff *skb,
|
||||
struct bpf_sk_buff_ptr *to_free) __ksym;
|
||||
extern void bpf_qdisc_watchdog_schedule(struct Qdisc *sch, __u64 expire,
|
||||
__u64 delta_ns) __ksym;
|
||||
extern void bpf_qdisc_bstats_update(struct Qdisc *sch,
|
||||
const struct sk_buff *skb) __ksym;
|
||||
extern void bpf_kfree_skb(struct sk_buff *skb) __ksym;
|
||||
|
||||
const volatile __u64 rate_kbps = 1024;
|
||||
const volatile __u32 queue_limit = 256;
|
||||
|
||||
struct pacer_stats stats;
|
||||
|
||||
struct packet_node {
|
||||
__u64 eligible_ns;
|
||||
__u32 packet_len;
|
||||
struct sk_buff __kptr *skb;
|
||||
struct bpf_list_node node;
|
||||
};
|
||||
|
||||
private(A) struct bpf_spin_lock queue_lock;
|
||||
private(A) struct bpf_list_head packet_queue __contains(packet_node, node);
|
||||
private(A) __u64 next_departure_ns;
|
||||
|
||||
static struct qdisc_skb_cb *qdisc_skb_cb(const struct sk_buff *skb)
|
||||
{
|
||||
return (struct qdisc_skb_cb *)skb->cb;
|
||||
}
|
||||
|
||||
static __u32 qdisc_packet_len(const struct sk_buff *skb)
|
||||
{
|
||||
return qdisc_skb_cb(skb)->pkt_len;
|
||||
}
|
||||
|
||||
SEC("struct_ops/egress_pacer_enqueue")
|
||||
int BPF_PROG(egress_pacer_enqueue, struct sk_buff *skb, struct Qdisc *sch,
|
||||
struct bpf_sk_buff_ptr *to_free)
|
||||
@@ -141,15 +189,7 @@ drop:
|
||||
__sync_fetch_and_add(&stats.policy_dropped, 1);
|
||||
return NET_XMIT_DROP;
|
||||
}
|
||||
```
|
||||
|
||||
The interval formula means the first packet after an idle period departs immediately (because `next_departure_ns` will be in the past, so `eligible_ns = now`). Subsequent packets are spaced by serialized packet length at the configured rate.
|
||||
|
||||
### BPF qdisc: dequeue
|
||||
|
||||
The dequeue callback pops the head of the list. If the current time is before the packet's `eligible_ns`, the packet goes back to the front of the list and the qdisc watchdog is scheduled:
|
||||
|
||||
```c
|
||||
SEC("struct_ops/egress_pacer_dequeue")
|
||||
struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
{
|
||||
@@ -162,8 +202,9 @@ struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
bpf_spin_lock(&queue_lock);
|
||||
node = bpf_list_pop_front(&packet_queue);
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
if (!node)
|
||||
if (!node) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
packet = container_of(node, struct packet_node, node);
|
||||
now = bpf_ktime_get_ns();
|
||||
@@ -192,82 +233,533 @@ struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
__sync_fetch_and_add(&stats.bytes_dequeued, packet_len);
|
||||
return skb;
|
||||
}
|
||||
|
||||
SEC("struct_ops/egress_pacer_init")
|
||||
int BPF_PROG(egress_pacer_init, struct Qdisc *sch, struct nlattr *opt,
|
||||
struct netlink_ext_ack *extack)
|
||||
{
|
||||
(void)opt;
|
||||
(void)extack;
|
||||
sch->limit = queue_limit;
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("struct_ops/egress_pacer_reset")
|
||||
void BPF_PROG(egress_pacer_reset, struct Qdisc *sch)
|
||||
{
|
||||
int queued = sch->q.qlen;
|
||||
int i;
|
||||
|
||||
bpf_for(i, 0, queued) {
|
||||
struct bpf_list_node *node;
|
||||
struct packet_node *packet;
|
||||
struct sk_buff *skb = NULL;
|
||||
|
||||
bpf_spin_lock(&queue_lock);
|
||||
node = bpf_list_pop_front(&packet_queue);
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
if (!node)
|
||||
break;
|
||||
|
||||
packet = container_of(node, struct packet_node, node);
|
||||
skb = bpf_kptr_xchg(&packet->skb, skb);
|
||||
if (skb) {
|
||||
bpf_kfree_skb(skb);
|
||||
__sync_fetch_and_add(&stats.cleanup_dropped, 1);
|
||||
}
|
||||
bpf_obj_drop(packet);
|
||||
}
|
||||
|
||||
bpf_spin_lock(&queue_lock);
|
||||
next_departure_ns = 0;
|
||||
sch->q.qlen = 0;
|
||||
sch->qstats.backlog = 0;
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
}
|
||||
|
||||
SEC("struct_ops/egress_pacer_destroy")
|
||||
void BPF_PROG(egress_pacer_destroy, struct Qdisc *sch)
|
||||
{
|
||||
(void)sch;
|
||||
}
|
||||
|
||||
SEC(".struct_ops")
|
||||
struct Qdisc_ops pacer = {
|
||||
.enqueue = (void *)egress_pacer_enqueue,
|
||||
.dequeue = (void *)egress_pacer_dequeue,
|
||||
.init = (void *)egress_pacer_init,
|
||||
.reset = (void *)egress_pacer_reset,
|
||||
.destroy = (void *)egress_pacer_destroy,
|
||||
.id = "bpf_pacer",
|
||||
};
|
||||
```
|
||||
<!-- END FULL SOURCE -->
|
||||
|
||||
The user-space loader validates the command line, registers the BPF qdisc, installs it on one interface, waits for the requested duration, and removes it.
|
||||
|
||||
<!-- BEGIN FULL SOURCE: src/53-egress-pacer/egress_pacer.c -->
|
||||
```c
|
||||
// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
|
||||
#include <errno.h>
|
||||
#include <getopt.h>
|
||||
#include <linux/pkt_sched.h>
|
||||
#include <net/if.h>
|
||||
#include <signal.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
#include <bpf/libbpf.h>
|
||||
#include "egress_pacer.h"
|
||||
#include "egress_pacer.skel.h"
|
||||
|
||||
static volatile sig_atomic_t exiting;
|
||||
|
||||
static struct env {
|
||||
const char *interface;
|
||||
unsigned long long rate_kbps;
|
||||
unsigned int queue_limit;
|
||||
unsigned int duration;
|
||||
bool verbose;
|
||||
} env = {
|
||||
.rate_kbps = 1024,
|
||||
.queue_limit = 256,
|
||||
.duration = 10,
|
||||
};
|
||||
|
||||
static void handle_signal(int signal)
|
||||
{
|
||||
(void)signal;
|
||||
exiting = 1;
|
||||
}
|
||||
|
||||
static int libbpf_print_fn(enum libbpf_print_level level, const char *format,
|
||||
va_list args)
|
||||
{
|
||||
if (level == LIBBPF_DEBUG && !env.verbose)
|
||||
return 0;
|
||||
return vfprintf(stderr, format, args);
|
||||
}
|
||||
|
||||
static void usage(const char *program)
|
||||
{
|
||||
fprintf(stderr,
|
||||
"Usage: %s --interface IFACE [--rate-kbps KBPS] "
|
||||
"[--queue-limit PACKETS] [--duration SEC] [--verbose]\n\n"
|
||||
"Temporarily pace one interface with a bounded BPF qdisc.\n\n"
|
||||
"Options:\n"
|
||||
" -i, --interface IFACE interface to control (required)\n"
|
||||
" -r, --rate-kbps KBPS egress rate, 8-100000000 "
|
||||
"(default: 1024)\n"
|
||||
" -q, --queue-limit PACKETS queue capacity, 1-65535 "
|
||||
"(default: 256)\n"
|
||||
" -d, --duration SEC control window, 1-86400 "
|
||||
"(default: 10)\n"
|
||||
" -v, --verbose print libbpf diagnostics\n"
|
||||
" -h, --help show this help\n",
|
||||
program);
|
||||
}
|
||||
|
||||
static int parse_u64(const char *value, unsigned long long maximum,
|
||||
unsigned long long *result)
|
||||
{
|
||||
char *end = NULL;
|
||||
unsigned long long parsed;
|
||||
|
||||
errno = 0;
|
||||
parsed = strtoull(value, &end, 10);
|
||||
if (errno || end == value || *end || parsed > maximum)
|
||||
return -EINVAL;
|
||||
*result = parsed;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int parse_rate(const char *value)
|
||||
{
|
||||
unsigned long long parsed;
|
||||
|
||||
if (parse_u64(value, 100000000, &parsed) || parsed < 8) {
|
||||
fprintf(stderr, "invalid rate in Kbit/s: %s\n", value);
|
||||
return -EINVAL;
|
||||
}
|
||||
env.rate_kbps = parsed;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int parse_queue_limit(const char *value)
|
||||
{
|
||||
unsigned long long parsed;
|
||||
|
||||
if (parse_u64(value, 65535, &parsed) || parsed == 0) {
|
||||
fprintf(stderr, "invalid queue limit: %s\n", value);
|
||||
return -EINVAL;
|
||||
}
|
||||
env.queue_limit = parsed;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int parse_duration(const char *value)
|
||||
{
|
||||
unsigned long long parsed;
|
||||
|
||||
if (parse_u64(value, 86400, &parsed) || parsed == 0) {
|
||||
fprintf(stderr, "invalid duration in seconds: %s\n", value);
|
||||
return -EINVAL;
|
||||
}
|
||||
env.duration = parsed;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int parse_option(int option, const char *program)
|
||||
{
|
||||
switch (option) {
|
||||
case 'i':
|
||||
env.interface = optarg;
|
||||
return 0;
|
||||
case 'r':
|
||||
return parse_rate(optarg);
|
||||
case 'q':
|
||||
return parse_queue_limit(optarg);
|
||||
case 'd':
|
||||
return parse_duration(optarg);
|
||||
case 'v':
|
||||
env.verbose = true;
|
||||
return 0;
|
||||
case 'h':
|
||||
usage(program);
|
||||
exit(0);
|
||||
default:
|
||||
return -EINVAL;
|
||||
}
|
||||
}
|
||||
|
||||
static int parse_args(int argc, char **argv)
|
||||
{
|
||||
static const struct option options[] = {
|
||||
{ "interface", required_argument, NULL, 'i' },
|
||||
{ "rate-kbps", required_argument, NULL, 'r' },
|
||||
{ "queue-limit", required_argument, NULL, 'q' },
|
||||
{ "duration", required_argument, NULL, 'd' },
|
||||
{ "verbose", no_argument, NULL, 'v' },
|
||||
{ "help", no_argument, NULL, 'h' },
|
||||
{},
|
||||
};
|
||||
int error, option;
|
||||
|
||||
while ((option = getopt_long(argc, argv, "i:r:q:d:vh", options, NULL)) != -1) {
|
||||
error = parse_option(option, argv[0]);
|
||||
if (error)
|
||||
return error;
|
||||
}
|
||||
|
||||
if (!env.interface) {
|
||||
fprintf(stderr, "--interface is required\n");
|
||||
return -EINVAL;
|
||||
}
|
||||
if (optind != argc)
|
||||
return -EINVAL;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static long long monotonic_milliseconds(void)
|
||||
{
|
||||
struct timespec timestamp;
|
||||
|
||||
if (clock_gettime(CLOCK_MONOTONIC, ×tamp))
|
||||
return -errno;
|
||||
return timestamp.tv_sec * 1000LL + timestamp.tv_nsec / 1000000;
|
||||
}
|
||||
|
||||
static int wait_for_duration(void)
|
||||
{
|
||||
struct timespec interval = { .tv_nsec = 100000000 };
|
||||
long long deadline, now;
|
||||
|
||||
now = monotonic_milliseconds();
|
||||
if (now < 0)
|
||||
return (int)now;
|
||||
deadline = now + env.duration * 1000LL;
|
||||
|
||||
while (!exiting) {
|
||||
now = monotonic_milliseconds();
|
||||
if (now < 0)
|
||||
return (int)now;
|
||||
if (now >= deadline)
|
||||
break;
|
||||
if (nanosleep(&interval, NULL) && errno != EINTR)
|
||||
return -errno;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv)
|
||||
{
|
||||
struct egress_pacer_bpf *skel = NULL;
|
||||
struct pacer_stats final_stats = {};
|
||||
struct bpf_tc_hook hook = {
|
||||
.sz = sizeof(hook),
|
||||
.attach_point = BPF_TC_QDISC,
|
||||
.parent = TC_H_ROOT,
|
||||
.handle = TC_H_MAKE(1 << 16, 0),
|
||||
.qdisc = "bpf_pacer",
|
||||
};
|
||||
bool qdisc_created = false;
|
||||
unsigned int ifindex;
|
||||
int cleanup_err;
|
||||
int err;
|
||||
|
||||
err = parse_args(argc, argv);
|
||||
if (err) {
|
||||
usage(argv[0]);
|
||||
return 1;
|
||||
}
|
||||
|
||||
ifindex = if_nametoindex(env.interface);
|
||||
if (!ifindex) {
|
||||
fprintf(stderr, "interface does not exist: %s\n", env.interface);
|
||||
return 1;
|
||||
}
|
||||
hook.ifindex = ifindex;
|
||||
|
||||
libbpf_set_print(libbpf_print_fn);
|
||||
signal(SIGINT, handle_signal);
|
||||
signal(SIGTERM, handle_signal);
|
||||
|
||||
skel = egress_pacer_bpf__open();
|
||||
if (!skel) {
|
||||
fprintf(stderr, "failed to open BPF skeleton\n");
|
||||
return 1;
|
||||
}
|
||||
skel->rodata->rate_kbps = env.rate_kbps;
|
||||
skel->rodata->queue_limit = env.queue_limit;
|
||||
|
||||
err = egress_pacer_bpf__load(skel);
|
||||
if (err) {
|
||||
fprintf(stderr,
|
||||
"failed to load BPF qdisc: %s\n"
|
||||
"This tool requires Linux 6.16+, CONFIG_NET_SCH_BPF, "
|
||||
"BTF, and BPF JIT.\n",
|
||||
strerror(-err));
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
err = egress_pacer_bpf__attach(skel);
|
||||
if (err) {
|
||||
fprintf(stderr, "failed to register bpf_pacer qdisc: %s\n",
|
||||
strerror(-err));
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
err = bpf_tc_hook_create(&hook);
|
||||
if (err) {
|
||||
if (err == -EEXIST)
|
||||
fprintf(stderr,
|
||||
"refusing to replace the existing root qdisc on %s\n",
|
||||
env.interface);
|
||||
else
|
||||
fprintf(stderr, "failed to attach bpf_pacer to %s: %s\n",
|
||||
env.interface, strerror(-err));
|
||||
goto cleanup;
|
||||
}
|
||||
qdisc_created = true;
|
||||
|
||||
printf("READY interface=%s rate_kbps=%llu queue_limit=%u duration=%u\n",
|
||||
env.interface, env.rate_kbps, env.queue_limit, env.duration);
|
||||
fflush(stdout);
|
||||
|
||||
err = wait_for_duration();
|
||||
|
||||
cleanup:
|
||||
if (qdisc_created) {
|
||||
cleanup_err = bpf_tc_hook_destroy(&hook);
|
||||
if (cleanup_err) {
|
||||
fprintf(stderr, "failed to remove bpf_pacer from %s: %s\n",
|
||||
env.interface, strerror(-cleanup_err));
|
||||
if (!err)
|
||||
err = cleanup_err;
|
||||
}
|
||||
}
|
||||
if (skel && skel->bss)
|
||||
final_stats = skel->bss->stats;
|
||||
if (qdisc_created) {
|
||||
printf("SUMMARY enqueued=%llu dequeued=%llu policy_dropped=%llu "
|
||||
"cleanup_dropped=%llu bytes_dequeued=%llu max_qlen=%llu\n",
|
||||
final_stats.enqueued, final_stats.dequeued,
|
||||
final_stats.policy_dropped, final_stats.cleanup_dropped,
|
||||
final_stats.bytes_dequeued, final_stats.max_qlen);
|
||||
}
|
||||
egress_pacer_bpf__destroy(skel);
|
||||
return err != 0;
|
||||
}
|
||||
```
|
||||
<!-- END FULL SOURCE -->
|
||||
|
||||
## How enqueue turns a rate into time
|
||||
|
||||
`egress_pacer_enqueue` first reads `qdisc_skb_cb(skb)->pkt_len`. The queue limit is measured in packets, while the rate calculation uses this qdisc packet length. A full queue goes directly to the drop path. A failed `bpf_obj_new` allocation uses the same path, so both cases increment `policy_dropped`.
|
||||
|
||||
For an accepted packet, `bpf_obj_new` allocates a `packet_node`, and `bpf_kptr_xchg` transfers the skb into its `__kptr` field. The node now carries the packet length and an `eligible_ns` timestamp. BPF list ownership and skb ownership move together, which lets reset release both objects later.
|
||||
|
||||
The pacing interval is the packet's serialization time at the requested rate:
|
||||
|
||||
```text
|
||||
interval_ns = packet_len * 8 * 1,000,000 / rate_kbps
|
||||
eligible_ns = max(next_departure_ns, now)
|
||||
next_departure_ns = eligible_ns + interval_ns
|
||||
```
|
||||
|
||||
The watchdog (`bpf_qdisc_watchdog_schedule`) asks the qdisc core to retry dequeue when the head packet becomes eligible. This provides pacing without busy-polling in the BPF program.
|
||||
The first packet after an idle period sees an old `next_departure_ns`, so it may leave immediately. Every later packet advances the clock by its own serialized length. This is an aggregate FIFO pacer, not a token bucket. It has no separate burst budget.
|
||||
|
||||
### BPF qdisc: reset
|
||||
The spin lock protects the list, `next_departure_ns`, qlen, and backlog updates. Successful insertion increments `enqueued`, and `max_qlen` records the largest queue seen during the run.
|
||||
|
||||
On qdisc removal, the `reset` callback drains all retained skbs, frees the `packet_node` objects, and zeroes qlen/backlog. Packets freed here are counted in `cleanup_dropped`, distinct from the policy drops during normal operation.
|
||||
## How dequeue waits without spinning
|
||||
|
||||
### User-space lifecycle
|
||||
`egress_pacer_dequeue` pops the first node and compares its timestamp with `bpf_ktime_get_ns()`. An eligible packet leaves the BPF node through another `bpf_kptr_xchg`. The callback reduces qlen and backlog, frees the empty node, updates the qdisc byte statistics, and increments `dequeued` and `bytes_dequeued`.
|
||||
|
||||
The loader (`egress_pacer.c`) follows this sequence:
|
||||
An early packet takes a different path. The callback pushes the node back to the front, calls `bpf_qdisc_watchdog_schedule(sch, expire, 0)`, and returns `NULL`. The qdisc core can retry when `eligible_ns` arrives, so the BPF program never loops while waiting for time to pass.
|
||||
|
||||
1. Parse arguments and resolve the interface name to an ifindex.
|
||||
2. Open the BPF skeleton and set `rodata->rate_kbps` and `rodata->queue_limit`.
|
||||
3. Load the BPF object (verifier runs here).
|
||||
4. Attach the struct_ops map, registering `bpf_pacer` as an available qdisc type.
|
||||
5. Create the qdisc at the interface root via `bpf_tc_hook_create` with `BPF_TC_QDISC` and `TC_H_ROOT`. If the interface already has a root qdisc, the call returns `-EEXIST` and the tool prints the refusal and exits.
|
||||
6. Print `READY`, then sleep in 100 ms increments until the duration expires or a signal arrives.
|
||||
7. Destroy the qdisc via `bpf_tc_hook_destroy`, read final BSS counters, print `SUMMARY`, and destroy the skeleton.
|
||||
## Reset keeps cleanup drops separate
|
||||
|
||||
SIGINT and SIGTERM share a handler that sets a flag and breaks the sleep loop. The fixture exercises duration-based normal exit and SIGTERM cleanup. It does not exercise SIGINT separately.
|
||||
Removing a qdisc can happen while packets are still queued. `egress_pacer_reset` walks the current qlen with `bpf_for`, removes every node, releases each skb with `bpf_kfree_skb`, and drops the node object. Packets released here increment `cleanup_dropped`, not `policy_dropped`.
|
||||
|
||||
### Statistics
|
||||
That distinction makes the final counters useful. `policy_dropped` describes admission failures while the pacer was running. `cleanup_dropped` describes packets discarded because the qdisc was removed with a backlog. Reset also clears `next_departure_ns`, qlen, and backlog.
|
||||
|
||||
The `pacer_stats` struct in BSS holds:
|
||||
The BSS `pacer_stats` fields have the following meanings:
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| `enqueued` | packets successfully queued |
|
||||
| `dequeued` | packets transmitted |
|
||||
| `policy_dropped` | packets dropped at the queue limit or after node allocation failure |
|
||||
| `cleanup_dropped` | packets drained during reset |
|
||||
| `bytes_dequeued` | total bytes of dequeued packets |
|
||||
| `max_qlen` | peak queue occupancy |
|
||||
| --- | --- |
|
||||
| `enqueued` | Packets successfully added to the FIFO |
|
||||
| `dequeued` | Packets returned for transmission |
|
||||
| `policy_dropped` | Packets rejected at the queue limit or after node allocation failed |
|
||||
| `cleanup_dropped` | Packets drained by qdisc reset |
|
||||
| `bytes_dequeued` | Total qdisc packet bytes returned by dequeue |
|
||||
| `max_qlen` | Peak packet occupancy of the FIFO |
|
||||
|
||||
The loader reads these from `skel->bss->stats` after destroying the qdisc.
|
||||
## User space owns the qdisc lifecycle
|
||||
|
||||
The loader sets `skel->rodata->rate_kbps` and `skel->rodata->queue_limit` before the verifier loads the BPF object. Attaching the skeleton registers the `struct_ops` implementation. `bpf_tc_hook_create` then asks libbpf to create `bpf_pacer` at `BPF_TC_QDISC` and `TC_H_ROOT` on the selected ifindex.
|
||||
|
||||
An existing root qdisc makes that create call return `-EEXIST`. The loader prints a refusal instead of inspecting, stacking, replacing, saving, or restoring the existing qdisc. This conservative rule is why the example belongs on an interface you created for the test.
|
||||
|
||||
After printing `READY`, the loader sleeps in 100 ms intervals until the duration expires or a signal handler sets `exiting`. SIGINT and SIGTERM share that handler. The integration fixture exercises duration-based exit and SIGTERM, but it does not run a separate SIGINT case.
|
||||
|
||||
Cleanup calls `bpf_tc_hook_destroy` before reading `skel->bss->stats`. Destroying the qdisc invokes reset for any remaining queue. The loader prints `SUMMARY` only after those cleanup drops have been counted, then destroys the skeleton and unregisters the `struct_ops` link.
|
||||
|
||||
## Compilation and execution
|
||||
|
||||
Build the BPF object and loader without root:
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
make clean
|
||||
make -j2
|
||||
```
|
||||
|
||||
The safest first run is the integration fixture. It creates and deletes a disposable `epac_tx`/`epac_rx` veth pair, so this command requires root:
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
The final test binary was run in the reused `bpf-benchmark` KVM guest with a clean `7.0.0-rc2+` kernel from source commit `a03114efd0720dff230388f7e160e427e54ea31b`. The captured output was:
|
||||
|
||||
```console
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8
|
||||
TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded
|
||||
```
|
||||
|
||||
The captured run is a functional smoke test, not a throughput or pacing-precision benchmark. The fixture bursts 40 raw 1024-byte EtherType `0x88B5` frames into a 64 Kbit/s pacer with an 8-packet queue. In this run, 31 attempts were counted as `policy_dropped` when the queue filled, while 9 packets were paced and received over 1024 ms. These numbers describe this fixture only.
|
||||
|
||||
Before that burst, the fixture installs a conflicting `pfifo` and confirms that the loader refuses to replace it. A second burst leaves packets queued before SIGTERM. That run requires a nonzero `cleanup_dropped` value and checks `enqueued = dequeued + cleanup_dropped`.
|
||||
|
||||
The last case sends SIGKILL. On the tested guest, the uncatchable signal left `bpf_pacer` installed. The fixture removes it explicitly with `tc`, verifies recovery, and deletes the veth pair in a Python `finally` block.
|
||||
|
||||
To use the loader manually, first create a dedicated interface that you control. Inspect its root qdisc before installation:
|
||||
|
||||
```bash
|
||||
tc qdisc show dev veth-service
|
||||
```
|
||||
|
||||
Then run the pacer for 30 seconds:
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
```
|
||||
|
||||
The command accepts the following options:
|
||||
|
||||
```text
|
||||
Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKETS] [--duration SEC] [--verbose]
|
||||
```
|
||||
|
||||
| Option | Range | Default |
|
||||
| --- | --- | --- |
|
||||
| `--interface` | Required | None |
|
||||
| `--rate-kbps` | 8 to 100000000 | 1024 |
|
||||
| `--queue-limit` | 1 to 65535 packets | 256 |
|
||||
| `--duration` | 1 to 86400 seconds | 10 |
|
||||
| `--verbose` | Flag | Off |
|
||||
|
||||
When the qdisc is active, the loader prints `READY`. It prints the final counters in `SUMMARY` after removal. The following block shows only the output shape. Its ellipses are placeholders, not captured values.
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
If the interface already has a root qdisc, the loader exits with this diagnostic:
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- Linux kernel 6.16 or later. BPF qdisc support entered in [this commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb).
|
||||
- libbpf 1.6.0 or later for BPF qdisc TC hook support ([libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)). The repository vendors libbpf and bpftool under `src/third_party`.
|
||||
- Architecture: x86_64 (tested).
|
||||
- The tool needs root privileges to load BPF and create or destroy a qdisc.
|
||||
- The fixture needs root privileges, Python 3, raw packet sockets, and `iproute2` (`ip` and `tc`).
|
||||
- The explicit recovery path needs `tc`.
|
||||
- Linux 6.16 or newer. [The BPF qdisc merge commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb) introduced the required kernel support.
|
||||
- libbpf 1.6.0 or newer for the BPF qdisc TC hook. The repository vendors libbpf and bpftool under `src/third_party`, and the relevant [libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df) documents the API.
|
||||
- An x86_64 system for the tested configuration.
|
||||
- Root privileges to load BPF and create or destroy the qdisc.
|
||||
- Root privileges, Python 3, raw packet sockets, and `iproute2` tools `ip` and `tc` for the integration fixture.
|
||||
- The `tc` command for the explicit abnormal-exit recovery path.
|
||||
- BTF and BPF JIT enabled.
|
||||
- Kernel configs: `CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_NET_SCHED=y`, `CONFIG_NET_SCH_BPF=y`.
|
||||
- `CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_NET_SCHED=y`, and `CONFIG_NET_SCH_BPF=y`.
|
||||
- A controlled network interface. A veth pair is sufficient.
|
||||
|
||||
## Limitations
|
||||
## Limits and recovery
|
||||
|
||||
- One aggregate FIFO. No fairness, per-flow isolation, per-cgroup policy, classes, priorities, ECN, burst budget, or congestion-control integration.
|
||||
- Queue limit is in packets. Rate accounting uses `qdisc_skb_cb(skb)->pkt_len`.
|
||||
- The first packet after an idle period departs immediately.
|
||||
- Rate and queue limit are fixed for one invocation.
|
||||
- One root qdisc instance per loader process due to global queue and BSS state.
|
||||
- Refuses every existing root qdisc without inspecting, preserving, stacking, or restoring it.
|
||||
- Duration-bounded CLI. Not a daemon, controller, metrics exporter, or persistent policy manager.
|
||||
- Duration-based normal exit and SIGTERM are tested cleanup paths. SIGINT uses the same handler but was not exercised separately. On the tested guest, SIGKILL left the qdisc installed. Cleanup is not guaranteed after an uncatchable signal or abnormal process failure, so inspect the interface and use the recovery command below. Host-crash and reboot behavior were not tested.
|
||||
- KVM test validates behavior but is not a throughput or precision benchmark. It does not account for driver/hardware offload, GSO/TSO, or real NIC timing.
|
||||
This lesson implements one aggregate FIFO. It has no fairness, per-flow isolation, per-cgroup policy, classes, priorities, ECN, burst budget, or congestion-control integration. The queue limit counts packets, and rate accounting uses `qdisc_skb_cb(skb)->pkt_len`. The first packet after an idle period may depart immediately.
|
||||
|
||||
## Recovery
|
||||
Rate and queue limit stay fixed for one invocation. Global queue and BSS state also limit one loader process to one root qdisc instance. The loader refuses every existing root qdisc without inspecting, preserving, stacking, or restoring it.
|
||||
|
||||
After an abnormal exit, verify with:
|
||||
The CLI is duration bounded. It is not a daemon, controller, metrics exporter, or persistent policy manager. Duration-based exit and SIGTERM cleanup were tested. SIGINT uses the same handler but was not tested separately. Cleanup is not guaranteed after SIGKILL or another abnormal process failure, and host-crash and reboot behavior were not tested.
|
||||
|
||||
The KVM fixture checks behavior only. It does not cover throughput, pacing precision on a real NIC, driver or hardware offload, GSO/TSO, or real hardware timing.
|
||||
|
||||
After an abnormal exit, inspect the controlled interface:
|
||||
|
||||
```bash
|
||||
tc qdisc show dev IFACE
|
||||
```
|
||||
|
||||
If `bpf_pacer` is still present on a controlled interface, remove it with:
|
||||
If `bpf_pacer` is still present, remove it with:
|
||||
|
||||
```bash
|
||||
sudo tc qdisc del dev IFACE root
|
||||
```
|
||||
|
||||
This changes traffic scheduling on the interface. Only do this on an interface you control.
|
||||
Deleting a qdisc changes traffic scheduling. Run this command only on an interface you control.
|
||||
|
||||
## Summary
|
||||
|
||||
This example turns the Linux 6.16 BPF qdisc interface into a bounded, short-lived egress pacer. You saw how BPF objects retain skbs, how serialization time becomes a departure schedule, how the watchdog waits without spinning, and how reset separates policy drops from cleanup drops. The same structure can support richer policies, but a real service would need explicit fairness, concurrency, persistence, and failure-recovery design.
|
||||
|
||||
> If you'd like to dive deeper into eBPF, check out our tutorial repository at <https://github.com/eunomia-bpf/bpf-developer-tutorial> or visit our website at <https://eunomia.dev/tutorials/>.
|
||||
|
||||
## References
|
||||
|
||||
@@ -277,10 +769,3 @@ This changes traffic scheduling on the interface. Only do this on an interface y
|
||||
- [Upstream BPF FIFO qdisc selftest](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c)
|
||||
- [Upstream BPF qdisc test runner](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c)
|
||||
- [libbpf 1.7.0 release](https://github.com/libbpf/libbpf/releases/tag/v1.7.0)
|
||||
|
||||
## More
|
||||
|
||||
You can now install a short-lived bounded pacer on a controlled interface, observe its queue and drop accounting, refuse unsafe root replacement, and recover the interface after abnormal termination. This example is part of the eBPF Developer Tutorial. For more examples and the full tutorial, visit:
|
||||
|
||||
- <https://github.com/eunomia-bpf/bpf-developer-tutorial>
|
||||
- <https://eunomia.dev/tutorials/>
|
||||
|
||||
@@ -1,103 +1,149 @@
|
||||
# eBPF 实践教程：使用 BPF Qdisc 对出口流量进行限速
|
||||
# eBPF 教程：用 BPF Qdisc 控制出口流量
|
||||
|
||||
你有一个实验环境、测试台或通过专用 veth/TAP 接口运行的服务。某个组件突然发出流量突发，队列无限制地增长，延迟飙升，而你对丢包情况一无所知。你需要一个按指定时长运行的出口限速器，它能报告队列和字节统计，并在正常退出或处理 SIGINT、SIGTERM 时移除自身。不可捕获的信号仍需使用下文的恢复流程。
|
||||
实验服务突然向 veth 接口发送一阵流量。队列随之增长，延迟开始抖动，部分报文消失了，却没有留下清晰的丢包统计。遇到这种情况，我们需要把突发限制在已知速率内，给队列设定上限，还要知道报文究竟已经发出，还是在队列中被丢弃。本课实现的 `egress_pacer` 就是这样一个小工具。它使用 Linux 6.16 引入的 BPF `struct_ops` 注册一个 FIFO qdisc，为每个报文计算允许发送的时间，并在运行结束时报告入队、出队、字节数和丢包统计。加载器发现已有 root qdisc 时会直接拒绝安装，不会覆盖现有配置。
|
||||
|
||||
本教程构建的工具正是做这件事。它利用 Linux 6.16 的 `struct_ops` BPF qdisc 基础设施注册一个有界 FIFO，以你指定的总速率限制出口流量。这里的限速通过按报文序列化时间进行整形（pacing）实现，并将入队/出队/丢弃统计暴露给用户态。
|
||||
这个示例面向受控的 veth、TAP 或 IFB 接口，不是通用物理网卡上的生产调度器。读完后，你会理解 BPF qdisc 如何持有 skb、qdisc watchdog 如何避免忙等待，以及用户态如何在一个有时限的实验中安装并移除 qdisc。
|
||||
|
||||
这不是生产级工具，不适用于通用物理网卡。它的设计目标是受控接口（veth、TAP、IFB），由操作者做出明确的生命周期决定。加载器拒绝替换任何已存在的 root qdisc。
|
||||
完整项目位于 [`53-egress-pacer` 源码目录](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer)。
|
||||
|
||||
[查看本课的完整源码](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer)。
|
||||
## 为什么要在 qdisc 中限速
|
||||
|
||||
## 快速开始
|
||||
Linux 会让出口报文经过队列规则，也就是 qdisc。普通 TC classifier 可以检查报文并返回处理动作，但报文队列仍由另一个内核 qdisc 管理。[TC classifier 教程](../20-tc/README.zh.md)采用的就是这种方式。
|
||||
|
||||
无需 root 即可编译：
|
||||
本例走的是另一条路径。BPF `struct_ops` 对象直接实现 `struct Qdisc_ops`，把 `bpf_pacer` 注册成 root qdisc。报文能否入队、队首何时可以发送，以及 reset 时如何释放残留 skb，都由这个 BPF 程序决定。
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
make clean
|
||||
make -j2
|
||||
```
|
||||
这种控制能力也带来了生命周期风险。替换 root qdisc 会改变实时流量，所以加载器把任何已有 root qdisc 都视为冲突。程序只运行指定时长，并在正常结束或处理 SIGINT、SIGTERM 后移除 qdisc。SIGKILL 无法捕获，可能留下已经安装的 qdisc，因此文末还给出了显式恢复方法。
|
||||
|
||||
首次运行时，建议让集成测试自动创建并删除一次性 veth 对。此步骤需要 root 权限：
|
||||
## 一个报文如何经过整个工具
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
内核态与用户态按照下面的顺序协作：
|
||||
|
||||
如果已经创建并控制一个专用接口，也可以手动运行加载器。请先通过 `tc qdisc show dev veth-service` 检查接口；如果内核报告已有 root qdisc，加载器会拒绝安装：
|
||||
1. 加载器解析受控接口，把速率和报文上限写入 BPF 只读数据，然后加载 BPF object。
|
||||
2. 挂载 `struct_ops` map 后，内核注册 `bpf_pacer`。加载器再通过 libbpf 的 qdisc TC hook，把它创建在 `TC_H_ROOT`。
|
||||
3. 流量到来前，qdisc 的 `init` 回调把 `queue_limit` 写入 `sch->limit`。
|
||||
4. `egress_pacer_enqueue` 在队列达到上限时丢包，否则把 skb 移交给一个 BPF queue node，并根据报文长度和配置速率计算允许发送时间。
|
||||
5. `egress_pacer_dequeue` 返回已经到达发送时间的 skb。如果队首报文还不能发送，回调会把它放回去，并让 qdisc watchdog 在正确时间重试。
|
||||
6. 用户态移除 qdisc 时，`egress_pacer_reset` 释放仍在队列中的 skb。加载器随后读取最终 BSS 计数器并打印 `SUMMARY`。
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
```
|
||||
整个设计有意保持简单。一个加载器只管理一个 root qdisc、一个 FIFO 和一组全局统计。
|
||||
|
||||
当 qdisc 激活后，工具输出 `READY`；退出时则输出 `SUMMARY` 统计。下面只展示输出格式，其中的省略号是占位符，并非实测值：
|
||||
## 完整源码
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
完整用法：
|
||||
|
||||
```text
|
||||
Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKETS] [--duration SEC] [--verbose]
|
||||
```
|
||||
|
||||
| 选项 | 取值范围 | 默认值 |
|
||||
|------|----------|--------|
|
||||
| `--interface` | 必选 | 无 |
|
||||
| `--rate-kbps` | 8 到 100000000 | 1024 |
|
||||
| `--queue-limit` | 1 到 65535 个报文 | 256 |
|
||||
| `--duration` | 1 到 86400 秒 | 10 |
|
||||
| `--verbose` | 开关 | 关闭 |
|
||||
|
||||
如果接口已有 root qdisc，工具会拒绝并输出：
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## 验证过的测试输出
|
||||
|
||||
仓库包含一个确定性集成测试。以下输出在 KVM 客户机（内核 `7.0.0-rc2+`）中于宿主机编译后采集：
|
||||
|
||||
```console
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8
|
||||
TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded
|
||||
```
|
||||
|
||||
这是功能性冒烟测试，不是基准性能数据。测试向 64 Kbit/s、队列容量 8 的 pacer 突发 40 个 1024 字节的 EtherType `0x88B5` 原始帧。本次运行中，队列填满后有 31 个计入 `policy_dropped`，另有 9 个经过整形后交付。不要将这些数字推广到其他场景。
|
||||
|
||||
在一个可随意使用的 Linux 6.16+ 系统上自行运行：
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
该测试会创建并删除 `epac_tx`/`epac_rx` veth 对。它先安装一个 `pfifo` 以验证冲突拒绝行为，再发送突发流量。第一轮检查有界丢包、字节和报文计数、整形效果及正常退出清理。
|
||||
|
||||
第二轮在发送突发后保留队列积压，再发送 SIGTERM。测试要求 `cleanup_dropped` 非零，并验证 `enqueued = dequeued + cleanup_dropped`。最后一轮确认 SIGKILL 会在测试客户机上留下 `bpf_pacer`，随后通过 `tc` 显式删除 qdisc、验证恢复结果，并在 `finally` 块中删除 veth 对。
|
||||
|
||||
## 工作原理
|
||||
|
||||
本工具由两部分组成：实现 `struct Qdisc_ops` 的 BPF 对象，以及管理 qdisc 生命周期的用户态加载器。
|
||||
|
||||
与[普通 TC classifier/action 示例](../20-tc/README.zh.md)不同，后者由 classifier 在独立的内核 qdisc 上返回处理动作。本例的 struct_ops 程序会注册 `bpf_pacer`，并将它自身安装为 root qdisc。报文到达前，`init` 回调会把配置的 `queue_limit` 写入 `sch->limit`。
|
||||
|
||||
### BPF qdisc：入队
|
||||
|
||||
当报文到达 qdisc 时，入队回调检查队列长度是否超过 `sch->limit`。如果队列已满，报文被丢弃并累加 `policy_dropped`。`bpf_obj_new` 分配失败时也会进入同一丢弃和计数路径。否则：
|
||||
|
||||
1. 通过 `bpf_obj_new` 分配一个 `packet_node`。
|
||||
2. 通过 `bpf_kptr_xchg` 将 skb 转移到节点中。
|
||||
3. 计算报文的允许发送时间戳 `eligible_ns = max(next_departure_ns, now)`，再将 `next_departure_ns` 推进一个序列化间隔，即 `(packet_len * 8 * 1,000,000 / rate_kbps)` 纳秒。
|
||||
4. 将节点推入 BPF 链表，更新 qlen 和 backlog。
|
||||
|
||||
以下代码摘自 BPF 源码（`egress_pacer.bpf.c`）：
|
||||
共享头文件定义了 BPF 程序与用户态共同使用的统计结构。
|
||||
|
||||
<!-- BEGIN FULL SOURCE: src/53-egress-pacer/egress_pacer.h -->
|
||||
```c
|
||||
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
|
||||
#ifndef __EGRESS_PACER_H
|
||||
#define __EGRESS_PACER_H
|
||||
|
||||
struct pacer_stats {
|
||||
unsigned long long enqueued;
|
||||
unsigned long long dequeued;
|
||||
unsigned long long policy_dropped;
|
||||
unsigned long long cleanup_dropped;
|
||||
unsigned long long bytes_dequeued;
|
||||
unsigned long long max_qlen;
|
||||
};
|
||||
|
||||
#endif /* __EGRESS_PACER_H */
|
||||
```
|
||||
<!-- END FULL SOURCE -->
|
||||
|
||||
仓库当前生成的 header 早于部分 BPF qdisc 定义。下面这个本地兼容头文件补齐了所需声明，不会修改仓库全局 vendored header。
|
||||
|
||||
<!-- BEGIN FULL SOURCE: src/53-egress-pacer/bpf_experimental.h -->
|
||||
```c
|
||||
/* SPDX-License-Identifier: GPL-2.0 */
|
||||
#ifndef __EGRESS_PACER_BPF_EXPERIMENTAL_H
|
||||
#define __EGRESS_PACER_BPF_EXPERIMENTAL_H
|
||||
|
||||
#include <bpf/bpf_core_read.h>
|
||||
|
||||
#define __contains(name, node) \
|
||||
__attribute__((btf_decl_tag("contains:" #name ":" #node)))
|
||||
|
||||
extern void *bpf_obj_new_impl(__u64 local_type_id, void *meta) __ksym;
|
||||
#define bpf_obj_new(type) \
|
||||
((type *)bpf_obj_new_impl(bpf_core_type_id_local(type), NULL))
|
||||
|
||||
extern void bpf_obj_drop_impl(void *kptr, void *meta) __ksym;
|
||||
#define bpf_obj_drop(kptr) bpf_obj_drop_impl(kptr, NULL)
|
||||
|
||||
extern int bpf_list_push_front_impl(struct bpf_list_head *head,
|
||||
struct bpf_list_node *node,
|
||||
void *meta, __u64 off) __ksym;
|
||||
#define bpf_list_push_front(head, node) \
|
||||
bpf_list_push_front_impl(head, node, NULL, 0)
|
||||
|
||||
extern int bpf_list_push_back_impl(struct bpf_list_head *head,
|
||||
struct bpf_list_node *node,
|
||||
void *meta, __u64 off) __ksym;
|
||||
#define bpf_list_push_back(head, node) \
|
||||
bpf_list_push_back_impl(head, node, NULL, 0)
|
||||
|
||||
extern struct bpf_list_node *
|
||||
bpf_list_pop_front(struct bpf_list_head *head) __ksym;
|
||||
|
||||
#endif /* __EGRESS_PACER_BPF_EXPERIMENTAL_H */
|
||||
```
|
||||
<!-- END FULL SOURCE -->
|
||||
|
||||
内核态程序实现 qdisc operation、报文队列、限速时钟和统计计数。
|
||||
|
||||
<!-- BEGIN FULL SOURCE: src/53-egress-pacer/egress_pacer.bpf.c -->
|
||||
```c
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
#include "vmlinux.h"
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include <bpf/bpf_tracing.h>
|
||||
#include "bpf_experimental.h"
|
||||
#include "egress_pacer.h"
|
||||
|
||||
char LICENSE[] SEC("license") = "GPL";
|
||||
|
||||
#define NET_XMIT_SUCCESS 0x00
|
||||
#define NET_XMIT_DROP 0x01
|
||||
#define private(name) SEC(".data." #name) __hidden __attribute__((aligned(8)))
|
||||
|
||||
struct bpf_sk_buff_ptr {
|
||||
struct sk_buff *skb;
|
||||
};
|
||||
|
||||
extern void bpf_qdisc_skb_drop(struct sk_buff *skb,
|
||||
struct bpf_sk_buff_ptr *to_free) __ksym;
|
||||
extern void bpf_qdisc_watchdog_schedule(struct Qdisc *sch, __u64 expire,
|
||||
__u64 delta_ns) __ksym;
|
||||
extern void bpf_qdisc_bstats_update(struct Qdisc *sch,
|
||||
const struct sk_buff *skb) __ksym;
|
||||
extern void bpf_kfree_skb(struct sk_buff *skb) __ksym;
|
||||
|
||||
const volatile __u64 rate_kbps = 1024;
|
||||
const volatile __u32 queue_limit = 256;
|
||||
|
||||
struct pacer_stats stats;
|
||||
|
||||
struct packet_node {
|
||||
__u64 eligible_ns;
|
||||
__u32 packet_len;
|
||||
struct sk_buff __kptr *skb;
|
||||
struct bpf_list_node node;
|
||||
};
|
||||
|
||||
private(A) struct bpf_spin_lock queue_lock;
|
||||
private(A) struct bpf_list_head packet_queue __contains(packet_node, node);
|
||||
private(A) __u64 next_departure_ns;
|
||||
|
||||
static struct qdisc_skb_cb *qdisc_skb_cb(const struct sk_buff *skb)
|
||||
{
|
||||
return (struct qdisc_skb_cb *)skb->cb;
|
||||
}
|
||||
|
||||
static __u32 qdisc_packet_len(const struct sk_buff *skb)
|
||||
{
|
||||
return qdisc_skb_cb(skb)->pkt_len;
|
||||
}
|
||||
|
||||
SEC("struct_ops/egress_pacer_enqueue")
|
||||
int BPF_PROG(egress_pacer_enqueue, struct sk_buff *skb, struct Qdisc *sch,
|
||||
struct bpf_sk_buff_ptr *to_free)
|
||||
@@ -141,15 +187,7 @@ drop:
|
||||
__sync_fetch_and_add(&stats.policy_dropped, 1);
|
||||
return NET_XMIT_DROP;
|
||||
}
|
||||
```
|
||||
|
||||
间隔公式意味着空闲后的第一个报文可以立即发出（因为 `next_departure_ns` 已经过去，`eligible_ns = now`）。后续报文按配置速率和序列化报文长度间隔发送。
|
||||
|
||||
### BPF qdisc：出队
|
||||
|
||||
出队回调弹出链表头部。如果当前时间早于报文的 `eligible_ns`，报文被放回链表头部，并调度 qdisc watchdog：
|
||||
|
||||
```c
|
||||
SEC("struct_ops/egress_pacer_dequeue")
|
||||
struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
{
|
||||
@@ -162,8 +200,9 @@ struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
bpf_spin_lock(&queue_lock);
|
||||
node = bpf_list_pop_front(&packet_queue);
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
if (!node)
|
||||
if (!node) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
packet = container_of(node, struct packet_node, node);
|
||||
now = bpf_ktime_get_ns();
|
||||
@@ -192,95 +231,539 @@ struct sk_buff *BPF_PROG(egress_pacer_dequeue, struct Qdisc *sch)
|
||||
__sync_fetch_and_add(&stats.bytes_dequeued, packet_len);
|
||||
return skb;
|
||||
}
|
||||
|
||||
SEC("struct_ops/egress_pacer_init")
|
||||
int BPF_PROG(egress_pacer_init, struct Qdisc *sch, struct nlattr *opt,
|
||||
struct netlink_ext_ack *extack)
|
||||
{
|
||||
(void)opt;
|
||||
(void)extack;
|
||||
sch->limit = queue_limit;
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("struct_ops/egress_pacer_reset")
|
||||
void BPF_PROG(egress_pacer_reset, struct Qdisc *sch)
|
||||
{
|
||||
int queued = sch->q.qlen;
|
||||
int i;
|
||||
|
||||
bpf_for(i, 0, queued) {
|
||||
struct bpf_list_node *node;
|
||||
struct packet_node *packet;
|
||||
struct sk_buff *skb = NULL;
|
||||
|
||||
bpf_spin_lock(&queue_lock);
|
||||
node = bpf_list_pop_front(&packet_queue);
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
if (!node)
|
||||
break;
|
||||
|
||||
packet = container_of(node, struct packet_node, node);
|
||||
skb = bpf_kptr_xchg(&packet->skb, skb);
|
||||
if (skb) {
|
||||
bpf_kfree_skb(skb);
|
||||
__sync_fetch_and_add(&stats.cleanup_dropped, 1);
|
||||
}
|
||||
bpf_obj_drop(packet);
|
||||
}
|
||||
|
||||
bpf_spin_lock(&queue_lock);
|
||||
next_departure_ns = 0;
|
||||
sch->q.qlen = 0;
|
||||
sch->qstats.backlog = 0;
|
||||
bpf_spin_unlock(&queue_lock);
|
||||
}
|
||||
|
||||
SEC("struct_ops/egress_pacer_destroy")
|
||||
void BPF_PROG(egress_pacer_destroy, struct Qdisc *sch)
|
||||
{
|
||||
(void)sch;
|
||||
}
|
||||
|
||||
SEC(".struct_ops")
|
||||
struct Qdisc_ops pacer = {
|
||||
.enqueue = (void *)egress_pacer_enqueue,
|
||||
.dequeue = (void *)egress_pacer_dequeue,
|
||||
.init = (void *)egress_pacer_init,
|
||||
.reset = (void *)egress_pacer_reset,
|
||||
.destroy = (void *)egress_pacer_destroy,
|
||||
.id = "bpf_pacer",
|
||||
};
|
||||
```
|
||||
<!-- END FULL SOURCE -->
|
||||
|
||||
用户态加载器负责检查参数、注册 BPF qdisc、把它安装到一个接口，等待指定时间后再将其移除。
|
||||
|
||||
<!-- BEGIN FULL SOURCE: src/53-egress-pacer/egress_pacer.c -->
|
||||
```c
|
||||
// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
|
||||
#include <errno.h>
|
||||
#include <getopt.h>
|
||||
#include <linux/pkt_sched.h>
|
||||
#include <net/if.h>
|
||||
#include <signal.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
#include <bpf/libbpf.h>
|
||||
#include "egress_pacer.h"
|
||||
#include "egress_pacer.skel.h"
|
||||
|
||||
static volatile sig_atomic_t exiting;
|
||||
|
||||
static struct env {
|
||||
const char *interface;
|
||||
unsigned long long rate_kbps;
|
||||
unsigned int queue_limit;
|
||||
unsigned int duration;
|
||||
bool verbose;
|
||||
} env = {
|
||||
.rate_kbps = 1024,
|
||||
.queue_limit = 256,
|
||||
.duration = 10,
|
||||
};
|
||||
|
||||
static void handle_signal(int signal)
|
||||
{
|
||||
(void)signal;
|
||||
exiting = 1;
|
||||
}
|
||||
|
||||
static int libbpf_print_fn(enum libbpf_print_level level, const char *format,
|
||||
va_list args)
|
||||
{
|
||||
if (level == LIBBPF_DEBUG && !env.verbose)
|
||||
return 0;
|
||||
return vfprintf(stderr, format, args);
|
||||
}
|
||||
|
||||
static void usage(const char *program)
|
||||
{
|
||||
fprintf(stderr,
|
||||
"Usage: %s --interface IFACE [--rate-kbps KBPS] "
|
||||
"[--queue-limit PACKETS] [--duration SEC] [--verbose]\n\n"
|
||||
"Temporarily pace one interface with a bounded BPF qdisc.\n\n"
|
||||
"Options:\n"
|
||||
" -i, --interface IFACE interface to control (required)\n"
|
||||
" -r, --rate-kbps KBPS egress rate, 8-100000000 "
|
||||
"(default: 1024)\n"
|
||||
" -q, --queue-limit PACKETS queue capacity, 1-65535 "
|
||||
"(default: 256)\n"
|
||||
" -d, --duration SEC control window, 1-86400 "
|
||||
"(default: 10)\n"
|
||||
" -v, --verbose print libbpf diagnostics\n"
|
||||
" -h, --help show this help\n",
|
||||
program);
|
||||
}
|
||||
|
||||
static int parse_u64(const char *value, unsigned long long maximum,
|
||||
unsigned long long *result)
|
||||
{
|
||||
char *end = NULL;
|
||||
unsigned long long parsed;
|
||||
|
||||
errno = 0;
|
||||
parsed = strtoull(value, &end, 10);
|
||||
if (errno || end == value || *end || parsed > maximum)
|
||||
return -EINVAL;
|
||||
*result = parsed;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int parse_rate(const char *value)
|
||||
{
|
||||
unsigned long long parsed;
|
||||
|
||||
if (parse_u64(value, 100000000, &parsed) || parsed < 8) {
|
||||
fprintf(stderr, "invalid rate in Kbit/s: %s\n", value);
|
||||
return -EINVAL;
|
||||
}
|
||||
env.rate_kbps = parsed;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int parse_queue_limit(const char *value)
|
||||
{
|
||||
unsigned long long parsed;
|
||||
|
||||
if (parse_u64(value, 65535, &parsed) || parsed == 0) {
|
||||
fprintf(stderr, "invalid queue limit: %s\n", value);
|
||||
return -EINVAL;
|
||||
}
|
||||
env.queue_limit = parsed;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int parse_duration(const char *value)
|
||||
{
|
||||
unsigned long long parsed;
|
||||
|
||||
if (parse_u64(value, 86400, &parsed) || parsed == 0) {
|
||||
fprintf(stderr, "invalid duration in seconds: %s\n", value);
|
||||
return -EINVAL;
|
||||
}
|
||||
env.duration = parsed;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int parse_option(int option, const char *program)
|
||||
{
|
||||
switch (option) {
|
||||
case 'i':
|
||||
env.interface = optarg;
|
||||
return 0;
|
||||
case 'r':
|
||||
return parse_rate(optarg);
|
||||
case 'q':
|
||||
return parse_queue_limit(optarg);
|
||||
case 'd':
|
||||
return parse_duration(optarg);
|
||||
case 'v':
|
||||
env.verbose = true;
|
||||
return 0;
|
||||
case 'h':
|
||||
usage(program);
|
||||
exit(0);
|
||||
default:
|
||||
return -EINVAL;
|
||||
}
|
||||
}
|
||||
|
||||
static int parse_args(int argc, char **argv)
|
||||
{
|
||||
static const struct option options[] = {
|
||||
{ "interface", required_argument, NULL, 'i' },
|
||||
{ "rate-kbps", required_argument, NULL, 'r' },
|
||||
{ "queue-limit", required_argument, NULL, 'q' },
|
||||
{ "duration", required_argument, NULL, 'd' },
|
||||
{ "verbose", no_argument, NULL, 'v' },
|
||||
{ "help", no_argument, NULL, 'h' },
|
||||
{},
|
||||
};
|
||||
int error, option;
|
||||
|
||||
while ((option = getopt_long(argc, argv, "i:r:q:d:vh", options, NULL)) != -1) {
|
||||
error = parse_option(option, argv[0]);
|
||||
if (error)
|
||||
return error;
|
||||
}
|
||||
|
||||
if (!env.interface) {
|
||||
fprintf(stderr, "--interface is required\n");
|
||||
return -EINVAL;
|
||||
}
|
||||
if (optind != argc)
|
||||
return -EINVAL;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static long long monotonic_milliseconds(void)
|
||||
{
|
||||
struct timespec timestamp;
|
||||
|
||||
if (clock_gettime(CLOCK_MONOTONIC, ×tamp))
|
||||
return -errno;
|
||||
return timestamp.tv_sec * 1000LL + timestamp.tv_nsec / 1000000;
|
||||
}
|
||||
|
||||
static int wait_for_duration(void)
|
||||
{
|
||||
struct timespec interval = { .tv_nsec = 100000000 };
|
||||
long long deadline, now;
|
||||
|
||||
now = monotonic_milliseconds();
|
||||
if (now < 0)
|
||||
return (int)now;
|
||||
deadline = now + env.duration * 1000LL;
|
||||
|
||||
while (!exiting) {
|
||||
now = monotonic_milliseconds();
|
||||
if (now < 0)
|
||||
return (int)now;
|
||||
if (now >= deadline)
|
||||
break;
|
||||
if (nanosleep(&interval, NULL) && errno != EINTR)
|
||||
return -errno;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv)
|
||||
{
|
||||
struct egress_pacer_bpf *skel = NULL;
|
||||
struct pacer_stats final_stats = {};
|
||||
struct bpf_tc_hook hook = {
|
||||
.sz = sizeof(hook),
|
||||
.attach_point = BPF_TC_QDISC,
|
||||
.parent = TC_H_ROOT,
|
||||
.handle = TC_H_MAKE(1 << 16, 0),
|
||||
.qdisc = "bpf_pacer",
|
||||
};
|
||||
bool qdisc_created = false;
|
||||
unsigned int ifindex;
|
||||
int cleanup_err;
|
||||
int err;
|
||||
|
||||
err = parse_args(argc, argv);
|
||||
if (err) {
|
||||
usage(argv[0]);
|
||||
return 1;
|
||||
}
|
||||
|
||||
ifindex = if_nametoindex(env.interface);
|
||||
if (!ifindex) {
|
||||
fprintf(stderr, "interface does not exist: %s\n", env.interface);
|
||||
return 1;
|
||||
}
|
||||
hook.ifindex = ifindex;
|
||||
|
||||
libbpf_set_print(libbpf_print_fn);
|
||||
signal(SIGINT, handle_signal);
|
||||
signal(SIGTERM, handle_signal);
|
||||
|
||||
skel = egress_pacer_bpf__open();
|
||||
if (!skel) {
|
||||
fprintf(stderr, "failed to open BPF skeleton\n");
|
||||
return 1;
|
||||
}
|
||||
skel->rodata->rate_kbps = env.rate_kbps;
|
||||
skel->rodata->queue_limit = env.queue_limit;
|
||||
|
||||
err = egress_pacer_bpf__load(skel);
|
||||
if (err) {
|
||||
fprintf(stderr,
|
||||
"failed to load BPF qdisc: %s\n"
|
||||
"This tool requires Linux 6.16+, CONFIG_NET_SCH_BPF, "
|
||||
"BTF, and BPF JIT.\n",
|
||||
strerror(-err));
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
err = egress_pacer_bpf__attach(skel);
|
||||
if (err) {
|
||||
fprintf(stderr, "failed to register bpf_pacer qdisc: %s\n",
|
||||
strerror(-err));
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
err = bpf_tc_hook_create(&hook);
|
||||
if (err) {
|
||||
if (err == -EEXIST)
|
||||
fprintf(stderr,
|
||||
"refusing to replace the existing root qdisc on %s\n",
|
||||
env.interface);
|
||||
else
|
||||
fprintf(stderr, "failed to attach bpf_pacer to %s: %s\n",
|
||||
env.interface, strerror(-err));
|
||||
goto cleanup;
|
||||
}
|
||||
qdisc_created = true;
|
||||
|
||||
printf("READY interface=%s rate_kbps=%llu queue_limit=%u duration=%u\n",
|
||||
env.interface, env.rate_kbps, env.queue_limit, env.duration);
|
||||
fflush(stdout);
|
||||
|
||||
err = wait_for_duration();
|
||||
|
||||
cleanup:
|
||||
if (qdisc_created) {
|
||||
cleanup_err = bpf_tc_hook_destroy(&hook);
|
||||
if (cleanup_err) {
|
||||
fprintf(stderr, "failed to remove bpf_pacer from %s: %s\n",
|
||||
env.interface, strerror(-cleanup_err));
|
||||
if (!err)
|
||||
err = cleanup_err;
|
||||
}
|
||||
}
|
||||
if (skel && skel->bss)
|
||||
final_stats = skel->bss->stats;
|
||||
if (qdisc_created) {
|
||||
printf("SUMMARY enqueued=%llu dequeued=%llu policy_dropped=%llu "
|
||||
"cleanup_dropped=%llu bytes_dequeued=%llu max_qlen=%llu\n",
|
||||
final_stats.enqueued, final_stats.dequeued,
|
||||
final_stats.policy_dropped, final_stats.cleanup_dropped,
|
||||
final_stats.bytes_dequeued, final_stats.max_qlen);
|
||||
}
|
||||
egress_pacer_bpf__destroy(skel);
|
||||
return err != 0;
|
||||
}
|
||||
```
|
||||
<!-- END FULL SOURCE -->
|
||||
|
||||
## 入队时如何把速率换算成时间
|
||||
|
||||
`egress_pacer_enqueue` 先读取 `qdisc_skb_cb(skb)->pkt_len`。队列上限以报文数计算，速率则使用 qdisc 记录的报文长度。队列已满时，程序直接进入丢弃路径。`bpf_obj_new` 分配失败也会走同一路径，因此两种情况都会累加 `policy_dropped`。
|
||||
|
||||
报文被接受后，`bpf_obj_new` 创建一个 `packet_node`，`bpf_kptr_xchg` 再把 skb 转移到节点的 `__kptr` 字段。节点同时保存报文长度和 `eligible_ns` 时间戳。BPF list 与 skb 的所有权一起移动，后续 reset 才能可靠释放这两类对象。
|
||||
|
||||
限速间隔就是该报文在配置速率下的序列化时间：
|
||||
|
||||
```text
|
||||
interval_ns = packet_len * 8 * 1,000,000 / rate_kbps
|
||||
eligible_ns = max(next_departure_ns, now)
|
||||
next_departure_ns = eligible_ns + interval_ns
|
||||
```
|
||||
|
||||
内核通过 qdisc watchdog（`bpf_qdisc_watchdog_schedule`）在队首报文满足发送条件时重试 dequeue。BPF 程序因此无需忙轮询即可完成整形。
|
||||
接口空闲一段时间后，`next_departure_ns` 已经落后于当前时间，第一个报文便可以立即离开。后续报文根据各自长度继续推进时钟。这里实现的是聚合 FIFO pacer，不是 token bucket，也没有单独的突发预算。
|
||||
|
||||
### BPF qdisc：reset
|
||||
这里用 spin lock 保护 list、`next_departure_ns`、qlen 与 backlog 的更新。节点成功插入后，程序增加 `enqueued`，并用 `max_qlen` 记录本次运行见到的队列峰值。
|
||||
|
||||
移除 qdisc 时，`reset` 回调排空所有残留的 skb，释放 `packet_node` 对象，将 qlen/backlog 归零。在此阶段释放的报文计入 `cleanup_dropped`，区别于正常运行期间的策略丢弃。
|
||||
## 出队时如何避免忙等待
|
||||
|
||||
### 用户态生命周期
|
||||
`egress_pacer_dequeue` 取出第一个节点，并把它的时间戳与 `bpf_ktime_get_ns()` 比较。时间已经到达时，另一次 `bpf_kptr_xchg` 把 skb 从 BPF node 中移出。回调减少 qlen 和 backlog，释放空节点，更新 qdisc 字节统计，再累加 `dequeued` 与 `bytes_dequeued`。
|
||||
|
||||
加载器（`egress_pacer.c`）按以下流程执行：
|
||||
如果报文还不能发送，回调会把节点放回队首，调用 `bpf_qdisc_watchdog_schedule(sch, expire, 0)`，然后返回 `NULL`。qdisc core 会在 `eligible_ns` 到来时重试，BPF 程序不需要循环等待时间流逝。
|
||||
|
||||
1. 解析参数，将接口名转换为 ifindex。
|
||||
2. 打开 BPF skeleton，设置 `rodata->rate_kbps` 和 `rodata->queue_limit`。
|
||||
3. 加载 BPF 对象（验证器在此运行）。
|
||||
4. attach struct_ops map，将 `bpf_pacer` 注册为可用的 qdisc 类型。
|
||||
5. 通过 `bpf_tc_hook_create`（带 `BPF_TC_QDISC` 和 `TC_H_ROOT`）在接口 root 创建 qdisc。若接口已有 root qdisc，返回 `-EEXIST`，工具输出拒绝信息并退出。
|
||||
6. 打印 `READY`，以 100 ms 间隔休眠直到持续时间到期或信号到达。
|
||||
7. 通过 `bpf_tc_hook_destroy` 销毁 qdisc，读取最终 BSS 计数器，打印 `SUMMARY`，销毁 skeleton。
|
||||
## 为什么要单独记录 reset 清理丢包
|
||||
|
||||
程序为 SIGINT 和 SIGTERM 注册了同一个信号处理器。处理器设置标志位来中断休眠循环。集成测试覆盖按时长正常退出和 SIGTERM 清理，但没有单独执行 SIGINT 路径。
|
||||
移除 qdisc 时，队列中可能仍有报文。`egress_pacer_reset` 按当前 qlen 运行 `bpf_for`，依次移除节点，用 `bpf_kfree_skb` 释放 skb，再释放 node。这个阶段释放的报文累加到 `cleanup_dropped`，不会混入 `policy_dropped`。
|
||||
|
||||
### 统计字段
|
||||
两个字段回答的是不同问题。`policy_dropped` 表示 pacer 运行期间的准入失败，`cleanup_dropped` 表示带着 backlog 移除 qdisc 时丢弃的报文。reset 最后还会清零 `next_departure_ns`、qlen 和 backlog。
|
||||
|
||||
内核 BSS 中的 `pacer_stats` 结构体包含：
|
||||
内核 BSS 中的 `pacer_stats` 包含下面六项统计：
|
||||
|
||||
| 字段 | 含义 |
|
||||
|------|------|
|
||||
| `enqueued` | 成功入队的报文数 |
|
||||
| `dequeued` | 已发送的报文数 |
|
||||
| `policy_dropped` | 队列满或节点分配失败时丢弃的报文数 |
|
||||
| `cleanup_dropped` | reset 阶段排空的报文数 |
|
||||
| `bytes_dequeued` | 已发送报文的总字节数 |
|
||||
| `max_qlen` | 队列峰值占用 |
|
||||
| --- | --- |
|
||||
| `enqueued` | 成功进入 FIFO 的报文数 |
|
||||
| `dequeued` | 已返回给发送路径的报文数 |
|
||||
| `policy_dropped` | 队列达到上限或 node 分配失败时拒绝的报文数 |
|
||||
| `cleanup_dropped` | qdisc reset 时排空的报文数 |
|
||||
| `bytes_dequeued` | dequeue 返回的 qdisc 报文总字节数 |
|
||||
| `max_qlen` | FIFO 的报文占用峰值 |
|
||||
|
||||
加载器在销毁 qdisc 后从 `skel->bss->stats` 读取这些字段。
|
||||
## 用户态如何管理 qdisc 生命周期
|
||||
|
||||
验证器加载 BPF object 之前，加载器会设置 `skel->rodata->rate_kbps` 与 `skel->rodata->queue_limit`。挂载 skeleton 会注册 `struct_ops` 实现，随后 `bpf_tc_hook_create` 在指定 ifindex 的 `BPF_TC_QDISC`、`TC_H_ROOT` 位置创建 `bpf_pacer`。
|
||||
|
||||
已有 root qdisc 会让创建调用返回 `-EEXIST`。加载器不会检查、堆叠、替换、保存或恢复现有 qdisc，只会输出拒绝信息。因此，手动运行时必须选择自己创建的测试接口。
|
||||
|
||||
工具打印 `READY` 后，以 100 ms 为间隔休眠，直到持续时间结束或信号处理器设置 `exiting`。SIGINT 和 SIGTERM 共用这个 handler。集成测试覆盖按时长退出与 SIGTERM，没有另外执行一遍 SIGINT。
|
||||
|
||||
清理流程先调用 `bpf_tc_hook_destroy`，再读取 `skel->bss->stats`。销毁 qdisc 会让 reset 处理残留队列，因此 `SUMMARY` 中已经包含清理阶段的丢包。最后，加载器销毁 skeleton，并注销 `struct_ops` link。
|
||||
|
||||
## 编译和运行
|
||||
|
||||
编译 BPF object 与加载器不需要 root：
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
make clean
|
||||
make -j2
|
||||
```
|
||||
|
||||
第一次运行最好使用集成测试。它会创建并删除一次性的 `epac_tx`/`epac_rx` veth 对，因此下面的命令需要 root：
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
最终测试程序运行在复用的 `bpf-benchmark` KVM 客户机中。客户机使用干净的 `7.0.0-rc2+` 内核，源码 commit 为 `a03114efd0720dff230388f7e160e427e54ea31b`。实际采集到的输出如下：
|
||||
|
||||
```console
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
SUMMARY enqueued=9 dequeued=9 policy_dropped=31 cleanup_dropped=0 bytes_dequeued=9216 max_qlen=8
|
||||
TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64.0
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded
|
||||
```
|
||||
|
||||
这是一项功能性冒烟测试，不是吞吐量或限速精度 benchmark。测试向速率 64 Kbit/s、队列上限 8 的 pacer 突发发送 40 个 1024 字节的 EtherType `0x88B5` 原始帧。在本次运行中，队列填满时有 31 次发送计入 `policy_dropped`，另外 9 个报文经过限速后在 1024 ms 内到达接收端。这些数字只描述当前 fixture。
|
||||
|
||||
突发测试之前，fixture 会安装一个冲突的 `pfifo`，确认加载器不会将其替换。第二轮先制造 backlog，再发送 SIGTERM。测试要求 `cleanup_dropped` 非零，并验证 `enqueued = dequeued + cleanup_dropped`。
|
||||
|
||||
最后一轮发送 SIGKILL。在测试客户机上，这个不可捕获的信号留下了 `bpf_pacer`。fixture 随后使用 `tc` 显式删除 qdisc、验证接口恢复，并在 Python `finally` 块中删除 veth 对。
|
||||
|
||||
手动运行前，请先创建一个专用接口，并确认自己拥有它。先检查 root qdisc：
|
||||
|
||||
```bash
|
||||
tc qdisc show dev veth-service
|
||||
```
|
||||
|
||||
接下来让 pacer 运行 30 秒：
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
```
|
||||
|
||||
命令行用法如下：
|
||||
|
||||
```text
|
||||
Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKETS] [--duration SEC] [--verbose]
|
||||
```
|
||||
|
||||
| 选项 | 取值范围 | 默认值 |
|
||||
| --- | --- | --- |
|
||||
| `--interface` | 必选 | 无 |
|
||||
| `--rate-kbps` | 8 到 100000000 | 1024 |
|
||||
| `--queue-limit` | 1 到 65535 个报文 | 256 |
|
||||
| `--duration` | 1 到 86400 秒 | 10 |
|
||||
| `--verbose` | 开关 | 关闭 |
|
||||
|
||||
加载器会在 qdisc 生效后打印 `READY`。移除 qdisc 后，最终统计出现在 `SUMMARY` 中。下面只展示输出格式，省略号是占位符，不是实测值。
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
接口已有 root qdisc 时，加载器退出并打印：
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## 运行要求
|
||||
|
||||
- Linux 内核 6.16 及以上。BPF qdisc 支持通过[此提交](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)合入。
|
||||
- libbpf 1.6.0 及以上用于 BPF qdisc TC hook 支持（[libbpf 提交](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)）。仓库在 `src/third_party` 下集成 libbpf 和 bpftool。
|
||||
- 架构：x86_64（已测试）。
|
||||
- 工具需要 root 权限加载 BPF，以及创建或销毁 qdisc。
|
||||
- 集成测试需要 root 权限、Python 3、原始报文套接字和 `iproute2`（`ip` 与 `tc`）。
|
||||
- 显式恢复路径需要 `tc`。
|
||||
- 需要开启 BTF 和 BPF JIT。
|
||||
- 内核配置：`CONFIG_BPF=y`、`CONFIG_BPF_SYSCALL=y`、`CONFIG_BPF_JIT=y`、`CONFIG_DEBUG_INFO_BTF=y`、`CONFIG_NET_SCHED=y`、`CONFIG_NET_SCH_BPF=y`。
|
||||
- 需要受控网络接口，veth 对即可满足。
|
||||
- Linux 6.16 或更新版本。[BPF qdisc 合入 commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)引入了所需内核支持。
|
||||
- libbpf 1.6.0 或更新版本，用于 BPF qdisc TC hook。仓库在 `src/third_party` 中集成了 libbpf 与 bpftool，相关 [libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)记录了这个 API。
|
||||
- 已测试的系统架构为 x86_64。
|
||||
- 加载 BPF、创建和销毁 qdisc 需要 root 权限。
|
||||
- 集成测试还需要 root、Python 3、原始报文 socket，以及 `iproute2` 提供的 `ip` 和 `tc`。
|
||||
- 异常退出后的显式恢复需要 `tc` 命令。
|
||||
- 内核需要启用 BTF 与 BPF JIT。
|
||||
- 内核配置需要包含 `CONFIG_BPF=y`、`CONFIG_BPF_SYSCALL=y`、`CONFIG_BPF_JIT=y`、`CONFIG_DEBUG_INFO_BTF=y`、`CONFIG_NET_SCHED=y` 和 `CONFIG_NET_SCH_BPF=y`。
|
||||
- 需要一个受控网络接口，veth 对已经足够。
|
||||
|
||||
## 限制
|
||||
## 适用边界与恢复方法
|
||||
|
||||
- 单一聚合 FIFO。没有公平调度、每流隔离、每 cgroup 策略、分类、优先级、ECN、突发预算或拥塞控制集成。
|
||||
- 队列限制以报文数计量，速率计算使用 `qdisc_skb_cb(skb)->pkt_len`。
|
||||
- 空闲后的第一个报文可以立即发出。
|
||||
- 速率和队列限制在一次调用中固定不变。
|
||||
- 由于全局队列和 BSS 状态，每个加载器进程只能运行一个 root qdisc 实例。
|
||||
- 拒绝所有已存在的 root qdisc，不检查、不保留、不堆叠、不恢复。
|
||||
- 有限时间的 CLI 工具，不是守护进程、控制器、指标导出器或持久策略管理器。
|
||||
- 程序按时长正常退出和处理 SIGTERM 时的清理路径已经过测试。SIGINT 使用同一处理器，但没有单独执行。在测试客户机上，SIGKILL 会留下已安装的 qdisc。不可捕获的信号或进程异常终止后不保证完成清理，因此需要检查接口并执行下文的恢复命令。宿主机崩溃和重启行为未经过测试。
|
||||
- KVM 测试验证行为正确性，不是吞吐量或精度基准测试，不考虑驱动/硬件卸载、GSO/TSO 或真实网卡时序。
|
||||
本课只实现一个聚合 FIFO，没有公平调度、每流隔离、每 cgroup 策略、class、优先级、ECN、突发预算或拥塞控制集成。队列上限统计报文数，速率计算使用 `qdisc_skb_cb(skb)->pkt_len`。接口空闲后的第一个报文可以立即发出。
|
||||
|
||||
## 恢复
|
||||
一次运行中的速率和队列上限保持固定。全局 queue 与 BSS 状态也意味着一个加载器进程只能管理一个 root qdisc instance。任何已有 root qdisc 都会被拒绝，加载器不会检查、保留、堆叠或恢复它。
|
||||
|
||||
异常退出后，验证 qdisc 状态：
|
||||
这个 CLI 只运行指定时长，不是 daemon、controller、metrics exporter 或持久策略管理器。测试覆盖了按时长退出和 SIGTERM 清理。SIGINT 使用同一个 handler，但没有单独测试。SIGKILL 或其他异常终止后不保证完成清理，宿主机崩溃与重启后的行为也没有测试。
|
||||
|
||||
这项 KVM fixture 只验证功能，不覆盖真实网卡上的吞吐量、限速精度、驱动或硬件 offload、GSO/TSO，也不覆盖真实硬件时序。
|
||||
|
||||
程序异常退出后，先检查受控接口：
|
||||
|
||||
```bash
|
||||
tc qdisc show dev IFACE
|
||||
```
|
||||
|
||||
如果 `bpf_pacer` 仍存在于受控接口上，移除它：
|
||||
如果 `bpf_pacer` 仍然存在，可以将其移除：
|
||||
|
||||
```bash
|
||||
sudo tc qdisc del dev IFACE root
|
||||
```
|
||||
|
||||
此操作会改变接口的流量调度。只在你控制的接口上执行。
|
||||
删除 qdisc 会改变接口的流量调度。只应在自己控制的接口上执行这条命令。
|
||||
|
||||
## 总结
|
||||
|
||||
这个示例把 Linux 6.16 的 BPF qdisc 接口变成了一个有界、短时运行的出口 pacer。我们分析了 BPF object 如何持有 skb、报文序列化时间如何形成发送计划、watchdog 如何避免忙等待，以及 reset 如何区分策略丢包与清理丢包。相同结构可以承载更丰富的策略，但真实服务还需要明确设计公平性、并发状态、持久化和故障恢复。
|
||||
|
||||
> 如果你想继续深入学习 eBPF，可以查看我们的[教程仓库](https://github.com/eunomia-bpf/bpf-developer-tutorial)，或访问 [eunomia 教程网站](https://eunomia.dev/tutorials/)。
|
||||
|
||||
## 参考资料
|
||||
|
||||
- [Linux BPF qdisc 合入提交](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)
|
||||
- [Qdisc watchdog 后续提交](https://github.com/torvalds/linux/commit/7a2dafda950b78611dc441c83d105dfdc7082681)
|
||||
- [Linux BPF qdisc 合入 commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb)
|
||||
- [Qdisc watchdog 后续 commit](https://github.com/torvalds/linux/commit/7a2dafda950b78611dc441c83d105dfdc7082681)
|
||||
- [libbpf qdisc TC hook 支持](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df)
|
||||
- [上游 BPF FIFO qdisc 自测程序](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c)
|
||||
- [上游 BPF qdisc 测试运行器](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c)
|
||||
- [libbpf 1.7.0 发布](https://github.com/libbpf/libbpf/releases/tag/v1.7.0)
|
||||
|
||||
## 更多内容
|
||||
|
||||
现在，你可以在受控接口上安装短期有界 pacer，观察队列和丢包计数，拒绝不安全的 root qdisc 替换，并在异常退出后恢复接口。本示例是 eBPF 开发者教程的一部分。更多示例和完整教程请访问：
|
||||
|
||||
- <https://github.com/eunomia-bpf/bpf-developer-tutorial>
|
||||
- <https://eunomia.dev/tutorials/>
|
||||
- [上游 BPF FIFO qdisc selftest](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/progs/bpf_qdisc_fifo.c)
|
||||
- [上游 BPF qdisc test runner](https://github.com/torvalds/linux/blob/v7.1/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c)
|
||||
- [libbpf 1.7.0 发布说明](https://github.com/libbpf/libbpf/releases/tag/v1.7.0)
|
||||
|
||||
@@ -68,7 +68,7 @@ GPU:
|
||||
- [lesson 42-xdp-loadbalancer](42-xdp-loadbalancer/README.zh.md) eBPF 开发者教程： 简单的 XDP 负载均衡器
|
||||
- [lesson 46-xdp-test](46-xdp-test/README.zh.md) eBPF 实例教程：构建高性能 XDP 数据包生成器
|
||||
- [lesson 50-tcx](50-tcx/README.zh.md) eBPF 入门实践教程第五十篇：使用 TCX Link 实现可组合的流量控制
|
||||
- [lesson 53-egress-pacer](53-egress-pacer/README.zh.md) eBPF 实践教程：使用 BPF Qdisc 对出口流量进行限速
|
||||
- [lesson 53-egress-pacer](53-egress-pacer/README.zh.md) eBPF 教程：用 BPF Qdisc 控制出口流量
|
||||
|
||||
|
||||
追踪:
|
||||
|
||||
@@ -57,7 +57,7 @@
|
||||
| [eBPF 教程：进程级能源监控与功耗分析](48-energy/README.zh.md) | 5.8 | 必需特性 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_EVENTS=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_TRACEPOINTS=y | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 教程：无需内核补丁修复故障的 HID 设备](49-hid/README.zh.md) | 6.3 | 教程文档 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_HID_BPF=y, CONFIG_UHID=y\|m | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 入门实践教程第五十篇：使用 TCX Link 实现可组合的流量控制](50-tcx/README.zh.md) | 6.6 | 教程文档 | x86_64, arm64 | 可选 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_NET_XGRESS=y | 网络接口 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实践教程：使用 BPF Qdisc 对出口流量进行限速](53-egress-pacer/README.zh.md) | 6.16 | 教程文档 | x86_64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_NET_SCHED=y, CONFIG_NET_SCH_BPF=y | 网络接口 | 需要 | CI 构建测试 |
|
||||
| [eBPF 教程：用 BPF Qdisc 控制出口流量](53-egress-pacer/README.zh.md) | 6.16 | 教程文档 | x86_64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y, CONFIG_NET_SCHED=y, CONFIG_NET_SCH_BPF=y | 网络接口 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实例教程：基于 cgroup 的策略控制](cgroup/README.zh.md) | 5.8 | 必需特性 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_CGROUPS=y, CONFIG_CGROUP_BPF=y, CONFIG_CGROUP_DEVICE=y, CONFIG_DEBUG_INFO_BTF=y | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 实例教程：BPF Arena 零拷贝共享内存](features/bpf_arena/README.zh.md) | 6.9 | 教程文档 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BTF=y | 无 | 需要 | CI 构建测试 |
|
||||
| [eBPF 教程：BPF 迭代器用于内核数据导出](features/bpf_iters/README.zh.md) | 5.8 | 教程文档 | x86_64, arm64 | 需要 | CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_DEBUG_INFO_BTF=y | 无 | 需要 | CI 构建测试 |
|
||||
|
||||
Reference in New Issue
Block a user