mirror of
https://github.com/eunomia-bpf/bpf-developer-tutorial.git
synced 2026-10-29 19:58:08 +08:00
Strengthen the egress pacer tutorial narrative
This commit is contained in:
LinuxDev9002
@@ -1,24 +1,24 @@
|
||||
# eBPF Tutorial by Example: Build an Egress Pacer with a BPF Qdisc
|
||||
|
||||
Have you ever wanted to slow down traffic from one test interface and see exactly when each packet becomes ready to leave? Most TC BPF examples cannot show that part. They inspect an skb and return an action, but a separate queueing discipline still owns the packet queue and the send time.
|
||||
Want to throttle a virtual link in your test environment? Say, squeeze a veth down to 64 Kbit/s and watch how many packets queue up before they start dropping. Ordinary TC BPF examples cannot do this. They inspect packets and return actions, but a separate qdisc still decides when packets actually leave.
|
||||
|
||||
Linux 6.16 lets BPF go one step further. A `struct_ops` program can implement the qdisc itself. In this tutorial, we will build `egress_pacer`, a small FIFO pacer for a disposable veth, TAP, or IFB interface. You give it a rate and a queue limit. It holds each skb until its departure time arrives, then reports how many packets were queued, sent, or dropped.
|
||||
Linux 6.16 hands that layer to BPF as well. You write a set of `struct_ops` callbacks and the kernel treats them as a real root qdisc. This tutorial builds `egress_pacer`, a small FIFO pacer that runs on a veth, TAP, or IFB. Give it a rate and a queue limit. It holds packets, releases them on schedule, and reports enqueue, dequeue, and drop statistics at the end.
|
||||
|
||||
We keep the design intentionally small. There are no classes, fairness rules, ECN, or burst controls. That makes it easier to follow the parts a qdisc must get right: skb ownership, queue accounting, pacing time, watchdog wakeups, and cleanup. Treat it as a lab tool, not as a production scheduler for a physical NIC.
|
||||
We keep the design deliberately minimal: no classes, no fairness, no ECN, no burst controls. The goal is to pull out the few things a qdisc must get right: skb ownership, queue accounting, departure time, watchdog wakeups, and teardown cleanup. Treat this as a lab tool. Do not attach it to a physical NIC or use it for production scheduling.
|
||||
|
||||
> The complete source code: <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer>
|
||||
|
||||
## What Changes When BPF Becomes the Qdisc?
|
||||
## Why Ordinary TC BPF Cannot Control Send Time
|
||||
|
||||
In an ordinary TC example, the BPF program sees a packet while another qdisc owns the queue. [Lesson 20](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/20-tc) uses that familiar model. `egress_pacer` goes one layer deeper. Its `struct_ops` object implements `struct Qdisc_ops`, so the BPF program becomes the root qdisc and takes responsibility for every skb it accepts.
|
||||
An ordinary TC BPF program stands beside the queue. It can inspect a packet and decide whether to pass or drop it, but another qdisc still owns that packet. [Lesson 20](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/20-tc) shows this familiar path. `egress_pacer` takes a different position. It implements `struct Qdisc_ops` through `struct_ops` and registers itself as the root qdisc. Once the kernel hands it an skb, the BPF program owns both queueing and release.
|
||||
|
||||
Follow one packet through the example. User space loads the BPF object with a rate and queue limit, registers `bpf_pacer`, and attaches it at `TC_H_ROOT`. The `enqueue` callback moves the skb into a BPF-owned node and assigns a departure time. The `dequeue` callback either returns the packet or asks the qdisc watchdog to wake up when the packet is ready. When user space removes the qdisc, `reset` releases anything still waiting.
|
||||
Follow one packet and the design becomes easier to see. User space sets the rate and queue limit, loads `bpf_pacer`, and attaches it at `TC_H_ROOT`. On arrival, `enqueue` puts the skb into a BPF-managed node and computes the earliest time it may leave. When `dequeue` runs, it returns an eligible skb or sets the watchdog so the kernel comes back later. User space eventually removes the qdisc, and `reset` disposes of packets that are still waiting. Every piece of code below belongs to this path.
|
||||
|
||||
## Building the Pacer
|
||||
## Start the Pacer with Shared Data
|
||||
|
||||
We will build the pacer one component at a time. The first component is the small statistics contract shared by BPF and user space.
|
||||
Before looking at callbacks, we need one contract between the two halves. BPF manages the queue. User space installs it and reports the result. Both sides must agree on the exact layout of the counters.
|
||||
|
||||
### Sharing Statistics with User Space
|
||||
### Define the Statistics Once
|
||||
|
||||
```c
|
||||
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
|
||||
@@ -37,13 +37,13 @@ struct pacer_stats {
|
||||
#endif /* __EGRESS_PACER_H */
|
||||
```
|
||||
|
||||
The BPF program updates these six counters while packets move through the queue. `enqueued` and `dequeued` cover the normal path. `policy_dropped` counts packets rejected because the queue is full or a node cannot be allocated. `cleanup_dropped` counts packets still waiting when the qdisc is reset. The last two fields record the number of bytes sent and the largest queue depth seen during the run.
|
||||
These six fields cover the possible outcomes for a packet. `enqueued` and `dequeued` count successful entry and exit. A full queue or failed node allocation increases `policy_dropped`. A packet that is still waiting when the qdisc is removed increases `cleanup_dropped`. `bytes_dequeued` adds up the bytes returned to the transmit path, while `max_qlen` remembers the deepest point reached by the queue.
|
||||
|
||||
The loader reads the same structure from `skel->bss->stats` after it removes the qdisc. Keeping this definition in one header prevents the BPF program and the loader from disagreeing about the layout.
|
||||
The counters live in BSS. After removing the qdisc, the loader reads them through `skel->bss->stats`. Including one header on both sides prevents a field-order or size mismatch from turning valid counters into nonsense.
|
||||
|
||||
### Adding the Missing BPF Declarations
|
||||
### Supply the Qdisc Declarations Missing from Older Headers
|
||||
|
||||
The repository's generated headers predate a few BPF qdisc definitions. This local compatibility header supplies those declarations without changing the vendored global headers.
|
||||
This example needs graph-object kfuncs that are absent from the repository's current generated headers. The compatibility header below adds only the BPF qdisc declarations used by this lesson. It leaves the shared vendored headers alone.
|
||||
|
||||
```c
|
||||
/* SPDX-License-Identifier: GPL-2.0 */
|
||||
@@ -80,13 +80,13 @@ bpf_list_pop_front(struct bpf_list_head *head) __ksym;
|
||||
#endif /* __EGRESS_PACER_BPF_EXPERIMENTAL_H */
|
||||
```
|
||||
|
||||
This file does not implement pacing. It exposes the graph-object kfuncs used by the real qdisc: `bpf_obj_new` and `bpf_obj_drop` manage packet nodes, while `bpf_list_push_back` and `bpf_list_pop_front` manage the FIFO. The `__contains` BTF tag tells the verifier which node type belongs to the list head.
|
||||
There is no pacing policy in this file. It only provides the tools needed to build the queue. `bpf_obj_new` creates a packet node, and `bpf_obj_drop` releases that node when it is empty. `bpf_list_push_back` and `bpf_list_pop_front` arrange the nodes as a FIFO. The BTF tag produced by `__contains` tells the verifier which node type belongs inside the list head, so it can track the graph safely.
|
||||
|
||||
Keeping these declarations local matters because they are compatibility glue for this example, not a replacement for the repository's generated libbpf headers. Once those headers provide the same definitions, this file can disappear without changing the pacing algorithm.
|
||||
Keeping the compatibility layer inside this lesson also separates it from the scheduler itself. Once the generated libbpf headers expose the same declarations, this file can be removed without touching the qdisc algorithm.
|
||||
|
||||
### Implementing the Qdisc Callbacks
|
||||
### Implement the Complete Qdisc Lifecycle in BPF
|
||||
|
||||
Now we can look at the kernel-mode program. It implements the qdisc operations, packet queue, pacing clock, and counters.
|
||||
Now we can read the complete kernel-side program. One file contains the qdisc callbacks, FIFO, pacing clock, and counter updates. The main path has only three stages: `enqueue` takes an skb, `dequeue` returns it at the right time, and `reset` handles anything that never left.
|
||||
|
||||
```c
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
@@ -288,13 +288,15 @@ struct Qdisc_ops pacer = {
|
||||
};
|
||||
```
|
||||
|
||||
#### Taking Ownership of an skb
|
||||
The full source is long, but ownership makes it manageable. We will follow the skb across each callback.
|
||||
|
||||
The `init` callback first copies `queue_limit` into `sch->limit`. From then on, `egress_pacer_enqueue` must either take ownership of each skb or reject it. It reads `qdisc_skb_cb(skb)->pkt_len`, checks the packet limit, and allocates a `packet_node` with `bpf_obj_new`. A full queue and an allocation failure use the same drop path, so both increase `policy_dropped`.
|
||||
#### `enqueue`: Own the skb and Schedule Its Departure
|
||||
|
||||
For an accepted packet, `bpf_kptr_xchg` moves the skb into the node's `__kptr` field. The node now owns the packet and stores its qdisc length and `eligible_ns` timestamp. While holding `queue_lock`, the program adds the node to the FIFO and updates qlen, backlog, `enqueued`, and `max_qlen` together.
|
||||
During initialization, `init` copies the selected `queue_limit` into `sch->limit`. Every later call to `egress_pacer_enqueue` must give the incoming skb a definite outcome. The callback reads the qdisc-visible length from `qdisc_skb_cb(skb)->pkt_len`, checks for space, and creates a `packet_node` with `bpf_obj_new`. A full queue and a failed allocation both mean that the program cannot accept ownership, so they share one drop path and increment `policy_dropped`.
|
||||
|
||||
The departure time comes from the serialization time of this packet at the requested rate:
|
||||
The actual ownership transfer happens through `bpf_kptr_xchg`. After it succeeds, the node's `__kptr` field holds the skb and the node is responsible for it. The node also stores the packet length and its `eligible_ns` timestamp. Under `queue_lock`, the program pushes that node to the FIFO tail and updates qlen, backlog, `enqueued`, and `max_qlen` in the same critical section. The list and its accounting therefore move together.
|
||||
|
||||
The delay is not a fixed interval. It comes from the time this packet would occupy the link at the selected rate:
|
||||
|
||||
```text
|
||||
interval_ns = packet_len * 8 * 1,000,000 / rate_kbps
|
||||
@@ -302,32 +304,32 @@ eligible_ns = max(next_departure_ns, now)
|
||||
next_departure_ns = eligible_ns + interval_ns
|
||||
```
|
||||
|
||||
If the queue was idle, `next_departure_ns` is already in the past, so the first packet can leave immediately. Every later packet advances the clock by its own length. This creates one aggregate FIFO schedule. It is not a token bucket and does not have a separate burst budget.
|
||||
After an idle period, `next_departure_ns` is behind the current time, so the next packet can leave at once. While the FIFO stays busy, every following packet advances the clock by its own length. The result is one aggregate FIFO schedule. It is not a token bucket, and there is no separate burst allowance.
|
||||
|
||||
#### Waiting Without Busy Looping
|
||||
#### `dequeue`: Hand Early Packets to the Watchdog
|
||||
|
||||
`egress_pacer_dequeue` pops the first node and compares its `eligible_ns` with `bpf_ktime_get_ns()`. If the time has arrived, `bpf_kptr_xchg` moves the skb out of the node. The callback reduces qlen and backlog, frees the empty node, updates the qdisc byte statistics, and increases `dequeued` and `bytes_dequeued`.
|
||||
`egress_pacer_dequeue` only needs to examine the front node. It pops that node and compares `eligible_ns` with the current value from `bpf_ktime_get_ns()`. If the packet is ready, a second `bpf_kptr_xchg` returns the skb to the qdisc. The callback then reduces qlen and backlog, drops the empty node, updates the qdisc's byte counters, and increments `dequeued` and `bytes_dequeued`.
|
||||
|
||||
If the packet is early, the callback puts the node back at the front and calls `bpf_qdisc_watchdog_schedule(sch, expire, 0)`. Returning `NULL` tells the qdisc core that nothing is ready. The core wakes the qdisc at `eligible_ns`, so the BPF program never spins while it waits.
|
||||
An early packet cannot leave, but the callback must not wait in a loop. It puts the node back at the front, schedules `bpf_qdisc_watchdog_schedule(sch, expire, 0)`, and returns `NULL`. The qdisc core now knows there is nothing ready and calls dequeue again around `eligible_ns`. No BPF code runs while the queue waits.
|
||||
|
||||
#### Releasing Packets During Reset
|
||||
#### `reset`: Finish the Ownership Story
|
||||
|
||||
Removal completes the ownership story. `egress_pacer_reset` uses `bpf_for` to drain the current qlen. It releases every retained skb with `bpf_kfree_skb` and drops each node object. These packets increase `cleanup_dropped`, while admission failures during normal enqueue increase `policy_dropped`. Reset also clears the pacing clock, qlen, and backlog.
|
||||
Removing a qdisc does not make its queued skbs disappear. `egress_pacer_reset` uses `bpf_for` to pop as many nodes as the current qlen reports. It releases every retained skb with `bpf_kfree_skb`, then drops the node object. Those packets count toward `cleanup_dropped`, separate from admission failures in `policy_dropped`. Once the list is empty, reset clears the pacing clock, qlen, and backlog as well.
|
||||
|
||||
Together, the BPF callbacks update the six counters we saw in the shared header:
|
||||
Each counter now has a clear update site and a concrete packet outcome:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `enqueued` | Packets successfully added to the FIFO |
|
||||
| `enqueued` | Packets that entered the FIFO |
|
||||
| `dequeued` | Packets returned for transmission |
|
||||
| `policy_dropped` | Packets rejected at the queue limit or after node allocation failed |
|
||||
| `cleanup_dropped` | Packets drained by qdisc reset |
|
||||
| `bytes_dequeued` | Total qdisc packet bytes returned by dequeue |
|
||||
| `policy_dropped` | Packets rejected at the queue limit or after allocation failed |
|
||||
| `cleanup_dropped` | Packets drained during qdisc reset |
|
||||
| `bytes_dequeued` | Total qdisc-packet bytes returned by dequeue |
|
||||
| `max_qlen` | Peak packet occupancy of the FIFO |
|
||||
|
||||
### Loading and Removing the Qdisc
|
||||
### Install and Remove the Qdisc from User Space
|
||||
|
||||
The user-space loader validates the command line, registers the BPF qdisc, installs it on one interface, waits for the requested duration, and removes it.
|
||||
The kernel-side file defines how the qdisc behaves. The user-space program below supplies its configuration, loads and registers it, installs it on one interface, and removes it when the run ends.
|
||||
|
||||
```c
|
||||
// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
|
||||
@@ -628,17 +630,19 @@ cleanup:
|
||||
}
|
||||
```
|
||||
|
||||
#### Keeping the Experiment on One Interface
|
||||
With the whole loader visible, we can follow the order from configuration to teardown.
|
||||
|
||||
Before `egress_pacer_bpf__load()`, the CLI writes the chosen rate and queue limit into `skel->rodata->rate_kbps` and `skel->rodata->queue_limit`. Loading invokes the verifier, attaching the skeleton registers the `struct_ops` implementation, and `bpf_tc_hook_create` creates `bpf_pacer` at `BPF_TC_QDISC` and `TC_H_ROOT` on the selected ifindex.
|
||||
#### Keep Every Operation on the Target Interface
|
||||
|
||||
The create request is exclusive. If another root qdisc already exists and creation returns `-EEXIST`, the loader refuses to inspect, replace, stack, save, or restore it. This conservative behavior is why the example belongs on an interface created for the experiment.
|
||||
After parsing arguments, the loader writes the rate and queue limit into `skel->rodata->rate_kbps` and `skel->rodata->queue_limit`. This must happen before `egress_pacer_bpf__load()` because rodata becomes immutable after loading. Once the verifier accepts the program, attaching the skeleton registers the `struct_ops` implementation. Only then does `bpf_tc_hook_create` create the `bpf_pacer` root qdisc at `BPF_TC_QDISC` and `TC_H_ROOT` for the target ifindex.
|
||||
|
||||
After `READY`, the loader waits in 100 ms intervals until the duration expires or a signal handler sets `exiting`. Normal completion, SIGINT, and SIGTERM enter the same cleanup path. That path calls `bpf_tc_hook_destroy` before reading `skel->bss->stats`, so reset-time drops appear in `SUMMARY`. If removal fails, the loader prints the error and the counters available at that point, returns failure, and leaves the interface for explicit inspection. Destroying the skeleton then unregisters the `struct_ops` link.
|
||||
Creation is exclusive. If the interface already has a root qdisc, the kernel returns `-EEXIST` and the loader stops. It does not guess whether replacement is safe, and it does not try to save and restore the existing setup. That conservative rule limits the damage this tutorial can do, but it also means you should use an interface created for the experiment.
|
||||
|
||||
## Compilation and execution
|
||||
After installation, the loader prints `READY` and checks the duration and `exiting` flag every 100 ms. A normal timeout, SIGINT, and SIGTERM all reach the same cleanup label. Cleanup order matters. The program calls `bpf_tc_hook_destroy` first, which triggers reset, and reads `skel->bss->stats` afterward. That is why `SUMMARY` includes packets dropped during reset. If destroy fails, the loader reports the error and any counters still available, returns failure, and leaves the interface in place for inspection. Destroying the skeleton then unregisters the `struct_ops` link.
|
||||
|
||||
Build the BPF object and loader with the lesson's [`Makefile`](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/Makefile). Compilation does not need root:
|
||||
## Build, Run, and Watch the Queue
|
||||
|
||||
The lesson's [`Makefile`](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/Makefile) builds the BPF object, skeleton, and user-space loader. This step does not need root:
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
@@ -646,13 +650,13 @@ make clean
|
||||
make -j2
|
||||
```
|
||||
|
||||
The safest first run uses the [integration fixture](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/tests/test_egress_pacer.py). It creates and removes a disposable `epac_tx`/`epac_rx` veth pair, so the runtime step needs root:
|
||||
Do not risk an existing interface on the first run. Use the [integration fixture](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/tests/test_egress_pacer.py), which creates a disposable `epac_tx`/`epac_rx` veth pair and deletes it when the test ends. Loading BPF and changing a qdisc require root:
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
The example was tested on x86_64 with a `7.0.0-rc2+` kernel built from commit `a03114efd0720dff230388f7e160e427e54ea31b`. The fixture produced:
|
||||
Here is one completed run. It used x86_64 and a `7.0.0-rc2+` kernel built from commit `a03114efd0720dff230388f7e160e427e54ea31b`:
|
||||
|
||||
```console
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
@@ -661,17 +665,17 @@ TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded
|
||||
```
|
||||
|
||||
This is a functional smoke test, not a throughput or pacing-precision benchmark. The fixture sends 40 raw 1024-byte EtherType `0x88B5` frames into a 64 Kbit/s pacer with an 8-packet queue. In this run, 9 packets arrived over 1024 ms and 31 attempts entered the `policy_dropped` path. Those values describe this fixture and this run.
|
||||
This output answers whether the main paths work. It does not measure throughput or pacing precision. The fixture sends 40 raw 1024-byte EtherType `0x88B5` frames into a 64 Kbit/s queue that can hold 8 packets. In this run, the receiver saw 9 packets over 1024 ms and another 31 send attempts reached `policy_dropped`. The exact numbers can change with timing and environment.
|
||||
|
||||
The same fixture also covers the less pleasant paths. It installs a conflicting `pfifo` and checks that the loader leaves it alone. It creates a backlog before SIGTERM, requires a nonzero `cleanup_dropped`, and checks `enqueued = dequeued + cleanup_dropped`. Finally, it sends SIGKILL. The uncatchable signal left `bpf_pacer` installed on this guest, so the fixture removed it with `tc`, verified recovery, and deleted the veth pair from a Python `finally` block.
|
||||
The fixture also exercises failure and cleanup. It installs a conflicting `pfifo` and confirms that the loader refuses to replace it. It builds a backlog, sends SIGTERM, requires `cleanup_dropped` to be nonzero, and checks `enqueued = dequeued + cleanup_dropped`. The final case sends SIGKILL, which the process cannot catch and may leave the qdisc behind. The fixture removes that qdisc with `tc`, confirms recovery, and deletes the veth pair from a Python `finally` block.
|
||||
|
||||
For a manual run, use only an interface you control. Inspect its root qdisc first:
|
||||
Once the automatic test makes sense, you can inspect the behavior manually. Choose only an interface you fully control and check its current root qdisc first:
|
||||
|
||||
```bash
|
||||
tc qdisc show dev veth-service
|
||||
```
|
||||
|
||||
Then start the pacer for 30 seconds:
|
||||
If that interface is safe to use, start the pacer for 30 seconds:
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
@@ -689,53 +693,50 @@ Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKET
|
||||
| `--duration` | 1 to 86400 seconds | 10 |
|
||||
| `--verbose` | Flag | Off |
|
||||
|
||||
The loader prints `READY` after installation and `SUMMARY` after successful removal. The ellipses below show the output shape rather than captured values:
|
||||
`READY` means installation succeeded. After the duration ends and removal succeeds, the loader prints `SUMMARY`. The ellipses below show the output format, not measured values:
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
An existing root qdisc makes exclusive creation fail with:
|
||||
If another root qdisc is present, the loader fails before changing it and explains the refusal:
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## Requirements and limits
|
||||
## Where This Example Runs, and What It Leaves Out
|
||||
|
||||
- Linux 6.16 or newer with BTF and BPF JIT enabled. The [BPF qdisc merge commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb) introduced the required support.
|
||||
- `CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_NET_SCHED=y`, and `CONFIG_NET_SCH_BPF=y`.
|
||||
- libbpf 1.6.0 or newer for the BPF qdisc TC hook. The repository vendors libbpf and bpftool under `src/third_party`. The [libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df) and [1.6.0 release](https://github.com/libbpf/libbpf/releases/tag/v1.6.0) document the version boundary.
|
||||
- Root privileges to load BPF and create or destroy a qdisc.
|
||||
- For the fixture, Python 3, raw packet sockets, and the `ip` and `tc` tools from `iproute2`.
|
||||
- A controlled interface. A veth pair is sufficient. The tested configuration is x86_64.
|
||||
The BPF qdisc support used here starts with Linux 6.16. The [BPF qdisc merge commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb) contains the core change. The kernel must also enable BTF and the BPF JIT, with at least `CONFIG_BPF=y`, `CONFIG_BPF_SYSCALL=y`, `CONFIG_BPF_JIT=y`, `CONFIG_DEBUG_INFO_BTF=y`, `CONFIG_NET_SCHED=y`, and `CONFIG_NET_SCH_BPF=y`.
|
||||
|
||||
The scheduler has one aggregate FIFO. It provides no fairness, per-flow isolation, per-cgroup policy, classes, priorities, ECN, congestion-control integration, or token-bucket burst budget. The packet limit counts packets, while rate accounting uses `qdisc_skb_cb(skb)->pkt_len`. An idle queue lets its first packet leave immediately.
|
||||
The TC hook for a BPF qdisc also needs libbpf 1.6.0 or newer. This repository vendors libbpf and bpftool under `src/third_party`; the relevant [libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df) and [1.6.0 release](https://github.com/libbpf/libbpf/releases/tag/v1.6.0) mark that boundary. Runtime operations require root. The fixture additionally needs Python 3, raw packet sockets, and the `ip` and `tc` tools from `iproute2`. One controlled veth pair is enough. The configuration tested here is x86_64.
|
||||
|
||||
Rate and queue limit remain fixed for one invocation. Global queue and BSS state limit one loader process to one root qdisc instance. The duration-bounded CLI is not a daemon, controller, metrics exporter, or persistent policy manager.
|
||||
The design boundary is just as important as the version boundary. This scheduler has one aggregate FIFO. It has no fairness, per-flow isolation, per-cgroup policy, classes, priorities, ECN, congestion-control integration, or token-bucket burst allowance. The queue limit counts packets, and the timing calculation uses `qdisc_skb_cb(skb)->pkt_len`. When the FIFO is idle, its next packet can leave immediately.
|
||||
|
||||
The fixture tested duration-based exit and SIGTERM cleanup. SIGINT uses the same handler but did not receive a separate test. SIGKILL and other abnormal failures may leave the qdisc installed. Host crashes, reboots, GSO/TSO, driver or hardware offload, real-NIC pacing precision, throughput, and hardware timing were not tested.
|
||||
Rate and queue limit are fixed when the object loads and cannot change during a run. The queue and counters use global BSS state, so one loader process manages one root qdisc instance. This duration-bounded CLI is not a daemon, controller, metrics exporter, or persistent policy manager.
|
||||
|
||||
After an abnormal exit, inspect the interface:
|
||||
The tested cleanup paths are duration-based exit and SIGTERM. SIGINT shares the same handler but was not run as a separate case. SIGKILL and other abnormal failures may still leave the qdisc installed. This lesson also does not test system crashes or reboots, GSO/TSO, driver or hardware offload, real-NIC pacing precision, throughput, or hardware timing. Do not extend the smoke-test result to those cases.
|
||||
|
||||
After an abnormal exit, inspect the interface before trying another install:
|
||||
|
||||
```bash
|
||||
tc qdisc show dev IFACE
|
||||
```
|
||||
|
||||
If `bpf_pacer` remains, remove it only from an interface you control:
|
||||
Only when the remaining qdisc is `bpf_pacer` and you fully control the interface should you remove it:
|
||||
|
||||
```bash
|
||||
sudo tc qdisc del dev IFACE root
|
||||
```
|
||||
|
||||
Deleting a qdisc changes traffic scheduling, so never run this recovery command on an interface you do not control.
|
||||
This command directly changes traffic scheduling. Never use it on a production interface or one whose purpose you do not know.
|
||||
|
||||
## Summary
|
||||
## What to Take Away
|
||||
|
||||
This lesson used a small FIFO to expose the full BPF qdisc lifecycle. `enqueue` takes ownership of an skb and assigns its departure time, the watchdog wakes `dequeue` without a busy loop, and `reset` accounts for packets still queued during removal. Those mechanics are the reusable part. Fairness, concurrent state, dynamic policy, persistence, and stronger recovery would require a larger scheduler.
|
||||
This deliberately small FIFO walked through the full BPF qdisc lifecycle. `enqueue` accepts an skb and schedules it from the packet length. The watchdog lets `dequeue` wait without a busy loop. `reset` closes the ownership path for skbs still queued during removal. Those ownership and timing rules are the reusable part of the lesson. Fairness, concurrent state, dynamic policy, persistence, and stronger recovery belong in a more complete scheduler.
|
||||
|
||||
> If you'd like to dive deeper into eBPF, check out our tutorial repository at <https://github.com/eunomia-bpf/bpf-developer-tutorial>.
|
||||
> To keep learning eBPF by building complete examples, explore the [tutorial repository](https://github.com/eunomia-bpf/bpf-developer-tutorial).
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -1,24 +1,24 @@
|
||||
# eBPF 教程：用 BPF Qdisc 实现出口限速
|
||||
|
||||
你有没有遇到过这样的情况：只想限制一个测试接口的出口速率，同时看清每个报文究竟什么时候可以发出去？常见的 TC BPF 示例看不到这一段。它们可以检查 skb 并返回处理动作，但报文怎样排队、什么时候发送，仍由另一个 qdisc 决定。
|
||||
想给测试环境里的一条虚拟网线限个速？比方说，把 veth 压到 64 Kbit/s，看看丢包之前队列能塞多少报文。常见的 TC BPF 示例做不到这一点：它们可以检查报文、决定放行还是丢弃，但报文什么时候发出去，仍由另一个 qdisc 来安排。
|
||||
|
||||
Linux 6.16 让 BPF 可以再向前一步：`struct_ops` 程序能够直接实现 qdisc。本课会用它构建 `egress_pacer`，一个运行在临时 veth、TAP 或 IFB 接口上的小型 FIFO 限速器。你只要给出速率和队列上限，它就会保存 skb，等到发送时间后再交还给网络栈，最后报告入队、发送和丢包统计。
|
||||
Linux 6.16 把这一层也交给了 BPF。你可以用 `struct_ops` 写一个真正的 qdisc，自己管 skb 队列，自己算发送时间。本课会用它构建 `egress_pacer`：一个运行在 veth、TAP 或 IFB 上的小型 FIFO 限速器。给出速率和队列上限，它就把报文存起来，到点再放，最后报告入队、发送和丢包统计。
|
||||
|
||||
这个例子故意没有加入 class、公平调度、ECN 和 burst 控制。这样我们才能把 qdisc 必须做对的事情看清楚：skb 所有权、队列统计、发送时间、watchdog 唤醒和清理。请把它当作实验工具，不要直接用作物理网卡上的生产调度器。
|
||||
这个例子故意只做聚合 FIFO，没有 class、公平调度、ECN、burst 控制。目的是把 qdisc 必须做对的几件事拎出来讲清楚：skb 所有权、队列统计、发送时间、watchdog 唤醒和 reset 清理。用它做实验可以，但别往物理网卡上挂，也别当生产调度器。
|
||||
|
||||
> 完整源码：<https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/53-egress-pacer>
|
||||
|
||||
## 当 BPF 自己成为 Qdisc
|
||||
## 为什么普通 TC BPF 管不到发送时间
|
||||
|
||||
在普通的 TC 示例中，BPF 程序可以看到报文，但队列仍由另一个 qdisc 管理。[第 20 课](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/20-tc) 用的就是这种模式。`egress_pacer` 再向下走一层。它通过 `struct_ops` 实现 `struct Qdisc_ops`，把自己注册为 root qdisc，并对接收的每个 skb 负责。
|
||||
普通 TC BPF 程序站在队列旁边。它能读写报文，也能决定放行或丢弃，却不拥有队列。[第 20 课](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/20-tc) 展示的就是这条常用路径。`egress_pacer` 换了一个位置：它用 `struct_ops` 实现 `struct Qdisc_ops`，自己注册成 root qdisc。内核把 skb 交过来之后，排队和发送就都由这段 BPF 程序负责。
|
||||
|
||||
跟着一个报文走一遍就容易理解了。用户态先写入速率和队列上限，加载并注册 `bpf_pacer`，再把它挂到 `TC_H_ROOT`。`enqueue` 回调把 skb 移进 BPF 管理的节点，并计算发送时间。`dequeue` 回调要么返回报文，要么请 qdisc watchdog 等到报文可以发送时再来。用户态移除 qdisc 后，`reset` 会释放仍在队列里的报文。
|
||||
先跟着一个报文走一遍。用户态设置速率和队列上限，加载 `bpf_pacer`，再把它挂到 `TC_H_ROOT`。报文到达时，`enqueue` 把 skb 装进 BPF 管理的节点，并算出它最早可以离开的时间。轮到 `dequeue` 时，时间到了就交还 skb；时间没到就设置 watchdog，等内核稍后再来。最后，用户态移除 qdisc，`reset` 收走仍在队列里的报文。后面的代码都围绕这条路径展开。
|
||||
|
||||
## 一步步构建 Pacer
|
||||
## 从共享数据开始搭起 Pacer
|
||||
|
||||
我们按组件来构建这个限速器。先从 BPF 程序和用户态共同使用的统计结构开始。
|
||||
先别急着看回调。BPF 程序负责排队，用户态负责安装和汇报结果，两边必须先约定一份相同的统计数据布局。
|
||||
|
||||
### 让 BPF 和用户态共享统计信息
|
||||
### 用一个头文件约定统计格式
|
||||
|
||||
```c
|
||||
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
|
||||
@@ -37,13 +37,13 @@ struct pacer_stats {
|
||||
#endif /* __EGRESS_PACER_H */
|
||||
```
|
||||
|
||||
BPF 程序会在报文经过队列时更新这六个计数器。`enqueued` 和 `dequeued` 记录正常的入队与出队；`policy_dropped` 记录队列已满或节点分配失败造成的拒绝；`cleanup_dropped` 记录 reset 时仍留在队列里的报文。最后两个字段分别保存已经发送的字节数和本次运行观察到的最大队长。
|
||||
这六个字段把报文的一生分成几种结果。成功进入和离开 FIFO 的报文分别计入 `enqueued`、`dequeued`。队列已满或节点分配失败时，`policy_dropped` 增加；移除 qdisc 时还没发出的报文则进入 `cleanup_dropped`。`bytes_dequeued` 累计实际交还给发送路径的字节数，`max_qlen` 记住队列最深的一刻。
|
||||
|
||||
loader 移除 qdisc 后，会从 `skel->bss->stats` 读取同一个结构。把定义放在共享头文件中，可以避免 BPF 程序和用户态对字段布局产生不同理解。
|
||||
这些计数器位于 BSS。qdisc 被移除后，加载器直接从 `skel->bss->stats` 读取它们。两端包含同一个头文件，就不会因为字段顺序或大小不同而把统计值读错。
|
||||
|
||||
### 补齐缺少的 BPF 声明
|
||||
### 给旧版头文件补上 qdisc 所需声明
|
||||
|
||||
仓库中较早生成的头文件缺少部分 BPF qdisc 定义。下面这个本地兼容头文件补齐所需声明，不会修改仓库全局集成的头文件。
|
||||
这个示例需要 graph object 相关的 kfunc，但仓库现有的生成头文件还没有这些 BPF qdisc 声明。下面的兼容头文件只在本课目录内补齐接口，不会改动仓库统一使用的头文件。
|
||||
|
||||
```c
|
||||
/* SPDX-License-Identifier: GPL-2.0 */
|
||||
@@ -80,13 +80,13 @@ bpf_list_pop_front(struct bpf_list_head *head) __ksym;
|
||||
#endif /* __EGRESS_PACER_BPF_EXPERIMENTAL_H */
|
||||
```
|
||||
|
||||
这个文件本身不做限速。它只是把真正的 qdisc 会用到的 graph-object kfunc 暴露出来：`bpf_obj_new` 和 `bpf_obj_drop` 管理报文节点的生命周期，`bpf_list_push_back` 和 `bpf_list_pop_front` 管理 FIFO。`__contains` 生成的 BTF tag 则告诉 verifier，list head 中保存的是哪一种节点。
|
||||
这里没有限速算法，只有构建队列所需的工具。`bpf_obj_new` 创建报文节点，`bpf_obj_drop` 在节点用完后释放它。`bpf_list_push_back` 和 `bpf_list_pop_front` 让这些节点组成 FIFO。`__contains` 生成 BTF tag，告诉 verifier 这个 list head 里装的是哪一种节点；没有这层类型信息，verifier 无法安全地跟踪链表对象。
|
||||
|
||||
这些声明放在教程目录内，是因为它们只是当前仓库的兼容胶水，不应该取代仓库统一生成的 libbpf 头文件。以后生成的头文件补齐同样的定义时，删除这个文件也不会影响限速算法。
|
||||
把兼容代码留在本课目录还有一个好处：它和限速逻辑是分开的。等仓库生成的 libbpf 头文件提供同样的声明后，可以直接删掉这层胶水，qdisc 的实现不需要跟着改。
|
||||
|
||||
### 实现 Qdisc 回调
|
||||
### 在 BPF 中实现整个 Qdisc 生命周期
|
||||
|
||||
接下来进入内核态程序。它实现 qdisc 回调、报文队列、限速时钟和统计计数。
|
||||
现在来看内核态的完整程序。一个文件里放着 qdisc 回调、FIFO、限速时钟和统计更新。代码较长，但主线只有三步：`enqueue` 接管 skb，`dequeue` 等到合适时间再交还 skb，`reset` 处理没来得及发送的 skb。
|
||||
|
||||
```c
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
@@ -288,13 +288,13 @@ struct Qdisc_ops pacer = {
|
||||
};
|
||||
```
|
||||
|
||||
#### 接管一个 skb
|
||||
#### `enqueue`：接管 skb，并排好发送时间
|
||||
|
||||
`init` 回调先把 `queue_limit` 写入 `sch->limit`。从这一步开始，`egress_pacer_enqueue` 必须明确处理每个 skb：要么接管它，要么拒绝它。程序读取 `qdisc_skb_cb(skb)->pkt_len`，检查队列上限，再用 `bpf_obj_new` 分配 `packet_node`。队列已满和节点分配失败会进入同一条 drop 路径，因此都会增加 `policy_dropped`。
|
||||
初始化时，`init` 把用户选择的 `queue_limit` 放进 `sch->limit`。此后每次调用 `egress_pacer_enqueue`，程序都必须给传入的 skb 一个明确结果。它先从 `qdisc_skb_cb(skb)->pkt_len` 取得 qdisc 看到的报文长度，再检查队列是否还有位置，最后用 `bpf_obj_new` 创建 `packet_node`。队列已满或节点创建失败都表示无法接管这个 skb，因此走同一条丢弃路径，并增加 `policy_dropped`。
|
||||
|
||||
报文可以入队时，`bpf_kptr_xchg` 会把 skb 转移到节点的 `__kptr` 字段。此后节点拥有这个 skb，同时保存报文长度和 `eligible_ns` 时间戳。程序持有 `queue_lock` 时，把节点加入 FIFO，并一起更新 qlen、backlog、`enqueued` 和 `max_qlen`。
|
||||
真正的所有权转移发生在 `bpf_kptr_xchg`。调用成功后，skb 被放进节点的 `__kptr` 字段，节点从此负责保管它。节点还记录报文长度和 `eligible_ns`。程序随后拿住 `queue_lock`，把节点推到 FIFO 尾部，并在同一个临界区内更新 qlen、backlog、`enqueued` 和 `max_qlen`，避免链表内容与计数彼此脱节。
|
||||
|
||||
发送时间来自这个报文在目标速率下的序列化时间：
|
||||
`eligible_ns` 不是固定间隔。程序按报文长度计算它在目标速率下需要占用多少发送时间：
|
||||
|
||||
```text
|
||||
interval_ns = packet_len * 8 * 1,000,000 / rate_kbps
|
||||
@@ -302,19 +302,19 @@ eligible_ns = max(next_departure_ns, now)
|
||||
next_departure_ns = eligible_ns + interval_ns
|
||||
```
|
||||
|
||||
如果队列刚才是空的，`next_departure_ns` 已经落在当前时间之前，所以第一个报文可以立即离开。后续报文再按各自长度推进时钟。这样得到的是一个聚合 FIFO 发送计划，不是 token bucket，也没有单独的突发额度。
|
||||
队列空闲一段时间后，`next_departure_ns` 会早于当前时间，因此新来的第一个报文不必额外等待。只要 FIFO 中持续有数据，后面的报文就按自己的长度继续推动这只时钟。这是一个聚合 FIFO 时间表，不是 token bucket，也没有可以另外消耗的 burst 额度。
|
||||
|
||||
#### 等待发送，但不忙等
|
||||
#### `dequeue`：时间未到就交给 watchdog
|
||||
|
||||
`egress_pacer_dequeue` 先取出队首节点，再比较它的 `eligible_ns` 和 `bpf_ktime_get_ns()`。发送时间已经到达时，`bpf_kptr_xchg` 会把 skb 从节点中取出。回调随后减少 qlen 和 backlog，释放空节点，更新 qdisc 字节统计，并增加 `dequeued` 和 `bytes_dequeued`。
|
||||
`egress_pacer_dequeue` 每次只看 FIFO 最前面的节点。它取出节点，把 `eligible_ns` 与 `bpf_ktime_get_ns()` 返回的当前时间比较。报文已经可以发送时，第二次 `bpf_kptr_xchg` 把 skb 从节点交回 qdisc。程序接着扣减 qlen 和 backlog，释放已经空掉的节点，更新 qdisc 自身的字节统计，再增加 `dequeued` 和 `bytes_dequeued`。
|
||||
|
||||
如果时间还没到，回调会把节点放回队首，调用 `bpf_qdisc_watchdog_schedule(sch, expire, 0)`，然后返回 `NULL`。这会告诉 qdisc 核心当前没有报文可发。核心会在 `eligible_ns` 到来时重新唤醒队列，BPF 程序不需要自己循环等待。
|
||||
如果队首报文来早了，程序不能把它发走，也不能挡在回调里忙等。它把节点放回队首，用 `bpf_qdisc_watchdog_schedule(sch, expire, 0)` 预约下一次唤醒，然后返回 `NULL`。qdisc 核心据此知道当前没有可发送的报文，并在 `eligible_ns` 附近再次调用 dequeue。等待期间不需要运行 BPF 代码。
|
||||
|
||||
#### 在 Reset 中释放报文
|
||||
#### `reset`：为队列里剩下的 skb 收尾
|
||||
|
||||
移除 qdisc 时，skb 所有权也必须有明确去处。`egress_pacer_reset` 使用 `bpf_for` 排空当前 qlen，通过 `bpf_kfree_skb` 释放每个残留 skb，再销毁节点。这些报文计入 `cleanup_dropped`，正常 enqueue 期间的准入失败则计入 `policy_dropped`。reset 最后还会清零限速时钟、qlen 和 backlog。
|
||||
移除 qdisc 不会让队列中的 skb 自动消失。`egress_pacer_reset` 按当前 qlen 用 `bpf_for` 逐个取出节点，调用 `bpf_kfree_skb` 释放其中的 skb，再销毁节点对象。这里清掉的报文计入 `cleanup_dropped`，与 enqueue 阶段的 `policy_dropped` 分开。链表排空后，reset 还要把限速时钟、qlen 和 backlog 一起归零。
|
||||
|
||||
到这里，BPF 回调已经用到了共享头文件中的全部六项统计：
|
||||
这样一来，每个计数器都有清楚的更新位置，也能对应到一种可观察的结果：
|
||||
|
||||
| 字段 | 含义 |
|
||||
| --- | --- |
|
||||
@@ -325,9 +325,9 @@ next_departure_ns = eligible_ns + interval_ns
|
||||
| `bytes_dequeued` | dequeue 返回的 qdisc 报文总字节数 |
|
||||
| `max_qlen` | FIFO 的报文占用峰值 |
|
||||
|
||||
### 加载并移除 Qdisc
|
||||
### 用用户态程序安装和移除 Qdisc
|
||||
|
||||
用户态加载器负责检查参数、注册 BPF qdisc、把它安装到一个接口，等待指定时间后再将其移除。
|
||||
内核态代码只定义 qdisc 怎样工作。下面的用户态程序负责把配置写进去，完成加载和注册，把 qdisc 安装到指定接口，并在运行时间结束后安全移除它。
|
||||
|
||||
```c
|
||||
// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
|
||||
@@ -628,17 +628,17 @@ cleanup:
|
||||
}
|
||||
```
|
||||
|
||||
#### 把实验限制在一个接口内
|
||||
#### 从配置到清理，始终只操作目标接口
|
||||
|
||||
调用 `egress_pacer_bpf__load()` 之前，命令行程序会把速率和队列上限写入 `skel->rodata->rate_kbps` 与 `skel->rodata->queue_limit`。加载阶段由内核验证器检查程序，挂载 skeleton 会注册 `struct_ops` 实现，随后 `bpf_tc_hook_create` 在目标 ifindex 的 `BPF_TC_QDISC`、`TC_H_ROOT` 位置创建 `bpf_pacer`。
|
||||
加载器解析完参数后，先把速率和队列上限写入 `skel->rodata->rate_kbps` 与 `skel->rodata->queue_limit`。这必须发生在 `egress_pacer_bpf__load()` 之前，因为 rodata 在加载后就不能再改。内核验证程序通过后，attach skeleton 会注册 `struct_ops` 实现。最后，`bpf_tc_hook_create` 才在目标 ifindex 的 `BPF_TC_QDISC`、`TC_H_ROOT` 位置创建名为 `bpf_pacer` 的 root qdisc。
|
||||
|
||||
创建操作采用独占模式。如果接口已经有 root qdisc，操作会返回 `-EEXIST`。加载器不会查看、替换、叠加、保存或恢复它，而是直接拒绝继续。因此，这个示例只能用于专门为实验创建的接口。
|
||||
创建请求带着独占语义。目标接口已经有 root qdisc 时，内核返回 `-EEXIST`，程序立即停止。它不会猜测现有配置能否替换，也不会尝试保存后恢复。这种保守做法减少了教程代码的破坏面，同时也意味着你应该只把它交给专门创建的测试接口。
|
||||
|
||||
打印 `READY` 后，加载器每隔 100 ms 检查一次，直到运行时间结束或信号处理函数设置 `exiting`。正常结束、SIGINT 和 SIGTERM 最终都会进入同一条清理路径。程序先调用 `bpf_tc_hook_destroy`，再读取 `skel->bss->stats`，因此 reset 阶段丢弃的报文也会出现在 `SUMMARY` 中。如果移除失败，程序会打印错误和当时还能读取到的统计，返回失败，并把接口留给用户检查。最后销毁 skeleton 时，`struct_ops` link 也会注销。
|
||||
安装成功后，程序打印 `READY`，再每隔 100 ms 检查运行时间和 `exiting` 标志。时间到、SIGINT 和 SIGTERM 都汇入同一个 cleanup 标签。清理顺序很重要：先调用 `bpf_tc_hook_destroy` 触发 reset，再读取 `skel->bss->stats`，`SUMMARY` 才能包含 reset 清掉的报文。如果 destroy 失败，程序打印错误和仍能读取的计数，返回失败，并保留接口现场供检查。随后销毁 skeleton，注销 `struct_ops` link。
|
||||
|
||||
## 编译和运行
|
||||
## 编译、运行，再观察队列行为
|
||||
|
||||
使用本课的 [`Makefile`](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/Makefile) 编译 BPF 对象和加载器，这一步不需要 root：
|
||||
本课的 [`Makefile`](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/Makefile) 会生成 BPF 对象、skeleton 和用户态加载器。编译本身不需要 root：
|
||||
|
||||
```bash
|
||||
cd src/53-egress-pacer
|
||||
@@ -646,13 +646,13 @@ make clean
|
||||
make -j2
|
||||
```
|
||||
|
||||
第一次运行最好使用[集成测试程序](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/tests/test_egress_pacer.py)。它会自动创建并删除一次性的 `epac_tx`/`epac_rx` veth 对，实际加载 BPF 和操作 qdisc 时需要 root：
|
||||
第一次尝试不要拿现有接口冒险。直接运行[集成测试程序](https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/53-egress-pacer/tests/test_egress_pacer.py)，让它创建一次性的 `epac_tx`/`epac_rx` veth 对，完成测试后再删除。加载 BPF 和修改 qdisc 需要 root：
|
||||
|
||||
```bash
|
||||
sudo python3 tests/test_egress_pacer.py ./egress_pacer
|
||||
```
|
||||
|
||||
本课在 x86_64 环境中完成测试，使用从 commit `a03114efd0720dff230388f7e160e427e54ea31b` 构建的 `7.0.0-rc2+` 内核。测试输出如下：
|
||||
下面是一次已经完成的运行。环境为 x86_64，内核是从 commit `a03114efd0720dff230388f7e160e427e54ea31b` 构建的 `7.0.0-rc2+`：
|
||||
|
||||
```console
|
||||
READY interface=epac_tx rate_kbps=64 queue_limit=8 duration=2
|
||||
@@ -661,17 +661,17 @@ TEST-SUMMARY attempts=40 received=9 send_errors=31 span_ms=1024 observed_kbps=64
|
||||
PASS: conflict refusal, bounded drops, pacing, accounting, normal/signal cleanup, and SIGKILL recovery succeeded
|
||||
```
|
||||
|
||||
这是一项功能性冒烟测试，不是吞吐量或限速精度基准。测试程序向速率为 64 Kbit/s、上限为 8 个报文的队列发送 40 个 1024 字节的 EtherType `0x88B5` 原始帧。本次运行收到 9 个报文，首尾时间相隔 1024 ms，另外 31 次发送进入 `policy_dropped` 路径。这些数字只代表这次测试。
|
||||
这段输出只能回答“主要路径能否正常工作”，不能用来评价吞吐量或限速精度。测试向一个 64 Kbit/s、最多容纳 8 个报文的队列连续发送 40 个 1024 字节 EtherType `0x88B5` 原始帧。本次运行中，接收端看到 9 个报文，首尾相隔 1024 ms；另外 31 次发送走到 `policy_dropped`。换一个系统或时序，这些具体数字可能不同。
|
||||
|
||||
同一个测试程序也覆盖了异常路径。它先安装冲突的 `pfifo`，确认加载器不会替换现有 qdisc。随后，测试程序制造 backlog 并发送 SIGTERM，要求 `cleanup_dropped` 大于零，同时验证 `enqueued = dequeued + cleanup_dropped`。最后一轮发送 SIGKILL。这个无法捕获的信号在测试客户机中留下了 `bpf_pacer`，测试程序因此用 `tc` 将其删除，确认接口恢复，并在 Python `finally` 块中清理 veth 对。
|
||||
测试并不只看正常退出。它会预先安装一个冲突的 `pfifo`，确认加载器拒绝覆盖；还会制造 backlog 后发送 SIGTERM，检查 `cleanup_dropped` 大于零，并验证 `enqueued = dequeued + cleanup_dropped`。最后的 SIGKILL 无法被进程捕获，qdisc 因而可能残留。测试程序会用 `tc` 删除它，确认接口可以恢复，并在 Python 的 `finally` 块中清掉 veth 对。
|
||||
|
||||
手动运行时，只能选择自己控制的接口。先查看它当前的 root qdisc：
|
||||
理解自动测试后，也可以手动观察。只选择完全由你控制的接口，并先确认它现在使用什么 root qdisc：
|
||||
|
||||
```bash
|
||||
tc qdisc show dev veth-service
|
||||
```
|
||||
|
||||
然后让 pacer 运行 30 秒：
|
||||
确认接口可以安全使用后，让 pacer 持续 30 秒：
|
||||
|
||||
```bash
|
||||
sudo ./egress_pacer --interface veth-service --rate-kbps 64000 --queue-limit 256 --duration 30
|
||||
@@ -689,53 +689,50 @@ Usage: ./egress_pacer --interface IFACE [--rate-kbps KBPS] [--queue-limit PACKET
|
||||
| `--duration` | 1 到 86400 秒 | 10 |
|
||||
| `--verbose` | 开关 | 关闭 |
|
||||
|
||||
qdisc 安装完成后，加载器会打印 `READY`。成功移除后，它会打印 `SUMMARY`。下面的省略号只表示输出格式，不是实测值：
|
||||
看到 `READY` 就表示 qdisc 已经安装。运行结束并成功移除后，程序打印 `SUMMARY`。下面只展示输出格式，其中的省略号不是测量结果：
|
||||
|
||||
```text
|
||||
READY interface=veth-service rate_kbps=64000 queue_limit=256 duration=30
|
||||
SUMMARY enqueued=... dequeued=... policy_dropped=... cleanup_dropped=... bytes_dequeued=... max_qlen=...
|
||||
```
|
||||
|
||||
如果接口已经存在 root qdisc，独占创建会失败，并输出：
|
||||
如果 root qdisc 已被占用，程序会在改变接口之前失败，并明确说明拒绝替换：
|
||||
|
||||
```text
|
||||
refusing to replace the existing root qdisc on IFACE
|
||||
```
|
||||
|
||||
## 运行要求和适用边界
|
||||
## 在什么环境里运行，以及它没有做什么
|
||||
|
||||
- Linux 6.16 或更新版本，并启用 BTF 和 BPF JIT。[BPF qdisc 合入 commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb) 引入了所需支持。
|
||||
- 内核配置包含 `CONFIG_BPF=y`、`CONFIG_BPF_SYSCALL=y`、`CONFIG_BPF_JIT=y`、`CONFIG_DEBUG_INFO_BTF=y`、`CONFIG_NET_SCHED=y` 和 `CONFIG_NET_SCH_BPF=y`。
|
||||
- libbpf 1.6.0 或更新版本，用于 BPF qdisc TC hook。仓库在 `src/third_party` 中集成了 libbpf 和 bpftool。相关 [libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df) 与 [1.6.0 发布说明](https://github.com/libbpf/libbpf/releases/tag/v1.6.0) 记录了版本边界。
|
||||
- 加载 BPF、创建和销毁 qdisc 需要 root 权限。
|
||||
- 集成测试还需要 Python 3、原始报文套接字，以及 `iproute2` 提供的 `ip` 和 `tc`。
|
||||
- 需要一个受控接口，veth 对已经足够。当前测试配置为 x86_64。
|
||||
这个功能从 Linux 6.16 开始可用。[BPF qdisc 合入 commit](https://github.com/torvalds/linux/commit/c8240344956e3f0b4e8f1d40ec3435e47040cacb) 包含了核心支持。内核还必须启用 BTF 和 BPF JIT，并至少包含 `CONFIG_BPF=y`、`CONFIG_BPF_SYSCALL=y`、`CONFIG_BPF_JIT=y`、`CONFIG_DEBUG_INFO_BTF=y`、`CONFIG_NET_SCHED=y` 与 `CONFIG_NET_SCH_BPF=y`。
|
||||
|
||||
这个调度器只有一个聚合 FIFO，不提供公平调度、每流隔离、每 cgroup 策略、类别、优先级、ECN、拥塞控制集成或 token bucket 的突发额度。队列上限按报文数计算，速率计算使用 `qdisc_skb_cb(skb)->pkt_len`。队列空闲时，第一个报文可以立即离开。
|
||||
BPF qdisc TC hook 需要 libbpf 1.6.0 或更新版本。仓库已经在 `src/third_party` 中集成 libbpf 和 bpftool；对应的 [libbpf commit](https://github.com/libbpf/libbpf/commit/f580871b429c550edf910a1b0d700510245351df) 和 [1.6.0 发布说明](https://github.com/libbpf/libbpf/releases/tag/v1.6.0) 给出了版本边界。运行时还要有 root 权限。集成测试另外依赖 Python 3、原始报文套接字，以及 `iproute2` 中的 `ip` 和 `tc`。一个受控的 veth 对就足够了；本课验证的架构是 x86_64。
|
||||
|
||||
每次运行的速率和队列上限都是固定的。全局队列与 BSS 状态也限制了一个加载器进程只能管理一个 root qdisc 实例。这个命令行程序只运行指定时长，不是守护进程、控制器、指标导出器或持久策略管理器。
|
||||
再看清它的能力边界。这个调度器只有一条聚合 FIFO，没有公平调度、每流隔离、每 cgroup 策略、class、优先级、ECN、拥塞控制集成，也没有 token bucket 的 burst 额度。队列上限按报文个数计算，发送间隔使用 `qdisc_skb_cb(skb)->pkt_len` 推导。FIFO 空闲时，新来的第一个报文可以立刻离开。
|
||||
|
||||
测试覆盖按时长退出和 SIGTERM 清理。SIGINT 使用同一个处理函数，但没有单独执行测试。SIGKILL 和其他异常终止可能留下 qdisc。宿主机崩溃、重启、GSO/TSO、驱动或硬件卸载、真实网卡的限速精度、吞吐量和硬件时序都不在本次测试范围内。
|
||||
速率和队列上限在每次加载时固定，运行中不能动态修改。队列和统计又是全局 BSS 状态，所以一个加载器进程只管理一个 root qdisc 实例。这个命令行程序会在指定时间后退出；它不是 daemon、控制器、指标导出器，也不会持久保存策略。
|
||||
|
||||
异常退出后，先检查接口：
|
||||
已经验证的清理路径包括到时退出和 SIGTERM。SIGINT 共用同一个处理函数，但没有单独跑一遍。SIGKILL 或其他异常终止仍可能留下 qdisc。本课也没有测试系统崩溃与重启、GSO/TSO、驱动或硬件卸载、真实网卡的限速精度、吞吐量和硬件时序。不要把冒烟测试的结果外推到这些场景。
|
||||
|
||||
进程异常消失后，先查看接口，而不是盲目再次安装：
|
||||
|
||||
```bash
|
||||
tc qdisc show dev IFACE
|
||||
```
|
||||
|
||||
如果 `bpf_pacer` 仍然存在，只能在自己控制的接口上将它删除：
|
||||
只有确认残留的是 `bpf_pacer`，并且接口完全由你控制时，才执行删除：
|
||||
|
||||
```bash
|
||||
sudo tc qdisc del dev IFACE root
|
||||
```
|
||||
|
||||
删除 qdisc 会改变接口的流量调度，因此只能在自己控制的接口上执行这条恢复命令。
|
||||
这条命令会直接改变流量调度。生产接口或用途不明的接口都不应该用它恢复。
|
||||
|
||||
## 总结
|
||||
## 小结
|
||||
|
||||
这个小型 FIFO 展示了完整的 BPF qdisc 生命周期。`enqueue` 接管 skb 并计算发送时间，watchdog 在不忙等的情况下重新唤醒 `dequeue`，`reset` 则处理移除时仍留在队列中的报文。真正可以复用的是这些机制。如果要加入公平性、并发状态、动态策略、持久化和更强的故障恢复，就需要设计一个更完整的调度器。
|
||||
我们用一个尽量小的 FIFO 走完了 BPF qdisc 的完整生命周期。`enqueue` 接过 skb，按报文长度排出发送时间；watchdog 让 `dequeue` 不必忙等；`reset` 为移除时仍在排队的 skb 收尾。这三段所有权和时序处理才是例子里最值得复用的部分。公平性、并发状态、动态策略、持久化和更强的故障恢复，则属于下一个更完整的调度器。
|
||||
|
||||
> 如果你想继续深入学习 eBPF，可以查看我们的[教程仓库](https://github.com/eunomia-bpf/bpf-developer-tutorial)。
|
||||
> 想继续动手学习 eBPF，可以在[教程仓库](https://github.com/eunomia-bpf/bpf-developer-tutorial)中找到更多完整示例。
|
||||
|
||||
## 参考资料
|
||||
|
||||
|
||||
Reference in New Issue
Block a user