feat: add tcx and bpf token tutorials (#203)

* feat: add tcx and bpf token tutorials * docs: auto-generate documentation * docs: rewrite tcx and bpf_token tutorials with richer content and consistent style Rewrote all 4 README files (EN/ZH for both tutorials) to match the existing tutorial style with detailed background, full code listings, step-by-step explanations, comparison tables, and proper references. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: replace em dashes with colons, commas, and parentheses Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: auto-generate documentation * ci: add tcx and bpf_token builds to CI; simplify execl in token_userns_demo - Add make targets for src/50-tcx and src/features/bpf_token to the test-libbpf CI workflow. - Replace four separate execl() calls with a single execv() using a dynamically built argv array, reducing complexity and eliminating CodeFactor command-injection false positives. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: fix mkdocs path in trigger-sync workflow Use .venv/bin/mkdocs instead of bare mkdocs, since make install puts it inside a virtualenv. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: LinuxDev9002 <linuxdev8883@example.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-10-31 12:45:23 +08:00 · 2026-03-10 21:23:23 -07:00
parent ee6d522e40
commit 3a722c03d5
22 changed files with 2212 additions and 1 deletions
--- a/src/50-tcx/README.md
+++ b/src/50-tcx/README.md
@@ -0,0 +1,240 @@
+# eBPF Tutorial by Example 50: Composable Traffic Control with TCX Links
+
+Ever tried attaching multiple BPF programs to the TC ingress path and got frustrated managing qdisc handles, filter priorities, and the `tc` CLI? Or needed one application's TC program to coexist safely with another's without accidentally overwriting it? Traditional `cls_bpf` attachment through `tc` works, but it inherits decades of queueing discipline plumbing that was never designed for the BPF-centric world. What if you could attach, order, and manage TC programs using the same link-based API that XDP and cgroup programs already enjoy?
+
+This is what **TCX** (Traffic Control eXtension) solves. Introduced by Daniel Borkmann and merged in Linux 6.6, TCX provides a lightweight, fd-based multi-program attach infrastructure for the TC ingress and egress data path. Programs get BPF link semantics (safe ownership, auto-detachment on close, and explicit ordering through `BPF_F_BEFORE` / `BPF_F_AFTER` flags) without touching a single qdisc or filter priority.
+
+In this tutorial, we'll attach two TCX ingress programs to the loopback interface, place one before the other, query the kernel's live chain state, and generate traffic to verify execution order.
+
+> The complete source code: <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/50-tcx>
+
+## Introduction to TCX: Why Classic TC Attachment Needed a Rethink
+
+### The Problem: Qdisc Plumbing and Unsafe Ownership
+
+Classic `tc` BPF attachment (`cls_bpf`) was bolted onto the existing Traffic Control framework. To attach a BPF program, you first needed a `clsact` qdisc on the interface, then added a filter with a handle and priority. This worked fine for a single operator, but created real problems in cloud-native environments where multiple applications need to attach TC programs to the same interface:
+
+1. **No ownership model**: A `tc filter del` from one application can accidentally remove another application's program. There's no protection against this because classic tc filters are identified by handle/priority, not by the process that created them.
+
+2. **Priority conflicts**: Two applications might pick the same priority number. The second attachment silently replaces the first.
+
+3. **Permanent attachment by default**: Classic tc filters persist until explicitly removed. If the application that attached a filter crashes without cleanup, the filter remains, potentially with stale program logic.
+
+4. **CLI dependency**: Even with libbpf, the attachment model was tied to netlink, the same mechanism the `tc` CLI uses. This meant your BPF application was sharing a control plane with every other tc user on the system.
+
+These issues became acute in projects like Cilium, where the BPF dataplane needs to coexist with third-party CNI plugins, observability agents, and security tools that all want to hook into TC.
+
+### The Solution: Link-Based Multi-Program Management
+
+TCX takes a fundamentally different approach. Instead of piggybacking on qdisc infrastructure, it provides a dedicated, qdisc-less extension point for BPF programs at the TC ingress and egress hooks. The key design principles:
+
+**BPF Link Semantics**: `bpf_program__attach_tcx()` creates a `BPF_LINK_TYPE_TCX` link. Like XDP links and cgroup links, TCX links give you safe ownership: the link is pinned to the file descriptor, auto-detaches when the fd is closed, and cannot be accidentally overridden by another application.
+
+**Explicit Ordering**: Instead of implicit priority numbers, you place programs relative to each other using `BPF_F_BEFORE` and `BPF_F_AFTER`. You can also use `BPF_F_REPLACE` to atomically swap a specific program. All operations support an `expected_revision` field that prevents race conditions during concurrent modifications.
+
+**Chain Return Codes**: TCX defines simplified return codes that make multi-program composition explicit:
+
+| Return Code | Value | Meaning |
+|-------------|-------|---------|
+| `TCX_NEXT` | -1 | Non-terminating; pass the packet to the next program in the chain |
+| `TCX_PASS` | 0 | Accept the packet and terminate the chain |
+| `TCX_DROP` | 2 | Drop the packet and terminate the chain |
+| `TCX_REDIRECT` | 7 | Redirect the packet and terminate the chain |
+
+Unknown return codes are mapped to `TCX_NEXT` for forward compatibility.
+
+**Coexistence with Classic TC**: TCX links can coexist with traditional `cls_bpf` filters on the same interface. The kernel runs TCX programs first, then falls through to classic `tcf_classify()` if present. This allows gradual migration from classic tc to TCX without a disruptive cutover.
+
+## Writing the eBPF Program
+
+Our BPF object contains two programs that demonstrate chain composition. Here is the complete source:
+
+```c
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_endian.h>
+#include <bpf/bpf_helpers.h>
+
+#ifndef TCX_NEXT
+#define TCX_NEXT -1
+#endif
+
+#ifndef TCX_PASS
+#define TCX_PASS 0
+#endif
+
+char LICENSE[] SEC("license") = "GPL";
+
+__u64 stats_hits;
+__u64 classifier_hits;
+__u32 last_len;
+__u16 last_protocol;
+__u32 last_ifindex;
+
+SEC("tcx/ingress")
+int tcx_stats(struct __sk_buff *skb)
+{
+	stats_hits++;
+	last_len = skb->len;
+	last_protocol = bpf_ntohs(skb->protocol);
+	last_ifindex = skb->ifindex;
+	return TCX_NEXT;
+}
+
+SEC("tcx/ingress")
+int tcx_classifier(struct __sk_buff *skb)
+{
+	classifier_hits++;
+	return TCX_PASS;
+}
+```
+
+Let's walk through this step by step.
+
+### Section Names: `SEC("tcx/ingress")`
+
+The `SEC("tcx/ingress")` annotation tells libbpf that this program should be attached to the TCX ingress hook rather than the classic TC classifier. This is not just a naming convention; libbpf maps this section name to `BPF_PROG_TYPE_SCHED_CLS` with the appropriate attach type for TCX. The corresponding egress variant is `SEC("tcx/egress")`.
+
+Note that `SEC("tc")`, `SEC("classifier")`, and `SEC("action")` are now considered deprecated by libbpf in favor of the `tcx/*` section names.
+
+### Global Variables as Counters
+
+Instead of using a BPF map for counters, we use global variables (`stats_hits`, `classifier_hits`, `last_len`, etc.). The libbpf skeleton exposes these through `skel->bss->stats_hits`, which makes the user-space code simpler. This is fine for a single-CPU demo; for production use, you would want per-CPU maps to avoid data races.
+
+### Return Codes: `TCX_NEXT` vs `TCX_PASS`
+
+This is the heart of TCX composition:
+
+- `tcx_stats` returns `TCX_NEXT`, which means "I've done my work, now pass the packet to the next program in the chain." The chain continues executing.
+- `tcx_classifier` returns `TCX_PASS`, which is a terminal verdict: the packet is accepted and no further programs in the chain run.
+
+If we had placed `tcx_classifier` *before* `tcx_stats` in the chain, `tcx_stats` would never execute because `TCX_PASS` terminates the chain. Ordering matters, and TCX makes it explicit.
+
+## User-Space Loader: Attaching and Querying the Chain
+
+The user-space code demonstrates three key TCX operations: attaching programs, ordering them relative to each other, and querying the live chain.
+
+### Step 1: Attach the First Program
+
+```c
+classifier_link = bpf_program__attach_tcx(skel->progs.tcx_classifier,
+					 ifindex, NULL);
+```
+
+This attaches `tcx_classifier` to the TCX ingress hook on the specified interface. Passing `NULL` for options means "use defaults", so the program gets appended to the chain. At this point, the chain has one program.
+
+### Step 2: Insert the Second Program *Before* the First
+
+```c
+LIBBPF_OPTS(bpf_tcx_opts, before_opts,
+	.flags = BPF_F_BEFORE,
+	.relative_fd = bpf_program__fd(skel->progs.tcx_classifier));
+
+stats_link = bpf_program__attach_tcx(skel->progs.tcx_stats,
+				    ifindex, &before_opts);
+```
+
+The `bpf_tcx_opts` structure tells the kernel to insert `tcx_stats` *before* `tcx_classifier` in the chain. The `.relative_fd` field identifies the reference point, which is the fd of the already-attached classifier program. After this, the chain is: `tcx_stats` → `tcx_classifier`.
+
+You could equivalently use `BPF_F_AFTER` with a different reference to achieve the same ordering. The important point is that you express the desired order directly, rather than hoping that two numeric priorities sort correctly.
+
+### Step 3: Query the Chain
+
+```c
+LIBBPF_OPTS(bpf_prog_query_opts, query);
+
+query.count = 8;
+query.prog_ids = prog_ids;
+query.link_ids = link_ids;
+
+err = bpf_prog_query_opts(ifindex, BPF_TCX_INGRESS, &query);
+```
+
+After attachment, the loader queries the kernel for the live chain state. The returned data includes:
+
+- **`revision`**: A monotonically increasing counter that changes on every chain modification. This is the value you would pass as `expected_revision` if you wanted to perform atomic updates.
+- **`prog_ids[]`**: The BPF program IDs in chain order.
+- **`link_ids[]`**: The corresponding BPF link IDs.
+
+This allows any observer to determine exactly which programs are attached and in what order, which is invaluable for debugging multi-program pipelines.
+
+### Step 4: Generate Traffic and Read Counters
+
+The loader sends a UDP packet to `127.0.0.1` (port 9, discard) to trigger the chain, waits briefly, then reads the global variables to verify both programs executed:
+
+```c
+printf("  tcx_stats hits      : %llu\n",
+       (unsigned long long)skel->bss->stats_hits);
+printf("  tcx_classifier hits : %llu\n",
+       (unsigned long long)skel->bss->classifier_hits);
+```
+
+If both counters are 1, the chain worked as expected: `tcx_stats` ran first (recording metadata and returning `TCX_NEXT`), then `tcx_classifier` ran second (counting the packet and returning `TCX_PASS`).
+
+## Compilation and Execution
+
+This example requires Linux 6.6+ with TCX support and a recent libbpf.
+
+```bash
+cd bpf-developer-tutorial/src/50-tcx
+make
+sudo ./tcx_demo -i lo
+```
+
+Expected output:
+
+```text
+Attached TCX programs to lo (ifindex=1)
+TCX ingress chain revision: 3
+  slot 0: prog_id=812 link_id=901
+  slot 1: prog_id=811 link_id=900
+
+Counters:
+  tcx_stats hits      : 1
+  tcx_classifier hits : 1
+  last ifindex        : 1
+  last protocol       : 0x0800
+  last length         : 46
+```
+
+The revision is 3 because the chain was modified twice: once when `tcx_classifier` was attached (revision went from 0 to 1), and once when `tcx_stats` was inserted before it (revision went to 2). The query itself increments the revision to 3.
+
+If you want to inspect the attach behavior without traffic, add `-n`:
+
+```bash
+sudo ./tcx_demo -i lo -n
+```
+
+Use `-v` to enable libbpf debug output, which is helpful for seeing the low-level BPF syscall sequence.
+
+## How This Differs from Lesson 20 (Classic TC)
+
+[Lesson 20-tc](../20-tc/README.md) teaches the classic TC data path: creating a `clsact` qdisc, attaching a `SEC("tc")` program as a filter, and using `__sk_buff` for packet inspection. That lesson is still valuable because the **packet processing model** is identical: TCX programs receive the same `__sk_buff` context and use the same helpers for packet parsing.
+
+What TCX replaces is the **control plane**:
+
+| Aspect | Classic TC (Lesson 20) | TCX (Lesson 50) |
+|--------|----------------------|-----------------|
+| Attach mechanism | Netlink / `tc` CLI | `bpf_program__attach_tcx()` |
+| Ownership | None; anyone can `tc filter del` | BPF link; auto-detaches on fd close |
+| Ordering | Implicit priority numbers | Explicit `BPF_F_BEFORE` / `BPF_F_AFTER` |
+| Multi-program | Manual priority management | Built-in chain with revision tracking |
+| Section name | `SEC("tc")` | `SEC("tcx/ingress")` / `SEC("tcx/egress")` |
+| Kernel requirement | Any modern kernel | Linux 6.6+ |
+
+If you are building new libbpf-based networking tools, TCX is the recommended interface. Cilium has already migrated from classic tc to TCX for its dataplane.
+
+## Summary
+
+In this tutorial, we learned how TCX modernizes TC program attachment by replacing qdisc-based plumbing with BPF link semantics. We attached two ingress programs, controlled their execution order with `BPF_F_BEFORE`, queried the live chain with `bpf_prog_query_opts()`, and verified that both programs executed in the correct order. TCX provides safe ownership, explicit ordering, revision-aware updates, and coexistence with classic TC, making it the foundation for composable, multi-program traffic control in modern eBPF applications.
+
+If you'd like to learn more about eBPF, visit our tutorial code repository at <https://github.com/eunomia-bpf/bpf-developer-tutorial> or website <https://eunomia.dev/tutorials/> for more examples and complete tutorials.
+
+## References
+
+- [TCX kernel commit: fd-based tcx multi-prog infra with link support](https://lore.kernel.org/bpf/20230707172455.7634-3-daniel@iogearbox.net/)
+- [BPF_PROG_TYPE_SCHED_CLS documentation](https://docs.ebpf.io/linux/program-type/BPF_PROG_TYPE_SCHED_CLS/)
+- [bpf_program__attach_tcx libbpf API](https://docs.ebpf.io/ebpf-library/libbpf/userspace/bpf_program__attach_tcx/)
+- [Cilium TCX & Netkit update (BPFConf 2024)](https://bpfconf.ebpf.io/bpfconf2024/bpfconf2024_material/tcx_netkit_update_and_global_sk_iter.pdf)
+- [Generic multi-prog API, tcx links and meta device (BPFConf 2023)](http://oldvger.kernel.org/bpfconf2023_material/tcx_meta_netdev_borkmann.pdf)
+- <https://docs.kernel.org/bpf/>