mirror of
https://github.com/eunomia-bpf/bpf-developer-tutorial.git
synced 2026-11-01 05:05:38 +08:00
rename README to chinese documents
This commit is contained in:
Yusheng Zheng
@@ -1,43 +1,50 @@
|
||||
# 借助 eBPF 和 BTF，让用户态也能一次编译、到处运行
|
||||
# Expanding eBPF Compile Once, Run Everywhere(CO-RE) to Userspace Compatibility
|
||||
|
||||
在现代 Linux 系统中，eBPF（扩展的 Berkeley Packet Filter）是一项强大而灵活的技术。它允许在内核中运行沙盒化程序，类似于虚拟机环境，为扩展内核功能提供了一种既安全又不会导致系统崩溃或安全风险的方法。
|
||||
> Yusheng
|
||||
|
||||
eBPF 中的 “co-re” 代表“一次编译、到处运行”。这是其关键特征之一，用于解决 eBPF 程序在不同内核版本间兼容性的主要挑战。eBPF 的 CO-RE 功能可以实现在不同的内核版本上运行同一 eBPF 程序，而无需重新编译。
|
||||
eBPF, short for extended Berkeley Packet Filter, is a powerful and versatile technology used in modern Linux systems. It allows for the running of sandboxed programs in a virtual machine-like environment within the kernel, providing a safe way to extend the capabilities of the kernel without the risk of crashing the system or compromising security.
|
||||
|
||||
利用 eBPF 的 Uprobe 功能，可以追踪用户空间应用程序并访问其内部数据结构。然而，用户空间应用程序的 CO-RE 实践目前尚不完善。本文将介绍一种新方法，利用 CO-RE 为用户空间应用程序确保 eBPF 程序在不同应用版本间的兼容性，从而避免了多次编译的需求。例如，在从加密流量中捕获 SSL/TLS 明文数据时，你或许不需要为每个版本的 OpenSSL 维护一个单独的 eBPF 程序。
|
||||
Co-RE, standing for 'Compile Once, Run Everywhere', tackles the critical issue of eBPF program compatibility across diverse kernel versions. This feature allows eBPF programs to run on various kernel versions without the need for recompilation, simplifying deployment and maintenance.
|
||||
|
||||
为了在用户空间应用程序中实现eBPF的“一次编译、到处运行”(Co-RE)特性，我们需要利用BPF类型格式(BTF)来克服传统eBPF程序的一些限制。这种方法的关键在于为用户空间程序提供与内核类似的类型信息和兼容性支持，从而使得eBPF程序能够更灵活地应对不同版本的用户空间应用和库。
|
||||
With eBPF Uprobe, you can also trace userspace applications and access their internal data structures. However, the CO-RE is not designed for userspace applications. This blog will introduce how to leverage CO-RE for user-space applications, ensuring eBPF Uprobe programs remain compatible across different application versions without the need for multiple compilations.
|
||||
|
||||
本文是eBPF开发者教程的一部分，详细内容可访问[https://eunomia.dev/tutorials/](https://eunomia.dev/tutorials/)。本文完整的代码请查看 <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe> 。
|
||||
This approach may be particularly beneficial for tracing applications like OpenSSL, where maintaining separate eBPF programs for each version is impractical. With userspace eBPF runtimes like bpftime, you can also expand the CO-RE to more usecases, including extensions, networking, and dynamic patching, providing versatile and efficient solutions.
|
||||
|
||||
## 为什么我们需要CO-RE？
|
||||
To implement the Co-RE feature of eBPF in user-space applications, we also need to utilize the BPF Type Format (BTF) to overcome some of the limitations of traditional eBPF programs. The key to this approach lies in providing user-space programs with similar type information and compatibility support as the kernel, thereby enabling eBPF programs to more flexibly handle different versions of user-space applications and libraries.
|
||||
|
||||
- **内核依赖性**：传统的eBPF程序和它们被编译的特定Linux内核版本紧密耦合。这是因为它们依赖于内核的特定内部数据结构和API，这些可能在内核版本间变化。
|
||||
- **可移植性问题**：如果你想在带有不同内核版本的不同Linux系统上运行一个eBPF程序，你通常需要为每个内核版本重新编译eBPF程序，这是一个麻烦而低效的过程。
|
||||
This article is part of the eBPF Developer Tutorial, and for more detailed content, you can visit [https://eunomia.dev/tutorials/](https://eunomia.dev/tutorials/). The source code is available on the [https://github.com/eunomia-bpf/bpf-developer-tutorial](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe).
|
||||
|
||||
### Co-RE的解决方案
|
||||
## Why we need CO-RE?
|
||||
|
||||
- **抽象内核依赖性**：Co-RE使eBPF程序更具可移植性，通过使用BPF类型格式(BTF)和重定位来抽象特定的内核依赖。
|
||||
- **BPF类型格式（BTF）**：BTF提供了关于内核中数据结构和函数的丰富类型信息。这些元数据允许eBPF程序在运行时理解内核结构的布局。
|
||||
- **重定位**：编译支持Co-RE的eBPF程序包含在加载时解析的重定位。这些重定位根据运行内核的实际布局和地址调整程序对内核数据结构和函数的引用。
|
||||
- **Kernel Dependencies**: Traditional eBPF programs are tightly coupled with the specific Linux kernel version they are compiled for. This is because they rely on specific internal data structures and kernel APIs which can change between kernel versions.
|
||||
- **Portability Issues**: If you wanted to run an eBPF program on different Linux systems with different kernel versions, you'd traditionally have to recompile the eBPF program for each kernel version, which is a cumbersome and inefficient process.
|
||||
|
||||
### Co-RE的优点
|
||||
### The Co-RE Solution
|
||||
|
||||
1. **编写一次，任何地方运行**：编译有Co-RE的eBPF程序可以在不同的内核版本上运行，无需重新编译。这大大简化了在多样环境中部署和维护eBPF程序。
|
||||
2. **安全和稳定**：Co-RE保持了eBPF的安全性，确保程序不会导致内核崩溃，遵守安全约束。
|
||||
3. **简单的开发**：开发者不需要关注每个内核版本的具体情况，这简化了eBPF程序的开发。
|
||||
- **Abstracting Kernel Dependencies**: Co-RE enables eBPF programs to be more portable by abstracting away specific kernel dependencies. This is achieved through the use of BPF Type Format (BTF) and relocations.
|
||||
- **BPF Type Format (BTF)**: BTF provides rich type information about data structures and functions in the kernel. This metadata allows eBPF programs to understand the layout of kernel structures at runtime.
|
||||
- **Relocations**: eBPF programs compiled with Co-RE support contain relocations that are resolved at load time. These relocations adjust the program's references to kernel data structures and functions according to the actual layout and addresses in the running kernel.
|
||||
|
||||
## 用户空间应用程序CO-RE的问题
|
||||
### Advantages of Co-RE
|
||||
|
||||
eBPF也支持追踪用户空间应用程序。Uprobe是一个用户空间探针，允许对用户空间程序进行动态仪表装置。探针位置包括函数入口、特定偏移和函数返回。
|
||||
1. **Write Once, Run Anywhere**: eBPF programs compiled with Co-RE can run on different kernel versions without the need for recompilation. This greatly simplifies the deployment and maintenance of eBPF programs in diverse environments.
|
||||
2. **Safety and Stability**: Co-RE maintains the safety guarantees of eBPF, ensuring that programs do not crash the kernel and adhere to security constraints.
|
||||
3. **Ease of Development**: Developers don't need to worry about the specifics of each kernel version, which simplifies the development of eBPF programs.
|
||||
|
||||
BTF是为内核设计的，生成自vmlinux，它可以帮助eBPF程序方便地兼容不同的内核版本。但是，用户空间应用程序也需要CO-RE。例如，SSL/TLS uprobe被广泛用于从加密流量中捕获明文数据。它是用用户空间库实现的，如OpenSSL、GnuTLS、NSS等。用户空间应用程序和库也有各种版本，如果我们需要为每个版本编译和维护eBPF程序，那就会很复杂。
|
||||
## Problem: userspace application CO-RE
|
||||
|
||||
下面是一些新的工具和方法，可以帮助我们为用户空间应用程序启用CO-RE。
|
||||
The eBPF also supports tracing userspace applications. Uprobe is a user-space probe that allows dynamic instrumentation in user-space programs. The probe locations include function entry, specific offsets, and function returns.
|
||||
|
||||
## 用户空间程序的BTF
|
||||
The BTF is designed for the kernel and generated from vmlinux, it can help the eBPF program to be easily compatible with different kernel versions.
|
||||
|
||||
The userspace application, however, also need CO-RE. For example, the SSL/TLS uprobe is widely used to capture the plaintext data from the encrypted traffic. It is implemented with the userspace library, such as OpenSSL, GnuTLS, NSS, etc. The userspace application and libraries also has different versions, it would be complex if we need to compile and maintain the eBPF program for each version.
|
||||
|
||||
Let's see what will happen if CO-RE is not enabled for userspace applications, and how the BTF from userspace applications can solve this.
|
||||
|
||||
## No BTF for userspace program
|
||||
|
||||
This is a simple uprobe example, it can capture the function call and arguments of the `add_test` function in the userspace program. You can add `#define BPF_NO_PRESERVE_ACCESS_INDEX` in the `uprobe.bpf.c` to make sure the eBPF program can be compiled without BTF for `struct data`.
|
||||
|
||||
这是一个简单的uprobe例子，它可以捕获用户空间程序的`add_test`函数的调用和参数。你可以在`uprobe.bpf.c`中添加`#define BPF_NO_PRESERVE_ACCESS_INDEX`来确保eBPF程序可以在没有`struct data`的BTF的情况下编译。
|
||||
|
||||
```c
|
||||
#define BPF_NO_GLOBAL_DATA
|
||||
@@ -65,9 +72,9 @@ int BPF_UPROBE(add_test, struct data *d)
|
||||
char LICENSE[] SEC("license") = "Dual BSD/GPL";
|
||||
```
|
||||
|
||||
然后，我们有两个不同版本的用户空间程序，`examples/btf-base`和`examples/btf-base-new`。两个版本中的struct `data`是不同的。
|
||||
Then, we have two different versions of the userspace program, `examples/btf-base` and `examples/btf-base-new`. The struct `data` is different in the two versions.
|
||||
|
||||
`examples/btf-base`：
|
||||
`examples/btf-base`:
|
||||
|
||||
```c
|
||||
// use a different struct
|
||||
@@ -88,7 +95,7 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
```
|
||||
|
||||
`examples/btf-base-new`：
|
||||
`examples/btf-base-new`:
|
||||
|
||||
```c
|
||||
struct data {
|
||||
@@ -109,57 +116,59 @@ int main(int argc, char **argv) {
|
||||
}
|
||||
```
|
||||
|
||||
我们可以使用pahole和clang来生成每个版本的btf。制作示例并生成btf:
|
||||
We can use pahole and clang to generate the btf for each version of userspace applications. The pahole tool can simply generate BTF from the debug info: <https://linux.die.net/man/1/pahole>
|
||||
|
||||
make examples and generate btf for them:
|
||||
|
||||
```sh
|
||||
make -C example # it's like: pahole --btf_encode_detached base.btf btf-base.o
|
||||
```
|
||||
|
||||
然后我们执行eBPF程序和用户空间程序。 对于 `btf-base`：
|
||||
The we execute the eBPF program with the userspace program. for `btf-base`:
|
||||
|
||||
```sh
|
||||
sudo ./uprobe examples/btf-base
|
||||
```
|
||||
|
||||
也是用户空间程序：
|
||||
And also the userspace program:
|
||||
|
||||
```console
|
||||
$ examples/btf-base
|
||||
add_test(&d) = 4
|
||||
```
|
||||
|
||||
我们将看到：
|
||||
We will see:
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe\
|
||||
<...>-25458 [000] ...11 27694.081465: bpf_trace_printk: add_test(&d) 1 + 3 = 4
|
||||
```
|
||||
|
||||
对于 `btf-base-new`：
|
||||
For `btf-base-new`:
|
||||
|
||||
```sh
|
||||
sudo ./uprobe examples/btf-base-new
|
||||
```
|
||||
|
||||
同时也是用户空间程序：
|
||||
And also the userspace program:
|
||||
|
||||
```console
|
||||
$ examples/btf-base-new
|
||||
add_test(&d) = 4
|
||||
```
|
||||
|
||||
但我们可以看到：
|
||||
But we will see:
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe\
|
||||
<...>-25809 [001] ...11 27828.314224: bpf_trace_printk: add_test(&d) 1 + 2 = 3
|
||||
```
|
||||
|
||||
结果是不同的，因为两个版本中的struct `data`是不同的。eBPF程序无法与不同版本的用户空间程序兼容，我们获取到了错误的结构体偏移量，也会导致我们追踪失败。
|
||||
The result is different, because the struct `data` is different in the two versions. The eBPF program can't be compatible with different versions of the userspace program, so we cannot get the correct information.
|
||||
|
||||
## 使用用户空间程序的BTF
|
||||
## Use BTF for userspace program
|
||||
|
||||
在`uprobe.bpf.c`中注释掉`#define BPF_NO_PRESERVE_ACCESS_INDEX` ，以确保eBPF程序可以以`struct data`的BTF编译。
|
||||
Comment the `#define BPF_NO_PRESERVE_ACCESS_INDEX` in the `uprobe.bpf.c` to make sure the eBPF program can be compiled with BTF for `struct data`.
|
||||
|
||||
```c
|
||||
#define BPF_NO_GLOBAL_DATA
|
||||
@@ -182,6 +191,7 @@ struct data {
|
||||
#pragma clang attribute pop
|
||||
#endif
|
||||
|
||||
|
||||
SEC("uprobe/examples/btf-base:add_test")
|
||||
int BPF_UPROBE(add_test, struct data *d)
|
||||
{
|
||||
@@ -195,15 +205,15 @@ int BPF_UPROBE(add_test, struct data *d)
|
||||
char LICENSE[] SEC("license") = "Dual BSD/GPL";
|
||||
```
|
||||
|
||||
`struct data`的记录在eBPF程序中被保留下来。然后，我们可以使用 `btf-base.btf`来编译eBPF程序。
|
||||
The record of `struct data` is preserved in the eBPF program. Then, we can use the `btf-base.btf` to compile the eBPF program.
|
||||
|
||||
将用户btf与内核btf合并，这样我们就有了一个完整的内核和用户空间的btf:
|
||||
Merge user btf with kernel btf, so we have a complete btf for the kernel and userspace:
|
||||
|
||||
```sh
|
||||
./merge-btf /sys/kernel/btf/vmlinux examples/base.btf target-base.btf
|
||||
```
|
||||
|
||||
然后我们使用用户空间程序执行eBPF程序。 对于 `btf-base`：
|
||||
Then we execute the eBPF program with the userspace program. for `btf-base`:
|
||||
|
||||
```console
|
||||
$ sudo ./uprobe examples/btf-base target-base.btf
|
||||
@@ -215,15 +225,16 @@ libbpf: prog 'add_test': relo #2: patched insn #11 (ALU/ALU64) imm 4 -> 4
|
||||
...
|
||||
```
|
||||
|
||||
执行用户空间程序并获取结果：
|
||||
Execute the userspace program and get result:
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
|
||||
[sudo] password for yunwei37:
|
||||
<...>-26740 [001] ...11 28180.156220: bpf_trace_printk: add_test(&d) 1 + 3 = 4
|
||||
|
||||
```
|
||||
|
||||
还可以对另一个版本的用户空间程序`btf-base-new`做同样的操作:
|
||||
Also, we do the same for another version of the userspace program `btf-base-new`:
|
||||
|
||||
```console
|
||||
$ ./merge-btf /sys/kernel/btf/vmlinux examples/base-new.btf target-base-new.btf
|
||||
@@ -245,7 +256,7 @@ libbpf: elf: symbol address match for 'add_test' in 'examples/btf-base-new': 0x1
|
||||
Successfully started! Press Ctrl+C to stop.
|
||||
```
|
||||
|
||||
结果是正确的：
|
||||
The result is correct:
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
|
||||
@@ -253,46 +264,50 @@ $ sudo cat /sys/kernel/debug/tracing/trace_pipe
|
||||
<...>-26740 [001] ...11 28180.156220: bpf_trace_printk: add_test(&d) 1 + 3 = 4
|
||||
```
|
||||
|
||||
我们的 eBPF 追踪程序也几乎不需要进行任何修改，只需要把包含 kernel 和用户态结构体偏移量的 BTF 加载进来即可。这和旧版本内核上没有 btf 信息的使用方式是一样的:
|
||||
For complete source code, you can visit [https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe) for more details.
|
||||
|
||||
The eBPF uprobe tracing program almost doesn't need any modifications. We just need to load the BTF containing the offsets of kernel and user-space structures. This is the same usage as enabling CO-RE on older kernel versions without BTF information:
|
||||
|
||||
```c
|
||||
LIBBPF_OPTS(bpf_object_open_opts , opts,
|
||||
);
|
||||
LIBBPF_OPTS(bpf_uprobe_opts, uprobe_opts);
|
||||
if (argc != 3 && argc != 2) {
|
||||
fprintf(stderr, "Usage: %s <example-name> [<external-btf>]\n", argv[0]);
|
||||
return 1;
|
||||
}
|
||||
if (argc == 3)
|
||||
opts.btf_custom_path = argv[2];
|
||||
LIBBPF_OPTS(bpf_object_open_opts , opts,
|
||||
);
|
||||
LIBBPF_OPTS(bpf_uprobe_opts, uprobe_opts);
|
||||
if (argc != 3 && argc != 2) {
|
||||
fprintf(stderr, "Usage: %s <example-name> [<external-btf>]\n", argv[0]);
|
||||
return 1;
|
||||
}
|
||||
if (argc == 3)
|
||||
opts.btf_custom_path = argv[2];
|
||||
|
||||
/* Set up libbpf errors and debug info callback */
|
||||
libbpf_set_print(libbpf_print_fn);
|
||||
/* Set up libbpf errors and debug info callback */
|
||||
libbpf_set_print(libbpf_print_fn);
|
||||
|
||||
/* Cleaner handling of Ctrl-C */
|
||||
signal(SIGINT, sig_handler);
|
||||
signal(SIGTERM, sig_handler);
|
||||
/* Cleaner handling of Ctrl-C */
|
||||
signal(SIGINT, sig_handler);
|
||||
signal(SIGTERM, sig_handler);
|
||||
|
||||
/* Load and verify BPF application */
|
||||
skel = uprobe_bpf__open_opts(&opts);
|
||||
if (!skel) {
|
||||
fprintf(stderr, "Failed to open and load BPF skeleton\n");
|
||||
return 1;
|
||||
}
|
||||
/* Load and verify BPF application */
|
||||
skel = uprobe_bpf__open_opts(&opts);
|
||||
if (!skel) {
|
||||
fprintf(stderr, "Failed to open and load BPF skeleton\n");
|
||||
return 1;
|
||||
}
|
||||
```
|
||||
|
||||
实际上，btf 实现重定向需要两个部分，一个是 bpf 程序带的编译时的 btf 信息，一个是内核的 btf 信息。在实际加载 ebpf 程序的时候，libbpf 会根据当前内核上准确的 btf 信息，来修改可能存在错误的 ebpf 指令，确保在不同内核版本上能够兼容。
|
||||
In fact, the BTF implementation for relocation requires two parts: the compile-time BTF information carried by the BPF program, and the BTF information of the kernel when loading the eBPF program. When actually loading the eBPF program, libbpf will modify potentially incorrect eBPF instructions based on the accurate BTF information of the current kernel, ensuring compatibility across different kernel versions.
|
||||
|
||||
有趣的是，实际上 libbpf 并不区分这些 btf 信息来自用户态程序还是内核，因此我们只要把用户态的重定向信息一起提供给 libbpf 进行重定向，问题就解决了。
|
||||
Interestingly, libbpf does not differentiate whether these BTF information come from user-space programs or the kernel. Therefore, by merging the user-space BTF information with kernel BTF and provide them to libbpf, the problem is solved.
|
||||
|
||||
本文的工具和完整的代码在 <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe> 开源。
|
||||
And also, since the relocation is happened in userspace loader(like libbpf), both kernel eBPF runtime and userspace eBPF runtimes(Such as bpftime) can benefit from the CO-RE. bpftime (<https://github.com/eunomia-bpf/bpftime>) is an open-source user-space eBPF runtime based on LLVM JIT/AOT. It enables the execution of eBPF programs in user space, compatible with kernel-space eBPF. While supporting uprobes, syscall trace, and general plugin extensions, it avoids the context switching between kernel and user spaces, thereby enhancing the execution efficiency of uprobe programs. With the support of libbpf and BTF, bpftime can also dynamically extend user-space applications, achieving compatibility across different versions of user-space programs.
|
||||
|
||||
## 结论
|
||||
For more details about BTF relocation, you may refer to <https://nakryiko.com/posts/bpf-core-reference-guide/>
|
||||
|
||||
- **灵活性和兼容性**：在用户空间eBPF程序中使用 BTF 大大增强了它们在不同版本的用户空间应用程序和库之间的灵活性和兼容性。
|
||||
- **简化了复杂性**：这种方法显著减少了维护不同版本的用户空间应用程序的eBPF程序的复杂性，因为它消除了需要多个程序版本的需要。
|
||||
- **更广泛的应用**：这种方法在性能监控、安全和用户空间应用程序的调试等方面也可能能有更广泛的应用。bpftime（https://github.com/eunomia-bpf/bpftime） 是一个开源的基于 LLVM JIT/AOT 的用户态 eBPF 运行时，它可以在用户态运行 eBPF 程序，和内核态的 eBPF 兼容。它在支持 uprobe、syscall trace 和一般的插件扩展的同时，避免了内核态和用户态之间的上下文切换，从而提高了 uprobe 程序的执行效率。借助 libbpf 和 btf 的支持，bpftime 也可以更加动态的扩展用户态应用程序，实现在不同用户态程序版本之间的兼容性。
|
||||
## Conclusion
|
||||
|
||||
这个示例展示了 eBPF 在实践中可以将其强大的 CO-RE 功能扩展到更动态地处理用户空间应用的不同版本变化。
|
||||
- **Flexibility and Compatibility**: The use of BTF in user-space eBPF programs greatly enhances their flexibility and compatibility across different versions of user-space applications and libraries.
|
||||
- **Reduced Complexity**: This approach significantly reduces the complexity involved in maintaining eBPF programs for different versions of user-space applications, as it eliminates the need for multiple program versions.
|
||||
- **Potential for Broader Application**: While your example focused on SSL/TLS monitoring, this methodology may has broader applications in performance monitoring, security, and debugging of user-space applications.
|
||||
|
||||
如果你想了解更多关于eBPF知识和实践，你可以访问我们的教程代码库<https://github.com/eunomia-bpf/bpf-developer-tutorial>或者网站<https://eunomia.dev/tutorials/>获得更多示例和完整教程。
|
||||
This example showcases a significant advancement in the practical application of eBPF, extending its powerful features to more dynamically handle user-space applications in a Linux environment. It's a compelling solution for software engineers and system administrators dealing with the complexities of modern Linux systems.
|
||||
|
||||
If you want to learn more about eBPF knowledge and practices, you can visit our tutorial code repository <https://github.com/eunomia-bpf/bpf-developer-tutorial> or website <https://eunomia.dev/tutorials/> to get more examples and complete tutorials.
|
||||
|
||||
298
src/38-btf-uprobe/README.zh.md
Normal file
298
src/38-btf-uprobe/README.zh.md
Normal file
@@ -0,0 +1,298 @@
|
||||
# 借助 eBPF 和 BTF，让用户态也能一次编译、到处运行
|
||||
|
||||
在现代 Linux 系统中，eBPF（扩展的 Berkeley Packet Filter）是一项强大而灵活的技术。它允许在内核中运行沙盒化程序，类似于虚拟机环境，为扩展内核功能提供了一种既安全又不会导致系统崩溃或安全风险的方法。
|
||||
|
||||
eBPF 中的 “co-re” 代表“一次编译、到处运行”。这是其关键特征之一，用于解决 eBPF 程序在不同内核版本间兼容性的主要挑战。eBPF 的 CO-RE 功能可以实现在不同的内核版本上运行同一 eBPF 程序，而无需重新编译。
|
||||
|
||||
利用 eBPF 的 Uprobe 功能，可以追踪用户空间应用程序并访问其内部数据结构。然而，用户空间应用程序的 CO-RE 实践目前尚不完善。本文将介绍一种新方法，利用 CO-RE 为用户空间应用程序确保 eBPF 程序在不同应用版本间的兼容性，从而避免了多次编译的需求。例如，在从加密流量中捕获 SSL/TLS 明文数据时，你或许不需要为每个版本的 OpenSSL 维护一个单独的 eBPF 程序。
|
||||
|
||||
为了在用户空间应用程序中实现eBPF的“一次编译、到处运行”(Co-RE)特性，我们需要利用BPF类型格式(BTF)来克服传统eBPF程序的一些限制。这种方法的关键在于为用户空间程序提供与内核类似的类型信息和兼容性支持，从而使得eBPF程序能够更灵活地应对不同版本的用户空间应用和库。
|
||||
|
||||
本文是eBPF开发者教程的一部分，详细内容可访问[https://eunomia.dev/tutorials/](https://eunomia.dev/tutorials/)。本文完整的代码请查看 <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe> 。
|
||||
|
||||
## 为什么我们需要CO-RE？
|
||||
|
||||
- **内核依赖性**：传统的eBPF程序和它们被编译的特定Linux内核版本紧密耦合。这是因为它们依赖于内核的特定内部数据结构和API，这些可能在内核版本间变化。
|
||||
- **可移植性问题**：如果你想在带有不同内核版本的不同Linux系统上运行一个eBPF程序，你通常需要为每个内核版本重新编译eBPF程序，这是一个麻烦而低效的过程。
|
||||
|
||||
### Co-RE的解决方案
|
||||
|
||||
- **抽象内核依赖性**：Co-RE使eBPF程序更具可移植性，通过使用BPF类型格式(BTF)和重定位来抽象特定的内核依赖。
|
||||
- **BPF类型格式（BTF）**：BTF提供了关于内核中数据结构和函数的丰富类型信息。这些元数据允许eBPF程序在运行时理解内核结构的布局。
|
||||
- **重定位**：编译支持Co-RE的eBPF程序包含在加载时解析的重定位。这些重定位根据运行内核的实际布局和地址调整程序对内核数据结构和函数的引用。
|
||||
|
||||
### Co-RE的优点
|
||||
|
||||
1. **编写一次，任何地方运行**：编译有Co-RE的eBPF程序可以在不同的内核版本上运行，无需重新编译。这大大简化了在多样环境中部署和维护eBPF程序。
|
||||
2. **安全和稳定**：Co-RE保持了eBPF的安全性，确保程序不会导致内核崩溃，遵守安全约束。
|
||||
3. **简单的开发**：开发者不需要关注每个内核版本的具体情况，这简化了eBPF程序的开发。
|
||||
|
||||
## 用户空间应用程序CO-RE的问题
|
||||
|
||||
eBPF也支持追踪用户空间应用程序。Uprobe是一个用户空间探针，允许对用户空间程序进行动态仪表装置。探针位置包括函数入口、特定偏移和函数返回。
|
||||
|
||||
BTF是为内核设计的，生成自vmlinux，它可以帮助eBPF程序方便地兼容不同的内核版本。但是，用户空间应用程序也需要CO-RE。例如，SSL/TLS uprobe被广泛用于从加密流量中捕获明文数据。它是用用户空间库实现的，如OpenSSL、GnuTLS、NSS等。用户空间应用程序和库也有各种版本，如果我们需要为每个版本编译和维护eBPF程序，那就会很复杂。
|
||||
|
||||
下面是一些新的工具和方法，可以帮助我们为用户空间应用程序启用CO-RE。
|
||||
|
||||
## 用户空间程序的BTF
|
||||
|
||||
这是一个简单的uprobe例子，它可以捕获用户空间程序的`add_test`函数的调用和参数。你可以在`uprobe.bpf.c`中添加`#define BPF_NO_PRESERVE_ACCESS_INDEX`来确保eBPF程序可以在没有`struct data`的BTF的情况下编译。
|
||||
|
||||
```c
|
||||
#define BPF_NO_GLOBAL_DATA
|
||||
#define BPF_NO_PRESERVE_ACCESS_INDEX
|
||||
#include <vmlinux.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include <bpf/bpf_tracing.h>
|
||||
|
||||
struct data {
|
||||
int a;
|
||||
int c;
|
||||
int d;
|
||||
};
|
||||
|
||||
SEC("uprobe/examples/btf-base:add_test")
|
||||
int BPF_UPROBE(add_test, struct data *d)
|
||||
{
|
||||
int a = 0, c = 0;
|
||||
bpf_probe_read_user(&a, sizeof(a), &d->a);
|
||||
bpf_probe_read_user(&c, sizeof(c), &d->c);
|
||||
bpf_printk("add_test(&d) %d + %d = %d\n", a, c, a + c);
|
||||
return a + c;
|
||||
}
|
||||
|
||||
char LICENSE[] SEC("license") = "Dual BSD/GPL";
|
||||
```
|
||||
|
||||
然后，我们有两个不同版本的用户空间程序，`examples/btf-base`和`examples/btf-base-new`。两个版本中的struct `data`是不同的。
|
||||
|
||||
`examples/btf-base`：
|
||||
|
||||
```c
|
||||
// use a different struct
|
||||
struct data {
|
||||
int a;
|
||||
int c;
|
||||
int d;
|
||||
};
|
||||
|
||||
int add_test(struct data *d) {
|
||||
return d->a + d->c;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
struct data d = {1, 3, 4};
|
||||
printf("add_test(&d) = %d\n", add_test(&d));
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
`examples/btf-base-new`：
|
||||
|
||||
```c
|
||||
struct data {
|
||||
int a;
|
||||
int b;
|
||||
int c;
|
||||
int d;
|
||||
};
|
||||
|
||||
int add_test(struct data *d) {
|
||||
return d->a + d->c;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
struct data d = {1, 2, 3, 4};
|
||||
printf("add_test(&d) = %d\n", add_test(&d));
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
我们可以使用pahole和clang来生成每个版本的btf。制作示例并生成btf:
|
||||
|
||||
```sh
|
||||
make -C example # it's like: pahole --btf_encode_detached base.btf btf-base.o
|
||||
```
|
||||
|
||||
然后我们执行eBPF程序和用户空间程序。 对于 `btf-base`：
|
||||
|
||||
```sh
|
||||
sudo ./uprobe examples/btf-base
|
||||
```
|
||||
|
||||
也是用户空间程序：
|
||||
|
||||
```console
|
||||
$ examples/btf-base
|
||||
add_test(&d) = 4
|
||||
```
|
||||
|
||||
我们将看到：
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe\
|
||||
<...>-25458 [000] ...11 27694.081465: bpf_trace_printk: add_test(&d) 1 + 3 = 4
|
||||
```
|
||||
|
||||
对于 `btf-base-new`：
|
||||
|
||||
```sh
|
||||
sudo ./uprobe examples/btf-base-new
|
||||
```
|
||||
|
||||
同时也是用户空间程序：
|
||||
|
||||
```console
|
||||
$ examples/btf-base-new
|
||||
add_test(&d) = 4
|
||||
```
|
||||
|
||||
但我们可以看到：
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe\
|
||||
<...>-25809 [001] ...11 27828.314224: bpf_trace_printk: add_test(&d) 1 + 2 = 3
|
||||
```
|
||||
|
||||
结果是不同的，因为两个版本中的struct `data`是不同的。eBPF程序无法与不同版本的用户空间程序兼容，我们获取到了错误的结构体偏移量，也会导致我们追踪失败。
|
||||
|
||||
## 使用用户空间程序的BTF
|
||||
|
||||
在`uprobe.bpf.c`中注释掉`#define BPF_NO_PRESERVE_ACCESS_INDEX` ，以确保eBPF程序可以以`struct data`的BTF编译。
|
||||
|
||||
```c
|
||||
#define BPF_NO_GLOBAL_DATA
|
||||
// #define BPF_NO_PRESERVE_ACCESS_INDEX
|
||||
#include <vmlinux.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include <bpf/bpf_tracing.h>
|
||||
|
||||
#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
|
||||
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
|
||||
#endif
|
||||
|
||||
struct data {
|
||||
int a;
|
||||
int c;
|
||||
int d;
|
||||
};
|
||||
|
||||
#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
|
||||
#pragma clang attribute pop
|
||||
#endif
|
||||
|
||||
SEC("uprobe/examples/btf-base:add_test")
|
||||
int BPF_UPROBE(add_test, struct data *d)
|
||||
{
|
||||
int a = 0, c = 0;
|
||||
bpf_probe_read_user(&a, sizeof(a), &d->a);
|
||||
bpf_probe_read_user(&c, sizeof(c), &d->c);
|
||||
bpf_printk("add_test(&d) %d + %d = %d\n", a, c, a + c);
|
||||
return a + c;
|
||||
}
|
||||
|
||||
char LICENSE[] SEC("license") = "Dual BSD/GPL";
|
||||
```
|
||||
|
||||
`struct data`的记录在eBPF程序中被保留下来。然后，我们可以使用 `btf-base.btf`来编译eBPF程序。
|
||||
|
||||
将用户btf与内核btf合并，这样我们就有了一个完整的内核和用户空间的btf:
|
||||
|
||||
```sh
|
||||
./merge-btf /sys/kernel/btf/vmlinux examples/base.btf target-base.btf
|
||||
```
|
||||
|
||||
然后我们使用用户空间程序执行eBPF程序。 对于 `btf-base`：
|
||||
|
||||
```console
|
||||
$ sudo ./uprobe examples/btf-base target-base.btf
|
||||
...
|
||||
libbpf: prog 'add_test': relo #1: patched insn #4 (ALU/ALU64) imm 0 -> 0
|
||||
libbpf: prog 'add_test': relo #2: <byte_off> [7] struct data.c (0:1 @ offset 4)
|
||||
libbpf: prog 'add_test': relo #2: matching candidate #0 <byte_off> [133110] struct data.c (0:1 @ offset 4)
|
||||
libbpf: prog 'add_test': relo #2: patched insn #11 (ALU/ALU64) imm 4 -> 4
|
||||
...
|
||||
```
|
||||
|
||||
执行用户空间程序并获取结果：
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
|
||||
[sudo] password for yunwei37:
|
||||
<...>-26740 [001] ...11 28180.156220: bpf_trace_printk: add_test(&d) 1 + 3 = 4
|
||||
```
|
||||
|
||||
还可以对另一个版本的用户空间程序`btf-base-new`做同样的操作:
|
||||
|
||||
```console
|
||||
$ ./merge-btf /sys/kernel/btf/vmlinux examples/base-new.btf target-base-new.btf
|
||||
$ sudo ./uprobe examples/btf-base-new target-base-new.btf
|
||||
....
|
||||
libbpf: sec 'uprobe/examples/btf-base:add_test': found 3 CO-RE relocations
|
||||
libbpf: CO-RE relocating [2] struct pt_regs: found target candidate [357] struct pt_regs in [vmlinux]
|
||||
libbpf: prog 'add_test': relo #0: <byte_off> [2] struct pt_regs.di (0:14 @ offset 112)
|
||||
libbpf: prog 'add_test': relo #0: matching candidate #0 <byte_off> [357] struct pt_regs.di (0:14 @ offset 112)
|
||||
libbpf: prog 'add_test': relo #0: patched insn #0 (LDX/ST/STX) off 112 -> 112
|
||||
libbpf: CO-RE relocating [7] struct data: found target candidate [133110] struct data in [vmlinux]
|
||||
libbpf: prog 'add_test': relo #1: <byte_off> [7] struct data.a (0:0 @ offset 0)
|
||||
libbpf: prog 'add_test': relo #1: matching candidate #0 <byte_off> [133110] struct data.a (0:0 @ offset 0)
|
||||
libbpf: prog 'add_test': relo #1: patched insn #4 (ALU/ALU64) imm 0 -> 0
|
||||
libbpf: prog 'add_test': relo #2: <byte_off> [7] struct data.c (0:1 @ offset 4)
|
||||
libbpf: prog 'add_test': relo #2: matching candidate #0 <byte_off> [133110] struct data.c (0:2 @ offset 8)
|
||||
libbpf: prog 'add_test': relo #2: patched insn #11 (ALU/ALU64) imm 4 -> 8
|
||||
libbpf: elf: symbol address match for 'add_test' in 'examples/btf-base-new': 0x1140
|
||||
Successfully started! Press Ctrl+C to stop.
|
||||
```
|
||||
|
||||
结果是正确的：
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
|
||||
[sudo] password for yunwei37:
|
||||
<...>-26740 [001] ...11 28180.156220: bpf_trace_printk: add_test(&d) 1 + 3 = 4
|
||||
```
|
||||
|
||||
我们的 eBPF 追踪程序也几乎不需要进行任何修改，只需要把包含 kernel 和用户态结构体偏移量的 BTF 加载进来即可。这和旧版本内核上没有 btf 信息的使用方式是一样的:
|
||||
|
||||
```c
|
||||
LIBBPF_OPTS(bpf_object_open_opts , opts,
|
||||
);
|
||||
LIBBPF_OPTS(bpf_uprobe_opts, uprobe_opts);
|
||||
if (argc != 3 && argc != 2) {
|
||||
fprintf(stderr, "Usage: %s <example-name> [<external-btf>]\n", argv[0]);
|
||||
return 1;
|
||||
}
|
||||
if (argc == 3)
|
||||
opts.btf_custom_path = argv[2];
|
||||
|
||||
/* Set up libbpf errors and debug info callback */
|
||||
libbpf_set_print(libbpf_print_fn);
|
||||
|
||||
/* Cleaner handling of Ctrl-C */
|
||||
signal(SIGINT, sig_handler);
|
||||
signal(SIGTERM, sig_handler);
|
||||
|
||||
/* Load and verify BPF application */
|
||||
skel = uprobe_bpf__open_opts(&opts);
|
||||
if (!skel) {
|
||||
fprintf(stderr, "Failed to open and load BPF skeleton\n");
|
||||
return 1;
|
||||
}
|
||||
```
|
||||
|
||||
实际上，btf 实现重定向需要两个部分，一个是 bpf 程序带的编译时的 btf 信息，一个是内核的 btf 信息。在实际加载 ebpf 程序的时候，libbpf 会根据当前内核上准确的 btf 信息，来修改可能存在错误的 ebpf 指令，确保在不同内核版本上能够兼容。
|
||||
|
||||
有趣的是，实际上 libbpf 并不区分这些 btf 信息来自用户态程序还是内核，因此我们只要把用户态的重定向信息一起提供给 libbpf 进行重定向，问题就解决了。
|
||||
|
||||
本文的工具和完整的代码在 <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe> 开源。
|
||||
|
||||
## 结论
|
||||
|
||||
- **灵活性和兼容性**：在用户空间eBPF程序中使用 BTF 大大增强了它们在不同版本的用户空间应用程序和库之间的灵活性和兼容性。
|
||||
- **简化了复杂性**：这种方法显著减少了维护不同版本的用户空间应用程序的eBPF程序的复杂性，因为它消除了需要多个程序版本的需要。
|
||||
- **更广泛的应用**：这种方法在性能监控、安全和用户空间应用程序的调试等方面也可能能有更广泛的应用。bpftime（https://github.com/eunomia-bpf/bpftime） 是一个开源的基于 LLVM JIT/AOT 的用户态 eBPF 运行时，它可以在用户态运行 eBPF 程序，和内核态的 eBPF 兼容。它在支持 uprobe、syscall trace 和一般的插件扩展的同时，避免了内核态和用户态之间的上下文切换，从而提高了 uprobe 程序的执行效率。借助 libbpf 和 btf 的支持，bpftime 也可以更加动态的扩展用户态应用程序，实现在不同用户态程序版本之间的兼容性。
|
||||
|
||||
这个示例展示了 eBPF 在实践中可以将其强大的 CO-RE 功能扩展到更动态地处理用户空间应用的不同版本变化。
|
||||
|
||||
如果你想了解更多关于eBPF知识和实践，你可以访问我们的教程代码库<https://github.com/eunomia-bpf/bpf-developer-tutorial>或者网站<https://eunomia.dev/tutorials/>获得更多示例和完整教程。
|
||||
@@ -1,313 +0,0 @@
|
||||
# Expanding eBPF Compile Once, Run Everywhere(CO-RE) to Userspace Compatibility
|
||||
|
||||
> Yusheng
|
||||
|
||||
eBPF, short for extended Berkeley Packet Filter, is a powerful and versatile technology used in modern Linux systems. It allows for the running of sandboxed programs in a virtual machine-like environment within the kernel, providing a safe way to extend the capabilities of the kernel without the risk of crashing the system or compromising security.
|
||||
|
||||
Co-RE, standing for 'Compile Once, Run Everywhere', tackles the critical issue of eBPF program compatibility across diverse kernel versions. This feature allows eBPF programs to run on various kernel versions without the need for recompilation, simplifying deployment and maintenance.
|
||||
|
||||
With eBPF Uprobe, you can also trace userspace applications and access their internal data structures. However, the CO-RE is not designed for userspace applications. This blog will introduce how to leverage CO-RE for user-space applications, ensuring eBPF Uprobe programs remain compatible across different application versions without the need for multiple compilations.
|
||||
|
||||
This approach may be particularly beneficial for tracing applications like OpenSSL, where maintaining separate eBPF programs for each version is impractical. With userspace eBPF runtimes like bpftime, you can also expand the CO-RE to more usecases, including extensions, networking, and dynamic patching, providing versatile and efficient solutions.
|
||||
|
||||
To implement the Co-RE feature of eBPF in user-space applications, we also need to utilize the BPF Type Format (BTF) to overcome some of the limitations of traditional eBPF programs. The key to this approach lies in providing user-space programs with similar type information and compatibility support as the kernel, thereby enabling eBPF programs to more flexibly handle different versions of user-space applications and libraries.
|
||||
|
||||
This article is part of the eBPF Developer Tutorial, and for more detailed content, you can visit [https://eunomia.dev/tutorials/](https://eunomia.dev/tutorials/). The source code is available on the [https://github.com/eunomia-bpf/bpf-developer-tutorial](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe).
|
||||
|
||||
## Why we need CO-RE?
|
||||
|
||||
- **Kernel Dependencies**: Traditional eBPF programs are tightly coupled with the specific Linux kernel version they are compiled for. This is because they rely on specific internal data structures and kernel APIs which can change between kernel versions.
|
||||
- **Portability Issues**: If you wanted to run an eBPF program on different Linux systems with different kernel versions, you'd traditionally have to recompile the eBPF program for each kernel version, which is a cumbersome and inefficient process.
|
||||
|
||||
### The Co-RE Solution
|
||||
|
||||
- **Abstracting Kernel Dependencies**: Co-RE enables eBPF programs to be more portable by abstracting away specific kernel dependencies. This is achieved through the use of BPF Type Format (BTF) and relocations.
|
||||
- **BPF Type Format (BTF)**: BTF provides rich type information about data structures and functions in the kernel. This metadata allows eBPF programs to understand the layout of kernel structures at runtime.
|
||||
- **Relocations**: eBPF programs compiled with Co-RE support contain relocations that are resolved at load time. These relocations adjust the program's references to kernel data structures and functions according to the actual layout and addresses in the running kernel.
|
||||
|
||||
### Advantages of Co-RE
|
||||
|
||||
1. **Write Once, Run Anywhere**: eBPF programs compiled with Co-RE can run on different kernel versions without the need for recompilation. This greatly simplifies the deployment and maintenance of eBPF programs in diverse environments.
|
||||
2. **Safety and Stability**: Co-RE maintains the safety guarantees of eBPF, ensuring that programs do not crash the kernel and adhere to security constraints.
|
||||
3. **Ease of Development**: Developers don't need to worry about the specifics of each kernel version, which simplifies the development of eBPF programs.
|
||||
|
||||
## Problem: userspace application CO-RE
|
||||
|
||||
The eBPF also supports tracing userspace applications. Uprobe is a user-space probe that allows dynamic instrumentation in user-space programs. The probe locations include function entry, specific offsets, and function returns.
|
||||
|
||||
The BTF is designed for the kernel and generated from vmlinux, it can help the eBPF program to be easily compatible with different kernel versions.
|
||||
|
||||
The userspace application, however, also need CO-RE. For example, the SSL/TLS uprobe is widely used to capture the plaintext data from the encrypted traffic. It is implemented with the userspace library, such as OpenSSL, GnuTLS, NSS, etc. The userspace application and libraries also has different versions, it would be complex if we need to compile and maintain the eBPF program for each version.
|
||||
|
||||
Let's see what will happen if CO-RE is not enabled for userspace applications, and how the BTF from userspace applications can solve this.
|
||||
|
||||
## No BTF for userspace program
|
||||
|
||||
This is a simple uprobe example, it can capture the function call and arguments of the `add_test` function in the userspace program. You can add `#define BPF_NO_PRESERVE_ACCESS_INDEX` in the `uprobe.bpf.c` to make sure the eBPF program can be compiled without BTF for `struct data`.
|
||||
|
||||
|
||||
```c
|
||||
#define BPF_NO_GLOBAL_DATA
|
||||
#define BPF_NO_PRESERVE_ACCESS_INDEX
|
||||
#include <vmlinux.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include <bpf/bpf_tracing.h>
|
||||
|
||||
struct data {
|
||||
int a;
|
||||
int c;
|
||||
int d;
|
||||
};
|
||||
|
||||
SEC("uprobe/examples/btf-base:add_test")
|
||||
int BPF_UPROBE(add_test, struct data *d)
|
||||
{
|
||||
int a = 0, c = 0;
|
||||
bpf_probe_read_user(&a, sizeof(a), &d->a);
|
||||
bpf_probe_read_user(&c, sizeof(c), &d->c);
|
||||
bpf_printk("add_test(&d) %d + %d = %d\n", a, c, a + c);
|
||||
return a + c;
|
||||
}
|
||||
|
||||
char LICENSE[] SEC("license") = "Dual BSD/GPL";
|
||||
```
|
||||
|
||||
Then, we have two different versions of the userspace program, `examples/btf-base` and `examples/btf-base-new`. The struct `data` is different in the two versions.
|
||||
|
||||
`examples/btf-base`:
|
||||
|
||||
```c
|
||||
// use a different struct
|
||||
struct data {
|
||||
int a;
|
||||
int c;
|
||||
int d;
|
||||
};
|
||||
|
||||
int add_test(struct data *d) {
|
||||
return d->a + d->c;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
struct data d = {1, 3, 4};
|
||||
printf("add_test(&d) = %d\n", add_test(&d));
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
`examples/btf-base-new`:
|
||||
|
||||
```c
|
||||
struct data {
|
||||
int a;
|
||||
int b;
|
||||
int c;
|
||||
int d;
|
||||
};
|
||||
|
||||
int add_test(struct data *d) {
|
||||
return d->a + d->c;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
struct data d = {1, 2, 3, 4};
|
||||
printf("add_test(&d) = %d\n", add_test(&d));
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
We can use pahole and clang to generate the btf for each version of userspace applications. The pahole tool can simply generate BTF from the debug info: <https://linux.die.net/man/1/pahole>
|
||||
|
||||
make examples and generate btf for them:
|
||||
|
||||
```sh
|
||||
make -C example # it's like: pahole --btf_encode_detached base.btf btf-base.o
|
||||
```
|
||||
|
||||
The we execute the eBPF program with the userspace program. for `btf-base`:
|
||||
|
||||
```sh
|
||||
sudo ./uprobe examples/btf-base
|
||||
```
|
||||
|
||||
And also the userspace program:
|
||||
|
||||
```console
|
||||
$ examples/btf-base
|
||||
add_test(&d) = 4
|
||||
```
|
||||
|
||||
We will see:
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe\
|
||||
<...>-25458 [000] ...11 27694.081465: bpf_trace_printk: add_test(&d) 1 + 3 = 4
|
||||
```
|
||||
|
||||
For `btf-base-new`:
|
||||
|
||||
```sh
|
||||
sudo ./uprobe examples/btf-base-new
|
||||
```
|
||||
|
||||
And also the userspace program:
|
||||
|
||||
```console
|
||||
$ examples/btf-base-new
|
||||
add_test(&d) = 4
|
||||
```
|
||||
|
||||
But we will see:
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe\
|
||||
<...>-25809 [001] ...11 27828.314224: bpf_trace_printk: add_test(&d) 1 + 2 = 3
|
||||
```
|
||||
|
||||
The result is different, because the struct `data` is different in the two versions. The eBPF program can't be compatible with different versions of the userspace program, so we cannot get the correct information.
|
||||
|
||||
## Use BTF for userspace program
|
||||
|
||||
Comment the `#define BPF_NO_PRESERVE_ACCESS_INDEX` in the `uprobe.bpf.c` to make sure the eBPF program can be compiled with BTF for `struct data`.
|
||||
|
||||
```c
|
||||
#define BPF_NO_GLOBAL_DATA
|
||||
// #define BPF_NO_PRESERVE_ACCESS_INDEX
|
||||
#include <vmlinux.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include <bpf/bpf_tracing.h>
|
||||
|
||||
#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
|
||||
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
|
||||
#endif
|
||||
|
||||
struct data {
|
||||
int a;
|
||||
int c;
|
||||
int d;
|
||||
};
|
||||
|
||||
#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
|
||||
#pragma clang attribute pop
|
||||
#endif
|
||||
|
||||
|
||||
SEC("uprobe/examples/btf-base:add_test")
|
||||
int BPF_UPROBE(add_test, struct data *d)
|
||||
{
|
||||
int a = 0, c = 0;
|
||||
bpf_probe_read_user(&a, sizeof(a), &d->a);
|
||||
bpf_probe_read_user(&c, sizeof(c), &d->c);
|
||||
bpf_printk("add_test(&d) %d + %d = %d\n", a, c, a + c);
|
||||
return a + c;
|
||||
}
|
||||
|
||||
char LICENSE[] SEC("license") = "Dual BSD/GPL";
|
||||
```
|
||||
|
||||
The record of `struct data` is preserved in the eBPF program. Then, we can use the `btf-base.btf` to compile the eBPF program.
|
||||
|
||||
Merge user btf with kernel btf, so we have a complete btf for the kernel and userspace:
|
||||
|
||||
```sh
|
||||
./merge-btf /sys/kernel/btf/vmlinux examples/base.btf target-base.btf
|
||||
```
|
||||
|
||||
Then we execute the eBPF program with the userspace program. for `btf-base`:
|
||||
|
||||
```console
|
||||
$ sudo ./uprobe examples/btf-base target-base.btf
|
||||
...
|
||||
libbpf: prog 'add_test': relo #1: patched insn #4 (ALU/ALU64) imm 0 -> 0
|
||||
libbpf: prog 'add_test': relo #2: <byte_off> [7] struct data.c (0:1 @ offset 4)
|
||||
libbpf: prog 'add_test': relo #2: matching candidate #0 <byte_off> [133110] struct data.c (0:1 @ offset 4)
|
||||
libbpf: prog 'add_test': relo #2: patched insn #11 (ALU/ALU64) imm 4 -> 4
|
||||
...
|
||||
```
|
||||
|
||||
Execute the userspace program and get result:
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
|
||||
[sudo] password for yunwei37:
|
||||
<...>-26740 [001] ...11 28180.156220: bpf_trace_printk: add_test(&d) 1 + 3 = 4
|
||||
|
||||
```
|
||||
|
||||
Also, we do the same for another version of the userspace program `btf-base-new`:
|
||||
|
||||
```console
|
||||
$ ./merge-btf /sys/kernel/btf/vmlinux examples/base-new.btf target-base-new.btf
|
||||
$ sudo ./uprobe examples/btf-base-new target-base-new.btf
|
||||
....
|
||||
libbpf: sec 'uprobe/examples/btf-base:add_test': found 3 CO-RE relocations
|
||||
libbpf: CO-RE relocating [2] struct pt_regs: found target candidate [357] struct pt_regs in [vmlinux]
|
||||
libbpf: prog 'add_test': relo #0: <byte_off> [2] struct pt_regs.di (0:14 @ offset 112)
|
||||
libbpf: prog 'add_test': relo #0: matching candidate #0 <byte_off> [357] struct pt_regs.di (0:14 @ offset 112)
|
||||
libbpf: prog 'add_test': relo #0: patched insn #0 (LDX/ST/STX) off 112 -> 112
|
||||
libbpf: CO-RE relocating [7] struct data: found target candidate [133110] struct data in [vmlinux]
|
||||
libbpf: prog 'add_test': relo #1: <byte_off> [7] struct data.a (0:0 @ offset 0)
|
||||
libbpf: prog 'add_test': relo #1: matching candidate #0 <byte_off> [133110] struct data.a (0:0 @ offset 0)
|
||||
libbpf: prog 'add_test': relo #1: patched insn #4 (ALU/ALU64) imm 0 -> 0
|
||||
libbpf: prog 'add_test': relo #2: <byte_off> [7] struct data.c (0:1 @ offset 4)
|
||||
libbpf: prog 'add_test': relo #2: matching candidate #0 <byte_off> [133110] struct data.c (0:2 @ offset 8)
|
||||
libbpf: prog 'add_test': relo #2: patched insn #11 (ALU/ALU64) imm 4 -> 8
|
||||
libbpf: elf: symbol address match for 'add_test' in 'examples/btf-base-new': 0x1140
|
||||
Successfully started! Press Ctrl+C to stop.
|
||||
```
|
||||
|
||||
The result is correct:
|
||||
|
||||
```console
|
||||
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
|
||||
[sudo] password for yunwei37:
|
||||
<...>-26740 [001] ...11 28180.156220: bpf_trace_printk: add_test(&d) 1 + 3 = 4
|
||||
```
|
||||
|
||||
For complete source code, you can visit [https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe](https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe) for more details.
|
||||
|
||||
The eBPF uprobe tracing program almost doesn't need any modifications. We just need to load the BTF containing the offsets of kernel and user-space structures. This is the same usage as enabling CO-RE on older kernel versions without BTF information:
|
||||
|
||||
```c
|
||||
LIBBPF_OPTS(bpf_object_open_opts , opts,
|
||||
);
|
||||
LIBBPF_OPTS(bpf_uprobe_opts, uprobe_opts);
|
||||
if (argc != 3 && argc != 2) {
|
||||
fprintf(stderr, "Usage: %s <example-name> [<external-btf>]\n", argv[0]);
|
||||
return 1;
|
||||
}
|
||||
if (argc == 3)
|
||||
opts.btf_custom_path = argv[2];
|
||||
|
||||
/* Set up libbpf errors and debug info callback */
|
||||
libbpf_set_print(libbpf_print_fn);
|
||||
|
||||
/* Cleaner handling of Ctrl-C */
|
||||
signal(SIGINT, sig_handler);
|
||||
signal(SIGTERM, sig_handler);
|
||||
|
||||
/* Load and verify BPF application */
|
||||
skel = uprobe_bpf__open_opts(&opts);
|
||||
if (!skel) {
|
||||
fprintf(stderr, "Failed to open and load BPF skeleton\n");
|
||||
return 1;
|
||||
}
|
||||
```
|
||||
|
||||
In fact, the BTF implementation for relocation requires two parts: the compile-time BTF information carried by the BPF program, and the BTF information of the kernel when loading the eBPF program. When actually loading the eBPF program, libbpf will modify potentially incorrect eBPF instructions based on the accurate BTF information of the current kernel, ensuring compatibility across different kernel versions.
|
||||
|
||||
Interestingly, libbpf does not differentiate whether these BTF information come from user-space programs or the kernel. Therefore, by merging the user-space BTF information with kernel BTF and provide them to libbpf, the problem is solved.
|
||||
|
||||
And also, since the relocation is happened in userspace loader(like libbpf), both kernel eBPF runtime and userspace eBPF runtimes(Such as bpftime) can benefit from the CO-RE. bpftime (<https://github.com/eunomia-bpf/bpftime>) is an open-source user-space eBPF runtime based on LLVM JIT/AOT. It enables the execution of eBPF programs in user space, compatible with kernel-space eBPF. While supporting uprobes, syscall trace, and general plugin extensions, it avoids the context switching between kernel and user spaces, thereby enhancing the execution efficiency of uprobe programs. With the support of libbpf and BTF, bpftime can also dynamically extend user-space applications, achieving compatibility across different versions of user-space programs.
|
||||
|
||||
For more details about BTF relocation, you may refer to <https://nakryiko.com/posts/bpf-core-reference-guide/>
|
||||
|
||||
## Conclusion
|
||||
|
||||
- **Flexibility and Compatibility**: The use of BTF in user-space eBPF programs greatly enhances their flexibility and compatibility across different versions of user-space applications and libraries.
|
||||
- **Reduced Complexity**: This approach significantly reduces the complexity involved in maintaining eBPF programs for different versions of user-space applications, as it eliminates the need for multiple program versions.
|
||||
- **Potential for Broader Application**: While your example focused on SSL/TLS monitoring, this methodology may has broader applications in performance monitoring, security, and debugging of user-space applications.
|
||||
|
||||
This example showcases a significant advancement in the practical application of eBPF, extending its powerful features to more dynamically handle user-space applications in a Linux environment. It's a compelling solution for software engineers and system administrators dealing with the complexities of modern Linux systems.
|
||||
|
||||
If you want to learn more about eBPF knowledge and practices, you can visit our tutorial code repository <https://github.com/eunomia-bpf/bpf-developer-tutorial> or website <https://eunomia.dev/tutorials/> to get more examples and complete tutorials.
|
||||
Reference in New Issue
Block a user