kernel/bpf-developer-tutorial
1
0
Fork 0
mirror of https://github.com/eunomia-bpf/bpf-developer-tutorial.git synced 2026-02-03 18:24:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
f56e271d1963c369f95249440d9ed9c045ce4c59
bpf-developer-tutorial/print.html

6088 lines
484 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE HTML>
<html lang="en" class="sidebar-visible no-js light">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>bpf-developer-tutorial</title>
<meta name="robots" content="noindex" />
<!-- Custom HTML head -->
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff" />
<link rel="icon" href="favicon.svg">
<link rel="shortcut icon" href="favicon.png">
<link rel="stylesheet" href="css/variables.css">
<link rel="stylesheet" href="css/general.css">
<link rel="stylesheet" href="css/chrome.css">
<link rel="stylesheet" href="css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" href="highlight.css">
<link rel="stylesheet" href="tomorrow-night.css">
<link rel="stylesheet" href="ayu-highlight.css">
<!-- Custom theme stylesheets -->
</head>
<body>
<div id="body-container">
<!-- Provide site root to javascript -->
<script>
var path_to_root = "";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";
</script>
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script>
try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script>
var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('no-js')
html.classList.remove('light')
html.classList.add(theme);
html.classList.add('js');
</script>
<!-- Hide / unhide sidebar before it is displayed -->
<script>
var html = document.querySelector('html');
var sidebar = null;
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
}
html.classList.remove('sidebar-visible');
html.classList.add("sidebar-" + sidebar);
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded affix "><li class="part-title">eBPF 实践教程：基于 libbpf 和 CO-RE</li><li class="chapter-item expanded "><a href="0-introduce/index.html"><strong aria-hidden="true">1.</strong> eBPF 入门开发实践教程一：介绍 eBPF 的基本概念、常见的开发工具</a></li><li class="chapter-item expanded "><a href="1-helloworld/index.html"><strong aria-hidden="true">2.</strong> eBPF 入门开发实践教程二：Hello World，基本框架和开发流程</a></li><li class="chapter-item expanded "><a href="2-kprobe-unlink/index.html"><strong aria-hidden="true">3.</strong> eBPF 入门开发实践教程二：在 eBPF 中使用 kprobe 监测捕获 unlink 系统调用</a></li><li class="chapter-item expanded "><a href="3-fentry-unlink/index.html"><strong aria-hidden="true">4.</strong> eBPF 入门开发实践教程三：在 eBPF 中使用 fentry 监测捕获 unlink 系统调用</a></li><li class="chapter-item expanded "><a href="4-opensnoop/index.html"><strong aria-hidden="true">5.</strong> eBPF 入门开发实践教程四：在 eBPF 中捕获进程打开文件的系统调用集合，使用全局变量过滤进程 pid</a></li><li class="chapter-item expanded "><a href="5-uprobe-bashreadline/index.html"><strong aria-hidden="true">6.</strong> eBPF 入门开发实践教程五：在 eBPF 中使用 uprobe 捕获 bash 的 readline 函数调用</a></li><li class="chapter-item expanded "><a href="6-sigsnoop/index.html"><strong aria-hidden="true">7.</strong> eBPF 入门开发实践教程六：捕获进程发送信号的系统调用集合，使用 hash map 保存状态</a></li><li class="chapter-item expanded "><a href="7-execsnoop/index.html"><strong aria-hidden="true">8.</strong> eBPF 入门实践教程七：捕获进程执行/退出时间，通过 perf event array 向用户态打印输出</a></li><li class="chapter-item expanded "><a href="8-exitsnoop/index.html"><strong aria-hidden="true">9.</strong> eBPF 入门开发实践教程八：在 eBPF 中使用 exitsnoop 监控进程退出事件，使用 ring buffer 向用户态打印输出</a></li><li class="chapter-item expanded "><a href="9-runqlat/index.html"><strong aria-hidden="true">10.</strong> eBPF 入门开发实践教程九：一个 Linux 内核 BPF 程序，通过柱状图来总结调度程序运行队列延迟，显示任务等待运行在 CPU 上的时间长度</a></li><li class="chapter-item expanded "><a href="10-hardirqs/index.html"><strong aria-hidden="true">11.</strong> eBPF 入门开发实践教程十：在 eBPF 中使用 hardirqs 或 softirqs 捕获中断事件</a></li><li class="chapter-item expanded "><a href="11-bootstrap/index.html"><strong aria-hidden="true">12.</strong> eBPF 入门开发实践教程十一：在 eBPF 中使用 bootstrap 开发用户态程序并跟踪 exec() 和 exit() 系统调用</a></li><li class="chapter-item expanded "><a href="13-tcpconnlat/index.html"><strong aria-hidden="true">13.</strong> eBPF入门实践教程：使用 libbpf-bootstrap 开发程序统计 TCP 连接延时</a></li><li class="chapter-item expanded "><a href="13-tcpconnlat/tcpconnlat.html"><strong aria-hidden="true">14.</strong> eBPF 入门实践教程：编写 eBPF 程序 tcpconnlat 测量 tcp 连接延时</a></li><li class="chapter-item expanded "><a href="14-tcpstates/index.html"><strong aria-hidden="true">15.</strong> eBPF入门实践教程：使用 libbpf-bootstrap 开发程序统计 TCP 连接延时</a></li><li class="chapter-item expanded "><a href="15-tcprtt/index.html"><strong aria-hidden="true">16.</strong> eBPF 入门实践教程：编写 eBPF 程序 Tcprtt 测量 TCP 连接的往返时间</a></li><li class="chapter-item expanded "><a href="16-memleak/index.html"><strong aria-hidden="true">17.</strong> eBPF 入门实践教程：编写 eBPF 程序 Memleak 监控内存泄漏</a></li><li class="chapter-item expanded "><a href="17-biopattern/index.html"><strong aria-hidden="true">18.</strong> eBPF 入门实践教程：编写 eBPF 程序 Biopattern: 统计随机/顺序磁盘 I/O</a></li><li class="chapter-item expanded "><a href="18-further-reading/index.html"><strong aria-hidden="true">19.</strong> 更多的参考资料</a></li><li class="chapter-item expanded "><a href="19-lsm-connect/index.html"><strong aria-hidden="true">20.</strong> eBPF 入门实践教程：使用 LSM 进行安全检测防御</a></li><li class="chapter-item expanded "><a href="20-tc/index.html"><strong aria-hidden="true">21.</strong> eBPF 入门实践教程：使用 eBPF 进行 tc 流量控制</a></li><li class="chapter-item expanded affix "><li class="part-title">bcc 开发者教程</li><li class="chapter-item expanded "><a href="bcc-documents/kernel-versions.html"><strong aria-hidden="true">22.</strong> BPF Features by Linux Kernel Version</a></li><li class="chapter-item expanded "><a href="bcc-documents/kernel_config.html"><strong aria-hidden="true">23.</strong> Kernel Configuration for BPF Features</a></li><li class="chapter-item expanded "><a href="bcc-documents/reference_guide.html"><strong aria-hidden="true">24.</strong> bcc Reference Guide</a></li><li class="chapter-item expanded "><a href="bcc-documents/special_filtering.html"><strong aria-hidden="true">25.</strong> Special Filtering</a></li><li class="chapter-item expanded "><a href="bcc-documents/tutorial.html"><strong aria-hidden="true">26.</strong> bcc Tutorial</a></li><li class="chapter-item expanded "><a href="bcc-documents/tutorial_bcc_python_developer.html"><strong aria-hidden="true">27.</strong> bcc Python Developer Tutorial</a></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle"></div>
</nav>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky bordered">
<div class="left-buttons">
<button id="sidebar-toggle" class="icon-button" type="button" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</button>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">bpf-developer-tutorial</h1>
<div class="right-buttons">
<a href="print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="ebpf-入门开发实践教程一介绍-ebpf-的基本概念常见的开发工具"><a class="header" href="#ebpf-入门开发实践教程一介绍-ebpf-的基本概念常见的开发工具">eBPF 入门开发实践教程一：介绍 eBPF 的基本概念、常见的开发工具</a></h1>
<h2 id="1-ebpf简介安全和有效地扩展内核"><a class="header" href="#1-ebpf简介安全和有效地扩展内核">1. eBPF简介：安全和有效地扩展内核</a></h2>
<p>eBPF 是一项革命性的技术，起源于 Linux 内核，可以在操作系统的内核中运行沙盒程序。它被用来安全和有效地扩展内核的功能，而不需要改变内核的源代码或加载内核模块。eBPF 通过允许在操作系统内运行沙盒程序，应用程序开发人员可以在运行时，可编程地向操作系统动态添加额外的功能。然后，操作系统保证安全和执行效率，就像在即时编译（JIT）编译器和验证引擎的帮助下进行本地编译一样。eBPF 程序在内核版本之间是可移植的，并且可以自动更新，从而避免了工作负载中断和节点重启。</p>
<p>今天，eBPF被广泛用于各类场景：在现代数据中心和云原生环境中，可以提供高性能的网络包处理和负载均衡；以非常低的资源开销，做到对多种细粒度指标的可观测性，帮助应用程序开发人员跟踪应用程序，为性能故障排除提供洞察力；保障应用程序和容器运行时的安全执行，等等。可能性是无穷的，而 eBPF 在操作系统内核中所释放的创新才刚刚开始[3]。</p>
<h3 id="ebpf-的未来内核的-javascript-可编程接口"><a class="header" href="#ebpf-的未来内核的-javascript-可编程接口">eBPF 的未来：内核的 JavaScript 可编程接口</a></h3>
<p>对于浏览器而言，JavaScript 的引入带来的可编程性开启了一场巨大的革命，使浏览器发展成为几乎独立的操作系统。现在让我们回到 eBPF：为了理解 eBPF 对 Linux 内核的可编程性影响，对 Linux 内核的结构以及它如何与应用程序和硬件进行交互有一个高层次的理解是有帮助的[4]。</p>
<p><img src="0-introduce/kernel-arch.webp" alt="kernel-arch" /></p>
<p>Linux 内核的主要目的是抽象出硬件或虚拟硬件，并提供一个一致的API（系统调用），允许应用程序运行和共享资源。为了实现这个目的，我们维护了一系列子系统和层，以分配这些责任[5]。每个子系统通常允许某种程度的配置，以考虑到用户的不同需求。如果不能配置所需的行为，就需要改变内核，从历史上看，改变内核的行为，或者让用户编写的程序能够在内核中运行，就有两种选择:</p>
<div class="table-wrapper"><table><thead><tr><th>本地支持内核模块</th><th>写一个内核模块</th></tr></thead><tbody>
<tr><td>改变内核源代码，并说服Linux内核社区相信这种改变是必要的。等待几年，让新的内核版本成为一种商品。</td><td>定期修复它，因为每个内核版本都可能破坏它。由于缺乏安全边界，冒着破坏你的Linux内核的风险</td></tr>
</tbody></table>
</div>
<p>实际上，两种方案都不常用，前者成本太高，后者则几乎没有可移植性。</p>
<p>有了 eBPF，就有了一个新的选择，可以重新编程 Linux 内核的行为，而不需要改变内核的源代码或加载内核模块，同时保证在不同内核版本之间一定程度上的行为一致性和兼容性、以及安全性[6]。为了实现这个目的，eBPF 程序也需要有一套对应的 API，允许用户定义的应用程序运行和共享资源 --- 换句话说，某种意义上讲 eBPF 虚拟机也提供了一套类似于系统调用的机制，借助 eBPF 和用户态通信的机制，Wasm 虚拟机和用户态应用也可以获得这套“系统调用”的完整使用权，一方面能可编程地扩展传统的系统调用的能力，另一方面能在网络、文件系统等许多层次实现更高效的可编程 IO 处理。</p>
<p><img src="0-introduce/new-os-model.jpg" alt="new-os" /></p>
<p>正如上图所示，当今的 Linux 内核正在向一个新的内核模型演化：用户定义的应用程序可以在内核态和用户态同时执行，用户态通过传统的系统调用访问系统资源，内核态则通过 BPF Helper Calls 和系统的各个部分完成交互。截止 2023 年初，内核中的 eBPF 虚拟机中已经有 220 多个Helper 系统接口，涵盖了非常多的应用场景。</p>
<p>值得注意的是，BPF Helper Call 和系统调用二者并不是竞争关系，它们的编程模型和有性能优势的场景完全不同，也不会完全替代对方。对 Wasm 和 Wasi 相关生态来说，情况也类似，专门设计的 wasi 接口需要经历一个漫长的标准化过程，但可能在特定场景能为用户态应用获取更佳的性能和可移植性保证，而 eBPF 在保证沙箱本质和可移植性的前提下，可以提供一个快速灵活的扩展系统接口的方案。</p>
<p>目前的 eBPF 仍然处于早期阶段，但是借助当前 eBPF 提供的内核接口和用户态交互的能力，经由 Wasm-bpf 的系统接口转换，Wasm 虚拟机中的应用已经几乎有能力获取内核以及用户态任意一个函数调用的数据和返回值（kprobe，uprobe...）；以很低的代价收集和理解所有系统调用，并获取所有网络操作的数据包和套接字级别的数据（tracepoint，socket...）；在网络包处理解决方案中添加额外的协议分析器，并轻松地编程任何转发逻辑（XDP，TC...），以满足不断变化的需求，而无需离开Linux内核的数据包处理环境。</p>
<p>不仅如此，eBPF 还有能力往用户空间任意进程的任意地址写入数据（bpf_probe_write_user[7]），有限度地修改内核函数的返回值（bpf_override_return[8]），甚至在内核态直接执行某些系统调用[9]；所幸的是，eBPF 在加载进内核之前对字节码会进行严格的安全检查，确保没有内存越界等操作，同时，许多可能会扩大攻击面、带来安全风险的功能都是需要在编译内核时明确选择启用才能使用的；在 Wasm 虚拟机将字节码加载进内核之前，也可以明确选择启用或者禁用某些 eBPF 功能，以确保沙箱的安全性。</p>
<h2 id="2-关于如何学习-ebpf-相关的开发的一些建议"><a class="header" href="#2-关于如何学习-ebpf-相关的开发的一些建议">2. 关于如何学习 eBPF 相关的开发的一些建议</a></h2>
<p>本文不会对 eBPF 的原理做更详细的介绍，不过这里有一个学习规划和参考资料，也许会有一些价值：</p>
<h3 id="ebpf-入门5-7h"><a class="header" href="#ebpf-入门5-7h">eBPF 入门（5-7h）</a></h3>
<ul>
<li>Google 或者其他搜索引擎查找：eBPF</li>
<li>询问 ChatGPT 之类的东西：eBPF 是什么？</li>
</ul>
<p>推荐：</p>
<ul>
<li>阅读 ebpf 简介：<a href="https://ebpf.io/">https://ebpf.io/</a> （30min）</li>
<li>简要了解一下 ebpf 内核相关文档：<a href="https://prototype-kernel.readthedocs.io/en/latest/bpf/">https://prototype-kernel.readthedocs.io/en/latest/bpf/</a> （知道有问题去哪里查询，30min）</li>
<li>阅读 ebpf 中文入门指南：<a href="https://www.modb.pro/db/391570">https://www.modb.pro/db/391570</a> （1h）</li>
<li>有大量的参考资料：<a href="https://github.com/zoidbergwill/awesome-ebpf">https://github.com/zoidbergwill/awesome-ebpf</a> （2-3h）</li>
<li>可以选自己感兴趣的 PPT 翻一翻：<a href="https://github.com/gojue/ebpf-slide">https://github.com/gojue/ebpf-slide</a> （1-2h）</li>
</ul>
<p>回答三个问题：</p>
<ol>
<li>了解 eBPF 是什么东西？为啥要有这个玩意，不能用内核模块？</li>
<li>它有什么功能？能在 Linux 内核里面完成哪些事情？有哪些 eBPF 程序的类型和 helper（不需要知道全部，但是需要知道去哪里找）？</li>
<li>能拿来做什么？比如说在哪些场景中进行运用？网络、安全、可观测性？</li>
</ol>
<h3 id="了解如何开发-ebpf-程序10-15h"><a class="header" href="#了解如何开发-ebpf-程序10-15h">了解如何开发 eBPF 程序（10-15h）</a></h3>
<p>了解并尝试一下 eBPF 开发框架：</p>
<ul>
<li>BCC 开发各类小工具的例子：<a href="https://github.com/iovisor/bcc/blob/master/docs/tutorial_bcc_python_developer.md">https://github.com/iovisor/bcc/blob/master/docs/tutorial_bcc_python_developer.md</a> （跑一遍，3-4h）</li>
<li>libbpf 的一些例子：<a href="https://github.com/libbpf/libbpf-bootstrap">https://github.com/libbpf/libbpf-bootstrap</a> （选感兴趣的运行一下，并阅读一下源代码，2h）</li>
<li>基于 libbpf 和 eunomia-bpf 的教程：<a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> （阅读 1-10 的部分，3-4h）</li>
</ul>
<p>其他开发框架：Go 语言或者 Rust 语言，请自行搜索并且尝试（0-2h）</p>
<p>有任何问题或者想了解的东西，不管是不是和本项目相关，都可以在本项目的 discussions 里面开始讨论。</p>
<p>回答一些问题，并且进行一些尝试（2-5h）：</p>
<ol>
<li>如何开发一个最简单的 eBPF 程序？</li>
<li>如何用 eBPF 追踪一个内核功能或函数？有很多种方法，举出对应的代码；</li>
<li>有哪些方案能通过用户态和内核态通信？如何从用户态向内核态传送信息？如何从内核态向用户态传递信息？举出代码示例；</li>
<li>编写一个你自己的 eBPF 程序，实现一个功能；</li>
<li>eBPF 程序的整个生命周期里面，分别在用户态和内核态做了哪些事情？</li>
</ol>
<h2 id="3-如何使用ebpf编程"><a class="header" href="#3-如何使用ebpf编程">3. 如何使用eBPF编程</a></h2>
<p>原始的eBPF程序编写是非常繁琐和困难的。为了改变这一现状，llvm于2015年推出了可以将由高级语言编写的代码编译为eBPF字节码的功能，同时，eBPF 社区将 <code>bpf()</code> 等原始的系统调用进行了初步地封装，给出了<code>libbpf</code>库。这些库会包含将字节码加载到内核中的函数以及一些其他的关键函数。在Linux的源码包的<code>samples/bpf/</code>目录下，有大量Linux提供的基于<code>libbpf</code>的eBPF样例代码。</p>
<p>一个典型的基于 <code>libbpf</code> 的eBPF程序具有<code>*_kern.c</code>和<code>*_user.c</code>两个文件，<code>*_kern.c</code>中书写在内核中的挂载点以及处理函数，<code>*_user.c</code>中书写用户态代码，完成内核态代码注入以及与用户交互的各种任务。 更为详细的教程可以参考<a href="https://www.bilibili.com/video/BV1f54y1h74r?spm_id_from=333.999.0.0">该视频</a>然而由于该方法仍然较难理解且入门存在一定的难度，因此现阶段的eBPF程序开发大多基于一些工具，比如：</p>
<ul>
<li>BCC</li>
<li>BPFtrace</li>
<li>libbpf-bootstrap</li>
<li>Go eBPF library</li>
</ul>
<p>以及还有比较新的工具，例如 <code>eunomia-bpf</code>.</p>
<h2 id="编写-ebpf-程序"><a class="header" href="#编写-ebpf-程序">编写 eBPF 程序</a></h2>
<p>eBPF 程序由内核态部分和用户态部分构成。内核态部分包含程序的实际逻辑，用户态部分负责加载和管理内核态部分。使用 eunomia-bpf 开发工具，只需编写内核态部分的代码。</p>
<p>内核态部分的代码需要符合 eBPF 的语法和指令集。eBPF 程序主要由若干个函数组成，每个函数都有其特定的作用。可以使用的函数类型包括：</p>
<ul>
<li>kprobe：插探函数，在指定的内核函数前或后执行。</li>
<li>tracepoint：跟踪点函数，在指定的内核跟踪点处执行。</li>
<li>raw_tracepoint：原始跟踪点函数，在指定的内核原始跟踪点处执行。</li>
<li>xdp：网络数据处理函数，拦截和处理网络数据包。</li>
<li>perf_event：性能事件函数，用于处理内核性能事件。</li>
<li>kretprobe：函数返回插探函数，在指定的内核函数返回时执行。</li>
<li>tracepoint_return：跟踪点函数返回，在指定的内核跟踪点返回时执行。</li>
<li>raw_tracepoint_return：原始跟踪点函数返回，在指定的内核原始跟踪</li>
</ul>
<h3 id="bcc"><a class="header" href="#bcc">BCC</a></h3>
<p>BCC全称为BPF Compiler Collection，该项目是一个python库，
包含了完整的编写、编译、和加载BPF程序的工具链，以及用于调试和诊断性能问题的工具。</p>
<p>自2015年发布以来，BCC经过上百位贡献者地不断完善后，目前已经包含了大量随时可用的跟踪工具。<a href="https://github.com/iovisor/bcc/blob/master/docs/tutorial.md">其官方项目库</a>
提供了一个方便上手的教程，用户可以快速地根据教程完成BCC入门工作。</p>
<p>用户可以在BCC上使用Python、Lua等高级语言进行编程。
相较于使用C语言直接编程，这些高级语言具有极大的便捷性，用户只需要使用C来设计内核中的
BPF程序，其余包括编译、解析、加载等工作在内，均可由BCC完成。</p>
<p>然而使用BCC存在一个缺点便是在于其兼容性并不好。基于BCC的
eBPF程序每次执行时候都需要进行编译，编译则需要用户配置相关的头文件和对应实现。在实际应用中，
相信大家也会有体会，编译依赖问题是一个很棘手的问题。也正是因此，在本项目的开发中我们放弃了BCC，
选择了可以做到一次编译-多次运行的libbpf-bootstrap工具。</p>
<h3 id="ebpf-go-library"><a class="header" href="#ebpf-go-library">eBPF Go library</a></h3>
<p>eBPF Go库提供了一个通用的eBPF库，它解耦了获取 eBPF 字节码的过程和 eBPF 程序的加载和管理，并实现了类似 libbpf 一样的 CO- 功能。eBPF程序通常是通过编写高级语言创建的，然后使用clang/LLVM编译器编译为eBPF字节码。</p>
<h3 id="libbpf"><a class="header" href="#libbpf">libbpf</a></h3>
<p><code>libbpf-bootstrap</code>是一个基于<code>libbpf</code>库的BPF开发脚手架，从其
<a href="https://github.com/libbpf/libbpf-bootstrap">github</a> 上可以得到其源码。</p>
<p><code>libbpf-bootstrap</code>综合了BPF社区过去多年的实践，为开发者提了一个现代化的、便捷的工作流，实
现了一次编译，重复使用的目的。</p>
<p>基于<code>libbpf-bootstrap</code>的BPF程序对于源文件有一定的命名规则，
用于生成内核态字节码的bpf文件以<code>.bpf.c</code>结尾，用户态加载字节码的文件以<code>.c</code>结尾，且这两个文件的
前缀必须相同。</p>
<p>基于<code>libbpf-bootstrap</code>的BPF程序在编译时会先将<code>*.bpf.c</code>文件编译为
对应的<code>.o</code>文件，然后根据此文件生成<code>skeleton</code>文件，即<code>*.skel.h</code>，这个文件会包含内核态中定义的一些
数据结构，以及用于装载内核态代码的关键函数。在用户态代码<code>include</code>此文件之后调用对应的装载函数即可将
字节码装载到内核中。同样的，<code>libbpf-bootstrap</code>也有非常完备的入门教程，用户可以在<a href="https://nakryiko.com/posts/libbpf-bootstrap/">该处</a>
得到详细的入门操作介绍。</p>
<h3 id="eunomia-bpf"><a class="header" href="#eunomia-bpf">eunomia-bpf</a></h3>
<p>开发、构建和分发 eBPF 一直以来都是一个高门槛的工作，使用 BCC、bpftrace 等工具开发效率高、可移植性好，但是分发部署时需要安装 LLVM、Clang等编译环境，每次运行的时候执行本地或远程编译过程，资源消耗较大；使用原生的 CO-RE libbpf时又需要编写不少用户态加载代码来帮助 eBPF 程序正确加载和从内核中获取上报的信息，同时对于 eBPF 程序的分发、管理也没有很好地解决方案。</p>
<p><a href="https://github.com/eunomia-bpf/eunomia-bpf">eunomia-bpf</a> 是一个开源的 eBPF 动态加载运行时和开发工具链，是为了简化 eBPF 程序的开发、构建、分发、运行而设计的，基于 libbpf 的 CO-RE 轻量级开发框架。</p>
<p>使用 eunomia-bpf ，可以：</p>
<ul>
<li>在编写 eBPF 程序或工具时只编写内核态代码，自动获取内核态导出信息，并作为模块动态加载；</li>
<li>使用 WASM 进行用户态交互程序的开发，在 WASM 虚拟机内部控制整个 eBPF 程序的加载和执行，以及处理相关数据；</li>
<li>eunomia-bpf 可以将预编译的 eBPF 程序打包为通用的 JSON 或 WASM 模块，跨架构和内核版本进行分发，无需重新编译即可动态加载运行。</li>
</ul>
<p>eunomia-bpf 由一个编译工具链和一个运行时库组成, 对比传统的 BCC、原生 libbpf 等框架，大幅简化了 eBPF 程序的开发流程，在大多数时候只需编写内核态代码，即可轻松构建、打包、发布完整的 eBPF 应用，同时内核态 eBPF 代码保证和主流的 libbpf, libbpfgo, libbpf-rs 等开发框架的 100% 兼容性。需要编写用户态代码的时候，也可以借助 Webassembly 实现通过多种语言进行用户态开发。和 bpftrace 等脚本工具相比, eunomia-bpf 保留了类似的便捷性, 同时不仅局限于 trace 方面, 可以用于更多的场景, 如网络、安全等等。</p>
<blockquote>
<ul>
<li>eunomia-bpf 项目 Github 地址: <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></li>
<li>gitee 镜像: <a href="https://gitee.com/anolis/eunomia">https://gitee.com/anolis/eunomia</a></li>
</ul>
</blockquote>
<h2 id="参考资料"><a class="header" href="#参考资料">参考资料</a></h2>
<ul>
<li>eBPF 介绍：<a href="https://ebpf.io/">https://ebpf.io/</a></li>
<li>BPF Compiler Collection (BCC)：<a href="https://github.com/iovisor/bcc">https://github.com/iovisor/bcc</a></li>
<li>eunomia-bpf：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></li>
</ul>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程二hello-world基本框架和开发流程"><a class="header" href="#ebpf-入门开发实践教程二hello-world基本框架和开发流程">eBPF 入门开发实践教程二：Hello World，基本框架和开发流程</a></h1>
<p>在本篇博客中，我们将深入探讨eBPF（Extended Berkeley Packet Filter）的基本框架和开发流程。eBPF是一种在Linux内核上运行的强大网络和性能分析工具，它为开发者提供了在内核运行时动态加载、更新和运行用户定义代码的能力。这使得开发者可以实现高效、安全的内核级别的网络监控、性能分析和故障排查等功能。</p>
<p>本文是eBPF入门开发实践教程的第二篇，我们将重点关注如何编写一个简单的eBPF程序，并通过实际例子演示整个开发流程。在阅读本教程之前，建议您先学习第一篇教程，以便对eBPF的基本概念有个大致的了解。</p>
<p>在开发eBPF程序时，有多种开发框架可供选择，如 BCC（BPF Compiler Collection）libbpf、cilium/ebpf、eunomia-bpf 等。虽然不同工具的特点各异，但它们的基本开发流程大致相同。在接下来的内容中，我们将深入了解这些流程，并以 Hello World 程序为例，带领读者逐步掌握eBPF开发的基本技巧。</p>
<p>本教程将帮助您了解eBPF程序的基本结构、编译和加载过程、用户空间与内核空间的交互方式以及调试与优化技巧。通过学习本教程，您将掌握eBPF开发的基本知识，并为后续进一步学习和实践奠定坚实的基础。</p>
<h2 id="ebpf开发环境准备与基本开发流程"><a class="header" href="#ebpf开发环境准备与基本开发流程">eBPF开发环境准备与基本开发流程</a></h2>
<p>在开始编写eBPF程序之前，我们需要准备一个合适的开发环境，并了解eBPF程序的基本开发流程。本部分将详细介绍这些内容。</p>
<h3 id="安装必要的软件和工具"><a class="header" href="#安装必要的软件和工具">安装必要的软件和工具</a></h3>
<p>要开发eBPF程序，您需要安装以下软件和工具：</p>
<ul>
<li>Linux 内核：由于eBPF是内核技术，因此您需要具备较新版本的Linux内核（推荐4.8及以上版本），以支持eBPF功能。</li>
<li>LLVM 和 Clang：这些工具用于编译eBPF程序。安装最新版本的LLVM和Clang可以确保您获得最佳的eBPF支持。</li>
</ul>
<p>eBPF 程序主要由两部分构成：内核态部分和用户态部分。内核态部分包含 eBPF 程序的实际逻辑，用户态部分负责加载、运行和监控内核态程序。当您选择了合适的开发框架后，如 BCC（BPF Compiler Collection）、libbpf、cilium/ebpf或eunomia-bpf等，您可以开始进行用户态和内核态程序的开发。以 BCC 工具为例，我们将介绍 eBPF 程序的基本开发流程：</p>
<p>当您选择了合适的开发框架后，如BCC（BPF Compiler Collection）、libbpf、cilium/ebpf或eunomia-bpf等，您可以开始进行用户态和内核态程序的开发。以BCC工具为例，我们将介绍eBPF程序的基本开发流程：</p>
<ol>
<li>安装BCC工具：根据您的Linux发行版，按照BCC官方文档的指南安装BCC工具和相关依赖。</li>
<li>编写eBPF程序（C语言）：使用C语言编写一个简单的eBPF程序，例如Hello World程序。该程序可以在内核空间执行并完成特定任务，如统计网络数据包数量。</li>
<li>编写用户态程序（Python或C等）：使用Python、C等语言编写用户态程序，用于加载、运行eBPF程序以及与之交互。在这个程序中，您需要使用BCC提供的API来加载和操作内核态的eBPF程序。</li>
<li>编译eBPF程序：使用BCC工具，将C语言编写的eBPF程序编译成内核可以执行的字节码。BCC会在运行时动态从源码编译eBPF程序。</li>
<li>加载并运行eBPF程序：在用户态程序中，使用BCC提供的API加载编译好的eBPF程序到内核空间，然后运行该程序。</li>
<li>与eBPF程序交互：用户态程序通过BCC提供的API与eBPF程序交互，实现数据收集、分析和展示等功能。例如，您可以使用BCC API读取eBPF程序中的map数据，以获取网络数据包统计信息。</li>
<li>卸载eBPF程序：当不再需要eBPF程序时，用户态程序应使用BCC API将其从内核空间卸载。</li>
<li>调试与优化：使用 bpftool 等工具进行eBPF程序的调试和优化，提高程序性能和稳定性。</li>
</ol>
<p>通过以上流程，您可以使用BCC工具开发、编译、运行和调试eBPF程序。请注意，其他框架（如libbpf、cilium/ebpf和eunomia-bpf）的开发流程大致相似但略有不同，因此在选择框架时，请参考相应的官方文档和示例。</p>
<p>通过这个过程，你可以开发出一个能够在内核中运行的 eBPF 程序。eunomia-bpf 是一个开源的 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。它基于 libbpf 的 CO-RE 轻量级开发框架，支持通过用户态 WASM 虚拟机控制 eBPF 程序的加载和执行，并将预编译的 eBPF 程序打包为通用的 JSON 或 WASM 模块进行分发。我们会使用 eunomia-bpf 进行演示。</p>
<h2 id="下载安装-eunomia-bpf-开发工具"><a class="header" href="#下载安装-eunomia-bpf-开发工具">下载安装 eunomia-bpf 开发工具</a></h2>
<p>可以通过以下步骤下载和安装 eunomia-bpf：</p>
<p>下载 ecli 工具，用于运行 eBPF 程序：</p>
<pre><code class="language-console">$ wget https://aka.pw/bpf-ecli -O ecli &amp;&amp; chmod +x ./ecli
$ ./ecli -h
Usage: ecli [--help] [--version] [--json] [--no-cache] url-and-args
</code></pre>
<p>下载编译器工具链，用于将 eBPF 内核代码编译为 config 文件或 WASM 模块：</p>
<pre><code class="language-console">$ wget https://github.com/eunomia-bpf/eunomia-bpf/releases/latest/download/ecc &amp;&amp; chmod +x ./ecc
$ ./ecc -h
eunomia-bpf compiler
Usage: ecc [OPTIONS] &lt;SOURCE_PATH&gt; [EXPORT_EVENT_HEADER]
....
</code></pre>
<p>也可以使用 docker 镜像进行编译：</p>
<pre><code class="language-console">$ docker run -it -v `pwd`/:/src/ yunwei37/ebpm:latest # 使用 docker 进行编译。`pwd` 应该包含 *.bpf.c 文件和 *.h 文件。
export PATH=PATH:~/.eunomia/bin
Compiling bpf object...
Packing ebpf object and config into /src/package.json...
</code></pre>
<h2 id="hello-world---minimal-ebpf-program"><a class="header" href="#hello-world---minimal-ebpf-program">Hello World - minimal eBPF program</a></h2>
<p>我们会先从一个简单的 eBPF 程序开始，它会在内核中打印一条消息。我们会使用 eunomia-bpf 的编译器工具链将其编译为 bpf 字节码文件，然后使用 ecli 工具加载并运行该程序。作为示例，我们可以暂时省略用户态程序的部分。</p>
<pre><code class="language-c">
```c
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
#define BPF_NO_GLOBAL_DATA
#include &lt;linux/bpf.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
typedef unsigned int u32;
typedef int pid_t;
const pid_t pid_filter = 0;
char LICENSE[] SEC(&quot;license&quot;) = &quot;Dual BSD/GPL&quot;;
SEC(&quot;tp/syscalls/sys_enter_write&quot;)
int handle_tp(void *ctx)
{
pid_t pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
if (pid_filter &amp;&amp; pid != pid_filter)
return 0;
bpf_printk(&quot;BPF triggered from PID %d.\n&quot;, pid);
return 0;
}
</code></pre>
<p>这段程序通过定义一个 handle_tp 函数并使用 SEC 宏把它附加到 sys_enter_write tracepoint（即在进入 write 系统调用时执行）。该函数通过使用 bpf_get_current_pid_tgid 和 bpf_printk 函数获取调用 write 系统调用的进程 ID，并在内核日志中打印出来。</p>
<ul>
<li><code>bpf_trace_printk()</code>： 一种将信息输出到trace_pipe(/sys/kernel/debug/tracing/trace_pipe)简单机制。 在一些简单用例中这样使用没有问题， but它也有一些限制：最多3 参数； 第一个参数必须是%s(即字符串)；同时trace_pipe在内核中全局共享，其他并行使用trace_pipe的程序有可能会将 trace_pipe 的输出扰乱。 一个更好的方式是通过 BPF_PERF_OUTPUT(), 稍后将会讲到。</li>
<li><code>void *ctx</code>：ctx本来是具体类型的参数， 但是由于我们这里没有使用这个参数，因此就将其写成void *类型。</li>
<li><code>return 0</code>;：必须这样，返回0 (如果要知道why, 参考 #139 <a href="https://github.com/iovisor/bcc/issues/139">https://github.com/iovisor/bcc/issues/139</a>)。</li>
</ul>
<p>要编译和运行这段程序，可以使用 ecc 工具和 ecli 命令。首先使用 ecc 编译程序：</p>
<pre><code class="language-console">$ ecc minimal.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
</code></pre>
<p>或使用 docker 镜像进行编译：</p>
<pre><code class="language-shell">docker run -it -v `pwd`/:/src/ yunwei37/ebpm:latest
</code></pre>
<p>然后使用 ecli 运行编译后的程序：</p>
<pre><code class="language-console">$ sudo ecli run ./package.json
Runing eBPF program...
</code></pre>
<p>运行这段程序后，可以通过查看 /sys/kernel/debug/tracing/trace_pipe 文件来查看 eBPF 程序的输出：</p>
<pre><code class="language-console">$ sudo cat /sys/kernel/debug/tracing/trace_pipe
&lt;...&gt;-3840345 [010] d... 3220701.101143: bpf_trace_printk: write system call from PID 3840345.
&lt;...&gt;-3840345 [010] d... 3220701.101143: bpf_trace_printk: write system call from PID 3840345.
</code></pre>
<h2 id="ebpf-程序的基本框架"><a class="header" href="#ebpf-程序的基本框架">eBPF 程序的基本框架</a></h2>
<p>如上所述， eBPF 程序的基本框架包括：</p>
<ul>
<li>包含头文件：需要包含 &lt;linux/bpf.h&gt; 和 &lt;bpf/bpf_helpers.h&gt; 等头文件。</li>
<li>定义许可证：需要定义许可证，通常使用 &quot;Dual BSD/GPL&quot;。</li>
<li>定义 BPF 函数：需要定义一个 BPF 函数，例如其名称为 handle_tp，其参数为 void *ctx，返回值为 int。通常用 C 语言编写。</li>
<li>使用 BPF 助手函数：在例如 BPF 函数中，可以使用 BPF 助手函数 bpf_get_current_pid_tgid() 和 bpf_printk()。</li>
<li>返回值</li>
</ul>
<h2 id="tracepoints"><a class="header" href="#tracepoints">tracepoints</a></h2>
<p>跟踪点（tracepoints）是内核静态插桩技术，跟踪点在技术上只是放置在内核源代码中的跟踪函数，实际上就是在源码中插入的一些带有控制条件的探测点，这些探测点允许事后再添加处理函数。比如在内核中，最常见的静态跟踪方法就是 printk，即输出日志。又比如：在系统调用、调度程序事件、文件系统操作和磁盘 I/O 的开始和结束时都有跟踪点。 于 2009 年在 Linux 2.6.32 版本中首次提供。跟踪点是一种稳定的 API，数量有限。</p>
<h2 id="github-模板轻松构建-ebpf-项目和开发环境"><a class="header" href="#github-模板轻松构建-ebpf-项目和开发环境">GitHub 模板：轻松构建 eBPF 项目和开发环境</a></h2>
<p>面对创建一个 eBPF 项目，您是否对如何开始搭建环境以及选择编程语言感到困惑？别担心，我们为您准备了一系列 GitHub 模板，以便您快速启动一个全新的eBPF项目。只需在GitHub上点击 <code>Use this template</code> 按钮，即可开始使用。</p>
<ul>
<li><a href="https://github.com/eunomia-bpf/libbpf-starter-template">https://github.com/eunomia-bpf/libbpf-starter-template</a>：基于C语言和 libbpf 框架的eBPF项目模板</li>
<li><a href="https://github.com/eunomia-bpf/cilium-ebpf-starter-template">https://github.com/eunomia-bpf/cilium-ebpf-starter-template</a>：基于C语言和cilium/ebpf框架的eBPF项目模板</li>
<li><a href="https://github.com/eunomia-bpf/libbpf-rs-starter-template">https://github.com/eunomia-bpf/libbpf-rs-starter-template</a>：基于Rust语言和libbpf-rs框架的eBPF项目模板</li>
<li><a href="https://github.com/eunomia-bpf/eunomia-template">https://github.com/eunomia-bpf/eunomia-template</a>：基于C语言和eunomia-bpf框架的eBPF项目模板</li>
</ul>
<p>这些启动模板包含以下功能：</p>
<ul>
<li>一个 Makefile，让您可以一键构建项目</li>
<li>一个 Dockerfile，用于为您的 eBPF 项目自动创建一个容器化环境并发布到 Github Packages</li>
<li>GitHub Actions，用于自动化构建、测试和发布流程</li>
<li>eBPF 开发所需的所有依赖项</li>
</ul>
<blockquote>
<p>通过将现有仓库设置为模板，您和其他人可以快速生成具有相同基础结构的新仓库，从而省去了手动创建和配置的繁琐过程。借助 GitHub 模板仓库，开发者可以专注于项目的核心功能和逻辑，而无需为基础设置和结构浪费时间。更多关于模板仓库的信息，请参阅官方文档：https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository</p>
</blockquote>
<h2 id="总结"><a class="header" href="#总结">总结</a></h2>
<p>eBPF 程序的开发和使用流程可以概括为如下几个步骤：</p>
<ul>
<li>定义 eBPF 程序的接口和类型：这包括定义 eBPF 程序的接口函数，定义和实现 eBPF 内核映射（maps）和共享内存（perf events），以及定义和使用 eBPF 内核帮助函数（helpers）。</li>
<li>编写 eBPF 程序的代码：这包括编写 eBPF 程序的主要逻辑，实现 eBPF 内核映射的读写操作，以及使用 eBPF 内核帮助函数。</li>
<li>编译 eBPF 程序：这包括使用 eBPF 编译器（例如 clang）将 eBPF 程序代码编译为 eBPF 字节码，并生成可执行的 eBPF 内核模块。ecc 本质上也是调用 clang 编译器来编译 eBPF 程序。</li>
<li>加载 eBPF 程序到内核：这包括将编译好的 eBPF 内核模块加载到 Linux 内核中，并将 eBPF 程序附加到指定的内核事件上。</li>
<li>使用 eBPF 程序：这包括监测 eBPF 程序的运行情况，并使用 eBPF 内核映射和共享内存进行数据交换和共享。</li>
<li>在实际开发中，还可能需要进行其他的步骤，例如配置编译和加载参数，管理 eBPF 内核模块和内核映射，以及使用其他高级功能等。</li>
</ul>
<p>需要注意的是，BPF 程序的执行是在内核空间进行的，因此需要使用特殊的工具和技术来编写、编译和调试 BPF 程序。eunomia-bpf 是一个开源的 BPF 编译器和工具包，它可以帮助开发者快速和简单地编写和运行 BPF 程序。</p>
<p>本教程的文档和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程二在-ebpf-中使用-kprobe-监测捕获-unlink-系统调用"><a class="header" href="#ebpf-入门开发实践教程二在-ebpf-中使用-kprobe-监测捕获-unlink-系统调用">eBPF 入门开发实践教程二：在 eBPF 中使用 kprobe 监测捕获 unlink 系统调用</a></h1>
<p>eBPF (Extended Berkeley Packet Filter) 是 Linux 内核上的一个强大的网络和性能分析工具。它允许开发者在内核运行时动态加载、更新和运行用户定义的代码。</p>
<p>本文是 eBPF 入门开发实践教程的第二篇，在 eBPF 中使用 kprobe 捕获 unlink 系统调用。本文会先讲解关于 kprobes 的基本概念和技术背景，然后介绍如何在 eBPF 中使用 kprobe 捕获 unlink 系统调用。</p>
<h2 id="kprobes-技术背景"><a class="header" href="#kprobes-技术背景">kprobes 技术背景</a></h2>
<p>开发人员在内核或者模块的调试过程中，往往会需要要知道其中的一些函数有无被调用、何时被调用、执行是否正确以及函数的入参和返回值是什么等等。比较简单的做法是在内核代码对应的函数中添加日志打印信息，但这种方式往往需要重新编译内核或模块，重新启动设备之类的，操作较为复杂甚至可能会破坏原有的代码执行过程。</p>
<p>而利用kprobes技术，用户可以定义自己的回调函数，然后在内核或者模块中几乎所有的函数中（有些函数是不可探测的，例如kprobes自身的相关实现函数，后文会有详细说明）动态的插入探测点，当内核执行流程执行到指定的探测函数时，会调用该回调函数，用户即可收集所需的信息了，同时内核最后还会回到原本的正常执行流程。如果用户已经收集足够的信息，不再需要继续探测，则同样可以动态地移除探测点。因此kprobes技术具有对内核执行流程影响小和操作方便的优点。</p>
<p>kprobes技术包括的3种探测手段分别时kprobe、jprobe和kretprobe。首先kprobe是最基本的探测方式，是实现后两种的基础，它可以在任意的位置放置探测点（就连函数内部的某条指令处也可以），它提供了探测点的调用前、调用后和内存访问出错3种回调方式，分别是pre_handler、post_handler和fault_handler，其中pre_handler函数将在被探测指令被执行前回调，post_handler会在被探测指令执行完毕后回调（注意不是被探测函数），fault_handler会在内存访问出错时被调用；jprobe基于kprobe实现，它用于获取被探测函数的入参值；最后kretprobe从名字中就可以看出其用途了，它同样基于kprobe实现，用于获取被探测函数的返回值。</p>
<p>kprobes的技术原理并不仅仅包含存软件的实现方案，它也需要硬件架构提供支持。其中涉及硬件架构相关的是CPU的异常处理和单步调试技术，前者用于让程序的执行流程陷入到用户注册的回调函数中去，而后者则用于单步执行被探测点指令，因此并不是所有的架构均支持，目前kprobes技术已经支持多种架构，包括i386、x86_64、ppc64、ia64、sparc64、arm、ppc和mips（有些架构实现可能并不完全，具体可参考内核的Documentation/kprobes.txt）。</p>
<p>kprobes的特点与使用限制：</p>
<ol>
<li>kprobes允许在同一个被被探测位置注册多个kprobe，但是目前jprobe却不可以；同时也不允许以其他的jprobe回调函数和kprobe的post_handler回调函数作为被探测点。</li>
<li>一般情况下，可以探测内核中的任何函数，包括中断处理函数。不过在kernel/kprobes.c和arch/*/kernel/kprobes.c程序中用于实现kprobes自身的函数是不允许被探测的，另外还有do_page_fault和notifier_call_chain；</li>
<li>如果以一个内联函数为探测点，则kprobes可能无法保证对该函数的所有实例都注册探测点。由于gcc可能会自动将某些函数优化为内联函数，因此可能无法达到用户预期的探测效果；</li>
<li>一个探测点的回调函数可能会修改被探测函数运行的上下文，例如通过修改内核的数据结构或者保存与struct pt_regs结构体中的触发探测器之前寄存器信息。因此kprobes可以被用来安装bug修复代码或者注入故障测试代码；</li>
<li>kprobes会避免在处理探测点函数时再次调用另一个探测点的回调函数，例如在printk()函数上注册了探测点，则在它的回调函数中可能再次调用printk函数，此时将不再触发printk探测点的回调，仅仅时增加了kprobe结构体中nmissed字段的数值；</li>
<li>在kprobes的注册和注销过程中不会使用mutex锁和动态的申请内存；</li>
<li>kprobes回调函数的运行期间是关闭内核抢占的，同时也可能在关闭中断的情况下执行，具体要视CPU架构而定。因此不论在何种情况下，在回调函数中不要调用会放弃CPU的函数（如信号量、mutex锁等）；</li>
<li>kretprobe通过替换返回地址为预定义的trampoline的地址来实现，因此栈回溯和gcc内嵌函数__builtin_return_address()调用将返回trampoline的地址而不是真正的被探测函数的返回地址；</li>
<li>如果一个函数的调用次数和返回次数不相等，则在类似这样的函数上注册kretprobe将可能不会达到预期的效果，例如do_exit()函数会存在问题，而do_execve()函数和do_fork()函数不会；</li>
<li>如果当在进入和退出一个函数时，CPU运行在非当前任务所有的栈上，那么往该函数上注册kretprobe可能会导致不可预料的后果，因此，kprobes不支持在X86_64的结构下为__switch_to()函数注册kretprobe，将直接返回-EINVAL。</li>
</ol>
<h2 id="kprobe-示例"><a class="header" href="#kprobe-示例">kprobe 示例</a></h2>
<p>完整代码如下：</p>
<pre><code class="language-c">#include &quot;vmlinux.h&quot;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#include &lt;bpf/bpf_core_read.h&gt;
char LICENSE[] SEC(&quot;license&quot;) = &quot;Dual BSD/GPL&quot;;
SEC(&quot;kprobe/do_unlinkat&quot;)
int BPF_KPROBE(do_unlinkat, int dfd, struct filename *name)
{
pid_t pid;
const char *filename;
pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
filename = BPF_CORE_READ(name, name);
bpf_printk(&quot;KPROBE ENTRY pid = %d, filename = %s\n&quot;, pid, filename);
return 0;
}
SEC(&quot;kretprobe/do_unlinkat&quot;)
int BPF_KRETPROBE(do_unlinkat_exit, long ret)
{
pid_t pid;
pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
bpf_printk(&quot;KPROBE EXIT: pid = %d, ret = %ld\n&quot;, pid, ret);
return 0;
}
</code></pre>
<p>这段代码是一个简单的 eBPF 程序，用于监测和捕获在 Linux 内核中执行的 unlink 系统调用。unlink 系统调用的功能是删除一个文件。这个 eBPF 程序通过使用 kprobe（内核探针）在 do_unlinkat 函数的入口和退出处放置钩子，实现对该系统调用的跟踪。</p>
<p>首先，我们导入必要的头文件，如 vmlinux.h，bpf_helpers.h，bpf_tracing.h 和 bpf_core_read.h。接着，我们定义许可证，以允许程序在内核中运行。</p>
<pre><code class="language-c">#include &quot;vmlinux.h&quot;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#include &lt;bpf/bpf_core_read.h&gt;
char LICENSE[] SEC(&quot;license&quot;) = &quot;Dual BSD/GPL&quot;;
</code></pre>
<p>接下来，我们定义一个名为 BPF_KPROBE(do_unlinkat) 的 kprobe，当进入 do_unlinkat 函数时，它会被触发。该函数接受两个参数：dfd（文件描述符）和 name（文件名结构体指针）。在这个 kprobe 中，我们获取当前进程的 PID（进程标识符），然后读取文件名。最后，我们使用 bpf_printk 函数在内核日志中打印 PID 和文件名。</p>
<pre><code class="language-c">SEC(&quot;kprobe/do_unlinkat&quot;)
int BPF_KPROBE(do_unlinkat, int dfd, struct filename *name)
{
pid_t pid;
const char *filename;
pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
filename = BPF_CORE_READ(name, name);
bpf_printk(&quot;KPROBE ENTRY pid = %d, filename = %s\n&quot;, pid, filename);
return 0;
}
</code></pre>
<p>接下来，我们定义一个名为 BPF_KRETPROBE(do_unlinkat_exit) 的 kretprobe，当从 do_unlinkat 函数退出时，它会被触发。这个 kretprobe 的目的是捕获函数的返回值（ret）。我们再次获取当前进程的 PID，并使用 bpf_printk 函数在内核日志中打印 PID 和返回值。</p>
<pre><code class="language-c">SEC(&quot;kretprobe/do_unlinkat&quot;)
int BPF_KRETPROBE(do_unlinkat_exit, long ret)
{
pid_t pid;
pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
bpf_printk(&quot;KPROBE EXIT: pid = %d, ret = %ld\n&quot;, pid, ret);
return 0;
}
</code></pre>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。</p>
<p>要编译这个程序，请使用 ecc 工具：</p>
<pre><code class="language-console">$ ecc kprobe-link.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
</code></pre>
<p>然后运行：</p>
<pre><code class="language-console">sudo ecli run package.json
</code></pre>
<p>在另外一个窗口中：</p>
<pre><code class="language-shell">touch test1
rm test1
touch test2
rm test2
</code></pre>
<p>在 /sys/kernel/debug/tracing/trace_pipe 文件中，应该能看到类似下面的 kprobe 演示输出：</p>
<pre><code class="language-shell">$ sudo cat /sys/kernel/debug/tracing/trace_pipe
rm-9346 [005] d..3 4710.951696: bpf_trace_printk: KPROBE ENTRY pid = 9346, filename = test1
rm-9346 [005] d..4 4710.951819: bpf_trace_printk: KPROBE EXIT: ret = 0
rm-9346 [005] d..3 4710.951852: bpf_trace_printk: KPROBE ENTRY pid = 9346, filename = test2
rm-9346 [005] d..4 4710.951895: bpf_trace_printk: KPROBE EXIT: ret = 0
</code></pre>
<h2 id="总结-1"><a class="header" href="#总结-1">总结</a></h2>
<p>通过本文的示例，我们学习了如何使用 eBPF 的 kprobe 和 kretprobe 捕获 unlink 系统调用。更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>本文是 eBPF 入门开发实践教程的第二篇。下一篇文章将介绍如何在 eBPF 中使用 fentry 监测捕获 unlink 系统调用。</p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程三在-ebpf-中使用-fentry-监测捕获-unlink-系统调用"><a class="header" href="#ebpf-入门开发实践教程三在-ebpf-中使用-fentry-监测捕获-unlink-系统调用">eBPF 入门开发实践教程三：在 eBPF 中使用 fentry 监测捕获 unlink 系统调用</a></h1>
<p>eBPF (Extended Berkeley Packet Filter) 是 Linux 内核上的一个强大的网络和性能分析工具。它允许开发者在内核运行时动态加载、更新和运行用户定义的代码。</p>
<p>本文是 eBPF 入门开发实践教程的第三篇，在 eBPF 中使用 fentry 捕获 unlink 系统调用。</p>
<h2 id="fentry"><a class="header" href="#fentry">Fentry</a></h2>
<p>fentry（function entry）和fexit（function exit）是eBPF（扩展的伯克利包过滤器）中的两种探针类型，用于在Linux内核函数的入口和退出处进行跟踪。它们允许开发者在内核函数执行的特定阶段收集信息、修改参数或观察返回值。这种跟踪和监控功能在性能分析、故障排查和安全分析等场景中非常有用。</p>
<p>与 kprobes 相比，fentry 和 fexit 程序有更高的性能和可用性。在这个例子中，我们可以直接访问函数的指针参数，就像在普通的 C 代码中一样，而不需要使用各种读取帮助程序。fexit 和 kretprobe 程序最大的区别在于，fexit 程序可以访问函数的输入参数和返回值，而 kretprobe 只能访问返回值。从 5.5 内核开始，fentry 和 fexit 对 eBPF 程序可用。</p>
<pre><code class="language-c">#include &quot;vmlinux.h&quot;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
char LICENSE[] SEC(&quot;license&quot;) = &quot;Dual BSD/GPL&quot;;
SEC(&quot;fentry/do_unlinkat&quot;)
int BPF_PROG(do_unlinkat, int dfd, struct filename *name)
{
pid_t pid;
pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
bpf_printk(&quot;fentry: pid = %d, filename = %s\n&quot;, pid, name-&gt;name);
return 0;
}
SEC(&quot;fexit/do_unlinkat&quot;)
int BPF_PROG(do_unlinkat_exit, int dfd, struct filename *name, long ret)
{
pid_t pid;
pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
bpf_printk(&quot;fexit: pid = %d, filename = %s, ret = %ld\n&quot;, pid, name-&gt;name, ret);
return 0;
}
</code></pre>
<p>这段程序是用C语言编写的eBPF（扩展的伯克利包过滤器）程序，它使用BPF的fentry和fexit探针来跟踪Linux内核函数do_unlinkat。在这个教程中，我们将以这段程序作为示例，让您学会如何在eBPF中使用fentry监测捕获unlink系统调用。</p>
<p>程序包含以下部分：</p>
<ol>
<li>包含头文件：包括vmlinux.h（用于访问内核数据结构）、bpf/bpf_helpers.h（包含eBPF帮助函数）、bpf/bpf_tracing.h（用于eBPF跟踪相关功能）。</li>
<li>定义许可证：这里定义了一个名为LICENSE的字符数组，包含许可证信息&quot;Dual BSD/GPL&quot;。</li>
<li>定义fentry探针：我们定义了一个名为BPF_PROG(do_unlinkat)的fentry探针，该探针在do_unlinkat函数的入口处被触发。这个探针获取当前进程的PID（进程ID）并将其与文件名一起打印到内核日志。</li>
<li>定义fexit探针：我们还定义了一个名为BPF_PROG(do_unlinkat_exit)的fexit探针，该探针在do_unlinkat函数的退出处被触发。与fentry探针类似，这个探针也会获取当前进程的PID并将其与文件名和返回值一起打印到内核日志。</li>
</ol>
<p>通过这个示例，您可以学习如何在eBPF中使用fentry和fexit探针来监控和捕获内核函数调用，例如在本教程中的unlink系统调用。</p>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。我们使用 eunomia-bpf 编译运行这个例子。</p>
<p>编译运行上述代码：</p>
<pre><code class="language-console">$ ecc fentry-link.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
$ sudo ecli run package.json
Runing eBPF program...
</code></pre>
<p>在另外一个窗口中：</p>
<pre><code class="language-shell">touch test_file
rm test_file
touch test_file2
rm test_file2
</code></pre>
<p>运行这段程序后，可以通过查看 /sys/kernel/debug/tracing/trace_pipe 文件来查看 eBPF 程序的输出：</p>
<pre><code class="language-console">$ sudo cat /sys/kernel/debug/tracing/trace_pipe
rm-9290 [004] d..2 4637.798698: bpf_trace_printk: fentry: pid = 9290, filename = test_file
rm-9290 [004] d..2 4637.798843: bpf_trace_printk: fexit: pid = 9290, filename = test_file, ret = 0
rm-9290 [004] d..2 4637.798698: bpf_trace_printk: fentry: pid = 9290, filename = test_file2
rm-9290 [004] d..2 4637.798843: bpf_trace_printk: fexit: pid = 9290, filename = test_file2, ret = 0
</code></pre>
<h2 id="总结-2"><a class="header" href="#总结-2">总结</a></h2>
<p>这段程序是一个 eBPF 程序，通过使用 fentry 和 fexit 捕获 do_unlinkat 和 do_unlinkat_exit 函数，并通过使用 bpf_get_current_pid_tgid 和 bpf_printk 函数获取调用 do_unlinkat 的进程 ID、文件名和返回值，并在内核日志中打印出来。</p>
<p>编译这个程序可以使用 ecc 工具，运行时可以使用 ecli 命令，并通过查看 /sys/kernel/debug/tracing/trace_pipe 文件查看 eBPF 程序的输出。更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程四在-ebpf-中捕获进程打开文件的系统调用集合使用全局变量过滤进程-pid"><a class="header" href="#ebpf-入门开发实践教程四在-ebpf-中捕获进程打开文件的系统调用集合使用全局变量过滤进程-pid">eBPF 入门开发实践教程四：在 eBPF 中捕获进程打开文件的系统调用集合，使用全局变量过滤进程 pid</a></h1>
<p>eBPF（Extended Berkeley Packet Filter）是一种内核执行环境，它可以让用户在内核中运行一些安全的、高效的程序。它通常用于网络过滤、性能分析、安全监控等场景。eBPF 之所以强大，是因为它能够在内核运行时捕获和修改数据包或者系统调用，从而实现对操作系统行为的监控和调整。</p>
<p>本文是 eBPF 入门开发实践教程的第四篇，主要介绍如何捕获进程打开文件的系统调用集合，并使用全局变量在 eBPF 中过滤进程 pid。</p>
<p>在 Linux 系统中，进程与文件之间的交互是通过系统调用来实现的。系统调用是用户态程序与内核态程序之间的接口，它们允许用户态程序请求内核执行特定操作。在本教程中，我们关注的是 sys_openat 系统调用，它是用于打开文件的。</p>
<p>当进程打开一个文件时，它会向内核发出 sys_openat 系统调用，并传递相关参数（例如文件路径、打开模式等）。内核会处理这个请求，并返回一个文件描述符（file descriptor），这个描述符将在后续的文件操作中用作引用。通过捕获 sys_openat 系统调用，我们可以了解进程在什么时候以及如何打开文件。</p>
<h2 id="在-ebpf-中捕获进程打开文件的系统调用集合"><a class="header" href="#在-ebpf-中捕获进程打开文件的系统调用集合">在 eBPF 中捕获进程打开文件的系统调用集合</a></h2>
<p>首先，我们需要编写一段 eBPF 程序来捕获进程打开文件的系统调用，具体实现如下：</p>
<pre><code class="language-c">#include &lt;vmlinux.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
/// @description &quot;Process ID to trace&quot;
const volatile int pid_target = 0;
SEC(&quot;tracepoint/syscalls/sys_enter_openat&quot;)
int tracepoint__syscalls__sys_enter_openat(struct trace_event_raw_sys_enter* ctx)
{
u64 id = bpf_get_current_pid_tgid();
u32 pid = id;
if (pid_target &amp;&amp; pid_target != pid)
return false;
// Use bpf_printk to print the process information
bpf_printk(&quot;Process ID: %d enter sys openat\n&quot;, pid);
return 0;
}
/// &quot;Trace open family syscalls.&quot;
char LICENSE[] SEC(&quot;license&quot;) = &quot;GPL&quot;;
</code></pre>
<p>这段 eBPF 程序实现了：</p>
<ol>
<li>引入头文件：&lt;vmlinux.h&gt; 包含了内核数据结构的定义，&lt;bpf/bpf_helpers.h&gt; 包含了 eBPF 程序所需的辅助函数。</li>
<li>定义全局变量 pid_target，用于过滤指定进程 ID。这里设为 0 表示捕获所有进程的 sys_openat 调用。</li>
<li>使用 SEC 宏定义一个 eBPF 程序，关联到 tracepoint &quot;tracepoint/syscalls/sys_enter_openat&quot;。这个 tracepoint 会在进程发起 sys_openat 系统调用时触发。</li>
<li>实现 eBPF 程序 tracepoint__syscalls__sys_enter_openat，它接收一个类型为 struct trace_event_raw_sys_enter 的参数 ctx。这个结构体包含了关于系统调用的信息。</li>
<li>使用 bpf_get_current_pid_tgid() 函数获取当前进程的 PID 和 TGID（线程组 ID）。由于我们只关心 PID，所以将其赋值给 u32 类型的变量 pid。</li>
<li>检查 pid_target 变量是否与当前进程的 pid 相等。如果 pid_target 不为 0 且与当前进程的 pid 不相等，则返回 false，不对该进程的 sys_openat 调用进行捕获。</li>
<li>使用 bpf_printk() 函数打印捕获到的进程 ID 和 sys_openat 调用的相关信息。这些信息将在用户空间通过 BPF 工具查看。</li>
<li>将程序许可证设置为 &quot;GPL&quot;，这是运行 eBPF 程序的必要条件。</li>
</ol>
<p>这个 eBPF 程序可以通过 libbpf 或 eunomia-bpf 等工具加载到内核并执行。它将捕获指定进程（或所有进程）的 sys_openat 系统调用，并在用户空间输出相关信息。</p>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。我们使用 eunomia-bpf 编译运行这个例子。</p>
<p>编译运行上述代码：</p>
<pre><code class="language-console">$ ecc opensnoop.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
$ sudo ecli run package.json
Runing eBPF program...
</code></pre>
<p>运行这段程序后，可以通过查看 /sys/kernel/debug/tracing/trace_pipe 文件来查看 eBPF 程序的输出：</p>
<pre><code class="language-console">$ sudo cat /sys/kernel/debug/tracing/trace_pipe
&lt;...&gt;-3840345 [010] d... 3220701.101179: bpf_trace_printk: Process ID: 3840345 enter sys openat
&lt;...&gt;-3840345 [010] d... 3220702.158000: bpf_trace_printk: Process ID: 3840345 enter sys openat
</code></pre>
<p>此时，我们已经能够捕获进程打开文件的系统调用了。</p>
<h2 id="使用全局变量在-ebpf-中过滤进程-pid"><a class="header" href="#使用全局变量在-ebpf-中过滤进程-pid">使用全局变量在 eBPF 中过滤进程 pid</a></h2>
<p>全局变量在 eBPF 程序中充当一种数据共享机制，它们允许用户态程序与 eBPF 程序之间进行数据交互。这在过滤特定条件或修改 eBPF 程序行为时非常有用。这种设计使得用户态程序能够在运行时动态地控制 eBPF 程序的行为。</p>
<p>在我们的例子中，全局变量 <code>pid_target</code> 用于过滤进程 PID。用户态程序可以设置此变量的值，以便在 eBPF 程序中只捕获与指定 PID 相关的 <code>sys_openat</code> 系统调用。</p>
<p>使用全局变量的原理是，全局变量在 eBPF 程序的数据段（data section）中定义并存储。当 eBPF 程序加载到内核并执行时，这些全局变量会保持在内核中，可以通过 BPF 系统调用进行访问。用户态程序可以使用 BPF 系统调用中的某些特性，如 bpf_obj_get_info_by_fd 和 bpf_obj_get_info，获取 eBPF 对象的信息，包括全局变量的位置和值。</p>
<p>可以通过执行 ecli -h 命令来查看 opensnoop 的帮助信息：</p>
<pre><code class="language-console">$ ecli package.json -h
Usage: opensnoop_bpf [--help] [--version] [--verbose] [--pid_target VAR]
Trace open family syscalls.
Optional arguments:
-h, --help shows help message and exits
-v, --version prints version information and exits
--verbose prints libbpf debug information
--pid_target Process ID to trace
Built with eunomia-bpf framework.
See https://github.com/eunomia-bpf/eunomia-bpf for more information.
</code></pre>
<p>可以通过 --pid_target 参数来指定要捕获的进程的 pid，例如：</p>
<pre><code class="language-console">$ sudo ./ecli run package.json --pid_target 618
Runing eBPF program...
</code></pre>
<p>运行这段程序后，可以通过查看 /sys/kernel/debug/tracing/trace_pipe 文件来查看 eBPF 程序的输出：</p>
<pre><code class="language-console">$ sudo cat /sys/kernel/debug/tracing/trace_pipe
&lt;...&gt;-3840345 [010] d... 3220701.101179: bpf_trace_printk: Process ID: 618 enter sys openat
&lt;...&gt;-3840345 [010] d... 3220702.158000: bpf_trace_printk: Process ID: 618 enter sys openat
</code></pre>
<h2 id="总结-3"><a class="header" href="#总结-3">总结</a></h2>
<p>本文介绍了如何使用 eBPF 程序来捕获进程打开文件的系统调用。在 eBPF 程序中，我们可以通过定义 tracepoint__syscalls__sys_enter_open 和 tracepoint__syscalls__sys_enter_openat 函数并使用 SEC 宏把它们附加到 sys_enter_open 和 sys_enter_openat 两个 tracepoint 来捕获进程打开文件的系统调用。我们可以使用 bpf_get_current_pid_tgid 函数获取调用 open 或 openat 系统调用的进程 ID，并使用 bpf_printk 函数在内核日志中打印出来。在 eBPF 程序中，我们还可以通过定义一个全局变量 pid_target 来指定要捕获的进程的 pid，从而过滤输出，只输出指定的进程的信息。</p>
<p>通过学习本教程，您应该对如何在 eBPF 中捕获和过滤特定进程的系统调用有了更深入的了解。这种方法在系统监控、性能分析和安全审计等场景中具有广泛的应用。</p>
<p>更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程五在-ebpf-中使用--uprobe-捕获-bash-的-readline-函数调用"><a class="header" href="#ebpf-入门开发实践教程五在-ebpf-中使用--uprobe-捕获-bash-的-readline-函数调用">eBPF 入门开发实践教程五：在 eBPF 中使用 uprobe 捕获 bash 的 readline 函数调用</a></h1>
<p>eBPF (Extended Berkeley Packet Filter) 是 Linux 内核上的一个强大的网络和性能分析工具，它允许开发者在内核运行时动态加载、更新和运行用户定义的代码。</p>
<p>本文是 eBPF 入门开发实践教程的第五篇，主要介绍如何使用 uprobe 捕获 bash 的 readline 函数调用。</p>
<h2 id="什么是uprobe"><a class="header" href="#什么是uprobe">什么是uprobe</a></h2>
<p>uprobe是一种用户空间探针，uprobe探针允许在用户空间程序中动态插桩，插桩位置包括：函数入口、特定偏移处，以及函数返回处。当我们定义uprobe时，内核会在附加的指令上创建快速断点指令（x86机器上为int3指令），当程序执行到该指令时，内核将触发事件，程序陷入到内核态，并以回调函数的方式调用探针函数，执行完探针函数再返回到用户态继续执行后序的指令。</p>
<p>uprobe基于文件，当一个二进制文件中的一个函数被跟踪时，所有使用到这个文件的进程都会被插桩，包括那些尚未启动的进程，这样就可以在全系统范围内跟踪系统调用。</p>
<p>uprobe适用于在用户态去解析一些内核态探针无法解析的流量，例如http2流量（报文header被编码，内核无法解码），https流量（加密流量，内核无法解密）。</p>
<h2 id="使用-uprobe-捕获-bash-的-readline-函数调用"><a class="header" href="#使用-uprobe-捕获-bash-的-readline-函数调用">使用 uprobe 捕获 bash 的 readline 函数调用</a></h2>
<p>uprobe 是一种用于捕获用户空间函数调用的 eBPF 的探针，我们可以通过它来捕获用户空间程序调用的系统函数。</p>
<p>例如，我们可以使用 uprobe 来捕获 bash 的 readline 函数调用，从而获取用户在 bash 中输入的命令行。示例代码如下：</p>
<pre><code class="language-c">#include &lt;vmlinux.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#define TASK_COMM_LEN 16
#define MAX_LINE_SIZE 80
/* Format of u[ret]probe section definition supporting auto-attach:
* u[ret]probe/binary:function[+offset]
*
* binary can be an absolute/relative path or a filename; the latter is resolved to a
* full binary path via bpf_program__attach_uprobe_opts.
*
* Specifying uprobe+ ensures we carry out strict matching; either &quot;uprobe&quot; must be
* specified (and auto-attach is not possible) or the above format is specified for
* auto-attach.
*/
SEC(&quot;uretprobe//bin/bash:readline&quot;)
int BPF_KRETPROBE(printret, const void *ret)
{
char str[MAX_LINE_SIZE];
char comm[TASK_COMM_LEN];
u32 pid;
if (!ret)
return 0;
bpf_get_current_comm(&amp;comm, sizeof(comm));
pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
bpf_probe_read_user_str(str, sizeof(str), ret);
bpf_printk(&quot;PID %d (%s) read: %s &quot;, pid, comm, str);
return 0;
};
char LICENSE[] SEC(&quot;license&quot;) = &quot;GPL&quot;;
</code></pre>
<p>这段代码的作用是在 bash 的 readline 函数返回时执行指定的 BPF_KRETPROBE 函数，即 printret 函数。</p>
<p>在 printret 函数中，我们首先获取了调用 readline 函数的进程的进程名称和进程 ID，然后通过 bpf_probe_read_user_str 函数读取了用户输入的命令行字符串，最后通过 bpf_printk 函数打印出进程 ID、进程名称和输入的命令行字符串。</p>
<p>除此之外，我们还需要通过 SEC 宏来定义 uprobe 探针，并使用 BPF_KRETPROBE 宏来定义探针函数。</p>
<p>在 SEC 宏中，我们需要指定 uprobe 的类型、要捕获的二进制文件的路径和要捕获的函数名称。例如，上面的代码中的 SEC 宏的定义如下：</p>
<pre><code class="language-c">SEC(&quot;uprobe//bin/bash:readline&quot;)
</code></pre>
<p>这表示我们要捕获的是 /bin/bash 二进制文件中的 readline 函数。</p>
<p>接下来，我们需要使用 BPF_KRETPROBE 宏来定义探针函数，例如：</p>
<pre><code class="language-c">BPF_KRETPROBE(printret, const void *ret)
</code></pre>
<p>这里的 printret 是探针函数的名称，const void *ret 是探针函数的参数，它代表被捕获的函数的返回值。</p>
<p>然后，我们使用了 bpf_get_current_comm 函数获取当前任务的名称，并将其存储在 comm 数组中。</p>
<pre><code class="language-c"> bpf_get_current_comm(&amp;comm, sizeof(comm));
</code></pre>
<p>使用 bpf_get_current_pid_tgid 函数获取当前进程的 PID，并将其存储在 pid 变量中。</p>
<pre><code class="language-c"> pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
</code></pre>
<p>使用 bpf_probe_read_user_str 函数从用户空间读取 readline 函数的返回值，并将其存储在 str 数组中。</p>
<pre><code class="language-c"> bpf_probe_read_user_str(str, sizeof(str), ret);
</code></pre>
<p>最后使用 bpf_printk 函数输出 PID、任务名称和用户输入的字符串。</p>
<pre><code class="language-c"> bpf_printk(&quot;PID %d (%s) read: %s &quot;, pid, comm, str);
</code></pre>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。我们使用 eunomia-bpf 编译运行这个例子。</p>
<p>编译运行上述代码：</p>
<pre><code class="language-console">$ ecc bashreadline.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
$ sudo ecli run package.json
Runing eBPF program...
</code></pre>
<p>运行这段程序后，可以通过查看 /sys/kernel/debug/tracing/trace_pipe 文件来查看 eBPF 程序的输出：</p>
<pre><code class="language-console">$ sudo cat /sys/kernel/debug/tracing/trace_pipe
bash-32969 [000] d..31 64001.375748: bpf_trace_printk: PID 32969 (bash) read: fff
bash-32969 [000] d..31 64002.056951: bpf_trace_printk: PID 32969 (bash) read: fff
</code></pre>
<p>可以看到，我们成功的捕获了 bash 的 readline 函数调用，并获取了用户在 bash 中输入的命令行。</p>
<h2 id="总结-4"><a class="header" href="#总结-4">总结</a></h2>
<p>在上述代码中，我们使用了 SEC 宏来定义了一个 uprobe 探针，它指定了要捕获的用户空间程序 (bin/bash) 和要捕获的函数 (readline)。此外，我们还使用了 BPF_KRETPROBE 宏来定义了一个用于处理 readline 函数返回值的回调函数 (printret)。该函数可以获取到 readline 函数的返回值，并将其打印到内核日志中。通过这样的方式，我们就可以使用 eBPF 来捕获 bash 的 readline 函数调用，并获取用户在 bash 中输入的命令行。</p>
<p>更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程六捕获进程发送信号的系统调用集合使用-hash-map-保存状态"><a class="header" href="#ebpf-入门开发实践教程六捕获进程发送信号的系统调用集合使用-hash-map-保存状态">eBPF 入门开发实践教程六：捕获进程发送信号的系统调用集合，使用 hash map 保存状态</a></h1>
<p>eBPF (Extended Berkeley Packet Filter) 是 Linux 内核上的一个强大的网络和性能分析工具，它允许开发者在内核运行时动态加载、更新和运行用户定义的代码。</p>
<p>本文是 eBPF 入门开发实践教程的第六篇，主要介绍如何实现一个 eBPF 工具，捕获进程发送信号的系统调用集合，使用 hash map 保存状态。</p>
<h2 id="sigsnoop"><a class="header" href="#sigsnoop">sigsnoop</a></h2>
<p>示例代码如下：</p>
<pre><code class="language-c">#include &lt;vmlinux.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#define MAX_ENTRIES 10240
#define TASK_COMM_LEN 16
struct event {
unsigned int pid;
unsigned int tpid;
int sig;
int ret;
char comm[TASK_COMM_LEN];
};
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_ENTRIES);
__type(key, __u32);
__type(value, struct event);
} values SEC(&quot;.maps&quot;);
static int probe_entry(pid_t tpid, int sig)
{
struct event event = {};
__u64 pid_tgid;
__u32 tid;
pid_tgid = bpf_get_current_pid_tgid();
tid = (__u32)pid_tgid;
event.pid = pid_tgid &gt;&gt; 32;
event.tpid = tpid;
event.sig = sig;
bpf_get_current_comm(event.comm, sizeof(event.comm));
bpf_map_update_elem(&amp;values, &amp;tid, &amp;event, BPF_ANY);
return 0;
}
static int probe_exit(void *ctx, int ret)
{
__u64 pid_tgid = bpf_get_current_pid_tgid();
__u32 tid = (__u32)pid_tgid;
struct event *eventp;
eventp = bpf_map_lookup_elem(&amp;values, &amp;tid);
if (!eventp)
return 0;
eventp-&gt;ret = ret;
bpf_printk(&quot;PID %d (%s) sent signal %d to PID %d, ret = %d&quot;,
eventp-&gt;pid, eventp-&gt;comm, eventp-&gt;sig, eventp-&gt;tpid, ret);
cleanup:
bpf_map_delete_elem(&amp;values, &amp;tid);
return 0;
}
SEC(&quot;tracepoint/syscalls/sys_enter_kill&quot;)
int kill_entry(struct trace_event_raw_sys_enter *ctx)
{
pid_t tpid = (pid_t)ctx-&gt;args[0];
int sig = (int)ctx-&gt;args[1];
return probe_entry(tpid, sig);
}
SEC(&quot;tracepoint/syscalls/sys_exit_kill&quot;)
int kill_exit(struct trace_event_raw_sys_exit *ctx)
{
return probe_exit(ctx, ctx-&gt;ret);
}
char LICENSE[] SEC(&quot;license&quot;) = &quot;Dual BSD/GPL&quot;;
</code></pre>
<p>上面的代码定义了一个 eBPF 程序，用于捕获进程发送信号的系统调用，包括 kill、tkill 和 tgkill。它通过使用 tracepoint 来捕获系统调用的进入和退出事件，并在这些事件发生时执行指定的探针函数，例如 probe_entry 和 probe_exit。</p>
<p>在探针函数中，我们使用 bpf_map 存储捕获的事件信息，包括发送信号的进程 ID、接收信号的进程 ID、信号值和系统调用的返回值。在系统调用退出时，我们将获取存储在 bpf_map 中的事件信息，并使用 bpf_printk 打印进程 ID、进程名称、发送的信号和系统调用的返回值。</p>
<p>最后，我们还需要使用 SEC 宏来定义探针，并指定要捕获的系统调用的名称，以及要执行的探针函数。</p>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。我们使用 eunomia-bpf 编译运行这个例子。</p>
<p>编译运行上述代码：</p>
<pre><code class="language-shell">docker run -it -v `pwd`/:/src/ yunwei37/ebpm:latest
</code></pre>
<p>或者</p>
<pre><code class="language-console">$ ecc sigsnoop.bpf.c
Compiling bpf object...
Generating export types...
Packing ebpf object and config into package.json...
$ sudo ecli run package.json
Runing eBPF program...
</code></pre>
<p>运行这段程序后，可以通过查看 /sys/kernel/debug/tracing/trace_pipe 文件来查看 eBPF 程序的输出：</p>
<pre><code class="language-console">$ sudo cat /sys/kernel/debug/tracing/trace_pipe
node-3517 [003] d..31 82575.798191: bpf_trace_printk: PID 3517 (node) sent signal 0 to PID 3427, ret = 0
node-15194 [003] d..31 82575.849227: bpf_trace_printk: PID 15194 (node) sent signal 0 to PID 3427, ret = 0
node-30016 [003] d..31 82576.001361: bpf_trace_printk: PID 30016 (node) sent signal 0 to PID 3427, ret = 0
cpptools-srv-38617 [002] d..31 82576.461085: bpf_trace_printk: PID 38617 (cpptools-srv) sent signal 0 to PID 30496, ret = 0
node-30040 [002] d..31 82576.467720: bpf_trace_printk: PID 30016 (node) sent signal 0 to PID 3427, ret = 0
</code></pre>
<h2 id="总结-5"><a class="header" href="#总结-5">总结</a></h2>
<p>本文主要介绍如何实现一个 eBPF 工具，捕获进程发送信号的系统调用集合，使用 hash map 保存状态。使用 hash map 需要定义一个结构体：</p>
<pre><code class="language-c">struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_ENTRIES);
__type(key, __u32);
__type(value, struct event);
} values SEC(&quot;.maps&quot;);
</code></pre>
<p>并使用一些对应的 API 进行访问，例如 bpf_map_lookup_elem、bpf_map_update_elem、bpf_map_delete_elem 等。</p>
<p>更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门实践教程七捕获进程执行退出时间通过-perf-event-array-向用户态打印输出"><a class="header" href="#ebpf-入门实践教程七捕获进程执行退出时间通过-perf-event-array-向用户态打印输出">eBPF 入门实践教程七：捕获进程执行/退出时间，通过 perf event array 向用户态打印输出</a></h1>
<p>eBPF (Extended Berkeley Packet Filter) 是 Linux 内核上的一个强大的网络和性能分析工具，它允许开发者在内核运行时动态加载、更新和运行用户定义的代码。</p>
<p>本文是 eBPF 入门开发实践教程的第七篇，主要介绍如何捕获 Linux 内核中进程执行的事件，并且通过 perf event array 向用户态命令行打印输出，不需要再通过查看 /sys/kernel/debug/tracing/trace_pipe 文件来查看 eBPF 程序的输出。通过 perf event array 向用户态发送信息之后，可以进行复杂的数据处理和分析。</p>
<h2 id="perf-buffer"><a class="header" href="#perf-buffer">perf buffer</a></h2>
<p>eBPF 提供了两个环形缓冲区，可以用来将信息从 eBPF 程序传输到用户区控制器。第一个是perf环形缓冲区，，它至少从内核v4.15开始就存在了。第二个是后来引入的 BPF 环形缓冲区。本文只考虑perf环形缓冲区。</p>
<h2 id="execsnoop"><a class="header" href="#execsnoop">execsnoop</a></h2>
<p>通过 perf event array 向用户态命令行打印输出，需要编写一个头文件，一个 C 源文件。示例代码如下：</p>
<p>头文件：execsnoop.h</p>
<pre><code class="language-c">#ifndef __EXECSNOOP_H
#define __EXECSNOOP_H
#define TASK_COMM_LEN 16
struct event {
int pid;
int ppid;
int uid;
int retval;
bool is_exit;
char comm[TASK_COMM_LEN];
};
#endif /* __EXECSNOOP_H */
</code></pre>
<p>源文件：execsnoop.bpf.c</p>
<pre><code class="language-c">// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
#include &lt;vmlinux.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_core_read.h&gt;
#include &quot;execsnoop.h&quot;
struct {
__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
__uint(key_size, sizeof(u32));
__uint(value_size, sizeof(u32));
} events SEC(&quot;.maps&quot;);
SEC(&quot;tracepoint/syscalls/sys_enter_execve&quot;)
int tracepoint__syscalls__sys_enter_execve(struct trace_event_raw_sys_enter* ctx)
{
u64 id;
pid_t pid, tgid;
struct event event={0};
struct task_struct *task;
uid_t uid = (u32)bpf_get_current_uid_gid();
id = bpf_get_current_pid_tgid();
pid = (pid_t)id;
tgid = id &gt;&gt; 32;
event.pid = tgid;
event.uid = uid;
task = (struct task_struct*)bpf_get_current_task();
event.ppid = BPF_CORE_READ(task, real_parent, tgid);
bpf_get_current_comm(&amp;event.comm, sizeof(event.comm));
bpf_perf_event_output(ctx, &amp;events, BPF_F_CURRENT_CPU, &amp;event, sizeof(event));
return 0;
}
char LICENSE[] SEC(&quot;license&quot;) = &quot;GPL&quot;;
</code></pre>
<p>这段代码定义了个 eBPF 程序，用于捕获进程执行 execve 系统调用的入口。</p>
<p>在入口程序中，我们首先获取了当前进程的进程 ID 和用户 ID，然后通过 bpf_get_current_task 函数获取了当前进程的 task_struct 结构体，并通过 bpf_probe_read_str 函数读取了进程名称。最后，我们通过 bpf_perf_event_output 函数将进程执行事件输出到 perf buffer。</p>
<p>使用这段代码，我们就可以捕获 Linux 内核中进程执行的事件, 并分析进程的执行情况。</p>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。我们使用 eunomia-bpf 编译运行这个例子。</p>
<p>使用容器编译：</p>
<pre><code class="language-shell">docker run -it -v `pwd`/:/src/ yunwei37/ebpm:latest
</code></pre>
<p>或者使用 ecc 编译：</p>
<pre><code class="language-shell">ecc execsnoop.bpf.c execsnoop.h
</code></pre>
<p>运行</p>
<pre><code class="language-console">$ sudo ./ecli run package.json
TIME PID PPID UID COMM
21:28:30 40747 3517 1000 node
21:28:30 40748 40747 1000 sh
21:28:30 40749 3517 1000 node
21:28:30 40750 40749 1000 sh
21:28:30 40751 3517 1000 node
21:28:30 40752 40751 1000 sh
21:28:30 40753 40752 1000 cpuUsage.sh
</code></pre>
<h2 id="总结-6"><a class="header" href="#总结-6">总结</a></h2>
<p>本文介绍了如何捕获 Linux 内核中进程执行的事件，并且通过 perf event array 向用户态命令行打印输出，通过 perf event array 向用户态发送信息之后，可以进行复杂的数据处理和分析。在 libbpf 对应的内核态代码中，定义这样一个结构体和对应的头文件：</p>
<pre><code class="language-c">struct {
__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
__uint(key_size, sizeof(u32));
__uint(value_size, sizeof(u32));
} events SEC(&quot;.maps&quot;);
</code></pre>
<p>就可以往用户态直接发送信息。</p>
<p>更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程八在-ebpf-中使用-exitsnoop-监控进程退出事件使用-ring-buffer-向用户态打印输出"><a class="header" href="#ebpf-入门开发实践教程八在-ebpf-中使用-exitsnoop-监控进程退出事件使用-ring-buffer-向用户态打印输出">eBPF 入门开发实践教程八：在 eBPF 中使用 exitsnoop 监控进程退出事件，使用 ring buffer 向用户态打印输出</a></h1>
<p>eBPF (Extended Berkeley Packet Filter) 是 Linux 内核上的一个强大的网络和性能分析工具。它允许开发者在内核运行时动态加载、更新和运行用户定义的代码。</p>
<p>本文是 eBPF 入门开发实践教程的第八篇，在 eBPF 中使用 exitsnoop 监控进程退出事件。</p>
<h2 id="ring-buffer"><a class="header" href="#ring-buffer">ring buffer</a></h2>
<p>现在有一个新的 BPF 数据结构可用。BPF 环形缓冲区（ring buffer）。它解决了 BPF perf buffer（当今从内核向用户空间发送数据的事实上的标准）的内存效率和事件重排问题，同时达到或超过了它的性能。它既提供了与 perf buffer 兼容以方便迁移，又有新的保留/提交API，具有更好的可用性。另外，合成和真实世界的基准测试表明，在几乎所有的情况下，所以考虑将其作为从BPF程序向用户空间发送数据的默认选择。</p>
<h3 id="bpf-ringbuf-vs-bpf-perfbuf"><a class="header" href="#bpf-ringbuf-vs-bpf-perfbuf">BPF ringbuf vs BPF perfbuf</a></h3>
<p>今天，只要BPF程序需要将收集到的数据发送到用户空间进行后处理和记录，它通常会使用BPF perf buffer（perfbuf）来实现。Perfbuf 是每个CPU循环缓冲区的集合，它允许在内核和用户空间之间有效地交换数据。它在实践中效果很好，但由于其按CPU设计，它有两个主要的缺点，在实践中被证明是不方便的：内存的低效使用和事件的重新排序。</p>
<p>为了解决这些问题，从Linux 5.8开始，BPF提供了一个新的BPF数据结构（BPF map）。BPF环形缓冲区（ringbuf）。它是一个多生产者、单消费者（MPSC）队列，可以同时在多个CPU上安全共享。</p>
<p>BPF ringbuf 支持来自 BPF perfbuf 的熟悉的功能:</p>
<ul>
<li>变长的数据记录。</li>
<li>能够通过内存映射区域有效地从用户空间读取数据，而不需要额外的内存拷贝和/或进入内核的系统调用。</li>
<li>既支持epoll通知，又能以绝对最小的延迟进行忙环操作。</li>
</ul>
<p>同时，BPF ringbuf解决了BPF perfbuf的以下问题:</p>
<ul>
<li>内存开销。</li>
<li>数据排序。</li>
<li>浪费的工作和额外的数据复制。</li>
</ul>
<h2 id="exitsnoop"><a class="header" href="#exitsnoop">exitsnoop</a></h2>
<p>本文是 eBPF 入门开发实践教程的第八篇，在 eBPF 中使用 exitsnoop 监控进程退出事件，并使用 ring buffer 向用户态打印输出。</p>
<p>使用 ring buffer 向用户态打印输出的步骤和 perf buffer 类似，首先需要定义一个头文件：</p>
<p>头文件：exitsnoop.h</p>
<pre><code class="language-c">#ifndef __BOOTSTRAP_H
#define __BOOTSTRAP_H
#define TASK_COMM_LEN 16
#define MAX_FILENAME_LEN 127
struct event {
int pid;
int ppid;
unsigned exit_code;
unsigned long long duration_ns;
char comm[TASK_COMM_LEN];
};
#endif /* __BOOTSTRAP_H */
</code></pre>
<p>源文件：exitsnoop.bpf.c</p>
<pre><code class="language-c">#include &quot;vmlinux.h&quot;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#include &lt;bpf/bpf_core_read.h&gt;
#include &quot;exitsnoop.h&quot;
char LICENSE[] SEC(&quot;license&quot;) = &quot;Dual BSD/GPL&quot;;
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 256 * 1024);
} rb SEC(&quot;.maps&quot;);
SEC(&quot;tp/sched/sched_process_exit&quot;)
int handle_exit(struct trace_event_raw_sched_process_template* ctx)
{
struct task_struct *task;
struct event *e;
pid_t pid, tid;
u64 id, ts, *start_ts, duration_ns = 0;
/* get PID and TID of exiting thread/process */
id = bpf_get_current_pid_tgid();
pid = id &gt;&gt; 32;
tid = (u32)id;
/* ignore thread exits */
if (pid != tid)
return 0;
/* reserve sample from BPF ringbuf */
e = bpf_ringbuf_reserve(&amp;rb, sizeof(*e), 0);
if (!e)
return 0;
/* fill out the sample with data */
task = (struct task_struct *)bpf_get_current_task();
e-&gt;duration_ns = duration_ns;
e-&gt;pid = pid;
e-&gt;ppid = BPF_CORE_READ(task, real_parent, tgid);
e-&gt;exit_code = (BPF_CORE_READ(task, exit_code) &gt;&gt; 8) &amp; 0xff;
bpf_get_current_comm(&amp;e-&gt;comm, sizeof(e-&gt;comm));
/* send data to user-space for post-processing */
bpf_ringbuf_submit(e, 0);
return 0;
}
</code></pre>
<p>这段代码是一个 BPF 程序，用于监控 Linux 系统中的进程退出事件。</p>
<p>该程序通过注册一个 tracepoint，来监控进程退出事件。Tracepoint 是一种内核特性，允许内核模块获取特定事件的通知。在本程序中，注册的 tracepoint 是“tp/sched/sched_process_exit”，表示该程序监控的是进程退出事件。</p>
<p>当系统中发生进程退出事件时，BPF 程序会捕获该事件，并调用“handle_exit”函数来处理它。该函数首先检查当前退出事件是否是进程退出事件（而不是线程退出事件），然后在 BPF 环形缓冲区（“rb”）中保留一个事件结构体，并填充该结构体中的其他信息，例如进程 ID、进程名称、退出代码和退出信号等信息。最后，该函数还会调用 BPF 的“perf_event_output”函数，将捕获的事件发送给用户空间程序。</p>
<p>总而言之，这段代码是一个 BPF 程序，用于监控 Linux 系统中的进程退出事件.</p>
<h2 id="compile-and-run"><a class="header" href="#compile-and-run">Compile and Run</a></h2>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。我们使用 eunomia-bpf 编译运行这个例子。</p>
<p>Compile:</p>
<pre><code class="language-shell">docker run -it -v `pwd`/:/src/ yunwei37/ebpm:latest
</code></pre>
<p>Or</p>
<pre><code class="language-console">$ ecc exitsnoop.bpf.c exitsnoop.h
Compiling bpf object...
Generating export types...
Packing ebpf object and config into package.json...
</code></pre>
<p>Run:</p>
<pre><code class="language-console">$ sudo ./ecli run package.json
TIME PID PPID EXIT_CODE DURATION_NS COMM
21:40:09 42050 42049 0 0 which
21:40:09 42049 3517 0 0 sh
21:40:09 42052 42051 0 0 ps
21:40:09 42051 3517 0 0 sh
21:40:09 42055 42054 0 0 sed
21:40:09 42056 42054 0 0 cat
21:40:09 42057 42054 0 0 cat
21:40:09 42058 42054 0 0 cat
21:40:09 42059 42054 0 0 cat
</code></pre>
<h2 id="总结-7"><a class="header" href="#总结-7">总结</a></h2>
<p>本文介绍了如何使用 eunomia-bpf 开发一个简单的 BPF 程序，该程序可以监控 Linux 系统中的进程退出事件, 并将捕获的事件通过 ring buffer 发送给用户空间程序。在本文中，我们使用 eunomia-bpf 编译运行了这个例子。如果你想了解更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程九一个-linux-内核-bpf-程序通过柱状图来总结调度程序运行队列延迟显示任务等待运行在-cpu-上的时间长度"><a class="header" href="#ebpf-入门开发实践教程九一个-linux-内核-bpf-程序通过柱状图来总结调度程序运行队列延迟显示任务等待运行在-cpu-上的时间长度">eBPF 入门开发实践教程九：一个 Linux 内核 BPF 程序，通过柱状图来总结调度程序运行队列延迟，显示任务等待运行在 CPU 上的时间长度</a></h1>
<p>eBPF (Extended Berkeley Packet Filter) 是 Linux 内核上的一个强大的网络和性能分析工具。它允许开发者在内核运行时动态加载、更新和运行用户定义的代码。</p>
<h2 id="runqlat是什么"><a class="header" href="#runqlat是什么">runqlat是什么？</a></h2>
<p>bcc-tools 是一组用于在 Linux 系统上使用 BPF 程序的工具。runqlat 是 bcc-tools 中的一个工具，用于分析 Linux 系统的调度性能。具体来说，runqlat 用于测量一个任务在被调度到 CPU 上运行之前在运行队列中等待的时间。这些信息对于识别性能瓶颈和提高 Linux 内核调度算法的整体效率非常有用。</p>
<h2 id="runqlat-原理"><a class="header" href="#runqlat-原理">runqlat 原理</a></h2>
<p>runqlat 使用内核跟踪点和函数探针的结合来测量进程在运行队列中的时间。当进程被排队时，trace_enqueue 函数会在一个映射中记录时间戳。当进程被调度到 CPU 上运行时，handle_switch 函数会检索时间戳，并计算当前时间与排队时间之间的时间差。这个差值（或 delta）然后用于更新进程的直方图，该直方图记录运行队列延迟的分布。该直方图可用于分析 Linux 内核的调度性能。</p>
<h2 id="runqlat-代码实现"><a class="header" href="#runqlat-代码实现">runqlat 代码实现</a></h2>
<p>首先我们需要编写一个源代码文件 runqlat.bpf.c:</p>
<pre><code class="language-c">// SPDX-License-Identifier: GPL-2.0
// Copyright (c) 2020 Wenbo Zhang
#include &lt;vmlinux.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_core_read.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#include &quot;runqlat.h&quot;
#include &quot;bits.bpf.h&quot;
#include &quot;maps.bpf.h&quot;
#include &quot;core_fixes.bpf.h&quot;
#define MAX_ENTRIES 10240
#define TASK_RUNNING 0
const volatile bool filter_cg = false;
const volatile bool targ_per_process = false;
const volatile bool targ_per_thread = false;
const volatile bool targ_per_pidns = false;
const volatile bool targ_ms = false;
const volatile pid_t targ_tgid = 0;
struct {
__uint(type, BPF_MAP_TYPE_CGROUP_ARRAY);
__type(key, u32);
__type(value, u32);
__uint(max_entries, 1);
} cgroup_map SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_ENTRIES);
__type(key, u32);
__type(value, u64);
} start SEC(&quot;.maps&quot;);
static struct hist zero;
/// @sample {&quot;interval&quot;: 1000, &quot;type&quot; : &quot;log2_hist&quot;}
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_ENTRIES);
__type(key, u32);
__type(value, struct hist);
} hists SEC(&quot;.maps&quot;);
static int trace_enqueue(u32 tgid, u32 pid)
{
u64 ts;
if (!pid)
return 0;
if (targ_tgid &amp;&amp; targ_tgid != tgid)
return 0;
ts = bpf_ktime_get_ns();
bpf_map_update_elem(&amp;start, &amp;pid, &amp;ts, BPF_ANY);
return 0;
}
static unsigned int pid_namespace(struct task_struct *task)
{
struct pid *pid;
unsigned int level;
struct upid upid;
unsigned int inum;
/* get the pid namespace by following task_active_pid_ns(),
* pid-&gt;numbers[pid-&gt;level].ns
*/
pid = BPF_CORE_READ(task, thread_pid);
level = BPF_CORE_READ(pid, level);
bpf_core_read(&amp;upid, sizeof(upid), &amp;pid-&gt;numbers[level]);
inum = BPF_CORE_READ(upid.ns, ns.inum);
return inum;
}
static int handle_switch(bool preempt, struct task_struct *prev, struct task_struct *next)
{
struct hist *histp;
u64 *tsp, slot;
u32 pid, hkey;
s64 delta;
if (filter_cg &amp;&amp; !bpf_current_task_under_cgroup(&amp;cgroup_map, 0))
return 0;
if (get_task_state(prev) == TASK_RUNNING)
trace_enqueue(BPF_CORE_READ(prev, tgid), BPF_CORE_READ(prev, pid));
pid = BPF_CORE_READ(next, pid);
tsp = bpf_map_lookup_elem(&amp;start, &amp;pid);
if (!tsp)
return 0;
delta = bpf_ktime_get_ns() - *tsp;
if (delta &lt; 0)
goto cleanup;
if (targ_per_process)
hkey = BPF_CORE_READ(next, tgid);
else if (targ_per_thread)
hkey = pid;
else if (targ_per_pidns)
hkey = pid_namespace(next);
else
hkey = -1;
histp = bpf_map_lookup_or_try_init(&amp;hists, &amp;hkey, &amp;zero);
if (!histp)
goto cleanup;
if (!histp-&gt;comm[0])
bpf_probe_read_kernel_str(&amp;histp-&gt;comm, sizeof(histp-&gt;comm),
next-&gt;comm);
if (targ_ms)
delta /= 1000000U;
else
delta /= 1000U;
slot = log2l(delta);
if (slot &gt;= MAX_SLOTS)
slot = MAX_SLOTS - 1;
__sync_fetch_and_add(&amp;histp-&gt;slots[slot], 1);
cleanup:
bpf_map_delete_elem(&amp;start, &amp;pid);
return 0;
}
SEC(&quot;raw_tp/sched_wakeup&quot;)
int BPF_PROG(handle_sched_wakeup, struct task_struct *p)
{
if (filter_cg &amp;&amp; !bpf_current_task_under_cgroup(&amp;cgroup_map, 0))
return 0;
return trace_enqueue(BPF_CORE_READ(p, tgid), BPF_CORE_READ(p, pid));
}
SEC(&quot;raw_tp/sched_wakeup_new&quot;)
int BPF_PROG(handle_sched_wakeup_new, struct task_struct *p)
{
if (filter_cg &amp;&amp; !bpf_current_task_under_cgroup(&amp;cgroup_map, 0))
return 0;
return trace_enqueue(BPF_CORE_READ(p, tgid), BPF_CORE_READ(p, pid));
}
SEC(&quot;raw_tp/sched_switch&quot;)
int BPF_PROG(handle_sched_switch, bool preempt, struct task_struct *prev, struct task_struct *next)
{
return handle_switch(preempt, prev, next);
}
char LICENSE[] SEC(&quot;license&quot;) = &quot;GPL&quot;;
</code></pre>
<p>然后我们需要定义一个头文件<code>runqlat.h</code>，用来给用户态处理从内核态上报的事件：</p>
<pre><code class="language-c">/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
#ifndef __RUNQLAT_H
#define __RUNQLAT_H
#define TASK_COMM_LEN 16
#define MAX_SLOTS 26
struct hist {
__u32 slots[MAX_SLOTS];
char comm[TASK_COMM_LEN];
};
#endif /* __RUNQLAT_H */
</code></pre>
<p>这是一个 Linux 内核 BPF 程序，旨在收集和报告运行队列的延迟。BPF 是 Linux 内核中一项技术，它允许将程序附加到内核中的特定点并进行安全高效的执行。这些程序可用于收集有关内核行为的信息，并实现自定义行为。这个 BPF 程序使用 BPF maps 来收集有关任务何时从内核的运行队列中排队和取消排队的信息，并记录任务在被安排执行之前在运行队列上等待的时间。然后，它使用这些信息生成直方图，显示不同组任务的运行队列延迟分布。这些直方图可用于识别和诊断内核调度行为中的性能问题。</p>
<h2 id="编译运行"><a class="header" href="#编译运行">编译运行</a></h2>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。我们使用 eunomia-bpf 编译运行这个例子。</p>
<p>Compile:</p>
<pre><code class="language-shell">docker run -it -v `pwd`/:/src/ yunwei37/ebpm:latest
</code></pre>
<p>或者</p>
<pre><code class="language-console">$ ecc runqlat.bpf.c runqlat.h
Compiling bpf object...
Generating export types...
Packing ebpf object and config into package.json...
</code></pre>
<p>Run:</p>
<pre><code class="language-console">$ sudo ecli run examples/bpftools/runqlat/package.json -h
Usage: runqlat_bpf [--help] [--version] [--verbose] [--filter_cg] [--targ_per_process] [--targ_per_thread] [--targ_per_pidns] [--targ_ms] [--targ_tgid VAR]
A simple eBPF program
Optional arguments:
-h, --help shows help message and exits
-v, --version prints version information and exits
--verbose prints libbpf debug information
--filter_cg set value of bool variable filter_cg
--targ_per_process set value of bool variable targ_per_process
--targ_per_thread set value of bool variable targ_per_thread
--targ_per_pidns set value of bool variable targ_per_pidns
--targ_ms set value of bool variable targ_ms
--targ_tgid set value of pid_t variable targ_tgid
Built with eunomia-bpf framework.
See https://github.com/eunomia-bpf/eunomia-bpf for more information.
$ sudo ecli run examples/bpftools/runqlat/package.json
key = 4294967295
comm = rcu_preempt
(unit) : count distribution
0 -&gt; 1 : 9 |**** |
2 -&gt; 3 : 6 |** |
4 -&gt; 7 : 12 |***** |
8 -&gt; 15 : 28 |************* |
16 -&gt; 31 : 40 |******************* |
32 -&gt; 63 : 83 |****************************************|
64 -&gt; 127 : 57 |*************************** |
128 -&gt; 255 : 19 |********* |
256 -&gt; 511 : 11 |***** |
512 -&gt; 1023 : 2 | |
1024 -&gt; 2047 : 2 | |
2048 -&gt; 4095 : 0 | |
4096 -&gt; 8191 : 0 | |
8192 -&gt; 16383 : 0 | |
16384 -&gt; 32767 : 1 | |
$ sudo ecli run examples/bpftools/runqlat/package.json --targ_per_process
key = 3189
comm = cpptools
(unit) : count distribution
0 -&gt; 1 : 0 | |
2 -&gt; 3 : 0 | |
4 -&gt; 7 : 0 | |
8 -&gt; 15 : 1 |*** |
16 -&gt; 31 : 2 |******* |
32 -&gt; 63 : 11 |****************************************|
64 -&gt; 127 : 8 |***************************** |
128 -&gt; 255 : 3 |********** |
</code></pre>
<h2 id="总结-8"><a class="header" href="#总结-8">总结</a></h2>
<p>runqlat 是一个 Linux 内核 BPF 程序，通过柱状图来总结调度程序运行队列延迟，显示任务等待运行在 CPU 上的时间长度。编译这个程序可以使用 ecc 工具，运行时可以使用 ecli 命令。</p>
<p>runqlat 是一种用于监控Linux内核中进程调度延迟的工具。它可以帮助您了解进程在内核中等待执行的时间，并根据这些信息优化进程调度，提高系统的性能。可以在 libbpf-tools 中找到最初的源代码：<a href="https://github.com/iovisor/bcc/blob/master/libbpf-tools/runqlat.bpf.c">https://github.com/iovisor/bcc/blob/master/libbpf-tools/runqlat.bpf.c</a></p>
<p>更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程十在-ebpf-中使用-hardirqs-或-softirqs-捕获中断事件"><a class="header" href="#ebpf-入门开发实践教程十在-ebpf-中使用-hardirqs-或-softirqs-捕获中断事件">eBPF 入门开发实践教程十：在 eBPF 中使用 hardirqs 或 softirqs 捕获中断事件</a></h1>
<p>eBPF (Extended Berkeley Packet Filter) 是 Linux 内核上的一个强大的网络和性能分析工具。它允许开发者在内核运行时动态加载、更新和运行用户定义的代码。</p>
<p>本文是 eBPF 入门开发实践教程的第十篇，在 eBPF 中。</p>
<h2 id="hardirqs是什么"><a class="header" href="#hardirqs是什么">hardirqs是什么？</a></h2>
<p>hardirqs 是 bcc-tools 工具包的一部分，该工具包是一组用于在 Linux 系统上执行系统跟踪和分析的实用程序。
hardirqs 是一种用于跟踪和分析 Linux 内核中的中断处理程序的工具。它使用 BPF（Berkeley Packet Filter）程序来收集有关中断处理程序的数据，
并可用于识别内核中的性能问题和其他与中断处理相关的问题。</p>
<h2 id="实现原理"><a class="header" href="#实现原理">实现原理</a></h2>
<p>在 Linux 内核中，每个中断处理程序都有一个唯一的名称，称为中断向量。hardirqs 通过检查每个中断处理程序的中断向量，来监控内核中的中断处理程序。当内核接收到一个中断时，它会查找与该中断相关的中断处理程序，并执行该程序。hardirqs 通过检查内核中执行的中断处理程序，来监控内核中的中断处理程序。另外，hardirqs 还可以通过注入 BPF 程序到内核中，来捕获内核中的中断处理程序。这样，hardirqs 就可以监控内核中执行的中断处理程序，并收集有关它们的信息。</p>
<h2 id="代码实现"><a class="header" href="#代码实现">代码实现</a></h2>
<pre><code class="language-c">// SPDX-License-Identifier: GPL-2.0
// Copyright (c) 2020 Wenbo Zhang
#include &lt;vmlinux.h&gt;
#include &lt;bpf/bpf_core_read.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#include &quot;hardirqs.h&quot;
#include &quot;bits.bpf.h&quot;
#include &quot;maps.bpf.h&quot;
#define MAX_ENTRIES 256
const volatile bool filter_cg = false;
const volatile bool targ_dist = false;
const volatile bool targ_ns = false;
const volatile bool do_count = false;
struct {
__uint(type, BPF_MAP_TYPE_CGROUP_ARRAY);
__type(key, u32);
__type(value, u32);
__uint(max_entries, 1);
} cgroup_map SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__uint(max_entries, 1);
__type(key, u32);
__type(value, u64);
} start SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_ENTRIES);
__type(key, struct irq_key);
__type(value, struct info);
} infos SEC(&quot;.maps&quot;);
static struct info zero;
static int handle_entry(int irq, struct irqaction *action)
{
if (filter_cg &amp;&amp; !bpf_current_task_under_cgroup(&amp;cgroup_map, 0))
return 0;
if (do_count) {
struct irq_key key = {};
struct info *info;
bpf_probe_read_kernel_str(&amp;key.name, sizeof(key.name), BPF_CORE_READ(action, name));
info = bpf_map_lookup_or_try_init(&amp;infos, &amp;key, &amp;zero);
if (!info)
return 0;
info-&gt;count += 1;
return 0;
} else {
u64 ts = bpf_ktime_get_ns();
u32 key = 0;
if (filter_cg &amp;&amp; !bpf_current_task_under_cgroup(&amp;cgroup_map, 0))
return 0;
bpf_map_update_elem(&amp;start, &amp;key, &amp;ts, BPF_ANY);
return 0;
}
}
static int handle_exit(int irq, struct irqaction *action)
{
struct irq_key ikey = {};
struct info *info;
u32 key = 0;
u64 delta;
u64 *tsp;
if (filter_cg &amp;&amp; !bpf_current_task_under_cgroup(&amp;cgroup_map, 0))
return 0;
tsp = bpf_map_lookup_elem(&amp;start, &amp;key);
if (!tsp)
return 0;
delta = bpf_ktime_get_ns() - *tsp;
if (!targ_ns)
delta /= 1000U;
bpf_probe_read_kernel_str(&amp;ikey.name, sizeof(ikey.name), BPF_CORE_READ(action, name));
info = bpf_map_lookup_or_try_init(&amp;infos, &amp;ikey, &amp;zero);
if (!info)
return 0;
if (!targ_dist) {
info-&gt;count += delta;
} else {
u64 slot;
slot = log2(delta);
if (slot &gt;= MAX_SLOTS)
slot = MAX_SLOTS - 1;
info-&gt;slots[slot]++;
}
return 0;
}
SEC(&quot;tp_btf/irq_handler_entry&quot;)
int BPF_PROG(irq_handler_entry_btf, int irq, struct irqaction *action)
{
return handle_entry(irq, action);
}
SEC(&quot;tp_btf/irq_handler_exit&quot;)
int BPF_PROG(irq_handler_exit_btf, int irq, struct irqaction *action)
{
return handle_exit(irq, action);
}
SEC(&quot;raw_tp/irq_handler_entry&quot;)
int BPF_PROG(irq_handler_entry, int irq, struct irqaction *action)
{
return handle_entry(irq, action);
}
SEC(&quot;raw_tp/irq_handler_exit&quot;)
int BPF_PROG(irq_handler_exit, int irq, struct irqaction *action)
{
return handle_exit(irq, action);
}
char LICENSE[] SEC(&quot;license&quot;) = &quot;GPL&quot;;
</code></pre>
<p>这是一个 BPF（Berkeley Packet Filter）程序。BPF 程序是小型程序，可以直接在 Linux 内核中运行，用于过滤和操纵网络流量。这个特定的程序似乎旨在收集内核中中断处理程序的统计信息。它定义了一些 maps （可以在 BPF 程序和内核的其他部分之间共享的数据结构）和两个函数：handle_entry 和 handle_exit。当内核进入和退出中断处理程序时，分别执行这些函数。handle_entry 函数用于跟踪中断处理程序被执行的次数，而 handle_exit 则用于测量中断处理程序中花费的时间。</p>
<h2 id="运行代码"><a class="header" href="#运行代码">运行代码</a></h2>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。我们使用 eunomia-bpf 编译运行这个例子。</p>
<p>要编译这个程序，请使用 ecc 工具：</p>
<pre><code class="language-console">$ ecc hardirqs.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
</code></pre>
<p>然后运行：</p>
<pre><code class="language-console">sudo ecli run ./package.json
</code></pre>
<h2 id="总结-9"><a class="header" href="#总结-9">总结</a></h2>
<p>更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门开发实践教程十一在-ebpf-中使用-bootstrap-开发用户态程序并跟踪-exec-和-exit-系统调用"><a class="header" href="#ebpf-入门开发实践教程十一在-ebpf-中使用-bootstrap-开发用户态程序并跟踪-exec-和-exit-系统调用">eBPF 入门开发实践教程十一：在 eBPF 中使用 bootstrap 开发用户态程序并跟踪 exec() 和 exit() 系统调用</a></h1>
<p>eBPF (Extended Berkeley Packet Filter) 是 Linux 内核上的一个强大的网络和性能分析工具。它允许开发者在内核运行时动态加载、更新和运行用户定义的代码。</p>
<h2 id="什么是bootstrap"><a class="header" href="#什么是bootstrap">什么是bootstrap?</a></h2>
<p>Bootstrap是一个工具，它使用BPF（Berkeley Packet Filter）程序跟踪执行exec()系统调用（使用SEC（“tp/sched/sched_process_exec”）handle_exit BPF程序），这大致对应于新进程的生成（忽略fork（）部分）。此外，它还跟踪exit（）（使用SEC（“tp/sched/sched_process_exit”）handle_exit BPF程序）以了解每个进程何时退出。这两个BPF程序共同工作，允许捕获有关任何新进程的有趣信息，例如二进制文件的文件名，以及测量进程的生命周期并在进程死亡时收集有趣的统计信息，例如退出代码或消耗的资源量等。我认为这是深入了解内核内部并观察事物如何真正运作的良好起点。</p>
<p>Bootstrap还使用argp API（libc的一部分）进行命令行参数解析。</p>
<h2 id="bootstrap"><a class="header" href="#bootstrap">Bootstrap</a></h2>
<p>TODO: 添加关于用户态的应用部分，以及关于 libbpf-boostrap 的完整介绍。也许可以参考类似：http://cn-sec.com/archives/1267522.html 的文档。</p>
<pre><code class="language-c">// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/* Copyright (c) 2020 Facebook */
#include &quot;vmlinux.h&quot;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#include &lt;bpf/bpf_core_read.h&gt;
#include &quot;bootstrap.h&quot;
char LICENSE[] SEC(&quot;license&quot;) = &quot;Dual BSD/GPL&quot;;
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 8192);
__type(key, pid_t);
__type(value, u64);
} exec_start SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 256 * 1024);
} rb SEC(&quot;.maps&quot;);
const volatile unsigned long long min_duration_ns = 0;
SEC(&quot;tp/sched/sched_process_exec&quot;)
int handle_exec(struct trace_event_raw_sched_process_exec *ctx)
{
struct task_struct *task;
unsigned fname_off;
struct event *e;
pid_t pid;
u64 ts;
/* remember time exec() was executed for this PID */
pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
ts = bpf_ktime_get_ns();
bpf_map_update_elem(&amp;exec_start, &amp;pid, &amp;ts, BPF_ANY);
/* don't emit exec events when minimum duration is specified */
if (min_duration_ns)
return 0;
/* reserve sample from BPF ringbuf */
e = bpf_ringbuf_reserve(&amp;rb, sizeof(*e), 0);
if (!e)
return 0;
/* fill out the sample with data */
task = (struct task_struct *)bpf_get_current_task();
e-&gt;exit_event = false;
e-&gt;pid = pid;
e-&gt;ppid = BPF_CORE_READ(task, real_parent, tgid);
bpf_get_current_comm(&amp;e-&gt;comm, sizeof(e-&gt;comm));
fname_off = ctx-&gt;__data_loc_filename &amp; 0xFFFF;
bpf_probe_read_str(&amp;e-&gt;filename, sizeof(e-&gt;filename), (void *)ctx + fname_off);
/* successfully submit it to user-space for post-processing */
bpf_ringbuf_submit(e, 0);
return 0;
}
SEC(&quot;tp/sched/sched_process_exit&quot;)
int handle_exit(struct trace_event_raw_sched_process_template* ctx)
{
struct task_struct *task;
struct event *e;
pid_t pid, tid;
u64 id, ts, *start_ts, duration_ns = 0;
/* get PID and TID of exiting thread/process */
id = bpf_get_current_pid_tgid();
pid = id &gt;&gt; 32;
tid = (u32)id;
/* ignore thread exits */
if (pid != tid)
return 0;
/* if we recorded start of the process, calculate lifetime duration */
start_ts = bpf_map_lookup_elem(&amp;exec_start, &amp;pid);
if (start_ts)
duration_ns = bpf_ktime_get_ns() - *start_ts;
else if (min_duration_ns)
return 0;
bpf_map_delete_elem(&amp;exec_start, &amp;pid);
/* if process didn't live long enough, return early */
if (min_duration_ns &amp;&amp; duration_ns &lt; min_duration_ns)
return 0;
/* reserve sample from BPF ringbuf */
e = bpf_ringbuf_reserve(&amp;rb, sizeof(*e), 0);
if (!e)
return 0;
/* fill out the sample with data */
task = (struct task_struct *)bpf_get_current_task();
e-&gt;exit_event = true;
e-&gt;duration_ns = duration_ns;
e-&gt;pid = pid;
e-&gt;ppid = BPF_CORE_READ(task, real_parent, tgid);
e-&gt;exit_code = (BPF_CORE_READ(task, exit_code) &gt;&gt; 8) &amp; 0xff;
bpf_get_current_comm(&amp;e-&gt;comm, sizeof(e-&gt;comm));
/* send data to user-space for post-processing */
bpf_ringbuf_submit(e, 0);
return 0;
}
</code></pre>
<p>这是一段使用BPF（Berkeley Packet Filter）的C程序，用于跟踪进程启动和退出事件，并显示有关它们的信息。BPF是一种强大的机制，允许您将称为BPF程序的小程序附加到Linux内核的各个部分。这些程序可用于过滤，监视或修改内核的行为。</p>
<p>程序首先定义一些常量，并包含一些头文件。然后定义了一个名为env的struct，用于存储一些程序选项，例如详细模式和进程报告的最小持续时间。</p>
<p>然后，程序定义了一个名为parse_arg的函数，用于解析传递给程序的命令行参数。它接受三个参数：一个表示正在解析的选项的整数key，一个表示选项参数的字符指针arg和一个表示当前解析状态的struct argp_state指针state。该函数处理选项并在env struct中设置相应的值。</p>
<p>然后，程序定义了一个名为sig_handler的函数，当被调用时会将全局标志exiting设置为true。这用于在接收到信号时允许程序干净地退出。</p>
<p>接下来，我们将继续描述这段代码中的其他部分。</p>
<p>程序定义了一个名为exec_start的BPF map，它的类型为BPF_MAP_TYPE_HASH，最大条目数为8192，键类型为pid_t，值类型为u64。</p>
<p>另外，程序还定义了一个名为rb的BPF map，它的类型为BPF_MAP_TYPE_RINGBUF，最大条目数为256 * 1024。</p>
<p>程序还定义了一个名为min_duration_ns的常量，其值为0。</p>
<p>程序定义了一个名为handle_exec的SEC（static evaluator of code）函数，它被附加到跟踪进程执行的BPF程序上。该函数记录为该PID执行exec（）的时间，并在指定了最小持续时间时不发出exec事件。如果未指定最小持续时间，则会从BPF ringbuf保留样本并使用数据填充样本，然后将其提交给用户空间进行后处理。</p>
<p>程序还定义了一个名为handle_exit的SEC函数，它被附加到跟踪进程退出的BPF程序上。该函数会在确定PID和TID后计算进程的生命周期，然后根据min_duration_ns的值决定是否发出退出事件。如果进程的生命周期足够长，则会从BPF ringbuf保留样本并使用数据填充样本，然后将其提交给用户空间进行后处理。</p>
<p>最后，主函数调用bpf_ringbuf_poll来轮询BPF ringbuf，并在接收到新的事件时处理该事件。这个函数会持续运行，直到全局标志exiting被设置为true，此时它会清理资源并退出。</p>
<h2 id="install-dependencies"><a class="header" href="#install-dependencies">Install Dependencies</a></h2>
<p>You will need <code>clang</code>, <code>libelf</code> and <code>zlib</code> to build the examples, package names may vary across distros.</p>
<p>On Ubuntu/Debian, you need:</p>
<pre><code class="language-shell">$ apt install clang libelf1 libelf-dev zlib1g-dev
</code></pre>
<p>On CentOS/Fedora, you need:</p>
<pre><code class="language-shell">$ dnf install clang elfutils-libelf elfutils-libelf-devel zlib-devel
</code></pre>
<h2 id="编译运行-1"><a class="header" href="#编译运行-1">编译运行</a></h2>
<p>编译运行上述代码：</p>
<pre><code class="language-console">$ ecc bootstrap.bpf.c bootstrap.h
Compiling bpf object...
Packing ebpf object and config into package.json...
$ sudo ecli run package.json
Runing eBPF program...
</code></pre>
<h2 id="总结-10"><a class="header" href="#总结-10">总结</a></h2>
<p>这是一个使用BPF的C程序，用于跟踪进程的启动和退出事件，并显示有关这些事件的信息。它通过使用argp API来解析命令行参数，并使用BPF maps 存储进程的信息，包括进程的PID和执行文件的文件名。程序还使用了SEC函数来附加BPF程序，以监视进程的执行和退出事件。最后，程序在终端中打印出启动和退出的进程信息。</p>
<p>编译这个程序可以使用 ecc 工具，运行时可以使用 ecli 命令。更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：https://github.com/eunomia-bpf/eunomia-bpf</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf入门实践教程使用-libbpf-bootstrap-开发程序统计-tcp-连接延时"><a class="header" href="#ebpf入门实践教程使用-libbpf-bootstrap-开发程序统计-tcp-连接延时">eBPF入门实践教程：使用 libbpf-bootstrap 开发程序统计 TCP 连接延时</a></h1>
<h2 id="背景"><a class="header" href="#背景">背景</a></h2>
<p>在互联网后端日常开发接口的时候中，不管你使用的是C、Java、PHP还是Golang，都避免不了需要调用mysql、redis等组件来获取数据，可能还需要执行一些rpc远程调用，或者再调用一些其它restful api。 在这些调用的底层，基本都是在使用TCP协议进行传输。这是因为在传输层协议中，TCP协议具备可靠的连接，错误重传，拥塞控制等优点，所以目前应用比UDP更广泛一些。但相对而言，tcp 连接也有一些缺点，例如建立连接的延时较长等。因此也会出现像 QUIC ，即 快速UDP网络连接 ( Quick UDP Internet Connections )这样的替代方案。</p>
<p>tcp 连接延时分析对于网络性能分析优化或者故障排查都能起到不少作用。</p>
<h2 id="tcpconnlat-的实现原理"><a class="header" href="#tcpconnlat-的实现原理">tcpconnlat 的实现原理</a></h2>
<p>tcpconnlat 这个工具跟踪执行活动TCP连接的内核函数 (例如，通过connect()系统调用），并显示本地测量的连接的延迟（时间），即从发送 SYN 到响应包的时间。</p>
<h3 id="tcp-连接原理"><a class="header" href="#tcp-连接原理">tcp 连接原理</a></h3>
<p>tcp 连接的整个过程如图所示：</p>
<p><img src="13-tcpconnlat/tcpconnlat1.png" alt="tcpconnlate" /></p>
<p>在这个连接过程中，我们来简单分析一下每一步的耗时：</p>
<ol>
<li>客户端发出SYNC包：客户端一般是通过connect系统调用来发出 SYN 的，这里牵涉到本机的系统调用和软中断的 CPU 耗时开销</li>
<li>SYN传到服务器：SYN从客户端网卡被发出，这是一次长途远距离的网络传输</li>
<li>服务器处理SYN包：内核通过软中断来收包，然后放到半连接队列中，然后再发出SYN/ACK响应。主要是 CPU 耗时开销</li>
<li>SYC/ACK传到客户端：长途网络跋涉</li>
<li>客户端处理 SYN/ACK：客户端内核收包并处理SYN后，经过几us的CPU处理，接着发出 ACK。同样是软中断处理开销</li>
<li>ACK传到服务器：长途网络跋涉</li>
<li>服务端收到ACK：服务器端内核收到并处理ACK，然后把对应的连接从半连接队列中取出来，然后放到全连接队列中。一次软中断CPU开销</li>
<li>服务器端用户进程唤醒：正在被accpet系统调用阻塞的用户进程被唤醒，然后从全连接队列中取出来已经建立好的连接。一次上下文切换的CPU开销</li>
</ol>
<p>在客户端视角，在正常情况下一次TCP连接总的耗时也就就大约是一次网络RTT的耗时。但在某些情况下，可能会导致连接时的网络传输耗时上涨、CPU处理开销增加、甚至是连接失败。这种时候在发现延时过长之后，就可以结合其他信息进行分析。</p>
<h3 id="ebpf-实现原理"><a class="header" href="#ebpf-实现原理">ebpf 实现原理</a></h3>
<p>在 TCP 三次握手的时候，Linux 内核会维护两个队列，分别是：</p>
<ul>
<li>半连接队列，也称 SYN 队列；</li>
<li>全连接队列，也称 accepet 队列；</li>
</ul>
<p>服务端收到客户端发起的 SYN 请求后，内核会把该连接存储到半连接队列，并向客户端响应 SYN+ACK，接着客户端会返回 ACK，服务端收到第三次握手的 ACK 后，内核会把连接从半连接队列移除，然后创建新的完全的连接，并将其添加到 accept 队列，等待进程调用 accept 函数时把连接取出来。</p>
<p>我们的 ebpf 代码实现在 <a href="https://github.com/yunwei37/Eunomia/blob/master/bpftools/tcpconnlat/tcpconnlat.bpf.c">https://github.com/yunwei37/Eunomia/blob/master/bpftools/tcpconnlat/tcpconnlat.bpf.c</a> 中：</p>
<p>它主要使用了 trace_tcp_rcv_state_process 和 kprobe/tcp_v4_connect 这样的跟踪点：</p>
<pre><code class="language-c">
SEC(&quot;kprobe/tcp_v4_connect&quot;)
int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
{
return trace_connect(sk);
}
SEC(&quot;kprobe/tcp_v6_connect&quot;)
int BPF_KPROBE(tcp_v6_connect, struct sock *sk)
{
return trace_connect(sk);
}
SEC(&quot;kprobe/tcp_rcv_state_process&quot;)
int BPF_KPROBE(tcp_rcv_state_process, struct sock *sk)
{
return handle_tcp_rcv_state_process(ctx, sk);
}
</code></pre>
<p>在 trace_connect 中，我们跟踪新的 tcp 连接，记录到达时间，并且把它加入 map 中：</p>
<pre><code class="language-c">struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 4096);
__type(key, struct sock *);
__type(value, struct piddata);
} start SEC(&quot;.maps&quot;);
static int trace_connect(struct sock *sk)
{
u32 tgid = bpf_get_current_pid_tgid() &gt;&gt; 32;
struct piddata piddata = {};
if (targ_tgid &amp;&amp; targ_tgid != tgid)
return 0;
bpf_get_current_comm(&amp;piddata.comm, sizeof(piddata.comm));
piddata.ts = bpf_ktime_get_ns();
piddata.tgid = tgid;
bpf_map_update_elem(&amp;start, &amp;sk, &amp;piddata, 0);
return 0;
}
</code></pre>
<p>在 handle_tcp_rcv_state_process 中，我们跟踪接收到的 tcp 数据包，从 map 从提取出对应的 connect 事件，并且计算延迟：</p>
<pre><code class="language-c">static int handle_tcp_rcv_state_process(void *ctx, struct sock *sk)
{
struct piddata *piddatap;
struct event event = {};
s64 delta;
u64 ts;
if (BPF_CORE_READ(sk, __sk_common.skc_state) != TCP_SYN_SENT)
return 0;
piddatap = bpf_map_lookup_elem(&amp;start, &amp;sk);
if (!piddatap)
return 0;
ts = bpf_ktime_get_ns();
delta = (s64)(ts - piddatap-&gt;ts);
if (delta &lt; 0)
goto cleanup;
event.delta_us = delta / 1000U;
if (targ_min_us &amp;&amp; event.delta_us &lt; targ_min_us)
goto cleanup;
__builtin_memcpy(&amp;event.comm, piddatap-&gt;comm,
sizeof(event.comm));
event.ts_us = ts / 1000;
event.tgid = piddatap-&gt;tgid;
event.lport = BPF_CORE_READ(sk, __sk_common.skc_num);
event.dport = BPF_CORE_READ(sk, __sk_common.skc_dport);
event.af = BPF_CORE_READ(sk, __sk_common.skc_family);
if (event.af == AF_INET) {
event.saddr_v4 = BPF_CORE_READ(sk, __sk_common.skc_rcv_saddr);
event.daddr_v4 = BPF_CORE_READ(sk, __sk_common.skc_daddr);
} else {
BPF_CORE_READ_INTO(&amp;event.saddr_v6, sk,
__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
BPF_CORE_READ_INTO(&amp;event.daddr_v6, sk,
__sk_common.skc_v6_daddr.in6_u.u6_addr32);
}
bpf_perf_event_output(ctx, &amp;events, BPF_F_CURRENT_CPU,
&amp;event, sizeof(event));
cleanup:
bpf_map_delete_elem(&amp;start, &amp;sk);
return 0;
}
</code></pre>
<h2 id="编译运行-2"><a class="header" href="#编译运行-2">编译运行</a></h2>
<ul>
<li><code>git clone https://github.com/libbpf/libbpf-bootstrap libbpf-bootstrap-cloned</code></li>
<li>将 <a href="13-tcpconnlat/libbpf-bootstrap">libbpf-bootstrap</a>目录下的文件复制到 <code>libbpf-bootstrap-cloned/examples/c</code>下</li>
<li>修改 <code>libbpf-bootstrap-cloned/examples/c/Makefile</code> ，在其 <code>APPS</code> 项后添加 <code>tcpconnlat</code></li>
<li>在 <code>libbpf-bootstrap-cloned/examples/c</code> 下运行 <code>make tcpconnlat</code></li>
<li><code>sudo ./tcpconnlat</code></li>
</ul>
<h2 id="效果"><a class="header" href="#效果">效果</a></h2>
<pre><code class="language-plain">root@yutong-VirtualBox:~/libbpf-bootstrap/examples/c# ./tcpconnlat
PID COMM IP SADDR DADDR DPORT LAT(ms)
222564 wget 4 192.168.88.15 110.242.68.3 80 25.29
222684 wget 4 192.168.88.15 167.179.101.42 443 246.76
222726 ssh 4 192.168.88.15 167.179.101.42 22 241.17
222774 ssh 4 192.168.88.15 1.15.149.151 22 25.31
</code></pre>
<h2 id="总结-11"><a class="header" href="#总结-11">总结</a></h2>
<p>通过上面的实验，我们可以看到，tcpconnlat 工具的实现原理是基于内核的TCP连接的跟踪，并且可以跟踪到 tcp 连接的延迟时间；除了命令行使用方式之外，还可以将其和容器、k8s 等元信息综合起来，通过 <code>prometheus</code> 和 <code>grafana</code> 等工具进行网络性能分析。</p>
<p>来源：<a href="https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcpconnlat.bpf.c">https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcpconnlat.bpf.c</a></p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门实践教程编写-ebpf-程序-tcpconnlat-测量-tcp-连接延时"><a class="header" href="#ebpf-入门实践教程编写-ebpf-程序-tcpconnlat-测量-tcp-连接延时">eBPF 入门实践教程：编写 eBPF 程序 tcpconnlat 测量 tcp 连接延时</a></h1>
<h2 id="代码解释"><a class="header" href="#代码解释">代码解释</a></h2>
<h3 id="背景-1"><a class="header" href="#背景-1">背景</a></h3>
<p>在互联网后端日常开发接口的时候中，不管你使用的是C、Java、PHP还是Golang，都避免不了需要调用mysql、redis等组件来获取数据，可能还需要执行一些rpc远程调用，或者再调用一些其它restful api。 在这些调用的底层，基本都是在使用TCP协议进行传输。这是因为在传输层协议中，TCP协议具备可靠的连接，错误重传，拥塞控制等优点，所以目前应用比UDP更广泛一些。但相对而言，tcp 连接也有一些缺点，例如建立连接的延时较长等。因此也会出现像 QUIC ，即 快速UDP网络连接 ( Quick UDP Internet Connections )这样的替代方案。</p>
<p>tcp 连接延时分析对于网络性能分析优化或者故障排查都能起到不少作用。</p>
<h3 id="tcpconnlat-的实现原理-1"><a class="header" href="#tcpconnlat-的实现原理-1">tcpconnlat 的实现原理</a></h3>
<p>tcpconnlat 这个工具跟踪执行活动TCP连接的内核函数 (例如，通过connect()系统调用），并显示本地测量的连接的延迟（时间），即从发送 SYN 到响应包的时间。</p>
<h3 id="tcp-连接原理-1"><a class="header" href="#tcp-连接原理-1">tcp 连接原理</a></h3>
<p>tcp 连接的整个过程如图所示：</p>
<p><img src="13-tcpconnlat/tcpconnlat1.png" alt="tcpconnlate" /></p>
<p>在这个连接过程中，我们来简单分析一下每一步的耗时：</p>
<ol>
<li>客户端发出SYNC包：客户端一般是通过connect系统调用来发出 SYN 的，这里牵涉到本机的系统调用和软中断的 CPU 耗时开销</li>
<li>SYN传到服务器：SYN从客户端网卡被发出，这是一次长途远距离的网络传输</li>
<li>服务器处理SYN包：内核通过软中断来收包，然后放到半连接队列中，然后再发出SYN/ACK响应。主要是 CPU 耗时开销</li>
<li>SYC/ACK传到客户端：长途网络跋涉</li>
<li>客户端处理 SYN/ACK：客户端内核收包并处理SYN后，经过几us的CPU处理，接着发出 ACK。同样是软中断处理开销</li>
<li>ACK传到服务器：长途网络跋涉</li>
<li>服务端收到ACK：服务器端内核收到并处理ACK，然后把对应的连接从半连接队列中取出来，然后放到全连接队列中。一次软中断CPU开销</li>
<li>服务器端用户进程唤醒：正在被accpet系统调用阻塞的用户进程被唤醒，然后从全连接队列中取出来已经建立好的连接。一次上下文切换的CPU开销</li>
</ol>
<p>在客户端视角，在正常情况下一次TCP连接总的耗时也就就大约是一次网络RTT的耗时。但在某些情况下，可能会导致连接时的网络传输耗时上涨、CPU处理开销增加、甚至是连接失败。这种时候在发现延时过长之后，就可以结合其他信息进行分析。</p>
<h3 id="ebpf-实现原理-1"><a class="header" href="#ebpf-实现原理-1">ebpf 实现原理</a></h3>
<p>在 TCP 三次握手的时候，Linux 内核会维护两个队列，分别是：</p>
<ul>
<li>半连接队列，也称 SYN 队列；</li>
<li>全连接队列，也称 accepet 队列；</li>
</ul>
<p>服务端收到客户端发起的 SYN 请求后，内核会把该连接存储到半连接队列，并向客户端响应 SYN+ACK，接着客户端会返回 ACK，服务端收到第三次握手的 ACK 后，内核会把连接从半连接队列移除，然后创建新的完全的连接，并将其添加到 accept 队列，等待进程调用 accept 函数时把连接取出来。</p>
<p>我们的 ebpf 代码实现在 <a href="https://github.com/yunwei37/Eunomia/blob/master/bpftools/tcpconnlat/tcpconnlat.bpf.c">https://github.com/yunwei37/Eunomia/blob/master/bpftools/tcpconnlat/tcpconnlat.bpf.c</a> 中：</p>
<p>它主要使用了 trace_tcp_rcv_state_process 和 kprobe/tcp_v4_connect 这样的跟踪点：</p>
<pre><code class="language-c">
SEC(&quot;kprobe/tcp_v4_connect&quot;)
int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
{
return trace_connect(sk);
}
SEC(&quot;kprobe/tcp_v6_connect&quot;)
int BPF_KPROBE(tcp_v6_connect, struct sock *sk)
{
return trace_connect(sk);
}
SEC(&quot;kprobe/tcp_rcv_state_process&quot;)
int BPF_KPROBE(tcp_rcv_state_process, struct sock *sk)
{
return handle_tcp_rcv_state_process(ctx, sk);
}
</code></pre>
<p>在 trace_connect 中，我们跟踪新的 tcp 连接，记录到达时间，并且把它加入 map 中：</p>
<pre><code class="language-c">struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 4096);
__type(key, struct sock *);
__type(value, struct piddata);
} start SEC(&quot;.maps&quot;);
static int trace_connect(struct sock *sk)
{
u32 tgid = bpf_get_current_pid_tgid() &gt;&gt; 32;
struct piddata piddata = {};
if (targ_tgid &amp;&amp; targ_tgid != tgid)
return 0;
bpf_get_current_comm(&amp;piddata.comm, sizeof(piddata.comm));
piddata.ts = bpf_ktime_get_ns();
piddata.tgid = tgid;
bpf_map_update_elem(&amp;start, &amp;sk, &amp;piddata, 0);
return 0;
}
</code></pre>
<p>在 handle_tcp_rcv_state_process 中，我们跟踪接收到的 tcp 数据包，从 map 从提取出对应的 connect 事件，并且计算延迟：</p>
<pre><code class="language-c">static int handle_tcp_rcv_state_process(void *ctx, struct sock *sk)
{
struct piddata *piddatap;
struct event event = {};
s64 delta;
u64 ts;
if (BPF_CORE_READ(sk, __sk_common.skc_state) != TCP_SYN_SENT)
return 0;
piddatap = bpf_map_lookup_elem(&amp;start, &amp;sk);
if (!piddatap)
return 0;
ts = bpf_ktime_get_ns();
delta = (s64)(ts - piddatap-&gt;ts);
if (delta &lt; 0)
goto cleanup;
event.delta_us = delta / 1000U;
if (targ_min_us &amp;&amp; event.delta_us &lt; targ_min_us)
goto cleanup;
__builtin_memcpy(&amp;event.comm, piddatap-&gt;comm,
sizeof(event.comm));
event.ts_us = ts / 1000;
event.tgid = piddatap-&gt;tgid;
event.lport = BPF_CORE_READ(sk, __sk_common.skc_num);
event.dport = BPF_CORE_READ(sk, __sk_common.skc_dport);
event.af = BPF_CORE_READ(sk, __sk_common.skc_family);
if (event.af == AF_INET) {
event.saddr_v4 = BPF_CORE_READ(sk, __sk_common.skc_rcv_saddr);
event.daddr_v4 = BPF_CORE_READ(sk, __sk_common.skc_daddr);
} else {
BPF_CORE_READ_INTO(&amp;event.saddr_v6, sk,
__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
BPF_CORE_READ_INTO(&amp;event.daddr_v6, sk,
__sk_common.skc_v6_daddr.in6_u.u6_addr32);
}
bpf_perf_event_output(ctx, &amp;events, BPF_F_CURRENT_CPU,
&amp;event, sizeof(event));
cleanup:
bpf_map_delete_elem(&amp;start, &amp;sk);
return 0;
}
</code></pre>
<h3 id="编译运行-3"><a class="header" href="#编译运行-3">编译运行</a></h3>
<p>TODO</p>
<h3 id="总结-12"><a class="header" href="#总结-12">总结</a></h3>
<p>通过上面的实验，我们可以看到，tcpconnlat 工具的实现原理是基于内核的TCP连接的跟踪，并且可以跟踪到 tcp 连接的延迟时间；除了命令行使用方式之外，还可以将其和容器、k8s 等元信息综合起来，通过 <code>prometheus</code> 和 <code>grafana</code> 等工具进行网络性能分析。</p>
<blockquote>
<p><code>Eunomia</code> 是一个使用 C/C++ 开发的基于 eBPF的轻量级，高性能云原生监控工具，旨在帮助用户了解容器的各项行为、监控可疑的容器安全事件，力求提供覆盖容器全生命周期的轻量级开源监控解决方案。它使用 <code>Linux</code> <code>eBPF</code> 技术在运行时跟踪您的系统和应用程序，并分析收集的事件以检测可疑的行为模式。目前，它包含性能分析、容器集群网络可视化分析*、容器安全感知告警、一键部署、持久化存储监控等功能，提供了多样化的 ebpf 追踪点。其核心导出器/命令行工具最小仅需要约 4MB 大小的二进制程序，即可在支持的 Linux 内核上启动。</p>
</blockquote>
<p>项目地址：<a href="https://github.com/yunwei37/Eunomia">https://github.com/yunwei37/Eunomia</a></p>
<h3 id="参考资料-1"><a class="header" href="#参考资料-1">参考资料</a></h3>
<ol>
<li><a href="http://kerneltravel.net/blog/2020/tcpconnlat/">http://kerneltravel.net/blog/2020/tcpconnlat/</a></li>
<li><a href="https://network.51cto.com/article/640631.html">https://network.51cto.com/article/640631.html</a></li>
</ol>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf入门实践教程使用-libbpf-bootstrap-开发程序统计-tcp-连接延时-1"><a class="header" href="#ebpf入门实践教程使用-libbpf-bootstrap-开发程序统计-tcp-连接延时-1">eBPF入门实践教程：使用 libbpf-bootstrap 开发程序统计 TCP 连接延时</a></h1>
<p><code>tcpstates</code> 是一个追踪当前系统上的TCP套接字的TCP状态的程序，主要通过跟踪内核跟踪点 <code>inet_sock_set_state</code> 来实现。统计数据通过 <code>perf_event</code>向用户态传输。</p>
<pre><code class="language-c">SEC(&quot;tracepoint/sock/inet_sock_set_state&quot;)
int handle_set_state(struct trace_event_raw_inet_sock_set_state *ctx)
</code></pre>
<p>在套接字改变状态处附加一个eBPF跟踪函数。</p>
<pre><code class="language-c"> if (ctx-&gt;protocol != IPPROTO_TCP)
return 0;
if (target_family &amp;&amp; target_family != family)
return 0;
if (filter_by_sport &amp;&amp; !bpf_map_lookup_elem(&amp;sports, &amp;sport))
return 0;
if (filter_by_dport &amp;&amp; !bpf_map_lookup_elem(&amp;dports, &amp;dport))
return 0;
</code></pre>
<p>跟踪函数被调用后，先判断当前改变状态的套接字是否满足我们需要的过滤条件，如果不满足则不进行记录。</p>
<pre><code class="language-c"> tsp = bpf_map_lookup_elem(&amp;timestamps, &amp;sk);
ts = bpf_ktime_get_ns();
if (!tsp)
delta_us = 0;
else
delta_us = (ts - *tsp) / 1000;
event.skaddr = (__u64)sk;
event.ts_us = ts / 1000;
event.delta_us = delta_us;
event.pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
event.oldstate = ctx-&gt;oldstate;
event.newstate = ctx-&gt;newstate;
event.family = family;
event.sport = sport;
event.dport = dport;
bpf_get_current_comm(&amp;event.task, sizeof(event.task));
if (family == AF_INET) {
bpf_probe_read_kernel(&amp;event.saddr, sizeof(event.saddr), &amp;sk-&gt;__sk_common.skc_rcv_saddr);
bpf_probe_read_kernel(&amp;event.daddr, sizeof(event.daddr), &amp;sk-&gt;__sk_common.skc_daddr);
} else { /* family == AF_INET6 */
bpf_probe_read_kernel(&amp;event.saddr, sizeof(event.saddr), &amp;sk-&gt;__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
bpf_probe_read_kernel(&amp;event.daddr, sizeof(event.daddr), &amp;sk-&gt;__sk_common.skc_v6_daddr.in6_u.u6_addr32);
}
</code></pre>
<p>使用状态改变相关填充event结构体。</p>
<ul>
<li>此处使用了<code>libbpf</code> 的 CO-RE 支持。</li>
</ul>
<pre><code class="language-c"> bpf_perf_event_output(ctx, &amp;events, BPF_F_CURRENT_CPU, &amp;event, sizeof(event));
</code></pre>
<p>将事件结构体发送至用户态程序。</p>
<pre><code class="language-c"> if (ctx-&gt;newstate == TCP_CLOSE)
bpf_map_delete_elem(&amp;timestamps, &amp;sk);
else
bpf_map_update_elem(&amp;timestamps, &amp;sk, &amp;ts, BPF_ANY);
</code></pre>
<p>根据这个TCP链接的新状态，决定是更新下时间戳记录还是不再记录它的时间戳。</p>
<h2 id="用户态程序"><a class="header" href="#用户态程序">用户态程序</a></h2>
<pre><code class="language-c"> while (!exiting) {
err = perf_buffer__poll(pb, PERF_POLL_TIMEOUT_MS);
if (err &lt; 0 &amp;&amp; err != -EINTR) {
warn(&quot;error polling perf buffer: %s\n&quot;, strerror(-err));
goto cleanup;
}
/* reset err to return 0 if exiting */
err = 0;
}
</code></pre>
<p>不停轮询内核程序所发过来的 <code>perf event</code>。</p>
<pre><code class="language-c">static void handle_event(void* ctx, int cpu, void* data, __u32 data_sz) {
char ts[32], saddr[26], daddr[26];
struct event* e = data;
struct tm* tm;
int family;
time_t t;
if (emit_timestamp) {
time(&amp;t);
tm = localtime(&amp;t);
strftime(ts, sizeof(ts), &quot;%H:%M:%S&quot;, tm);
printf(&quot;%8s &quot;, ts);
}
inet_ntop(e-&gt;family, &amp;e-&gt;saddr, saddr, sizeof(saddr));
inet_ntop(e-&gt;family, &amp;e-&gt;daddr, daddr, sizeof(daddr));
if (wide_output) {
family = e-&gt;family == AF_INET ? 4 : 6;
printf(
&quot;%-16llx %-7d %-16s %-2d %-26s %-5d %-26s %-5d %-11s -&gt; %-11s &quot;
&quot;%.3f\n&quot;,
e-&gt;skaddr, e-&gt;pid, e-&gt;task, family, saddr, e-&gt;sport, daddr,
e-&gt;dport, tcp_states[e-&gt;oldstate], tcp_states[e-&gt;newstate],
(double)e-&gt;delta_us / 1000);
} else {
printf(
&quot;%-16llx %-7d %-10.10s %-15s %-5d %-15s %-5d %-11s -&gt; %-11s %.3f\n&quot;,
e-&gt;skaddr, e-&gt;pid, e-&gt;task, saddr, e-&gt;sport, daddr, e-&gt;dport,
tcp_states[e-&gt;oldstate], tcp_states[e-&gt;newstate],
(double)e-&gt;delta_us / 1000);
}
}
static void handle_lost_events(void* ctx, int cpu, __u64 lost_cnt) {
warn(&quot;lost %llu events on CPU #%d\n&quot;, lost_cnt, cpu);
}
</code></pre>
<p>收到事件后所调用对应的处理函数并进行输出打印。</p>
<h2 id="编译运行-4"><a class="header" href="#编译运行-4">编译运行</a></h2>
<ul>
<li><code>git clone https://github.com/libbpf/libbpf-bootstrap libbpf-bootstrap-cloned</code></li>
<li>将 <a href="14-tcpstates/libbpf-bootstrap">libbpf-bootstrap</a>目录下的文件复制到 <code>libbpf-bootstrap-cloned/examples/c</code>下</li>
<li>修改 <code>libbpf-bootstrap-cloned/examples/c/Makefile</code> ，在其 <code>APPS</code> 项后添加 <code>tcpstates</code></li>
<li>在 <code>libbpf-bootstrap-cloned/examples/c</code> 下运行 <code>make tcpstates</code></li>
<li><code>sudo ./tcpstates</code></li>
</ul>
<h2 id="效果-1"><a class="header" href="#效果-1">效果</a></h2>
<pre><code class="language-plain">root@yutong-VirtualBox:~/libbpf-bootstrap/examples/c# ./tcpstates
SKADDR PID COMM LADDR LPORT RADDR RPORT OLDSTATE -&gt; NEWSTATE MS
ffff9bf61bb62bc0 164978 node 192.168.88.15 0 52.178.17.2 443 CLOSE -&gt; SYN_SENT 0.000
ffff9bf61bb62bc0 0 swapper/0 192.168.88.15 41596 52.178.17.2 443 SYN_SENT -&gt; ESTABLISHED 225.794
ffff9bf61bb62bc0 0 swapper/0 192.168.88.15 41596 52.178.17.2 443 ESTABLISHED -&gt; CLOSE_WAIT 901.454
ffff9bf61bb62bc0 164978 node 192.168.88.15 41596 52.178.17.2 443 CLOSE_WAIT -&gt; LAST_ACK 0.793
ffff9bf61bb62bc0 164978 node 192.168.88.15 41596 52.178.17.2 443 LAST_ACK -&gt; LAST_ACK 0.086
ffff9bf61bb62bc0 228759 kworker/u6 192.168.88.15 41596 52.178.17.2 443 LAST_ACK -&gt; CLOSE 0.193
ffff9bf6d8ee88c0 229832 redis-serv 0.0.0.0 6379 0.0.0.0 0 CLOSE -&gt; LISTEN 0.000
ffff9bf6d8ee88c0 229832 redis-serv 0.0.0.0 6379 0.0.0.0 0 LISTEN -&gt; CLOSE 1.763
ffff9bf7109d6900 88750 node 127.0.0.1 39755 127.0.0.1 50966 ESTABLISHED -&gt; FIN_WAIT1 0.000
</code></pre>
<p>对于输出的详细解释，详见 <a href="14-tcpstates/README.html">README.md</a></p>
<h2 id="总结-13"><a class="header" href="#总结-13">总结</a></h2>
<p>这里的代码修改自 <a href="https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcpstates.bpf.c">https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcpstates.bpf.c</a></p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门实践教程编写-ebpf-程序-tcprtt-测量-tcp-连接的往返时间"><a class="header" href="#ebpf-入门实践教程编写-ebpf-程序-tcprtt-测量-tcp-连接的往返时间">eBPF 入门实践教程：编写 eBPF 程序 Tcprtt 测量 TCP 连接的往返时间</a></h1>
<h2 id="背景-2"><a class="header" href="#背景-2">背景</a></h2>
<p>网络质量在互联网社会中是一个很重要的因素。导致网络质量差的因素有很多，可能是硬件因素导致，也可能是程序
写的不好导致。为了能更好地定位网络问题，<code>tcprtt</code> 工具被提出。它可以监测TCP链接的往返时间，从而分析
网络质量，帮助用户定位问题来源。</p>
<p>当有tcp链接建立时，该工具会自动根据当前系统的支持情况，选择合适的执行函数。
在执行函数中，<code>tcprtt</code>会收集tcp链接的各项基本信息，包括地址，源端口，目标端口，耗时
等等，并将其更新到直方图的map中。运行结束后通过用户态代码，展现给用户。</p>
<h2 id="编写-ebpf-程序-1"><a class="header" href="#编写-ebpf-程序-1">编写 eBPF 程序</a></h2>
<pre><code class="language-c">// SPDX-License-Identifier: GPL-2.0
// Copyright (c) 2021 Wenbo Zhang
#include &lt;vmlinux.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_core_read.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#include &lt;bpf/bpf_endian.h&gt;
#include &quot;tcprtt.h&quot;
#include &quot;bits.bpf.h&quot;
#include &quot;maps.bpf.h&quot;
char LICENSE[] SEC(&quot;license&quot;) = &quot;Dual BSD/GPL&quot;;
const volatile bool targ_laddr_hist = false;
const volatile bool targ_raddr_hist = false;
const volatile bool targ_show_ext = false;
const volatile __u16 targ_sport = 0;
const volatile __u16 targ_dport = 0;
const volatile __u32 targ_saddr = 0;
const volatile __u32 targ_daddr = 0;
const volatile bool targ_ms = false;
#define MAX_ENTRIES 10240
/// @sample {&quot;interval&quot;: 1000, &quot;type&quot; : &quot;log2_hist&quot;}
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, MAX_ENTRIES);
__type(key, u64);
__type(value, struct hist);
} hists SEC(&quot;.maps&quot;);
static struct hist zero;
SEC(&quot;fentry/tcp_rcv_established&quot;)
int BPF_PROG(tcp_rcv, struct sock *sk)
{
const struct inet_sock *inet = (struct inet_sock *)(sk);
struct tcp_sock *ts;
struct hist *histp;
u64 key, slot;
u32 srtt;
if (targ_sport &amp;&amp; targ_sport != inet-&gt;inet_sport)
return 0;
if (targ_dport &amp;&amp; targ_dport != sk-&gt;__sk_common.skc_dport)
return 0;
if (targ_saddr &amp;&amp; targ_saddr != inet-&gt;inet_saddr)
return 0;
if (targ_daddr &amp;&amp; targ_daddr != sk-&gt;__sk_common.skc_daddr)
return 0;
if (targ_laddr_hist)
key = inet-&gt;inet_saddr;
else if (targ_raddr_hist)
key = inet-&gt;sk.__sk_common.skc_daddr;
else
key = 0;
histp = bpf_map_lookup_or_try_init(&amp;hists, &amp;key, &amp;zero);
if (!histp)
return 0;
ts = (struct tcp_sock *)(sk);
srtt = BPF_CORE_READ(ts, srtt_us) &gt;&gt; 3;
if (targ_ms)
srtt /= 1000U;
slot = log2l(srtt);
if (slot &gt;= MAX_SLOTS)
slot = MAX_SLOTS - 1;
__sync_fetch_and_add(&amp;histp-&gt;slots[slot], 1);
if (targ_show_ext) {
__sync_fetch_and_add(&amp;histp-&gt;latency, srtt);
__sync_fetch_and_add(&amp;histp-&gt;cnt, 1);
}
return 0;
}
</code></pre>
<p>这段代码是基于eBPF的网络延迟分析工具，它通过hooking TCP协议栈中的tcp_rcv_established函数来统计TCP连接的RTT分布。下面是这段代码的主要工作原理：</p>
<ol>
<li>
<p>首先定义了一个名为&quot;hists&quot;的eBPF哈希表，用于保存RTT直方图数据。</p>
</li>
<li>
<p>当tcp_rcv_established函数被调用时，它首先从传入的socket结构体中获取TCP相关信息，包括本地/远程IP地址、本地/远程端口号以及TCP状态信息等。</p>
</li>
<li>
<p>接下来，代码会检查用户指定的条件是否匹配当前TCP连接。如果匹配失败，则直接返回。</p>
</li>
<li>
<p>如果匹配成功，则从&quot;hists&quot;哈希表中查找与本地/远程IP地址匹配的直方图数据。如果该IP地址的直方图不存在，则创建一个新的直方图并插入哈希表中。</p>
</li>
<li>
<p>接下来，代码会从socket结构体中获取当前TCP连接的RTT(srtt)，并根据用户设置的选项来将srtt值进行处理。如果用户设置了&quot;-ms&quot;选项，则将srtt值除以1000。</p>
</li>
<li>
<p>接着，代码会将srtt值转换为直方图的槽位(slot)，并将该槽位的计数器+1。</p>
</li>
<li>
<p>如果用户设置了&quot;-show-ext&quot;选项，则还会累加直方图的总延迟(latency)和计数(cnt)。</p>
</li>
</ol>
<h2 id="编译运行-5"><a class="header" href="#编译运行-5">编译运行</a></h2>
<p>eunomia-bpf 是一个结合 Wasm 的开源 eBPF 动态加载运行时和开发工具链，它的目的是简化 eBPF 程序的开发、构建、分发、运行。可以参考 <a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a> 下载和安装 ecc 编译工具链和 ecli 运行时。我们使用 eunomia-bpf 编译运行这个例子。</p>
<p>Compile:</p>
<pre><code class="language-shell">docker run -it -v `pwd`/:/src/ yunwei37/ebpm:latest
</code></pre>
<p>或者</p>
<pre><code class="language-console">$ ecc runqlat.bpf.c runqlat.h
Compiling bpf object...
Generating export types...
Packing ebpf object and config into package.json...
</code></pre>
<p>Run:</p>
<pre><code class="language-console">$ sudo ecli run package.json -h
A simple eBPF program
Usage: package.json [OPTIONS]
Options:
--verbose Whether to show libbpf debug information
--targ_laddr_hist Set value of `bool` variable targ_laddr_hist
--targ_raddr_hist Set value of `bool` variable targ_raddr_hist
--targ_show_ext Set value of `bool` variable targ_show_ext
--targ_sport &lt;targ_sport&gt; Set value of `__u16` variable targ_sport
--targ_dport &lt;targ_dport&gt; Set value of `__u16` variable targ_dport
--targ_saddr &lt;targ_saddr&gt; Set value of `__u32` variable targ_saddr
--targ_daddr &lt;targ_daddr&gt; Set value of `__u32` variable targ_daddr
--targ_ms Set value of `bool` variable targ_ms
-h, --help Print help
-V, --version Print version
Built with eunomia-bpf framework.
See https://github.com/eunomia-bpf/eunomia-bpf for more information.
$ sudo ecli run package.json
key = 0
latency = 0
cnt = 0
(unit) : count distribution
0 -&gt; 1 : 0 | |
2 -&gt; 3 : 0 | |
4 -&gt; 7 : 0 | |
8 -&gt; 15 : 0 | |
16 -&gt; 31 : 0 | |
32 -&gt; 63 : 0 | |
64 -&gt; 127 : 0 | |
128 -&gt; 255 : 0 | |
256 -&gt; 511 : 0 | |
512 -&gt; 1023 : 4 |******************** |
1024 -&gt; 2047 : 1 |***** |
2048 -&gt; 4095 : 0 | |
4096 -&gt; 8191 : 8 |****************************************|
key = 0
latency = 0
cnt = 0
(unit) : count distribution
0 -&gt; 1 : 0 | |
2 -&gt; 3 : 0 | |
4 -&gt; 7 : 0 | |
8 -&gt; 15 : 0 | |
16 -&gt; 31 : 0 | |
32 -&gt; 63 : 0 | |
64 -&gt; 127 : 0 | |
128 -&gt; 255 : 0 | |
256 -&gt; 511 : 0 | |
512 -&gt; 1023 : 11 |*************************** |
1024 -&gt; 2047 : 1 |** |
2048 -&gt; 4095 : 0 | |
4096 -&gt; 8191 : 16 |****************************************|
8192 -&gt; 16383 : 4 |********** |
</code></pre>
<h2 id="总结-14"><a class="header" href="#总结-14">总结</a></h2>
<p>tcprtt是一个基于eBPF的TCP延迟分析工具。通过hooking TCP协议栈中的tcp_rcv_established函数来统计TCP连接的RTT分布，可以对指定的TCP连接进行RTT分布统计，并将结果保存到eBPF哈希表中。同时，这个工具支持多种条件过滤和RTT分布数据扩展功能，以便用户可以更好地进行网络性能分析和调优。</p>
<p>更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门实践教程编写-ebpf-程序-memleak-监控内存泄漏"><a class="header" href="#ebpf-入门实践教程编写-ebpf-程序-memleak-监控内存泄漏">eBPF 入门实践教程：编写 eBPF 程序 Memleak 监控内存泄漏</a></h1>
<h2 id="背景-3"><a class="header" href="#背景-3">背景</a></h2>
<p>内存泄漏对于一个程序而言是一个很严重的问题。倘若放任一个存在内存泄漏的程序运行，久而久之
系统的内存会慢慢被耗尽，导致程序运行速度显著下降。为了避免这一情况，<code>memleak</code>工具被提出。
它可以跟踪并匹配内存分配和释放的请求，并且打印出已经被分配资源而又尚未释放的堆栈信息。</p>
<h2 id="实现原理-1"><a class="header" href="#实现原理-1">实现原理</a></h2>
<p><code>memleak</code> 的实现逻辑非常直观。它在我们常用的动态分配内存的函数接口路径上挂载了ebpf程序，
同时在free上也挂载了ebpf程序。在调用分配内存相关函数时，<code>memleak</code> 会记录调用者的pid，分配得到
内存的地址，分配得到的内存大小等基本数据。在free之后，<code>memeleak</code>则会去map中删除记录的对应的分配
信息。对于用户态常用的分配函数 <code>malloc</code>, <code>calloc</code> 等，<code>memleak</code>使用了 uporbe 技术实现挂载，对于
内核态的函数，比如 <code>kmalloc</code> 等，<code>memleak</code> 则使用了现有的 tracepoint 来实现。</p>
<h2 id="编写-ebpf-程序-2"><a class="header" href="#编写-ebpf-程序-2">编写 eBPF 程序</a></h2>
<pre><code class="language-c">struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, pid_t);
__type(value, u64);
__uint(max_entries, 10240);
} sizes SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, u64); /* address */
__type(value, struct alloc_info);
__uint(max_entries, ALLOCS_MAX_ENTRIES);
} allocs SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, u64); /* stack id */
__type(value, union combined_alloc_info);
__uint(max_entries, COMBINED_ALLOCS_MAX_ENTRIES);
} combined_allocs SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, u64);
__type(value, u64);
__uint(max_entries, 10240);
} memptrs SEC(&quot;.maps&quot;);
struct {
__uint(type, BPF_MAP_TYPE_STACK_TRACE);
__type(key, u32);
} stack_traces SEC(&quot;.maps&quot;);
struct alloc_info {
__u64 size;
__u64 timestamp_ns;
int stack_id;
};
union combined_alloc_info {
struct {
__u64 total_size : 40;
__u64 number_of_allocs : 24;
};
__u64 bits;
};
</code></pre>
<p>这段代码定义了memleak工具中使用的5个BPF Map：</p>
<ul>
<li>sizes用于记录程序中每个内存分配请求的大小；</li>
<li>allocs用于跟踪每个内存分配请求的详细信息，包括请求的大小、堆栈信息等；</li>
<li>combined_allocs的键是堆栈的唯一标识符(stack id)，值是一个combined_alloc_info联合体，用于记录该堆栈的内存分配总大小和内存分配数量；</li>
<li>memptrs用于跟踪每个内存分配请求返回的指针，以便在内存释放请求到来时找到对应的内存分配请求；</li>
<li>stack_traces是一个堆栈跟踪类型的哈希表，用于存储每个线程的堆栈信息（key为线程id，value为堆栈跟踪信息）以便在内存分配和释放请求到来时能够追踪和分析相应的堆栈信息。</li>
</ul>
<p>其中combined_alloc_info是一个联合体，其中包含一个结构体和一个unsigned long long类型的变量bits。结构体中的两个成员变量total_size和number_of_allocs分别表示总分配大小和分配的次数。其中40和24分别表示total_size和number_of_allocs这两个成员变量所占用的位数，用来限制其大小。通过这样的位数限制，可以节省combined_alloc_info结构的存储空间。同时，由于total_size和number_of_allocs在存储时是共用一个unsigned long long类型的变量bits，因此可以通过在成员变量bits上进行位运算来访问和修改total_size和number_of_allocs，从而避免了在程序中定义额外的变量和函数的复杂性。</p>
<pre><code class="language-c">static int gen_alloc_enter(size_t size)
{
if (size &lt; min_size || size &gt; max_size)
return 0;
if (sample_rate &gt; 1) {
if (bpf_ktime_get_ns() % sample_rate != 0)
return 0;
}
const pid_t pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
bpf_map_update_elem(&amp;sizes, &amp;pid, &amp;size, BPF_ANY);
if (trace_all)
bpf_printk(&quot;alloc entered, size = %lu\n&quot;, size);
return 0;
}
SEC(&quot;uprobe&quot;)
int BPF_KPROBE(malloc_enter, size_t size)
{
return gen_alloc_enter(size);
}
</code></pre>
<p>这个函数用于处理内存分配请求的进入事件。它会首先检查内存分配请求的大小是否在指定的范围内，如果不在范围内，则直接返回0表示不处理该事件。如果启用了采样率(sample_rate &gt; 1)，则该函数会采样内存分配请求的进入事件。如果当前时间戳不是采样周期的倍数，则也会直接返回0，表示不处理该事件。接下来，该函数会获取当前线程的PID并将其存储在pid变量中。然后，它会将当前线程的pid和请求的内存分配大小存储在sizes map中，以便后续收集和分析内存分配信息。如果开启了跟踪模式(trace_all)，该函数会通过bpf_printk打印日志信息，以便用户实时监控内存分配的情况。</p>
<p>最后定义了BPF_KPROBE(malloc_enter, size_t size)，它会在malloc函数被调用时被BPF uprobe拦截执行，并通过gen_alloc_enter来记录内存分配大小。</p>
<pre><code class="language-c">static void update_statistics_add(u64 stack_id, u64 sz)
{
union combined_alloc_info *existing_cinfo;
existing_cinfo = bpf_map_lookup_or_try_init(&amp;combined_allocs, &amp;stack_id, &amp;initial_cinfo);
if (!existing_cinfo)
return;
const union combined_alloc_info incremental_cinfo = {
.total_size = sz,
.number_of_allocs = 1
};
__sync_fetch_and_add(&amp;existing_cinfo-&gt;bits, incremental_cinfo.bits);
}
static int gen_alloc_exit2(void *ctx, u64 address)
{
const pid_t pid = bpf_get_current_pid_tgid() &gt;&gt; 32;
struct alloc_info info;
const u64* size = bpf_map_lookup_elem(&amp;sizes, &amp;pid);
if (!size)
return 0; // missed alloc entry
__builtin_memset(&amp;info, 0, sizeof(info));
info.size = *size;
bpf_map_delete_elem(&amp;sizes, &amp;pid);
if (address != 0) {
info.timestamp_ns = bpf_ktime_get_ns();
info.stack_id = bpf_get_stackid(ctx, &amp;stack_traces, stack_flags);
bpf_map_update_elem(&amp;allocs, &amp;address, &amp;info, BPF_ANY);
update_statistics_add(info.stack_id, info.size);
}
if (trace_all) {
bpf_printk(&quot;alloc exited, size = %lu, result = %lx\n&quot;,
info.size, address);
}
return 0;
}
static int gen_alloc_exit(struct pt_regs *ctx)
{
return gen_alloc_exit2(ctx, PT_REGS_RC(ctx));
}
SEC(&quot;uretprobe&quot;)
int BPF_KRETPROBE(malloc_exit)
{
return gen_alloc_exit(ctx);
}
</code></pre>
<p>gen_alloc_exit2函数会在内存释放时被调用，它用来记录内存释放的信息，并更新相关的 map。具体地，它首先通过 bpf_get_current_pid_tgid 来获取当前进程的 PID，并将其右移32位，获得PID值，然后使用 bpf_map_lookup_elem 查找 sizes map 中与该 PID 相关联的内存分配大小信息，并将其赋值给 info.size。如果找不到相应的 entry，则返回 0，表示在内存分配时没有记录到该 PID 相关的信息。接着，它会调用 __builtin_memset 来将 info 的所有字段清零，并调用 bpf_map_delete_elem 来删除 sizes map 中与该 PID 相关联的 entry。</p>
<p>如果 address 不为 0，则说明存在相应的内存分配信息，此时它会调用 bpf_ktime_get_ns 来获取当前时间戳，并将其赋值给 info.timestamp_ns。然后，它会调用 bpf_get_stackid 来获取当前函数调用堆栈的 ID，并将其赋值给 info.stack_id。最后，它会调用 bpf_map_update_elem 来将 address 和 info 相关联，即将 address 映射到 info。随后，它会调用 update_statistics_add 函数来更新 combined_allocs map 中与 info.stack_id 相关联的内存分配信息。</p>
<p>最后，如果 trace_all 为真，则会调用 bpf_printk 打印相关的调试信息。</p>
<p>update_statistics_add函数的主要作用是更新内存分配的统计信息，其中参数stack_id是当前内存分配的堆栈ID，sz是当前内存分配的大小。该函数首先通过bpf_map_lookup_or_try_init函数在combined_allocs map中查找与当前堆栈ID相关联的combined_alloc_info结构体，如果找到了，则将新的分配大小和分配次数加入到已有的combined_alloc_info结构体中；如果未找到，则使用initial_cinfo初始化一个新的combined_alloc_info结构体，并添加到combined_allocs map中。</p>
<p>更新combined_alloc_info结构体的方法是使用__sync_fetch_and_add函数，原子地将incremental_cinfo中的值累加到existing_cinfo中的值中。通过这种方式，即使多个线程同时调用update_statistics_add函数，也可以保证计数的正确性。</p>
<p>在gen_alloc_exit函数中，将ctx参数传递给gen_alloc_exit2函数，并将它的返回值作为自己的返回值。这里使用了PT_REGS_RC宏获取函数返回值。</p>
<p>最后定义的BPF_KRETPROBE(malloc_exit)是一个kretprobe类型的函数，用于在malloc函数返回时执行。并调用gen_alloc_exit函数跟踪内存分配和释放的请求。</p>
<pre><code class="language-c">static void update_statistics_del(u64 stack_id, u64 sz)
{
union combined_alloc_info *existing_cinfo;
existing_cinfo = bpf_map_lookup_elem(&amp;combined_allocs, &amp;stack_id);
if (!existing_cinfo) {
bpf_printk(&quot;failed to lookup combined allocs\n&quot;);
return;
}
const union combined_alloc_info decremental_cinfo = {
.total_size = sz,
.number_of_allocs = 1
};
__sync_fetch_and_sub(&amp;existing_cinfo-&gt;bits, decremental_cinfo.bits);
}
static int gen_free_enter(const void *address)
{
const u64 addr = (u64)address;
const struct alloc_info *info = bpf_map_lookup_elem(&amp;allocs, &amp;addr);
if (!info)
return 0;
bpf_map_delete_elem(&amp;allocs, &amp;addr);
update_statistics_del(info-&gt;stack_id, info-&gt;size);
if (trace_all) {
bpf_printk(&quot;free entered, address = %lx, size = %lu\n&quot;,
address, info-&gt;size);
}
return 0;
}
SEC(&quot;uprobe&quot;)
int BPF_KPROBE(free_enter, void *address)
{
return gen_free_enter(address);
}
</code></pre>
<p>gen_free_enter函数接收一个地址参数，该函数首先使用allocs map查找该地址对应的内存分配信息。如果未找到，则表示该地址没有被分配，该函数返回0。如果找到了对应的内存分配信息，则使用bpf_map_delete_elem从allocs map中删除该信息。</p>
<p>接下来，调用update_statistics_del函数用于更新内存分配的统计信息，它接收堆栈ID和内存块大小作为参数。首先在combined_allocs map中查找堆栈ID对应的内存分配统计信息。如果没有找到，则输出一条日志，表示查找失败，并且函数直接返回。如果找到了对应的内存分配统计信息，则使用原子操作从内存分配统计信息中减去该内存块大小和1（表示减少了1个内存块）。这是因为堆栈ID对应的内存块数量减少了1，而堆栈ID对应的内存块总大小也减少了该内存块的大小。</p>
<p>最后定义了一个bpf程序BPF_KPROBE(free_enter, void *address)会在进程调用free函数时执行。它会接收参数address，表示正在释放的内存块的地址，并调用gen_free_enter函数来处理该内存块的释放。</p>
<h2 id="编译运行-6"><a class="header" href="#编译运行-6">编译运行</a></h2>
<pre><code class="language-console">$ git clone https://github.com/iovisor/bcc.git --recurse-submodules
$ cd libbpf-tools/
$ make memleak
$ sudo ./memleak
using default object: libc.so.6
using page size: 4096
tracing kernel: true
Tracing outstanding memory allocs... Hit Ctrl-C to end
[17:17:27] Top 10 stacks with outstanding allocations:
1236992 bytes in 302 allocations from stack
0 [&lt;ffffffff812c8f43&gt;] &lt;null sym&gt;
1 [&lt;ffffffff812c8f43&gt;] &lt;null sym&gt;
2 [&lt;ffffffff812a9d42&gt;] &lt;null sym&gt;
3 [&lt;ffffffff812aa392&gt;] &lt;null sym&gt;
4 [&lt;ffffffff810df0cb&gt;] &lt;null sym&gt;
5 [&lt;ffffffff81edc3fd&gt;] &lt;null sym&gt;
6 [&lt;ffffffff82000b62&gt;] &lt;null sym&gt;
...
</code></pre>
<h2 id="总结-15"><a class="header" href="#总结-15">总结</a></h2>
<p>memleak是一个内存泄漏监控工具，可以用来跟踪内存分配和释放时间对应的调用栈信息。随着时间的推移，这个工具可以显示长期不被释放的内存。</p>
<p>这份代码来自于https://github.com/iovisor/bcc/blob/master/libbpf-tools/memleak.bpf.c</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门实践教程编写-ebpf-程序-biopattern-统计随机顺序磁盘-io"><a class="header" href="#ebpf-入门实践教程编写-ebpf-程序-biopattern-统计随机顺序磁盘-io">eBPF 入门实践教程：编写 eBPF 程序 Biopattern: 统计随机/顺序磁盘 I/O</a></h1>
<h2 id="背景-4"><a class="header" href="#背景-4">背景</a></h2>
<p>Biopattern 可以统计随机/顺序磁盘I/O次数的比例。</p>
<p>TODO</p>
<h2 id="实现原理-2"><a class="header" href="#实现原理-2">实现原理</a></h2>
<p>Biopattern 的ebpf代码在 tracepoint/block/block_rq_complete 挂载点下实现。在磁盘完成IO请求
后，程序会经过此挂载点。Biopattern 内部存有一张以设备号为主键的哈希表，当程序经过挂载点时, Biopattern
会获得操作信息，根据哈希表中该设备的上一次操作记录来判断本次操作是随机IO还是顺序IO，并更新操作计数。</p>
<h2 id="编写-ebpf-程序-3"><a class="header" href="#编写-ebpf-程序-3">编写 eBPF 程序</a></h2>
<p>TODO</p>
<h3 id="总结-16"><a class="header" href="#总结-16">总结</a></h3>
<p>Biopattern 可以展现随机/顺序磁盘I/O次数的比例，对于开发者把握整体I/O情况有较大帮助。</p>
<p>TODO</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="更多的参考资料"><a class="header" href="#更多的参考资料">更多的参考资料</a></h1>
<p>TODO</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门实践教程使用-lsm-进行安全检测防御"><a class="header" href="#ebpf-入门实践教程使用-lsm-进行安全检测防御">eBPF 入门实践教程：使用 LSM 进行安全检测防御</a></h1>
<h2 id="背景-5"><a class="header" href="#背景-5">背景</a></h2>
<p>LSM 从 Linux 2.6 开始成为官方内核的一个安全框架，基于此的安全实现包括 SELinux 和 AppArmor 等。在 Linux 5.7 引入 BPF LSM 后，系统开发人员已经能够自由地实现函数粒度的安全检查能力，本文就提供了这样一个案例：限制通过 socket connect 函数对特定 IPv4 地址进行访问的 BPF LSM 程序。（可见其控制精度是很高的）</p>
<h2 id="lsm-概述"><a class="header" href="#lsm-概述">LSM 概述</a></h2>
<p>LSM（Linux Security Modules）是 Linux 内核中用于支持各种计算机安全模型的框架。LSM 在 Linux 内核安全相关的关键路径上预置了一批 hook 点，从而实现了内核和安全模块的解耦，使不同的安全模块可以自由地在内核中加载/卸载，无需修改原有的内核代码就可以加入安全检查功能。</p>
<p>在过去，使用 LSM 主要通过配置已有的安全模块（如 SELinux 和 AppArmor）或编写自己的内核模块；而在 Linux 5.7 引入 BPF LSM 机制后，一切都变得不同了：现在，开发人员可以通过 eBPF 编写自定义的安全策略，并将其动态加载到内核中的 LSM 挂载点，而无需配置或编写内核模块。</p>
<p>现在 LSM 支持的 hook 点包括但不限于：</p>
<ul>
<li>对文件的打开、创建、删除和移动等；</li>
<li>文件系统的挂载；</li>
<li>对 task 和 process 的操作；</li>
<li>对 socket 的操作（创建、绑定 socket，发送和接收消息等）；</li>
</ul>
<p>更多 hook 点可以参考 <a href="https://github.com/torvalds/linux/blob/master/include/linux/lsm_hooks.h">lsm_hooks.h</a>。</p>
<h2 id="确认-bpf-lsm-是否可用"><a class="header" href="#确认-bpf-lsm-是否可用">确认 BPF LSM 是否可用</a></h2>
<p>首先，请确认内核版本高于 5.7。接下来，可以通过</p>
<pre><code class="language-console">$ cat /boot/config-$(uname -r) | grep BPF_LSM
CONFIG_BPF_LSM=y
</code></pre>
<p>判断是否内核是否支持 BPF LSM。上述条件都满足的情况下，可以通过</p>
<pre><code class="language-console">$ cat /sys/kernel/security/lsm
ndlock,lockdown,yama,integrity,apparmor
</code></pre>
<p>查看输出是否包含 bpf 选项，如果输出不包含（像上面的例子），可以通过修改 <code>/etc/default/grub</code>：</p>
<pre><code class="language-conf">GRUB_CMDLINE_LINUX=&quot;lsm=ndlock,lockdown,yama,integrity,apparmor,bpf&quot;
</code></pre>
<p>并通过 <code>update-grub2</code> 命令更新 grub 配置（不同系统的对应命令可能不同），然后重启系统。</p>
<h2 id="编写-ebpf-程序-4"><a class="header" href="#编写-ebpf-程序-4">编写 eBPF 程序</a></h2>
<pre><code class="language-C">// lsm-connect.bpf.c
#include &quot;vmlinux.h&quot;
#include &lt;bpf/bpf_core_read.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
char LICENSE[] SEC(&quot;license&quot;) = &quot;GPL&quot;;
#define EPERM 1
#define AF_INET 2
const __u32 blockme = 16843009; // 1.1.1.1 -&gt; int
SEC(&quot;lsm/socket_connect&quot;)
int BPF_PROG(restrict_connect, struct socket *sock, struct sockaddr *address, int addrlen, int ret)
{
// Satisfying &quot;cannot override a denial&quot; rule
if (ret != 0)
{
return ret;
}
// Only IPv4 in this example
if (address-&gt;sa_family != AF_INET)
{
return 0;
}
// Cast the address to an IPv4 socket address
struct sockaddr_in *addr = (struct sockaddr_in *)address;
// Where do you want to go?
__u32 dest = addr-&gt;sin_addr.s_addr;
bpf_printk(&quot;lsm: found connect to %d&quot;, dest);
if (dest == blockme)
{
bpf_printk(&quot;lsm: blocking %d&quot;, dest);
return -EPERM;
}
return 0;
}
</code></pre>
<p>这是一段 C 实现的 eBPF 内核侧代码，它会阻碍所有试图通过 socket 对 1.1.1.1 的连接操作，其中：</p>
<ul>
<li><code>SEC(&quot;lsm/socket_connect&quot;)</code> 宏指出该程序期望的挂载点；</li>
<li>程序通过 <code>BPF_PROG</code> 宏定义（详情可查看 <a href="https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/lib/bpf/bpf_tracing.h">tools/lib/bpf/bpf_tracing.h</a>）；</li>
<li><code>restrict_connect</code> 是 <code>BPF_PROG</code> 宏要求的程序名；</li>
<li><code>ret</code> 是该挂载点上（潜在的）当前函数之前的 LSM 检查程序的返回值；</li>
</ul>
<p>整个程序的思路不难理解：</p>
<ul>
<li>首先，若其他安全检查函数返回值不为 0（不通过），则无需检查，直接返回不通过；</li>
<li>接下来，判断是否为 IPV4 的连接请求，并比较试图连接的地址是否为 1.1.1.1；</li>
<li>若请求地址为 1.1.1.1 则拒绝连接，否则允许连接；</li>
</ul>
<p>在程序运行期间，所有通过 socket 的连接操作都会被输出到 <code>/sys/kernel/debug/tracing/trace_pipe</code>。</p>
<h2 id="编译运行-7"><a class="header" href="#编译运行-7">编译运行</a></h2>
<p>通过容器编译：</p>
<pre><code class="language-console">docker run -it -v `pwd`/:/src/ yunwei37/ebpm:latest
</code></pre>
<p>或是通过 <code>ecc</code> 编译：</p>
<pre><code class="language-console">$ ecc lsm-connect.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
</code></pre>
<p>并通过 <code>ecli</code> 运行：</p>
<pre><code class="language-console">$ sudo ecli run package.json
</code></pre>
<p>接下来，可以打开另一个 terminal，并尝试访问 1.1.1.1：</p>
<pre><code class="language-console">$ ping 1.1.1.1
ping: connect: Operation not permitted
$ curl 1.1.1.1
curl: (7) Couldn't connect to server
$ wget 1.1.1.1
--2023-04-23 08:41:18-- (try: 2) http://1.1.1.1/
Connecting to 1.1.1.1:80... failed: Operation not permitted.
Retrying.
</code></pre>
<p>同时，我们可以查看 <code>bpf_printk</code> 的输出：</p>
<pre><code class="language-console">$ sudo cat /sys/kernel/debug/tracing/trace_pipe
ping-7054 [000] d...1 6313.430872: bpf_trace_printk: lsm: found connect to 16843009
ping-7054 [000] d...1 6313.430874: bpf_trace_printk: lsm: blocking 16843009
curl-7058 [000] d...1 6316.346582: bpf_trace_printk: lsm: found connect to 16843009
curl-7058 [000] d...1 6316.346584: bpf_trace_printk: lsm: blocking 16843009
wget-7061 [000] d...1 6318.800698: bpf_trace_printk: lsm: found connect to 16843009
wget-7061 [000] d...1 6318.800700: bpf_trace_printk: lsm: blocking 16843009
</code></pre>
<h2 id="总结-17"><a class="header" href="#总结-17">总结</a></h2>
<p>本文介绍了如何使用 BPF LSM 来限制通过 socket 对特定 IPv4 地址的访问。我们可以通过修改 GRUB 配置文件来开启 LSM 的 BPF 挂载点。在 eBPF 程序中，我们通过 <code>BPF_PROG</code> 宏定义函数，并通过 <code>SEC</code> 宏指定挂载点；在函数实现上，遵循 LSM 安全检查模块中 &quot;cannot override a denial&quot; 的原则，并根据 socket 连接请求的目的地址对该请求进行限制。</p>
<p>更多的例子和详细的开发指南，请参考 eunomia-bpf 的官方文档：<a href="https://github.com/eunomia-bpf/eunomia-bpf">https://github.com/eunomia-bpf/eunomia-bpf</a></p>
<p>完整的教程和源代码已经全部开源，可以在 <a href="https://github.com/eunomia-bpf/bpf-developer-tutorial">https://github.com/eunomia-bpf/bpf-developer-tutorial</a> 中查看。</p>
<h2 id="参考"><a class="header" href="#参考">参考</a></h2>
<p><a href="https://github.com/leodido/demo-cloud-native-ebpf-day">https://github.com/leodido/demo-cloud-native-ebpf-day</a></p>
<p><a href="https://aya-rs.dev/book/programs/lsm/#writing-lsm-bpf-program">https://aya-rs.dev/book/programs/lsm/#writing-lsm-bpf-program</a></p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ebpf-入门实践教程使用-ebpf-进行-tc-流量控制"><a class="header" href="#ebpf-入门实践教程使用-ebpf-进行-tc-流量控制">eBPF 入门实践教程：使用 eBPF 进行 tc 流量控制</a></h1>
<h2 id="tc-程序示例"><a class="header" href="#tc-程序示例">tc 程序示例</a></h2>
<pre><code class="language-c">#include &lt;vmlinux.h&gt;
#include &lt;bpf/bpf_endian.h&gt;
#include &lt;bpf/bpf_helpers.h&gt;
#include &lt;bpf/bpf_tracing.h&gt;
#define TC_ACT_OK 0
#define ETH_P_IP 0x0800 /* Internet Protocol packet */
/// @tchook {&quot;ifindex&quot;:1, &quot;attach_point&quot;:&quot;BPF_TC_INGRESS&quot;}
/// @tcopts {&quot;handle&quot;:1, &quot;priority&quot;:1}
SEC(&quot;tc&quot;)
int tc_ingress(struct __sk_buff *ctx)
{
void *data_end = (void *)(__u64)ctx-&gt;data_end;
void *data = (void *)(__u64)ctx-&gt;data;
struct ethhdr *l2;
struct iphdr *l3;
if (ctx-&gt;protocol != bpf_htons(ETH_P_IP))
return TC_ACT_OK;
l2 = data;
if ((void *)(l2 + 1) &gt; data_end)
return TC_ACT_OK;
l3 = (struct iphdr *)(l2 + 1);
if ((void *)(l3 + 1) &gt; data_end)
return TC_ACT_OK;
bpf_printk(&quot;Got IP packet: tot_len: %d, ttl: %d&quot;, bpf_ntohs(l3-&gt;tot_len), l3-&gt;ttl);
return TC_ACT_OK;
}
char __license[] SEC(&quot;license&quot;) = &quot;GPL&quot;;
</code></pre>
<p>这段代码定义了一个 eBPF 程序，它可以通过 Linux TC（Transmission Control）来捕获数据包并进行处理。在这个程序中，我们限定了只捕获 IPv4 协议的数据包，然后通过 bpf_printk 函数打印出数据包的总长度和 Time-To-Live（TTL）字段的值。</p>
<p>需要注意的是，我们在代码中使用了一些 BPF 库函数，例如 bpf_htons 和 bpf_ntohs 函数，它们用于进行网络字节序和主机字节序之间的转换。此外，我们还使用了一些注释来为 TC 提供附加点和选项信息。例如，在这段代码的开头，我们使用了以下注释：</p>
<pre><code class="language-c">/// @tchook {&quot;ifindex&quot;:1, &quot;attach_point&quot;:&quot;BPF_TC_INGRESS&quot;}
/// @tcopts {&quot;handle&quot;:1, &quot;priority&quot;:1}
</code></pre>
<p>这些注释告诉 TC 将 eBPF 程序附加到网络接口的 ingress 附加点，并指定了 handle 和 priority 选项的值。</p>
<p>总之，这段代码实现了一个简单的 eBPF 程序，用于捕获数据包并打印出它们的信息。</p>
<h2 id="编译运行-8"><a class="header" href="#编译运行-8">编译运行</a></h2>
<pre><code class="language-console">docker run -it -v `pwd`/:/src/ yunwei37/ebpm:latest
</code></pre>
<p>or compile with <code>ecc</code>:</p>
<pre><code class="language-console">$ ecc tc.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
</code></pre>
<pre><code class="language-shell">$ sudo ecli run ./package.json
...
Successfully started! Please run `sudo cat /sys/kernel/debug/tracing/trace_pipe` to see output of the BPF program.
......
</code></pre>
<p>The <code>tc</code> output in <code>/sys/kernel/debug/tracing/trace_pipe</code> should look
something like this:</p>
<pre><code class="language-console">$ sudo cat /sys/kernel/debug/tracing/trace_pipe
node-1254811 [007] ..s1 8737831.671074: 0: Got IP packet: tot_len: 79, ttl: 64
sshd-1254728 [006] ..s1 8737831.674334: 0: Got IP packet: tot_len: 79, ttl: 64
sshd-1254728 [006] ..s1 8737831.674349: 0: Got IP packet: tot_len: 72, ttl: 64
node-1254811 [007] ..s1 8737831.674550: 0: Got IP packet: tot_len: 71, ttl: 64
</code></pre>
<h2 id="总结-18"><a class="header" href="#总结-18">总结</a></h2>
<p>TODO</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="bpf-features-by-linux-kernel-version"><a class="header" href="#bpf-features-by-linux-kernel-version">BPF Features by Linux Kernel Version</a></h1>
<h2 id="ebpf-support"><a class="header" href="#ebpf-support">eBPF support</a></h2>
<div class="table-wrapper"><table><thead><tr><th>Kernel version</th><th>Commit</th></tr></thead><tbody>
<tr><td>3.15</td><td><a href="https://github.com/torvalds/linux/commit/bd4cf0ed331a275e9bf5a49e6d0fd55dffc551b8"><code>bd4cf0ed331a</code></a></td></tr>
</tbody></table>
</div>
<h2 id="jit-compiling"><a class="header" href="#jit-compiling">JIT compiling</a></h2>
<p>The list of supported architectures for your kernel can be retrieved with:</p>
<pre><code class="language-sh">git grep HAVE_EBPF_JIT arch/
</code></pre>
<div class="table-wrapper"><table><thead><tr><th>Feature / Architecture</th><th>Kernel version</th><th>Commit</th></tr></thead><tbody>
<tr><td>x86_64</td><td>3.16</td><td><a href="https://github.com/torvalds/linux/commit/622582786c9e041d0bd52bde201787adeab249f8"><code>622582786c9e</code></a></td></tr>
<tr><td>ARM64</td><td>3.18</td><td><a href="https://github.com/torvalds/linux/commit/e54bcde3d69d40023ae77727213d14f920eb264a"><code>e54bcde3d69d</code></a></td></tr>
<tr><td>s390</td><td>4.1</td><td><a href="https://github.com/torvalds/linux/commit/054623105728b06852f077299e2bf1bf3d5f2b0b"><code>054623105728</code></a></td></tr>
<tr><td>Constant blinding for JIT machines</td><td>4.7</td><td><a href="https://github.com/torvalds/linux/commit/4f3446bb809f20ad56cadf712e6006815ae7a8f9"><code>4f3446bb809f</code></a></td></tr>
<tr><td>PowerPC64</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/156d0e290e969caba25f1851c52417c14d141b24"><code>156d0e290e96</code></a></td></tr>
<tr><td>Constant blinding - PowerPC64</td><td>4.9</td><td><a href="https://github.com/torvalds/linux/commit/b7b7013cac55d794940bd9cb7b7c55c9dececac4"><code>b7b7013cac55</code></a></td></tr>
<tr><td>Sparc64</td><td>4.12</td><td><a href="https://github.com/torvalds/linux/commit/7a12b5031c6b947cc13918237ae652b536243b76"><code>7a12b5031c6b</code></a></td></tr>
<tr><td>MIPS</td><td>4.13</td><td><a href="https://github.com/torvalds/linux/commit/f381bf6d82f032b7410185b35d000ea370ac706b"><code>f381bf6d82f0</code></a></td></tr>
<tr><td>ARM32</td><td>4.14</td><td><a href="https://github.com/torvalds/linux/commit/39c13c204bb1150d401e27d41a9d8b332be47c49"><code>39c13c204bb1</code></a></td></tr>
<tr><td>x86_32</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/03f5781be2c7b7e728d724ac70ba10799cc710d7"><code>03f5781be2c7</code></a></td></tr>
<tr><td>RISC-V RV64G</td><td>5.1</td><td><a href="https://github.com/torvalds/linux/commit/2353ecc6f91fd15b893fa01bf85a1c7a823ee4f2"><code>2353ecc6f91f</code></a></td></tr>
<tr><td>RISC-V RV32G</td><td>5.7</td><td><a href="https://github.com/torvalds/linux/commit/5f316b65e99f109942c556dc8790abd4c75bcb34"><code>5f316b65e99f</code></a></td></tr>
<tr><td>PowerPC32</td><td>5.13</td><td><a href="https://github.com/torvalds/linux/commit/51c66ad849a703d9bbfd7704c941827aed0fd9fd"><code>51c66ad849a7</code></a></td></tr>
<tr><td>LoongArch</td><td>6.1</td><td><a href="https://github.com/torvalds/linux/commit/5dc615520c4dfb358245680f1904bad61116648e"><code>5dc615520c4d</code></a></td></tr>
</tbody></table>
</div>
<h2 id="main-features"><a class="header" href="#main-features">Main features</a></h2>
<p>Several (but not all) of these <em>main features</em> translate to an eBPF program type.
The list of such program types supported in your kernel can be found in file
<a href="https://github.com/torvalds/linux/blob/master/include/uapi/linux/bpf.h"><code>include/uapi/linux/bpf.h</code></a>:</p>
<pre><code class="language-sh">git grep -W 'bpf_prog_type {' include/uapi/linux/bpf.h
</code></pre>
<div class="table-wrapper"><table><thead><tr><th>Feature</th><th>Kernel version</th><th>Commit</th></tr></thead><tbody>
<tr><td><code>AF_PACKET</code> (libpcap/tcpdump, <code>cls_bpf</code> classifier, netfilter's <code>xt_bpf</code>, team driver's load-balancing mode…)</td><td>3.15</td><td><a href="https://github.com/torvalds/linux/commit/bd4cf0ed331a275e9bf5a49e6d0fd55dffc551b8"><code>bd4cf0ed331a</code></a></td></tr>
<tr><td>Kernel helpers</td><td>3.15</td><td><a href="https://github.com/torvalds/linux/commit/bd4cf0ed331a275e9bf5a49e6d0fd55dffc551b8"><code>bd4cf0ed331a</code></a></td></tr>
<tr><td><code>bpf()</code> syscall</td><td>3.18</td><td><a href="https://github.com/torvalds/linux/commit/99c55f7d47c0dc6fc64729f37bf435abf43f4c60"><code>99c55f7d47c0</code></a></td></tr>
<tr><td>Maps (<em>a.k.a.</em> Tables; details below)</td><td>3.18</td><td><a href="https://github.com/torvalds/linux/commit/99c55f7d47c0dc6fc64729f37bf435abf43f4c60"><code>99c55f7d47c0</code></a></td></tr>
<tr><td>BPF attached to sockets</td><td>3.19</td><td><a href="https://github.com/torvalds/linux/commit/89aa075832b0da4402acebd698d0411dcc82d03e"><code>89aa075832b0</code></a></td></tr>
<tr><td>BPF attached to <code>kprobes</code></td><td>4.1</td><td><a href="https://github.com/torvalds/linux/commit/2541517c32be2531e0da59dfd7efc1ce844644f5"><code>2541517c32be</code></a></td></tr>
<tr><td><code>cls_bpf</code> / <code>act_bpf</code> for <code>tc</code></td><td>4.1</td><td><a href="https://github.com/torvalds/linux/commit/e2e9b6541dd4b31848079da80fe2253daaafb549"><code>e2e9b6541dd4</code></a></td></tr>
<tr><td>Tail calls</td><td>4.2</td><td><a href="https://github.com/torvalds/linux/commit/04fd61ab36ec065e194ab5e74ae34a5240d992bb"><code>04fd61ab36ec</code></a></td></tr>
<tr><td>Non-root programs on sockets</td><td>4.4</td><td><a href="https://github.com/torvalds/linux/commit/1be7f75d1668d6296b80bf35dcf6762393530afc"><code>1be7f75d1668</code></a></td></tr>
<tr><td>Persistent maps and programs (virtual FS)</td><td>4.4</td><td><a href="https://github.com/torvalds/linux/commit/b2197755b2633e164a439682fb05a9b5ea48f706"><code>b2197755b263</code></a></td></tr>
<tr><td><code>tc</code>'s <code>direct-action</code> (<code>da</code>) mode</td><td>4.4</td><td><a href="https://github.com/torvalds/linux/commit/045efa82ff563cd4e656ca1c2e354fa5bf6bbda4"><code>045efa82ff56</code></a></td></tr>
<tr><td><code>tc</code>'s <code>clsact</code> qdisc</td><td>4.5</td><td><a href="https://github.com/torvalds/linux/commit/1f211a1b929c804100e138c5d3d656992cfd5622"><code>1f211a1b929c</code></a></td></tr>
<tr><td>BPF attached to tracepoints</td><td>4.7</td><td><a href="https://github.com/torvalds/linux/commit/98b5c2c65c2951772a8fc661f50d675e450e8bce"><code>98b5c2c65c29</code></a></td></tr>
<tr><td>Direct packet access</td><td>4.7</td><td><a href="https://github.com/torvalds/linux/commit/969bf05eb3cedd5a8d4b7c346a85c2ede87a6d6d"><code>969bf05eb3ce</code></a></td></tr>
<tr><td>XDP (see below)</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/6a773a15a1e8874e5eccd2f29190c31085912c95"><code>6a773a15a1e8</code></a></td></tr>
<tr><td>BPF attached to perf events</td><td>4.9</td><td><a href="https://github.com/torvalds/linux/commit/0515e5999a466dfe6e1924f460da599bb6821487"><code>0515e5999a46</code></a></td></tr>
<tr><td>Hardware offload for <code>tc</code>'s <code>cls_bpf</code></td><td>4.9</td><td><a href="https://github.com/torvalds/linux/commit/332ae8e2f6ecda5e50c5c62ed62894963e3a83f5"><code>332ae8e2f6ec</code></a></td></tr>
<tr><td>Verifier exposure and internal hooks</td><td>4.9</td><td><a href="https://github.com/torvalds/linux/commit/13a27dfc669724564aafa2699976ee756029fed2"><code>13a27dfc6697</code></a></td></tr>
<tr><td>BPF attached to cgroups for socket filtering</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/0e33661de493db325435d565a4a722120ae4cbf3"><code>0e33661de493</code></a></td></tr>
<tr><td>Lightweight tunnel encapsulation</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/3a0af8fd61f90920f6fa04e4f1e9a6a73c1b4fd2"><code>3a0af8fd61f9</code></a></td></tr>
<tr><td><strong>e</strong>BPF support for <code>xt_bpf</code> module (iptables)</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/2c16d60332643e90d4fa244f4a706c454b8c7569"><code>2c16d6033264</code></a></td></tr>
<tr><td>BPF program tag</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/7bd509e311f408f7a5132fcdde2069af65fa05ae"><code>7bd509e311f4</code></a></td></tr>
<tr><td>Tracepoints to debug BPF</td><td>4.11 (removed in 4.18)</td><td><a href="https://github.com/torvalds/linux/commit/a67edbf4fb6deadcfe57a04a134abed4a5ba3bb5"><code>a67edbf4fb6d</code></a> <a href="https://github.com/torvalds/linux/commit/4d220ed0f8140c478ab7b0a14d96821da639b646"><code>4d220ed0f814</code></a></td></tr>
<tr><td>Testing / benchmarking BPF programs</td><td>4.12</td><td><a href="https://github.com/torvalds/linux/commit/1cf1cae963c2e6032aebe1637e995bc2f5d330f4"><code>1cf1cae963c2</code></a></td></tr>
<tr><td>BPF programs and maps IDs</td><td>4.13</td><td><a href="https://github.com/torvalds/linux/commit/dc4bb0e2356149aee4cdae061936f3bbdd45595c"><code>dc4bb0e23561</code></a></td></tr>
<tr><td>BPF support for <code>sock_ops</code></td><td>4.13</td><td><a href="https://github.com/torvalds/linux/commit/40304b2a1567fecc321f640ee4239556dd0f3ee0"><code>40304b2a1567</code></a></td></tr>
<tr><td>BPF support for skbs on sockets</td><td>4.14</td><td><a href="https://github.com/torvalds/linux/commit/b005fd189cec9407b700599e1e80e0552446ee79"><code>b005fd189cec</code></a></td></tr>
<tr><td>bpftool utility in kernel sources</td><td>4.15</td><td><a href="https://github.com/torvalds/linux/commit/71bb428fe2c19512ac671d5ee16ef3e73e1b49a8"><code>71bb428fe2c1</code></a></td></tr>
<tr><td>BPF attached to cgroups as device controller</td><td>4.15</td><td><a href="https://github.com/torvalds/linux/commit/ebc614f687369f9df99828572b1d85a7c2de3d92"><code>ebc614f68736</code></a></td></tr>
<tr><td>bpf2bpf function calls</td><td>4.16</td><td><a href="https://github.com/torvalds/linux/commit/cc8b0b92a1699bc32f7fec71daa2bfc90de43a4d"><code>cc8b0b92a169</code></a></td></tr>
<tr><td>BPF used for monitoring socket RX/TX data</td><td>4.17</td><td><a href="https://github.com/torvalds/linux/commit/4f738adba30a7cfc006f605707e7aee847ffefa0"><code>4f738adba30a</code></a></td></tr>
<tr><td>BPF attached to raw tracepoints</td><td>4.17</td><td><a href="https://github.com/torvalds/linux/commit/c4f6699dfcb8558d138fe838f741b2c10f416cf9"><code>c4f6699dfcb8</code></a></td></tr>
<tr><td>BPF attached to <code>bind()</code> system call</td><td>4.17</td><td><a href="https://github.com/torvalds/linux/commit/4fbac77d2d092b475dda9eea66da674369665427"><code>4fbac77d2d09</code></a> <a href="https://github.com/torvalds/linux/commit/aac3fc320d9404f2665a8b1249dc3170d5fa3caf"><code>aac3fc320d94</code></a></td></tr>
<tr><td>BPF attached to <code>connect()</code> system call</td><td>4.17</td><td><a href="https://github.com/torvalds/linux/commit/d74bad4e74ee373787a9ae24197c17b7cdc428d5"><code>d74bad4e74ee</code></a></td></tr>
<tr><td>BPF Type Format (BTF)</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/69b693f0aefa0ed521e8bd02260523b5ae446ad7"><code>69b693f0aefa</code></a></td></tr>
<tr><td>AF_XDP</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/fbfc504a24f53f7ebe128ab55cb5dba634f4ece8"><code>fbfc504a24f5</code></a></td></tr>
<tr><td>bpfilter</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/d2ba09c17a0647f899d6c20a11bab9e6d3382f07"><code>d2ba09c17a06</code></a></td></tr>
<tr><td>End.BPF action for seg6local LWT</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/004d4b274e2a1a895a0e5dc66158b90a7d463d44"><code>004d4b274e2a</code></a></td></tr>
<tr><td>BPF attached to LIRC devices</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/f4364dcfc86df7c1ca47b256eaf6b6d0cdd0d936"><code>f4364dcfc86d</code></a></td></tr>
<tr><td>Pass map values to map helpers</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/d71962f3e627b5941804036755c844fabfb65ff5"><code>d71962f3e627</code></a></td></tr>
<tr><td>BPF socket reuseport</td><td>4.19</td><td><a href="https://github.com/torvalds/linux/commit/2dbb9b9e6df67d444fbe425c7f6014858d337adf"><code>2dbb9b9e6df6</code></a></td></tr>
<tr><td>BPF flow dissector</td><td>4.20</td><td><a href="https://github.com/torvalds/linux/commit/d58e468b1112dcd1d5193c0a89ff9f98b5a3e8b9"><code>d58e468b1112</code></a></td></tr>
<tr><td>BPF 1M insn limit</td><td>5.2</td><td><a href="https://github.com/torvalds/linux/commit/c04c0d2b968ac45d6ef020316808ef6c82325a82"><code>c04c0d2b968a</code></a></td></tr>
<tr><td>BPF cgroup sysctl</td><td>5.2</td><td><a href="https://github.com/torvalds/linux/commit/7b146cebe30cb481b0f70d85779da938da818637"><code>7b146cebe30c</code></a></td></tr>
<tr><td>BPF raw tracepoint writable</td><td>5.2</td><td><a href="https://github.com/torvalds/linux/commit/9df1c28bb75217b244257152ab7d788bb2a386d0"><code>9df1c28bb752</code></a></td></tr>
<tr><td>BPF bounded loop</td><td>5.3</td><td><a href="https://github.com/torvalds/linux/commit/2589726d12a1b12eaaa93c7f1ea64287e383c7a5"><code>2589726d12a1</code></a></td></tr>
<tr><td>BPF trampoline</td><td>5.5</td><td><a href="https://github.com/torvalds/linux/commit/fec56f5890d93fc2ed74166c397dc186b1c25951"><code>fec56f5890d9</code></a></td></tr>
<tr><td>BPF LSM hook</td><td>5.7</td><td><a href="https://github.com/torvalds/linux/commit/fc611f47f2188ade2b48ff6902d5cce8baac0c58"><code>fc611f47f218</code></a> <a href="https://github.com/torvalds/linux/commit/641cd7b06c911c5935c34f24850ea18690649917"><code>641cd7b06c91</code></a></td></tr>
<tr><td>BPF iterator</td><td>5.8</td><td><a href="https://github.com/torvalds/linux/commit/180139dca8b38c858027b8360ee10064fdb2fbf7"><code>180139dca8b3</code></a></td></tr>
<tr><td>BPF socket lookup hook</td><td>5.9</td><td><a href="https://github.com/torvalds/linux/commit/e9ddbb7707ff5891616240026062b8c1e29864ca"><code>e9ddbb7707ff</code></a></td></tr>
<tr><td>Sleepable BPF programs</td><td>5.10</td><td><a href="https://github.com/torvalds/linux/commit/1e6c62a8821557720a9b2ea9617359b264f2f67c"><code>1e6c62a88215</code></a></td></tr>
</tbody></table>
</div>
<h3 id="program-types"><a class="header" href="#program-types">Program types</a></h3>
<div class="table-wrapper"><table><thead><tr><th>Program type</th><th>Kernel version</th><th>Commit</th><th>Enum</th></tr></thead><tbody>
<tr><td>Socket filter</td><td>3.19</td><td><a href="https://github.com/torvalds/linux/commit/ddd872bc3098f9d9abe1680a6b2013e59e3337f7"><code>ddd872bc3098</code></a></td><td>BPF_PROG_TYPE_SOCKET_FILTER</td></tr>
<tr><td>Kprobe</td><td>4.1</td><td><a href="https://github.com/torvalds/linux/commit/2541517c32be2531e0da59dfd7efc1ce844644f5"><code>2541517c32be</code></a></td><td>BPF_PROG_TYPE_KPROBE</td></tr>
<tr><td>traffic control (TC)</td><td>4.1</td><td><a href="https://github.com/torvalds/linux/commit/96be4325f443dbbfeb37d2a157675ac0736531a1"><code>96be4325f443</code></a></td><td>BPF_PROG_TYPE_SCHED_CLS</td></tr>
<tr><td>traffic control (TC)</td><td>4.1</td><td><a href="https://github.com/torvalds/linux/commit/94caee8c312d96522bcdae88791aaa9ebcd5f22c"><code>94caee8c312d</code></a></td><td>BPF_PROG_TYPE_SCHED_ACT</td></tr>
<tr><td>Tracepoint</td><td>4.7</td><td><a href="https://github.com/torvalds/linux/commit/98b5c2c65c2951772a8fc661f50d675e450e8bce"><code>98b5c2c65c29</code></a></td><td>BPF_PROG_TYPE_TRACEPOINT</td></tr>
<tr><td>XDP</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/6a773a15a1e8874e5eccd2f29190c31085912c95"><code>6a773a15a1e8</code></a></td><td>BPF_PROG_TYPE_XDP</td></tr>
<tr><td>Perf event</td><td>4.9</td><td><a href="https://github.com/torvalds/linux/commit/0515e5999a466dfe6e1924f460da599bb6821487"><code>0515e5999a46</code></a></td><td>BPF_PROG_TYPE_PERF_EVENT</td></tr>
<tr><td>cgroup socket filtering</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/0e33661de493db325435d565a4a722120ae4cbf3"><code>0e33661de493</code></a></td><td>BPF_PROG_TYPE_CGROUP_SKB</td></tr>
<tr><td>cgroup sock modification</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/61023658760032e97869b07d54be9681d2529e77"><code>610236587600</code></a></td><td>BPF_PROG_TYPE_CGROUP_SOCK</td></tr>
<tr><td>lightweight tunnel (IN)</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/3a0af8fd61f90920f6fa04e4f1e9a6a73c1b4fd2"><code>3a0af8fd61f9</code></a></td><td>BPF_PROG_TYPE_LWT_IN</td></tr>
<tr><td>lightweight tunnel (OUT)</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/3a0af8fd61f90920f6fa04e4f1e9a6a73c1b4fd2"><code>3a0af8fd61f9</code></a></td><td>BPF_PROG_TYPE_LWT_OUT</td></tr>
<tr><td>lightweight tunnel (XMIT)</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/3a0af8fd61f90920f6fa04e4f1e9a6a73c1b4fd2"><code>3a0af8fd61f9</code></a></td><td>BPF_PROG_TYPE_LWT_XMIT</td></tr>
<tr><td>cgroup sock ops (per conn)</td><td>4.13</td><td><a href="https://github.com/torvalds/linux/commit/40304b2a1567fecc321f640ee4239556dd0f3ee0"><code>40304b2a1567</code></a></td><td>BPF_PROG_TYPE_SOCK_OPS</td></tr>
<tr><td>stream parser / stream verdict</td><td>4.14</td><td><a href="https://github.com/torvalds/linux/commit/b005fd189cec9407b700599e1e80e0552446ee79"><code>b005fd189cec</code></a></td><td>BPF_PROG_TYPE_SK_SKB</td></tr>
<tr><td>cgroup device manager</td><td>4.15</td><td><a href="https://github.com/torvalds/linux/commit/ebc614f687369f9df99828572b1d85a7c2de3d92"><code>ebc614f68736</code></a></td><td>BPF_PROG_TYPE_CGROUP_DEVICE</td></tr>
<tr><td>socket msg verdict</td><td>4.17</td><td><a href="https://github.com/torvalds/linux/commit/4f738adba30a7cfc006f605707e7aee847ffefa0"><code>4f738adba30a</code></a></td><td>BPF_PROG_TYPE_SK_MSG</td></tr>
<tr><td>Raw tracepoint</td><td>4.17</td><td><a href="https://github.com/torvalds/linux/commit/c4f6699dfcb8558d138fe838f741b2c10f416cf9"><code>c4f6699dfcb8</code></a></td><td>BPF_PROG_TYPE_RAW_TRACEPOINT</td></tr>
<tr><td>socket binding</td><td>4.17</td><td><a href="https://github.com/torvalds/linux/commit/4fbac77d2d092b475dda9eea66da674369665427"><code>4fbac77d2d09</code></a></td><td>BPF_PROG_TYPE_CGROUP_SOCK_ADDR</td></tr>
<tr><td>LWT seg6local</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/004d4b274e2a1a895a0e5dc66158b90a7d463d44"><code>004d4b274e2a</code></a></td><td>BPF_PROG_TYPE_LWT_SEG6LOCAL</td></tr>
<tr><td>lirc devices</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/f4364dcfc86df7c1ca47b256eaf6b6d0cdd0d936"><code>f4364dcfc86d</code></a></td><td>BPF_PROG_TYPE_LIRC_MODE2</td></tr>
<tr><td>lookup SO_REUSEPORT socket</td><td>4.19</td><td><a href="https://github.com/torvalds/linux/commit/2dbb9b9e6df67d444fbe425c7f6014858d337adf"><code>2dbb9b9e6df6</code></a></td><td>BPF_PROG_TYPE_SK_REUSEPORT</td></tr>
<tr><td>flow dissector</td><td>4.20</td><td><a href="https://github.com/torvalds/linux/commit/d58e468b1112dcd1d5193c0a89ff9f98b5a3e8b9"><code>d58e468b1112</code></a></td><td>BPF_PROG_TYPE_FLOW_DISSECTOR</td></tr>
<tr><td>cgroup sysctl</td><td>5.2</td><td><a href="https://github.com/torvalds/linux/commit/7b146cebe30cb481b0f70d85779da938da818637"><code>7b146cebe30c</code></a></td><td>BPF_PROG_TYPE_CGROUP_SYSCTL</td></tr>
<tr><td>writable raw tracepoints</td><td>5.2</td><td><a href="https://github.com/torvalds/linux/commit/9df1c28bb75217b244257152ab7d788bb2a386d0"><code>9df1c28bb752</code></a></td><td>BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE</td></tr>
<tr><td>cgroup getsockopt/setsockopt</td><td>5.3</td><td><a href="https://github.com/torvalds/linux/commit/0d01da6afc5402f60325c5da31b22f7d56689b49"><code>0d01da6afc54</code></a></td><td>BPF_PROG_TYPE_CGROUP_SOCKOPT</td></tr>
<tr><td>Tracing (BTF/BPF trampoline)</td><td>5.5</td><td><a href="https://github.com/torvalds/linux/commit/f1b9509c2fb0ef4db8d22dac9aef8e856a5d81f6"><code>f1b9509c2fb0</code></a></td><td>BPF_PROG_TYPE_TRACING</td></tr>
<tr><td>struct ops</td><td>5.6</td><td><a href="https://github.com/torvalds/linux/commit/27ae7997a66174cb8afd6a75b3989f5e0c1b9e5a"><code>27ae7997a661</code></a></td><td>BPF_PROG_TYPE_STRUCT_OPS</td></tr>
<tr><td>extensions</td><td>5.6</td><td><a href="https://github.com/torvalds/linux/commit/be8704ff07d2374bcc5c675526f95e70c6459683"><code>be8704ff07d2</code></a></td><td>BPF_PROG_TYPE_EXT</td></tr>
<tr><td>LSM</td><td>5.7</td><td><a href="https://github.com/torvalds/linux/commit/fc611f47f2188ade2b48ff6902d5cce8baac0c58"><code>fc611f47f218</code></a></td><td>BPF_PROG_TYPE_LSM</td></tr>
<tr><td>lookup listening socket</td><td>5.9</td><td><a href="https://github.com/torvalds/linux/commit/e9ddbb7707ff5891616240026062b8c1e29864ca"><code>e9ddbb7707ff</code></a></td><td>BPF_PROG_TYPE_SK_LOOKUP</td></tr>
<tr><td>Allow executing syscalls</td><td>5.15</td><td><a href="https://github.com/torvalds/linux/commit/79a7f8bdb159d9914b58740f3d31d602a6e4aca8"><code>79a7f8bdb159</code></a></td><td>BPF_PROG_TYPE_SYSCALL</td></tr>
</tbody></table>
</div>
<h2 id="maps-aka-tables-in-bcc-lingo"><a class="header" href="#maps-aka-tables-in-bcc-lingo">Maps (<em>a.k.a.</em> Tables, in BCC lingo)</a></h2>
<h3 id="map-types"><a class="header" href="#map-types">Map types</a></h3>
<p>The list of map types supported in your kernel can be found in file
<a href="https://github.com/torvalds/linux/blob/master/include/uapi/linux/bpf.h"><code>include/uapi/linux/bpf.h</code></a>:</p>
<pre><code class="language-sh">git grep -W 'bpf_map_type {' include/uapi/linux/bpf.h
</code></pre>
<div class="table-wrapper"><table><thead><tr><th>Map type</th><th>Kernel version</th><th>Commit</th><th>Enum</th></tr></thead><tbody>
<tr><td>Hash</td><td>3.19</td><td><a href="https://github.com/torvalds/linux/commit/0f8e4bd8a1fc8c4185f1630061d0a1f2d197a475"><code>0f8e4bd8a1fc</code></a></td><td>BPF_MAP_TYPE_HASH</td></tr>
<tr><td>Array</td><td>3.19</td><td><a href="https://github.com/torvalds/linux/commit/28fbcfa08d8ed7c5a50d41a0433aad222835e8e3"><code>28fbcfa08d8e</code></a></td><td>BPF_MAP_TYPE_ARRAY</td></tr>
<tr><td>Prog array</td><td>4.2</td><td><a href="https://github.com/torvalds/linux/commit/04fd61ab36ec065e194ab5e74ae34a5240d992bb"><code>04fd61ab36ec</code></a></td><td>BPF_MAP_TYPE_PROG_ARRAY</td></tr>
<tr><td>Perf events</td><td>4.3</td><td><a href="https://github.com/torvalds/linux/commit/ea317b267e9d03a8241893aa176fba7661d07579"><code>ea317b267e9d</code></a></td><td>BPF_MAP_TYPE_PERF_EVENT_ARRAY</td></tr>
<tr><td>Per-CPU hash</td><td>4.6</td><td><a href="https://github.com/torvalds/linux/commit/824bd0ce6c7c43a9e1e210abf124958e54d88342"><code>824bd0ce6c7c</code></a></td><td>BPF_MAP_TYPE_PERCPU_HASH</td></tr>
<tr><td>Per-CPU array</td><td>4.6</td><td><a href="https://github.com/torvalds/linux/commit/a10423b87a7eae75da79ce80a8d9475047a674ee"><code>a10423b87a7e</code></a></td><td>BPF_MAP_TYPE_PERCPU_ARRAY</td></tr>
<tr><td>Stack trace</td><td>4.6</td><td><a href="https://github.com/torvalds/linux/commit/d5a3b1f691865be576c2bffa708549b8cdccda19"><code>d5a3b1f69186</code></a></td><td>BPF_MAP_TYPE_STACK_TRACE</td></tr>
<tr><td>cgroup array</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/4ed8ec521ed57c4e207ad464ca0388776de74d4b"><code>4ed8ec521ed5</code></a></td><td>BPF_MAP_TYPE_CGROUP_ARRAY</td></tr>
<tr><td>LRU hash</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/29ba732acbeece1e34c68483d1ec1f3720fa1bb3"><code>29ba732acbee</code></a> <a href="https://github.com/torvalds/linux/commit/3a08c2fd763450a927d1130de078d6f9e74944fb"><code>3a08c2fd7634</code></a></td><td>BPF_MAP_TYPE_LRU_HASH</td></tr>
<tr><td>LRU per-CPU hash</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/8f8449384ec364ba2a654f11f94e754e4ff719e0"><code>8f8449384ec3</code></a> <a href="https://github.com/torvalds/linux/commit/961578b63474d13ad0e2f615fcc2901c5197dda6"><code>961578b63474</code></a></td><td>BPF_MAP_TYPE_LRU_PERCPU_HASH</td></tr>
<tr><td>LPM trie (longest-prefix match)</td><td>4.11</td><td><a href="https://github.com/torvalds/linux/commit/b95a5c4db09bc7c253636cb84dc9b12c577fd5a0"><code>b95a5c4db09b</code></a></td><td>BPF_MAP_TYPE_LPM_TRIE</td></tr>
<tr><td>Array of maps</td><td>4.12</td><td><a href="https://github.com/torvalds/linux/commit/56f668dfe00dcf086734f1c42ea999398fad6572"><code>56f668dfe00d</code></a></td><td>BPF_MAP_TYPE_ARRAY_OF_MAPS</td></tr>
<tr><td>Hash of maps</td><td>4.12</td><td><a href="https://github.com/torvalds/linux/commit/bcc6b1b7ebf857a9fe56202e2be3361131588c15"><code>bcc6b1b7ebf8</code></a></td><td>BPF_MAP_TYPE_HASH_OF_MAPS</td></tr>
<tr><td>Netdevice references (array)</td><td>4.14</td><td><a href="https://github.com/torvalds/linux/commit/546ac1ffb70d25b56c1126940e5ec639c4dd7413"><code>546ac1ffb70d</code></a></td><td>BPF_MAP_TYPE_DEVMAP</td></tr>
<tr><td>Socket references (array)</td><td>4.14</td><td><a href="https://github.com/torvalds/linux/commit/174a79ff9515f400b9a6115643dafd62a635b7e6"><code>174a79ff9515</code></a></td><td>BPF_MAP_TYPE_SOCKMAP</td></tr>
<tr><td>CPU references</td><td>4.15</td><td><a href="https://github.com/torvalds/linux/commit/6710e1126934d8b4372b4d2f9ae1646cd3f151bf"><code>6710e1126934</code></a></td><td>BPF_MAP_TYPE_CPUMAP</td></tr>
<tr><td>AF_XDP socket (XSK) references</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/fbfc504a24f53f7ebe128ab55cb5dba634f4ece8"><code>fbfc504a24f5</code></a></td><td>BPF_MAP_TYPE_XSKMAP</td></tr>
<tr><td>Socket references (hashmap)</td><td>4.18</td><td><a href="https://github.com/torvalds/linux/commit/81110384441a59cff47430f20f049e69b98c17f4"><code>81110384441a</code></a></td><td>BPF_MAP_TYPE_SOCKHASH</td></tr>
<tr><td>cgroup storage</td><td>4.19</td><td><a href="https://github.com/torvalds/linux/commit/de9cbbaadba5adf88a19e46df61f7054000838f6"><code>de9cbbaadba5</code></a></td><td>BPF_MAP_TYPE_CGROUP_STORAGE</td></tr>
<tr><td>reuseport sockarray</td><td>4.19</td><td><a href="https://github.com/torvalds/linux/commit/5dc4c4b7d4e8115e7cde96a030f98cb3ab2e458c"><code>5dc4c4b7d4e8</code></a></td><td>BPF_MAP_TYPE_REUSEPORT_SOCKARRAY</td></tr>
<tr><td>precpu cgroup storage</td><td>4.20</td><td><a href="https://github.com/torvalds/linux/commit/b741f1630346defcbc8cc60f1a2bdae8b3b0036f"><code>b741f1630346</code></a></td><td>BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE</td></tr>
<tr><td>queue</td><td>4.20</td><td><a href="https://github.com/torvalds/linux/commit/f1a2e44a3aeccb3ff18d3ccc0b0203e70b95bd92"><code>f1a2e44a3aec</code></a></td><td>BPF_MAP_TYPE_QUEUE</td></tr>
<tr><td>stack</td><td>4.20</td><td><a href="https://github.com/torvalds/linux/commit/f1a2e44a3aeccb3ff18d3ccc0b0203e70b95bd92"><code>f1a2e44a3aec</code></a></td><td>BPF_MAP_TYPE_STACK</td></tr>
<tr><td>socket local storage</td><td>5.2</td><td><a href="https://github.com/torvalds/linux/commit/6ac99e8f23d4b10258406ca0dd7bffca5f31da9d"><code>6ac99e8f23d4</code></a></td><td>BPF_MAP_TYPE_SK_STORAGE</td></tr>
<tr><td>Netdevice references (hashmap)</td><td>5.4</td><td><a href="https://github.com/torvalds/linux/commit/6f9d451ab1a33728adb72d7ff66a7b374d665176"><code>6f9d451ab1a3</code></a></td><td>BPF_MAP_TYPE_DEVMAP_HASH</td></tr>
<tr><td>struct ops</td><td>5.6</td><td><a href="https://github.com/torvalds/linux/commit/85d33df357b634649ddbe0a20fd2d0fc5732c3cb"><code>85d33df357b6</code></a></td><td>BPF_MAP_TYPE_STRUCT_OPS</td></tr>
<tr><td>ring buffer</td><td>5.8</td><td><a href="https://github.com/torvalds/linux/commit/457f44363a8894135c85b7a9afd2bd8196db24ab"><code>457f44363a88</code></a></td><td>BPF_MAP_TYPE_RINGBUF</td></tr>
<tr><td>inode storage</td><td>5.10</td><td><a href="https://github.com/torvalds/linux/commit/8ea636848aca35b9f97c5b5dee30225cf2dd0fe6"><code>8ea636848aca</code></a></td><td>BPF_MAP_TYPE_INODE_STORAGE</td></tr>
<tr><td>task storage</td><td>5.11</td><td><a href="https://github.com/torvalds/linux/commit/4cf1bc1f10452065a29d576fc5693fc4fab5b919"><code>4cf1bc1f1045</code></a></td><td>BPF_MAP_TYPE_TASK_STORAGE</td></tr>
<tr><td>Bloom filter</td><td>5.16</td><td><a href="https://github.com/torvalds/linux/commit/9330986c03006ab1d33d243b7cfe598a7a3c1baa"><code>9330986c0300</code></a></td><td>BPF_MAP_TYPE_BLOOM_FILTER</td></tr>
<tr><td>user ringbuf</td><td>6.1</td><td><a href="https://github.com/torvalds/linux/commit/583c1f420173f7d84413a1a1fbf5109d798b4faa"><code>583c1f420173</code></a></td><td>BPF_MAP_TYPE_USER_RINGBUF</td></tr>
</tbody></table>
</div>
<h3 id="map-userspace-api"><a class="header" href="#map-userspace-api">Map userspace API</a></h3>
<p>Some (but not all) of these <em>API features</em> translate to a subcommand beginning with <code>BPF_MAP_</code>.
The list of subcommands supported in your kernel can be found in file
<a href="https://github.com/torvalds/linux/blob/master/include/uapi/linux/bpf.h"><code>include/uapi/linux/bpf.h</code></a>:</p>
<pre><code class="language-sh">git grep -W 'bpf_cmd {' include/uapi/linux/bpf.h
</code></pre>
<div class="table-wrapper"><table><thead><tr><th>Feature</th><th>Kernel version</th><th>Commit</th></tr></thead><tbody>
<tr><td>Basic operations (lookup, update, delete, <code>GET_NEXT_KEY</code>)</td><td>3.18</td><td><a href="https://github.com/torvalds/linux/commit/db20fd2b01087bdfbe30bce314a198eefedcc42e"><code>db20fd2b0108</code></a></td></tr>
<tr><td>Pass flags to <code>UPDATE_ELEM</code></td><td>3.19</td><td><a href="https://github.com/torvalds/linux/commit/3274f52073d88b62f3c5ace82ae9d48546232e72"><code>3274f52073d8</code></a></td></tr>
<tr><td>Pre-alloc map memory by default</td><td>4.6</td><td><a href="https://github.com/torvalds/linux/commit/6c90598174322b8888029e40dd84a4eb01f56afe"><code>6c9059817432</code></a></td></tr>
<tr><td>Pass <code>NULL</code> to <code>GET_NEXT_KEY</code></td><td>4.12</td><td><a href="https://github.com/torvalds/linux/commit/8fe45924387be6b5c1be59a7eb330790c61d5d10"><code>8fe45924387b</code></a></td></tr>
<tr><td>Creation: select NUMA node</td><td>4.14</td><td><a href="https://github.com/torvalds/linux/commit/96eabe7a40aa17e613cf3db2c742ee8b1fc764d0"><code>96eabe7a40aa</code></a></td></tr>
<tr><td>Restrict access from syscall side</td><td>4.15</td><td><a href="https://github.com/torvalds/linux/commit/6e71b04a82248ccf13a94b85cbc674a9fefe53f5"><code>6e71b04a8224</code></a></td></tr>
<tr><td>Creation: specify map name</td><td>4.15</td><td><a href="https://github.com/torvalds/linux/commit/ad5b177bd73f5107d97c36f56395c4281fb6f089"><code>ad5b177bd73f</code></a></td></tr>
<tr><td><code>LOOKUP_AND_DELETE_ELEM</code></td><td>4.20</td><td><a href="https://github.com/torvalds/linux/commit/bd513cd08f10cbe28856f99ae951e86e86803861"><code>bd513cd08f10</code></a></td></tr>
<tr><td>Creation: <code>BPF_F_ZERO_SEED</code></td><td>5.0</td><td><a href="https://github.com/torvalds/linux/commit/96b3b6c9091d23289721350e32c63cc8749686be"><code>96b3b6c9091d</code></a></td></tr>
<tr><td><code>BPF_F_LOCK</code> flag for lookup / update</td><td>5.1</td><td><a href="https://github.com/torvalds/linux/commit/96049f3afd50fe8db69fa0068cdca822e747b1e4"><code>96049f3afd50</code></a></td></tr>
<tr><td>Restrict access from BPF side</td><td>5.2</td><td><a href="https://github.com/torvalds/linux/commit/591fe9888d7809d9ee5c828020b6c6ae27c37229"><code>591fe9888d78</code></a></td></tr>
<tr><td><code>FREEZE</code></td><td>5.2</td><td><a href="https://github.com/torvalds/linux/commit/87df15de441bd4add7876ef584da8cabdd9a042a"><code>87df15de441b</code></a></td></tr>
<tr><td>mmap() support for array maps</td><td>5.5</td><td><a href="https://github.com/torvalds/linux/commit/fc9702273e2edb90400a34b3be76f7b08fa3344b"><code>fc9702273e2e</code></a></td></tr>
<tr><td><code>LOOKUP_BATCH</code></td><td>5.6</td><td><a href="https://github.com/torvalds/linux/commit/cb4d03ab499d4c040f4ab6fd4389d2b49f42b5a5"><code>cb4d03ab499d</code></a></td></tr>
<tr><td><code>UPDATE_BATCH</code>, <code>DELETE_BATCH</code></td><td>5.6</td><td><a href="https://github.com/torvalds/linux/commit/aa2e93b8e58e18442edfb2427446732415bc215e"><code>aa2e93b8e58e</code></a></td></tr>
<tr><td><code>LOOKUP_AND_DELETE_BATCH</code></td><td>5.6</td><td><a href="https://github.com/torvalds/linux/commit/057996380a42bb64ccc04383cfa9c0ace4ea11f0"><code>057996380a42</code></a></td></tr>
<tr><td><code>LOOKUP_AND_DELETE_ELEM</code> support for hash maps</td><td>5.14</td><td><a href="https://github.com/torvalds/linux/commit/3e87f192b405960c0fe83e0925bd0dadf4f8cf43"><code>3e87f192b405</code></a></td></tr>
</tbody></table>
</div>
<h2 id="xdp"><a class="header" href="#xdp">XDP</a></h2>
<p>An approximate list of drivers or components supporting XDP programs for your
kernel can be retrieved with:</p>
<pre><code class="language-sh">git grep -l XDP_SETUP_PROG drivers/
</code></pre>
<div class="table-wrapper"><table><thead><tr><th>Feature / Driver</th><th>Kernel version</th><th>Commit</th></tr></thead><tbody>
<tr><td>XDP core architecture</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/6a773a15a1e8874e5eccd2f29190c31085912c95"><code>6a773a15a1e8</code></a></td></tr>
<tr><td>Action: drop</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/6a773a15a1e8874e5eccd2f29190c31085912c95"><code>6a773a15a1e8</code></a></td></tr>
<tr><td>Action: pass on to stack</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/6a773a15a1e8874e5eccd2f29190c31085912c95"><code>6a773a15a1e8</code></a></td></tr>
<tr><td>Action: direct forwarding (on same port)</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/6ce96ca348a9e949f8c43f4d3e98db367d93cffd"><code>6ce96ca348a9</code></a></td></tr>
<tr><td>Direct packet data write</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/4acf6c0b84c91243c705303cd9ff16421914150d"><code>4acf6c0b84c9</code></a></td></tr>
<tr><td>Mellanox <code>mlx4</code> driver</td><td>4.8</td><td><a href="https://github.com/torvalds/linux/commit/47a38e155037f417c5740e24ccae6482aedf4b68"><code>47a38e155037</code></a></td></tr>
<tr><td>Mellanox <code>mlx5</code> driver</td><td>4.9</td><td><a href="https://github.com/torvalds/linux/commit/86994156c736978d113e7927455d4eeeb2128b9f"><code>86994156c736</code></a></td></tr>
<tr><td>Netronome <code>nfp</code> driver</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/ecd63a0217d5f1e8a92f7516f5586d1177b95de2"><code>ecd63a0217d5</code></a></td></tr>
<tr><td>QLogic (Cavium) <code>qed*</code> drivers</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/496e051709588f832d7a6a420f44f8642b308a87"><code>496e05170958</code></a></td></tr>
<tr><td><code>virtio_net</code> driver</td><td>4.10</td><td><a href="https://github.com/torvalds/linux/commit/f600b690501550b94e83e07295d9c8b9c4c39f4e"><code>f600b6905015</code></a></td></tr>
<tr><td>Broadcom <code>bnxt_en</code> driver</td><td>4.11</td><td><a href="https://github.com/torvalds/linux/commit/c6d30e8391b85e00eb544e6cf047ee0160ee9938"><code>c6d30e8391b8</code></a></td></tr>
<tr><td>Intel <code>ixgbe*</code> drivers</td><td>4.12</td><td><a href="https://github.com/torvalds/linux/commit/9247080816297de4e31abb684939c0e53e3a8a67"><code>924708081629</code></a></td></tr>
<tr><td>Cavium <code>thunderx</code> driver</td><td>4.12</td><td><a href="https://github.com/torvalds/linux/commit/05c773f52b96ef3fbc7d9bfa21caadc6247ef7a8"><code>05c773f52b96</code></a></td></tr>
<tr><td>Generic XDP</td><td>4.12</td><td><a href="https://github.com/torvalds/linux/commit/b5cdae3291f7be7a34e75affe4c0ec1f7f328b64"><code>b5cdae3291f7</code></a></td></tr>
<tr><td>Intel <code>i40e</code> driver</td><td>4.13</td><td><a href="https://github.com/torvalds/linux/commit/0c8493d90b6bb0f5c4fe9217db8f7203f24c0f28"><code>0c8493d90b6b</code></a></td></tr>
<tr><td>Action: redirect</td><td>4.14</td><td><a href="https://github.com/torvalds/linux/commit/6453073987ba392510ab6c8b657844a9312c67f7"><code>6453073987ba</code></a></td></tr>
<tr><td>Support for tap</td><td>4.14</td><td><a href="https://github.com/torvalds/linux/commit/761876c857cb2ef8489fbee01907151da902af91"><code>761876c857cb</code></a></td></tr>
<tr><td>Support for veth</td><td>4.14</td><td><a href="https://github.com/torvalds/linux/commit/d445516966dcb2924741b13b27738b54df2af01a"><code>d445516966dc</code></a></td></tr>
<tr><td>Intel <code>ixgbevf</code> driver</td><td>4.17</td><td><a href="https://github.com/torvalds/linux/commit/c7aec59657b60f3a29fc7d3274ebefd698879301"><code>c7aec59657b6</code></a></td></tr>
<tr><td>Freescale <code>dpaa2</code> driver</td><td>5.0</td><td><a href="https://github.com/torvalds/linux/commit/7e273a8ebdd3b83f94eb8b49fc8ee61464f47cc2"><code>7e273a8ebdd3</code></a></td></tr>
<tr><td>Socionext <code>netsec</code> driver</td><td>5.3</td><td><a href="https://github.com/torvalds/linux/commit/ba2b232108d3c2951bab02930a00f23b0cffd5af"><code>ba2b232108d3</code></a></td></tr>
<tr><td>TI <code>cpsw</code> driver</td><td>5.3</td><td><a href="https://github.com/torvalds/linux/commit/9ed4050c0d75768066a07cf66eef4f8dc9d79b52"><code>9ed4050c0d75</code></a></td></tr>
<tr><td>Intel <code>ice</code> driver</td><td>5.5</td><td><a href="https://github.com/torvalds/linux/commit/efc2214b6047b6f5b4ca53151eba62521b9452d6"><code>efc2214b6047</code></a></td></tr>
<tr><td>Solarflare <code>sfc</code> driver</td><td>5.5</td><td><a href="https://github.com/torvalds/linux/commit/eb9a36be7f3ec414700af9a616f035eda1f1e63e"><code>eb9a36be7f3e</code></a></td></tr>
<tr><td>Marvell <code>mvneta</code> driver</td><td>5.5</td><td><a href="https://github.com/torvalds/linux/commit/0db51da7a8e99f0803ec3a8e25c1a66234a219cb"><code>0db51da7a8e9</code></a></td></tr>
<tr><td>Microsoft <code>hv_netvsc</code> driver</td><td>5.6</td><td><a href="https://github.com/torvalds/linux/commit/351e1581395fcc7fb952bbd7dda01238f69968fd"><code>351e1581395f</code></a></td></tr>
<tr><td>Amazon <code>ena</code> driver</td><td>5.6</td><td><a href="https://github.com/torvalds/linux/commit/838c93dc5449e5d6378bae117b0a65a122cf7361"><code>838c93dc5449</code></a></td></tr>
<tr><td><code>xen-netfront</code> driver</td><td>5.9</td><td><a href="https://github.com/torvalds/linux/commit/6c5aa6fc4defc2a0977a2c59e4710d50fa1e834c"><code>6c5aa6fc4def</code></a></td></tr>
<tr><td>Intel <code>igb</code> driver</td><td>5.10</td><td><a href="https://github.com/torvalds/linux/commit/9cbc948b5a20c9c054d9631099c0426c16da546b"><code>9cbc948b5a20</code></a></td></tr>
</tbody></table>
</div>
<h2 id="helpers"><a class="header" href="#helpers">Helpers</a></h2>
<p>The list of helpers supported in your kernel can be found in file
<a href="https://github.com/torvalds/linux/blob/master/include/uapi/linux/bpf.h"><code>include/uapi/linux/bpf.h</code></a>:</p>
<pre><code class="language-sh">git grep ' FN(' include/uapi/linux/bpf.h
</code></pre>
<p>Alphabetical order</p>
<div class="table-wrapper"><table><thead><tr><th>Helper</th><th>Kernel version</th><th>License</th><th>Commit</th></tr></thead><tbody>
<tr><td><code>BPF_FUNC_bind()</code></td><td>4.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d74bad4e74ee373787a9ae24197c17b7cdc428d5"><code>d74bad4e74ee</code></a></td></tr>
<tr><td><code>BPF_FUNC_bprm_opts_set()</code></td><td>5.11</td><td></td><td><a href="https://github.com/torvalds/linux/commit/3f6719c7b62f0327c9091e26d0da10e65668229e"><code>3f6719c7b62f</code></a></td></tr>
<tr><td><code>BPF_FUNC_btf_find_by_name_kind()</code></td><td>5.14</td><td></td><td><a href="https://github.com/torvalds/linux/commit/3d78417b60fba249cc555468cb72d96f5cde2964"><code>3d78417b60fb</code></a></td></tr>
<tr><td><code>BPF_FUNC_cgrp_storage_delete()</code></td><td>6.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/c4bcfb38a95edb1021a53f2d0356a78120ecfbe4"><code>c4bcfb38a95e</code></a></td></tr>
<tr><td><code>BPF_FUNC_cgrp_storage_get()</code></td><td>6.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/c4bcfb38a95edb1021a53f2d0356a78120ecfbe4"><code>c4bcfb38a95e</code></a></td></tr>
<tr><td><code>BPF_FUNC_check_mtu()</code></td><td>5.12</td><td></td><td><a href="https://github.com/torvalds/linux/commit/34b2021cc61642d61c3cf943d9e71925b827941b"><code>34b2021cc616</code></a></td></tr>
<tr><td><code>BPF_FUNC_clone_redirect()</code></td><td>4.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/3896d655f4d491c67d669a15f275a39f713410f8"><code>3896d655f4d4</code></a></td></tr>
<tr><td><code>BPF_FUNC_copy_from_user()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/07be4c4a3e7a0db148e44b16c5190e753d1c8569"><code>07be4c4a3e7a</code></a></td></tr>
<tr><td><code>BPF_FUNC_copy_from_user_task()</code></td><td>5.18</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/376040e47334c6dc6a939a32197acceb00fe4acf"><code>376040e47334</code></a></td></tr>
<tr><td><code>BPF_FUNC_csum_diff()</code></td><td>4.6</td><td></td><td><a href="https://github.com/torvalds/linux/commit/7d672345ed295b1356a5d9f7111da1d1d7d65867"><code>7d672345ed29</code></a></td></tr>
<tr><td><code>BPF_FUNC_csum_level()</code></td><td>5.7</td><td></td><td><a href="https://github.com/torvalds/linux/commit/7cdec54f9713256bb170873a1fc5c75c9127c9d2"><code>7cdec54f9713</code></a></td></tr>
<tr><td><code>BPF_FUNC_csum_update()</code></td><td>4.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/36bbef52c7eb646ed6247055a2acd3851e317857"><code>36bbef52c7eb</code></a></td></tr>
<tr><td><code>BPF_FUNC_current_task_under_cgroup()</code></td><td>4.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/60d20f9195b260bdf0ac10c275ae9f6016f9c069"><code>60d20f9195b2</code></a></td></tr>
<tr><td><code>BPF_FUNC_d_path()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/6e22ab9da79343532cd3cde39df25e5a5478c692"><code>6e22ab9da793</code></a></td></tr>
<tr><td><code>BPF_FUNC_dynptr_data()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/34d4ef5775f776ec4b0d53a02d588bf3195cada6"><code>34d4ef5775f7</code></a></td></tr>
<tr><td><code>BPF_FUNC_dynptr_from_mem()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/263ae152e96253f40c2c276faad8629e096b3bad"><code>263ae152e962</code></a></td></tr>
<tr><td><code>BPF_FUNC_dynptr_read()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/13bbbfbea7598ea9f8d9c3d73bf053bb57f9c4b2"><code>13bbbfbea759</code></a></td></tr>
<tr><td><code>BPF_FUNC_dynptr_write()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/13bbbfbea7598ea9f8d9c3d73bf053bb57f9c4b2"><code>13bbbfbea759</code></a></td></tr>
<tr><td><code>BPF_FUNC_fib_lookup()</code></td><td>4.18</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/87f5fc7e48dd3175b30dd03b41564e1a8e136323"><code>87f5fc7e48dd</code></a></td></tr>
<tr><td><code>BPF_FUNC_find_vma()</code></td><td>5.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/7c7e3d31e7856a8260a254f8c71db416f7f9f5a1"><code>7c7e3d31e785</code></a></td></tr>
<tr><td><code>BPF_FUNC_for_each_map_elem()</code></td><td>5.13</td><td></td><td><a href="https://github.com/torvalds/linux/commit/69c087ba6225b574afb6e505b72cb75242a3d844"><code>69c087ba6225</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_attach_cookie()</code></td><td>5.15</td><td></td><td><a href="https://github.com/torvalds/linux/commit/7adfc6c9b315e174cf8743b21b7b691c8766791b"><code>7adfc6c9b315</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_branch_snapshot()</code></td><td>5.16</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/856c02dbce4f8d6a5644083db22c11750aa11481"><code>856c02dbce4f</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_current_ancestor_cgroup_id()</code></td><td>5.6</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b4490c5c4e023f09b7d27c9a9d3e7ad7d09ea6bf"><code>b4490c5c4e02</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_cgroup_classid()</code></td><td>4.3</td><td></td><td><a href="https://github.com/torvalds/linux/commit/8d20aabe1c76cccac544d9fcc3ad7823d9e98a2d"><code>8d20aabe1c76</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_current_cgroup_id()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/bf6fa2c893c5237b48569a13fa3c673041430b6c"><code>bf6fa2c893c5</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_current_comm()</code></td><td>4.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/ffeedafbf0236f03aeb2e8db273b3e5ae5f5bc89"><code>ffeedafbf023</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_current_pid_tgid()</code></td><td>4.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/ffeedafbf0236f03aeb2e8db273b3e5ae5f5bc89"><code>ffeedafbf023</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_current_task()</code></td><td>4.8</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/606274c5abd8e245add01bc7145a8cbb92b69ba8"><code>606274c5abd8</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_current_task_btf()</code></td><td>5.11</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/3ca1032ab7ab010eccb107aa515598788f7d93bb"><code>3ca1032ab7ab</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_current_uid_gid()</code></td><td>4.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/ffeedafbf0236f03aeb2e8db273b3e5ae5f5bc89"><code>ffeedafbf023</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_func_arg()</code></td><td>5.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f92c1e183604c20ce00eb889315fdaa8f2d9e509"><code>f92c1e183604</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_func_arg_cnt()</code></td><td>5.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f92c1e183604c20ce00eb889315fdaa8f2d9e509"><code>f92c1e183604</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_func_ip()</code></td><td>5.15</td><td></td><td><a href="https://github.com/torvalds/linux/commit/5d8b583d04aedb3bd5f6d227a334c210c7d735f9"><code>5d8b583d04ae</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_func_ret()</code></td><td>5.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f92c1e183604c20ce00eb889315fdaa8f2d9e509"><code>f92c1e183604</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_retval()</code></td><td>5.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b44123b4a3dcad4664d3a0f72c011ffd4c9c4d93"><code>b44123b4a3dc</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_hash_recalc()</code></td><td>4.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/13c5c240f789bbd2bcacb14a23771491485ae61f"><code>13c5c240f789</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_listener_sock()</code></td><td>5.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/dbafd7ddd62369b2f3926ab847cbf8fc40e800b7"><code>dbafd7ddd623</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_local_storage()</code></td><td>4.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/cd3394317653837e2eb5c5d0904a8996102af9fc"><code>cd3394317653</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_netns_cookie()</code></td><td>5.7</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f318903c0bf42448b4c884732df2bbb0ef7a2284"><code>f318903c0bf4</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_ns_current_pid_tgid()</code></td><td>5.7</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b4490c5c4e023f09b7d27c9a9d3e7ad7d09ea6bf"><code>b4490c5c4e02</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_numa_node_id()</code></td><td>4.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/2d0e30c30f84d08dc16f0f2af41f1b8a85f0755e"><code>2d0e30c30f84</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_prandom_u32()</code></td><td>4.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/03e69b508b6f7c51743055c9f61d1dfeadf4b635"><code>03e69b508b6f</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_route_realm()</code></td><td>4.4</td><td></td><td><a href="https://github.com/torvalds/linux/commit/c46646d0484f5d08e2bede9b45034ba5b8b489cc"><code>c46646d0484f</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_smp_processor_id()</code></td><td>4.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/c04167ce2ca0ecaeaafef006cb0d65cf01b68e42"><code>c04167ce2ca0</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_socket_cookie()</code></td><td>4.12</td><td></td><td><a href="https://github.com/torvalds/linux/commit/91b8270f2a4d1d9b268de90451cdca63a70052d6"><code>91b8270f2a4d</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_socket_uid()</code></td><td>4.12</td><td></td><td><a href="https://github.com/torvalds/linux/commit/6acc5c2910689fc6ee181bf63085c5efff6a42bd"><code>6acc5c291068</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_stack()</code></td><td>4.18</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/de2ff05f48afcde816ff4edb217417f62f624ab5"><code>de2ff05f48af</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_stackid()</code></td><td>4.6</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/d5a3b1f691865be576c2bffa708549b8cdccda19"><code>d5a3b1f69186</code></a></td></tr>
<tr><td><code>BPF_FUNC_get_task_stack()</code></td><td>5.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/fa28dcb82a38f8e3993b0fae9106b1a80b59e4f0"><code>fa28dcb82a38</code></a></td></tr>
<tr><td><code>BPF_FUNC_getsockopt()</code></td><td>4.15</td><td></td><td><a href="https://github.com/torvalds/linux/commit/cd86d1fd21025fdd6daf23d1288da405e7ad0ec6"><code>cd86d1fd2102</code></a></td></tr>
<tr><td><code>BPF_FUNC_ima_file_hash()</code></td><td>5.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/174b16946e39ebd369097e0f773536c91a8c1a4c"><code>174b16946e39</code></a></td></tr>
<tr><td><code>BPF_FUNC_ima_inode_hash()</code></td><td>5.11</td><td></td><td><a href="https://github.com/torvalds/linux/commit/27672f0d280a3f286a410a8db2004f46ace72a17"><code>27672f0d280a</code></a></td></tr>
<tr><td><code>BPF_FUNC_inode_storage_delete()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/8ea636848aca35b9f97c5b5dee30225cf2dd0fe6"><code>8ea636848aca</code></a></td></tr>
<tr><td><code>BPF_FUNC_inode_storage_get()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/8ea636848aca35b9f97c5b5dee30225cf2dd0fe6"><code>8ea636848aca</code></a></td></tr>
<tr><td><code>BPF_FUNC_jiffies64()</code></td><td>5.5</td><td></td><td><a href="https://github.com/torvalds/linux/commit/5576b991e9c1a11d2cc21c4b94fc75ec27603896"><code>5576b991e9c1</code></a></td></tr>
<tr><td><code>BPF_FUNC_kallsyms_lookup_name()</code></td><td>5.16</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d6aef08a872b9e23eecc92d0e92393473b13c497"><code>d6aef08a872b</code></a></td></tr>
<tr><td><code>BPF_FUNC_kptr_xchg()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/c0a5a21c25f37c9fd7b36072f9968cdff1e4aa13"><code>c0a5a21c25f3</code></a></td></tr>
<tr><td><code>BPF_FUNC_ktime_get_boot_ns()</code></td><td>5.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/71d19214776e61b33da48f7c1b46e522c7f78221"><code>71d19214776e</code></a></td></tr>
<tr><td><code>BPF_FUNC_ktime_get_coarse_ns()</code></td><td>5.11</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d055126180564a57fe533728a4e93d0cb53d49b3"><code>d05512618056</code></a></td></tr>
<tr><td><code>BPF_FUNC_ktime_get_ns()</code></td><td>4.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d9847d310ab4003725e6ed1822682e24bd406908"><code>d9847d310ab4</code></a></td></tr>
<tr><td><code>BPF_FUNC_ktime_get_tai_ns()</code></td><td>6.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/c8996c98f703b09afe77a1d247dae691c9849dc1"><code>c8996c98f703</code></a></td></tr>
<tr><td><code>BPF_FUNC_l3_csum_replace()</code></td><td>4.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/91bc4822c3d61b9bb7ef66d3b77948a4f9177954"><code>91bc4822c3d6</code></a></td></tr>
<tr><td><code>BPF_FUNC_l4_csum_replace()</code></td><td>4.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/91bc4822c3d61b9bb7ef66d3b77948a4f9177954"><code>91bc4822c3d6</code></a></td></tr>
<tr><td><code>BPF_FUNC_load_hdr_opt()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/0813a841566f0962a5551be7749b43c45f0022a0"><code>0813a841566f</code></a></td></tr>
<tr><td><code>BPF_FUNC_loop()</code></td><td>5.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/e6f2dd0f80674e9d5960337b3e9c2a242441b326"><code>e6f2dd0f8067</code></a></td></tr>
<tr><td><code>BPF_FUNC_lwt_push_encap()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/fe94cc290f535709d3c5ebd1e472dfd0aec7ee79"><code>fe94cc290f53</code></a></td></tr>
<tr><td><code>BPF_FUNC_lwt_seg6_action()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/fe94cc290f535709d3c5ebd1e472dfd0aec7ee79"><code>fe94cc290f53</code></a></td></tr>
<tr><td><code>BPF_FUNC_lwt_seg6_adjust_srh()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/fe94cc290f535709d3c5ebd1e472dfd0aec7ee79"><code>fe94cc290f53</code></a></td></tr>
<tr><td><code>BPF_FUNC_lwt_seg6_store_bytes()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/fe94cc290f535709d3c5ebd1e472dfd0aec7ee79"><code>fe94cc290f53</code></a></td></tr>
<tr><td><code>BPF_FUNC_map_delete_elem()</code></td><td>3.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d0003ec01c667b731c139e23de3306a8b328ccf5"><code>d0003ec01c66</code></a></td></tr>
<tr><td><code>BPF_FUNC_map_lookup_elem()</code></td><td>3.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d0003ec01c667b731c139e23de3306a8b328ccf5"><code>d0003ec01c66</code></a></td></tr>
<tr><td><code>BPF_FUNC_map_lookup_percpu_elem()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/07343110b293456d30393e89b86c4dee1ac051c8"><code>07343110b293</code></a></td></tr>
<tr><td><code>BPF_FUNC_map_peek_elem()</code></td><td>4.20</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f1a2e44a3aeccb3ff18d3ccc0b0203e70b95bd92"><code>f1a2e44a3aec</code></a></td></tr>
<tr><td><code>BPF_FUNC_map_pop_elem()</code></td><td>4.20</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f1a2e44a3aeccb3ff18d3ccc0b0203e70b95bd92"><code>f1a2e44a3aec</code></a></td></tr>
<tr><td><code>BPF_FUNC_map_push_elem()</code></td><td>4.20</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f1a2e44a3aeccb3ff18d3ccc0b0203e70b95bd92"><code>f1a2e44a3aec</code></a></td></tr>
<tr><td><code>BPF_FUNC_map_update_elem()</code></td><td>3.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d0003ec01c667b731c139e23de3306a8b328ccf5"><code>d0003ec01c66</code></a></td></tr>
<tr><td><code>BPF_FUNC_msg_apply_bytes()</code></td><td>4.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/2a100317c9ebc204a166f16294884fbf9da074ce"><code>2a100317c9eb</code></a></td></tr>
<tr><td><code>BPF_FUNC_msg_cork_bytes()</code></td><td>4.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/91843d540a139eb8070bcff8aa10089164436deb"><code>91843d540a13</code></a></td></tr>
<tr><td><code>BPF_FUNC_msg_pop_data()</code></td><td>5.0</td><td></td><td><a href="https://github.com/torvalds/linux/commit/7246d8ed4dcce23f7509949a77be15fa9f0e3d28"><code>7246d8ed4dcc</code></a></td></tr>
<tr><td><code>BPF_FUNC_msg_pull_data()</code></td><td>4.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/015632bb30daaaee64e1bcac07570860e0bf3092"><code>015632bb30da</code></a></td></tr>
<tr><td><code>BPF_FUNC_msg_push_data()</code></td><td>4.20</td><td></td><td><a href="https://github.com/torvalds/linux/commit/6fff607e2f14bd7c63c06c464a6f93b8efbabe28"><code>6fff607e2f14</code></a></td></tr>
<tr><td><code>BPF_FUNC_msg_redirect_hash()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/81110384441a59cff47430f20f049e69b98c17f4"><code>81110384441a</code></a></td></tr>
<tr><td><code>BPF_FUNC_msg_redirect_map()</code></td><td>4.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4f738adba30a7cfc006f605707e7aee847ffefa0"><code>4f738adba30a</code></a></td></tr>
<tr><td><code>BPF_FUNC_per_cpu_ptr()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/eaa6bcb71ef6ed3dc18fc525ee7e293b06b4882b"><code>eaa6bcb71ef6</code></a></td></tr>
<tr><td><code>BPF_FUNC_perf_event_output()</code></td><td>4.4</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/a43eec304259a6c637f4014a6d4767159b6a3aa3"><code>a43eec304259</code></a></td></tr>
<tr><td><code>BPF_FUNC_perf_event_read()</code></td><td>4.3</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/35578d7984003097af2b1e34502bc943d40c1804"><code>35578d798400</code></a></td></tr>
<tr><td><code>BPF_FUNC_perf_event_read_value()</code></td><td>4.15</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/908432ca84fc229e906ba164219e9ad0fe56f755"><code>908432ca84fc</code></a></td></tr>
<tr><td><code>BPF_FUNC_perf_prog_read_value()</code></td><td>4.15</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/4bebdc7a85aa400c0222b5329861e4ad9252f1e5"><code>4bebdc7a85aa</code></a></td></tr>
<tr><td><code>BPF_FUNC_probe_read()</code></td><td>4.1</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/2541517c32be2531e0da59dfd7efc1ce844644f5"><code>2541517c32be</code></a></td></tr>
<tr><td><code>BPF_FUNC_probe_read_kernel()</code></td><td>5.5</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/6ae08ae3dea2cfa03dd3665a3c8475c2d429ef47"><code>6ae08ae3dea2</code></a></td></tr>
<tr><td><code>BPF_FUNC_probe_read_kernel_str()</code></td><td>5.5</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/6ae08ae3dea2cfa03dd3665a3c8475c2d429ef47"><code>6ae08ae3dea2</code></a></td></tr>
<tr><td><code>BPF_FUNC_probe_read_user()</code></td><td>5.5</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/6ae08ae3dea2cfa03dd3665a3c8475c2d429ef47"><code>6ae08ae3dea2</code></a></td></tr>
<tr><td><code>BPF_FUNC_probe_read_user_str()</code></td><td>5.5</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/6ae08ae3dea2cfa03dd3665a3c8475c2d429ef47"><code>6ae08ae3dea2</code></a></td></tr>
<tr><td><code>BPF_FUNC_probe_read_str()</code></td><td>4.11</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/a5e8c07059d0f0b31737408711d44794928ac218"><code>a5e8c07059d0</code></a></td></tr>
<tr><td><code>BPF_FUNC_probe_write_user()</code></td><td>4.8</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/96ae52279594470622ff0585621a13e96b700600"><code>96ae52279594</code></a></td></tr>
<tr><td><code>BPF_FUNC_rc_keydown()</code></td><td>4.18</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/f4364dcfc86df7c1ca47b256eaf6b6d0cdd0d936"><code>f4364dcfc86d</code></a></td></tr>
<tr><td><code>BPF_FUNC_rc_pointer_rel()</code></td><td>5.0</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/01d3240a04f4c09392e13c77b54d4423ebce2d72"><code>01d3240a04f4</code></a></td></tr>
<tr><td><code>BPF_FUNC_rc_repeat()</code></td><td>4.18</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/f4364dcfc86df7c1ca47b256eaf6b6d0cdd0d936"><code>f4364dcfc86d</code></a></td></tr>
<tr><td><code>BPF_FUNC_read_branch_records()</code></td><td>5.6</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/fff7b64355eac6e29b50229ad1512315bc04b44e"><code>fff7b64355ea</code></a></td></tr>
<tr><td><code>BPF_FUNC_redirect()</code></td><td>4.4</td><td></td><td><a href="https://github.com/torvalds/linux/commit/27b29f63058d26c6c1742f1993338280d5a41dc6"><code>27b29f63058d</code></a></td></tr>
<tr><td><code>BPF_FUNC_redirect_map()</code></td><td>4.14</td><td></td><td><a href="https://github.com/torvalds/linux/commit/97f91a7cf04ff605845c20948b8a80e54cbd3376"><code>97f91a7cf04f</code></a></td></tr>
<tr><td><code>BPF_FUNC_redirect_neigh()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b4ab31414970a7a03a5d55d75083f2c101a30592"><code>b4ab31414970</code></a></td></tr>
<tr><td><code>BPF_FUNC_redirect_peer()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/9aa1206e8f48222f35a0c809f33b2f4aaa1e2661"><code>9aa1206e8f48</code></a></td></tr>
<tr><td><code>BPF_FUNC_reserve_hdr_opt()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/0813a841566f0962a5551be7749b43c45f0022a0"><code>0813a841566f</code></a></td></tr>
<tr><td><code>BPF_FUNC_ringbuf_discard()</code></td><td>5.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/457f44363a8894135c85b7a9afd2bd8196db24ab"><code>457f44363a88</code></a></td></tr>
<tr><td><code>BPF_FUNC_ringbuf_discard_dynptr()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/bc34dee65a65e9c920c420005b8a43f2a721a458"><code>bc34dee65a65</code></a></td></tr>
<tr><td><code>BPF_FUNC_ringbuf_output()</code></td><td>5.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/457f44363a8894135c85b7a9afd2bd8196db24ab"><code>457f44363a88</code></a></td></tr>
<tr><td><code>BPF_FUNC_ringbuf_query()</code></td><td>5.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/457f44363a8894135c85b7a9afd2bd8196db24ab"><code>457f44363a88</code></a></td></tr>
<tr><td><code>BPF_FUNC_ringbuf_reserve()</code></td><td>5.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/457f44363a8894135c85b7a9afd2bd8196db24ab"><code>457f44363a88</code></a></td></tr>
<tr><td><code>BPF_FUNC_ringbuf_reserve_dynptr()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/bc34dee65a65e9c920c420005b8a43f2a721a458"><code>bc34dee65a65</code></a></td></tr>
<tr><td><code>BPF_FUNC_ringbuf_submit()</code></td><td>5.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/457f44363a8894135c85b7a9afd2bd8196db24ab"><code>457f44363a88</code></a></td></tr>
<tr><td><code>BPF_FUNC_ringbuf_submit_dynptr()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/bc34dee65a65e9c920c420005b8a43f2a721a458"><code>bc34dee65a65</code></a></td></tr>
<tr><td><code>BPF_FUNC_send_signal()</code></td><td>5.3</td><td></td><td><a href="https://github.com/torvalds/linux/commit/8b401f9ed2441ad9e219953927a842d24ed051fc"><code>8b401f9ed244</code></a></td></tr>
<tr><td><code>BPF_FUNC_send_signal_thread()</code></td><td>5.5</td><td></td><td><a href="https://github.com/torvalds/linux/commit/8482941f09067da42f9c3362e15bfb3f3c19d610"><code>8482941f0906</code></a></td></tr>
<tr><td><code>BPF_FUNC_seq_printf()</code></td><td>5.7</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/492e639f0c222784e2e0f121966375f641c61b15"><code>492e639f0c22</code></a></td></tr>
<tr><td><code>BPF_FUNC_seq_printf_btf()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/eb411377aed9e27835e77ee0710ee8f4649958f3"><code>eb411377aed9</code></a></td></tr>
<tr><td><code>BPF_FUNC_seq_write()</code></td><td>5.7</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/492e639f0c222784e2e0f121966375f641c61b15"><code>492e639f0c22</code></a></td></tr>
<tr><td><code>BPF_FUNC_set_hash()</code></td><td>4.13</td><td></td><td><a href="https://github.com/torvalds/linux/commit/ded092cd73c2c56a394b936f86897f29b2e131c0"><code>ded092cd73c2</code></a></td></tr>
<tr><td><code>BPF_FUNC_set_hash_invalid()</code></td><td>4.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/7a4b28c6cc9ffac50f791b99cc7e46106436e5d8"><code>7a4b28c6cc9f</code></a></td></tr>
<tr><td><code>BPF_FUNC_set_retval()</code></td><td>5.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b44123b4a3dcad4664d3a0f72c011ffd4c9c4d93"><code>b44123b4a3dc</code></a></td></tr>
<tr><td><code>BPF_FUNC_setsockopt()</code></td><td>4.13</td><td></td><td><a href="https://github.com/torvalds/linux/commit/8c4b4c7e9ff0447995750d9329949fa082520269"><code>8c4b4c7e9ff0</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_ancestor_cgroup_id()</code></td><td>5.7</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f307fa2cb4c935f7f1ff0aeb880c7b44fb9a642b"><code>f307fa2cb4c9</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_assign()</code></td><td>5.6</td><td></td><td><a href="https://github.com/torvalds/linux/commit/cf7fbe660f2dbd738ab58aea8e9b0ca6ad232449"><code>cf7fbe660f2d</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_cgroup_id()</code></td><td>5.7</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f307fa2cb4c935f7f1ff0aeb880c7b44fb9a642b"><code>f307fa2cb4c9</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_fullsock()</code></td><td>5.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/46f8bc92758c6259bcf945e9216098661c1587cd"><code>46f8bc92758c</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_lookup_tcp()</code></td><td>4.20</td><td></td><td><a href="https://github.com/torvalds/linux/commit/6acc9b432e6714d72d7d77ec7c27f6f8358d0c71"><code>6acc9b432e67</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_lookup_udp()</code></td><td>4.20</td><td></td><td><a href="https://github.com/torvalds/linux/commit/6acc9b432e6714d72d7d77ec7c27f6f8358d0c71"><code>6acc9b432e67</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_redirect_hash()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/81110384441a59cff47430f20f049e69b98c17f4"><code>81110384441a</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_redirect_map()</code></td><td>4.14</td><td></td><td><a href="https://github.com/torvalds/linux/commit/174a79ff9515f400b9a6115643dafd62a635b7e6"><code>174a79ff9515</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_release()</code></td><td>4.20</td><td></td><td><a href="https://github.com/torvalds/linux/commit/6acc9b432e6714d72d7d77ec7c27f6f8358d0c71"><code>6acc9b432e67</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_select_reuseport()</code></td><td>4.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/2dbb9b9e6df67d444fbe425c7f6014858d337adf"><code>2dbb9b9e6df6</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_storage_delete()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/6ac99e8f23d4b10258406ca0dd7bffca5f31da9d"><code>6ac99e8f23d4</code></a></td></tr>
<tr><td><code>BPF_FUNC_sk_storage_get()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/6ac99e8f23d4b10258406ca0dd7bffca5f31da9d"><code>6ac99e8f23d4</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_adjust_room()</code></td><td>4.13</td><td></td><td><a href="https://github.com/torvalds/linux/commit/2be7e212d5419a400d051c84ca9fdd083e5aacac"><code>2be7e212d541</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_ancestor_cgroup_id()</code></td><td>4.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/7723628101aaeb1d723786747529b4ea65c5b5c5"><code>7723628101aa</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_change_head()</code></td><td>4.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/3a0af8fd61f90920f6fa04e4f1e9a6a73c1b4fd2"><code>3a0af8fd61f9</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_change_proto()</code></td><td>4.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/6578171a7ff0c31dc73258f93da7407510abf085"><code>6578171a7ff0</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_change_tail()</code></td><td>4.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/5293efe62df81908f2e90c9820c7edcc8e61f5e9"><code>5293efe62df8</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_change_type()</code></td><td>4.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d2485c4242a826fdf493fd3a27b8b792965b9b9e"><code>d2485c4242a8</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_cgroup_classid()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b426ce83baa7dff947fb354118d3133f2953aac8"><code>b426ce83baa7</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_cgroup_id()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/cb20b08ead401fd17627a36f035c0bf5bfee5567"><code>cb20b08ead40</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_ecn_set_ce()</code></td><td>5.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/f7c917ba11a67632a8452ea99fe132f626a7a2cc"><code>f7c917ba11a6</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_get_tunnel_key()</code></td><td>4.3</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d3aa45ce6b94c65b83971257317867db13e5f492"><code>d3aa45ce6b94</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_get_tunnel_opt()</code></td><td>4.6</td><td></td><td><a href="https://github.com/torvalds/linux/commit/14ca0751c96f8d3d0f52e8ed3b3236f8b34d3460"><code>14ca0751c96f</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_get_xfrm_state()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/12bed760a78da6e12ac8252fec64d019a9eac523"><code>12bed760a78d</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_load_bytes()</code></td><td>4.5</td><td></td><td><a href="https://github.com/torvalds/linux/commit/05c74e5e53f6cb07502c3e6a820f33e2777b6605"><code>05c74e5e53f6</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_load_bytes_relative()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4e1ec56cdc59746943b2acfab3c171b930187bbe"><code>4e1ec56cdc59</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_output()</code></td><td>5.5</td><td></td><td><a href="https://github.com/torvalds/linux/commit/a7658e1a4164ce2b9eb4a11aadbba38586e93bd6"><code>a7658e1a4164</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_pull_data()</code></td><td>4.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/36bbef52c7eb646ed6247055a2acd3851e317857"><code>36bbef52c7eb</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_set_tstamp()</code></td><td>5.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/9bb984f28d5bcb917d35d930fcfb89f90f9449fd"><code>9bb984f28d5b</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_set_tunnel_key()</code></td><td>4.3</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d3aa45ce6b94c65b83971257317867db13e5f492"><code>d3aa45ce6b94</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_set_tunnel_opt()</code></td><td>4.6</td><td></td><td><a href="https://github.com/torvalds/linux/commit/14ca0751c96f8d3d0f52e8ed3b3236f8b34d3460"><code>14ca0751c96f</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_store_bytes()</code></td><td>4.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/91bc4822c3d61b9bb7ef66d3b77948a4f9177954"><code>91bc4822c3d6</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_under_cgroup()</code></td><td>4.8</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4a482f34afcc162d8456f449b137ec2a95be60d8"><code>4a482f34afcc</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_vlan_pop()</code></td><td>4.3</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4e10df9a60d96ced321dd2af71da558c6b750078"><code>4e10df9a60d9</code></a></td></tr>
<tr><td><code>BPF_FUNC_skb_vlan_push()</code></td><td>4.3</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4e10df9a60d96ced321dd2af71da558c6b750078"><code>4e10df9a60d9</code></a></td></tr>
<tr><td><code>BPF_FUNC_skc_lookup_tcp()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/edbf8c01de5a104a71ed6df2bf6421ceb2836a8e"><code>edbf8c01de5a</code></a></td></tr>
<tr><td><code>BPF_FUNC_skc_to_mctcp_sock()</code></td><td>5.19</td><td></td><td><a href="https://github.com/torvalds/linux/commit/3bc253c2e652cf5f12cd8c00d80d8ec55d67d1a7"><code>3bc253c2e652</code></a></td></tr>
<tr><td><code>BPF_FUNC_skc_to_tcp_sock()</code></td><td>5.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/478cfbdf5f13dfe09cfd0b1cbac821f5e27f6108"><code>478cfbdf5f13</code></a></td></tr>
<tr><td><code>BPF_FUNC_skc_to_tcp_request_sock()</code></td><td>5.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/478cfbdf5f13dfe09cfd0b1cbac821f5e27f6108"><code>478cfbdf5f13</code></a></td></tr>
<tr><td><code>BPF_FUNC_skc_to_tcp_timewait_sock()</code></td><td>5.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/478cfbdf5f13dfe09cfd0b1cbac821f5e27f6108"><code>478cfbdf5f13</code></a></td></tr>
<tr><td><code>BPF_FUNC_skc_to_tcp6_sock()</code></td><td>5.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/af7ec13833619e17f03aa73a785a2f871da6d66b"><code>af7ec1383361</code></a></td></tr>
<tr><td><code>BPF_FUNC_skc_to_udp6_sock()</code></td><td>5.9</td><td></td><td><a href="https://github.com/torvalds/linux/commit/0d4fad3e57df2bf61e8ffc8d12a34b1caf9b8835"><code>0d4fad3e57df</code></a></td></tr>
<tr><td><code>BPF_FUNC_skc_to_unix_sock()</code></td><td>5.16</td><td></td><td><a href="https://github.com/torvalds/linux/commit/9eeb3aa33ae005526f672b394c1791578463513f"><code>9eeb3aa33ae0</code></a></td></tr>
<tr><td><code>BPF_FUNC_snprintf()</code></td><td>5.13</td><td></td><td><a href="https://github.com/torvalds/linux/commit/7b15523a989b63927c2bb08e9b5b0bbc10b58bef"><code>7b15523a989b</code></a></td></tr>
<tr><td><code>BPF_FUNC_snprintf_btf()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/c4d0bfb45068d853a478b9067a95969b1886a30f"><code>c4d0bfb45068</code></a></td></tr>
<tr><td><code>BPF_FUNC_sock_from_file()</code></td><td>5.11</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4f19cab76136e800a3f04d8c9aa4d8e770e3d3d8"><code>4f19cab76136</code></a></td></tr>
<tr><td><code>BPF_FUNC_sock_hash_update()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/81110384441a59cff47430f20f049e69b98c17f4"><code>81110384441a</code></a></td></tr>
<tr><td><code>BPF_FUNC_sock_map_update()</code></td><td>4.14</td><td></td><td><a href="https://github.com/torvalds/linux/commit/174a79ff9515f400b9a6115643dafd62a635b7e6"><code>174a79ff9515</code></a></td></tr>
<tr><td><code>BPF_FUNC_spin_lock()</code></td><td>5.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d83525ca62cf8ebe3271d14c36fb900c294274a2"><code>d83525ca62cf</code></a></td></tr>
<tr><td><code>BPF_FUNC_spin_unlock()</code></td><td>5.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d83525ca62cf8ebe3271d14c36fb900c294274a2"><code>d83525ca62cf</code></a></td></tr>
<tr><td><code>BPF_FUNC_store_hdr_opt()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/0813a841566f0962a5551be7749b43c45f0022a0"><code>0813a841566f</code></a></td></tr>
<tr><td><code>BPF_FUNC_strncmp()</code></td><td>5.17</td><td></td><td><a href="https://github.com/torvalds/linux/commit/c5fb19937455095573a19ddcbff32e993ed10e35"><code>c5fb19937455</code></a></td></tr>
<tr><td><code>BPF_FUNC_strtol()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d7a4cb9b6705a89937d12c8158a35a3145dc967a"><code>d7a4cb9b6705</code></a></td></tr>
<tr><td><code>BPF_FUNC_strtoul()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/d7a4cb9b6705a89937d12c8158a35a3145dc967a"><code>d7a4cb9b6705</code></a></td></tr>
<tr><td><code>BPF_FUNC_sys_bpf()</code></td><td>5.14</td><td></td><td><a href="https://github.com/torvalds/linux/commit/79a7f8bdb159d9914b58740f3d31d602a6e4aca8"><code>79a7f8bdb159</code></a></td></tr>
<tr><td><code>BPF_FUNC_sys_close()</code></td><td>5.14</td><td></td><td><a href="https://github.com/torvalds/linux/commit/3abea089246f76c1517b054ddb5946f3f1dbd2c0"><code>3abea089246f</code></a></td></tr>
<tr><td><code>BPF_FUNC_sysctl_get_current_value()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/1d11b3016cec4ed9770b98e82a61708c8f4926e7"><code>1d11b3016cec</code></a></td></tr>
<tr><td><code>BPF_FUNC_sysctl_get_name()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/808649fb787d918a48a360a668ee4ee9023f0c11"><code>808649fb787d</code></a></td></tr>
<tr><td><code>BPF_FUNC_sysctl_get_new_value()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4e63acdff864654cee0ac5aaeda3913798ee78f6"><code>4e63acdff864</code></a></td></tr>
<tr><td><code>BPF_FUNC_sysctl_set_new_value()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4e63acdff864654cee0ac5aaeda3913798ee78f6"><code>4e63acdff864</code></a></td></tr>
<tr><td><code>BPF_FUNC_tail_call()</code></td><td>4.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/04fd61ab36ec065e194ab5e74ae34a5240d992bb"><code>04fd61ab36ec</code></a></td></tr>
<tr><td><code>BPF_FUNC_task_pt_regs()</code></td><td>5.15</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/dd6e10fbd9fb86a571d925602c8a24bb4d09a2a7"><code>dd6e10fbd9f</code></a></td></tr>
<tr><td><code>BPF_FUNC_task_storage_delete()</code></td><td>5.11</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4cf1bc1f10452065a29d576fc5693fc4fab5b919"><code>4cf1bc1f1045</code></a></td></tr>
<tr><td><code>BPF_FUNC_task_storage_get()</code></td><td>5.11</td><td></td><td><a href="https://github.com/torvalds/linux/commit/4cf1bc1f10452065a29d576fc5693fc4fab5b919"><code>4cf1bc1f1045</code></a></td></tr>
<tr><td><code>BPF_FUNC_tcp_check_syncookie()</code></td><td>5.2</td><td></td><td><a href="https://github.com/torvalds/linux/commit/399040847084a69f345e0a52fd62f04654e0fce3"><code>399040847084</code></a></td></tr>
<tr><td><code>BPF_FUNC_tcp_gen_syncookie()</code></td><td>5.3</td><td></td><td><a href="https://github.com/torvalds/linux/commit/70d66244317e958092e9c971b08dd5b7fd29d9cb#diff-05da4bf36c7fbcd176254e1615d98b28"><code>70d66244317e</code></a></td></tr>
<tr><td><code>BPF_FUNC_tcp_raw_check_syncookie_ipv4()</code></td><td>6.0</td><td></td><td><a href="https://github.com/torvalds/linux/commit/33bf9885040c399cf6a95bd33216644126728e14"><code>33bf9885040c</code></a></td></tr>
<tr><td><code>BPF_FUNC_tcp_raw_check_syncookie_ipv6()</code></td><td>6.0</td><td></td><td><a href="https://github.com/torvalds/linux/commit/33bf9885040c399cf6a95bd33216644126728e14"><code>33bf9885040c</code></a></td></tr>
<tr><td><code>BPF_FUNC_tcp_raw_gen_syncookie_ipv4()</code></td><td>6.0</td><td></td><td><a href="https://github.com/torvalds/linux/commit/33bf9885040c399cf6a95bd33216644126728e14"><code>33bf9885040c</code></a></td></tr>
<tr><td><code>BPF_FUNC_tcp_raw_gen_syncookie_ipv6()</code></td><td>6.0</td><td></td><td><a href="https://github.com/torvalds/linux/commit/33bf9885040c399cf6a95bd33216644126728e14"><code>33bf9885040c</code></a></td></tr>
<tr><td><code>BPF_FUNC_tcp_send_ack()</code></td><td>5.5</td><td></td><td><a href="https://github.com/torvalds/linux/commit/206057fe020ac5c037d5e2dd6562a9bd216ec765"><code>206057fe020a</code></a></td></tr>
<tr><td><code>BPF_FUNC_tcp_sock()</code></td><td>5.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/655a51e536c09d15ffa3603b1b6fce2b45b85a1f"><code>655a51e536c0</code></a></td></tr>
<tr><td><code>BPF_FUNC_this_cpu_ptr()</code></td><td>5.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/63d9b80dcf2c67bc5ade61cbbaa09d7af21f43f1"><code>63d9b80dcf2c</code></a></td></tr>
<tr><td><code>BPF_FUNC_timer_init()</code></td><td>5.15</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b00628b1c7d595ae5b544e059c27b1f5828314b4"><code>b00628b1c7d5</code></a></td></tr>
<tr><td><code>BPF_FUNC_timer_set_callback()</code></td><td>5.15</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b00628b1c7d595ae5b544e059c27b1f5828314b4"><code>b00628b1c7d5</code></a></td></tr>
<tr><td><code>BPF_FUNC_timer_start()</code></td><td>5.15</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b00628b1c7d595ae5b544e059c27b1f5828314b4"><code>b00628b1c7d5</code></a></td></tr>
<tr><td><code>BPF_FUNC_timer_cancel()</code></td><td>5.15</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b00628b1c7d595ae5b544e059c27b1f5828314b4"><code>b00628b1c7d5</code></a></td></tr>
<tr><td><code>BPF_FUNC_trace_printk()</code></td><td>4.1</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/9c959c863f8217a2ff3d7c296e8223654d240569"><code>9c959c863f82</code></a></td></tr>
<tr><td><code>BPF_FUNC_trace_vprintk()</code></td><td>5.16</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/10aceb629e198429c849d5e995c3bb1ba7a9aaa3"><code>10aceb629e19</code></a></td></tr>
<tr><td><code>BPF_FUNC_user_ringbuf_drain()</code></td><td>6.1</td><td></td><td><a href="https://github.com/torvalds/linux/commit/20571567384428dfc9fe5cf9f2e942e1df13c2dd"><code>205715673844</code></a></td></tr>
<tr><td><code>BPF_FUNC_xdp_adjust_head()</code></td><td>4.10</td><td></td><td><a href="https://github.com/torvalds/linux/commit/17bedab2723145d17b14084430743549e6943d03"><code>17bedab27231</code></a></td></tr>
<tr><td><code>BPF_FUNC_xdp_adjust_meta()</code></td><td>4.15</td><td></td><td><a href="https://github.com/torvalds/linux/commit/de8f3a83b0a0fddb2cf56e7a718127e9619ea3da"><code>de8f3a83b0a0</code></a></td></tr>
<tr><td><code>BPF_FUNC_xdp_adjust_tail()</code></td><td>4.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b32cc5b9a346319c171e3ad905e0cddda032b5eb"><code>b32cc5b9a346</code></a></td></tr>
<tr><td><code>BPF_FUNC_xdp_get_buff_len()</code></td><td>5.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/0165cc817075cf701e4289838f1d925ff1911b3e"><code>0165cc817075</code></a></td></tr>
<tr><td><code>BPF_FUNC_xdp_load_bytes()</code></td><td>5.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/3f364222d032eea6b245780e845ad213dab28cdd"><code>3f364222d032</code></a></td></tr>
<tr><td><code>BPF_FUNC_xdp_store_bytes()</code></td><td>5.18</td><td></td><td><a href="https://github.com/torvalds/linux/commit/3f364222d032eea6b245780e845ad213dab28cdd"><code>3f364222d032</code></a></td></tr>
<tr><td><code>BPF_FUNC_xdp_output()</code></td><td>5.6</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/d831ee84bfc9173eecf30dbbc2553ae81b996c60"><code>d831ee84bfc9</code></a></td></tr>
<tr><td><code>BPF_FUNC_override_return()</code></td><td>4.16</td><td>GPL</td><td><a href="https://github.com/torvalds/linux/commit/9802d86585db91655c7d1929a4f6bbe0952ea88e"><code>9802d86585db</code></a></td></tr>
<tr><td><code>BPF_FUNC_sock_ops_cb_flags_set()</code></td><td>4.16</td><td></td><td><a href="https://github.com/torvalds/linux/commit/b13d880721729384757f235166068c315326f4a1"><code>b13d88072172</code></a></td></tr>
</tbody></table>
</div>
<p>Note: GPL-only BPF helpers require a GPL-compatible license. The current licenses considered GPL-compatible by the kernel are:</p>
<ul>
<li>GPL</li>
<li>GPL v2</li>
<li>GPL and additional rights</li>
<li>Dual BSD/GPL</li>
<li>Dual MIT/GPL</li>
<li>Dual MPL/GPL</li>
</ul>
<p>Check the list of GPL-compatible licenses in your <a href="https://github.com/torvalds/linux/blob/master/include/linux/license.h">kernel source code</a>.</p>
<h2 id="program-types-1"><a class="header" href="#program-types-1">Program Types</a></h2>
<p>The list of program types and supported helper functions can be retrieved with:</p>
<pre><code class="language-sh">git grep -W 'func_proto(enum bpf_func_id func_id' kernel/ net/ drivers/
</code></pre>
<div class="table-wrapper"><table><thead><tr><th>Program Type</th><th>Helper Functions</th></tr></thead><tbody>
<tr><td><code>BPF_PROG_TYPE_SOCKET_FILTER</code></td><td><code>BPF_FUNC_skb_load_bytes()</code> <br> <code>BPF_FUNC_skb_load_bytes_relative()</code> <br> <code>BPF_FUNC_get_socket_cookie()</code> <br> <code>BPF_FUNC_get_socket_uid()</code> <br> <code>BPF_FUNC_perf_event_output()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_KPROBE</code></td><td><code>BPF_FUNC_perf_event_output()</code> <br> <code>BPF_FUNC_get_stackid()</code> <br> <code>BPF_FUNC_get_stack()</code> <br> <code>BPF_FUNC_perf_event_read_value()</code> <br> <code>BPF_FUNC_override_return()</code> <br> <code>Tracing functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_SCHED_CLS</code> <br> <code>BPF_PROG_TYPE_SCHED_ACT</code></td><td><code>BPF_FUNC_skb_store_bytes()</code> <br> <code>BPF_FUNC_skb_load_bytes()</code> <br> <code>BPF_FUNC_skb_load_bytes_relative()</code> <br> <code>BPF_FUNC_skb_pull_data()</code> <br> <code>BPF_FUNC_csum_diff()</code> <br> <code>BPF_FUNC_csum_update()</code> <br> <code>BPF_FUNC_l3_csum_replace()</code> <br> <code>BPF_FUNC_l4_csum_replace()</code> <br> <code>BPF_FUNC_clone_redirect()</code> <br> <code>BPF_FUNC_get_cgroup_classid()</code> <br> <code>BPF_FUNC_skb_vlan_push()</code> <br> <code>BPF_FUNC_skb_vlan_pop()</code> <br> <code>BPF_FUNC_skb_change_proto()</code> <br> <code>BPF_FUNC_skb_change_type()</code> <br> <code>BPF_FUNC_skb_adjust_room()</code> <br> <code>BPF_FUNC_skb_change_tail()</code> <br> <code>BPF_FUNC_skb_get_tunnel_key()</code> <br> <code>BPF_FUNC_skb_set_tunnel_key()</code> <br> <code>BPF_FUNC_skb_get_tunnel_opt()</code> <br> <code>BPF_FUNC_skb_set_tunnel_opt()</code> <br> <code>BPF_FUNC_redirect()</code> <br> <code>BPF_FUNC_get_route_realm()</code> <br> <code>BPF_FUNC_get_hash_recalc()</code> <br> <code>BPF_FUNC_set_hash_invalid()</code> <br> <code>BPF_FUNC_set_hash()</code> <br> <code>BPF_FUNC_perf_event_output()</code> <br> <code>BPF_FUNC_get_smp_processor_id()</code> <br> <code>BPF_FUNC_skb_under_cgroup()</code> <br> <code>BPF_FUNC_get_socket_cookie()</code> <br> <code>BPF_FUNC_get_socket_uid()</code> <br> <code>BPF_FUNC_fib_lookup()</code> <br> <code>BPF_FUNC_skb_get_xfrm_state()</code> <br> <code>BPF_FUNC_skb_cgroup_id()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_TRACEPOINT</code></td><td><code>BPF_FUNC_perf_event_output()</code> <br> <code>BPF_FUNC_get_stackid()</code> <br> <code>BPF_FUNC_get_stack()</code> <br> <code>BPF_FUNC_d_path()</code> <br> <code>Tracing functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_XDP</code></td><td><code>BPF_FUNC_perf_event_output()</code> <br> <code>BPF_FUNC_get_smp_processor_id()</code> <br> <code>BPF_FUNC_csum_diff()</code> <br> <code>BPF_FUNC_xdp_adjust_head()</code> <br> <code>BPF_FUNC_xdp_adjust_meta()</code> <br> <code>BPF_FUNC_redirect()</code> <br> <code>BPF_FUNC_redirect_map()</code> <br> <code>BPF_FUNC_xdp_adjust_tail()</code> <br> <code>BPF_FUNC_fib_lookup()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_PERF_EVENT</code></td><td><code>BPF_FUNC_perf_event_output()</code> <br> <code>BPF_FUNC_get_stackid()</code> <br> <code>BPF_FUNC_get_stack()</code> <br> <code>BPF_FUNC_perf_prog_read_value()</code> <br> <code>Tracing functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_CGROUP_SKB</code></td><td><code>BPF_FUNC_skb_load_bytes()</code> <br> <code>BPF_FUNC_skb_load_bytes_relative()</code> <br> <code>BPF_FUNC_get_socket_cookie()</code> <br> <code>BPF_FUNC_get_socket_uid()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_CGROUP_SOCK</code></td><td><code>BPF_FUNC_get_current_uid_gid()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_LWT_IN</code></td><td><code>BPF_FUNC_lwt_push_encap()</code> <br> <code>LWT functions</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_LWT_OUT</code></td><td><code>LWT functions</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_LWT_XMIT</code></td><td><code>BPF_FUNC_skb_get_tunnel_key()</code> <br> <code>BPF_FUNC_skb_set_tunnel_key()</code> <br> <code>BPF_FUNC_skb_get_tunnel_opt()</code> <br> <code>BPF_FUNC_skb_set_tunnel_opt()</code> <br> <code>BPF_FUNC_redirect()</code> <br> <code>BPF_FUNC_clone_redirect()</code> <br> <code>BPF_FUNC_skb_change_tail()</code> <br> <code>BPF_FUNC_skb_change_head()</code> <br> <code>BPF_FUNC_skb_store_bytes()</code> <br> <code>BPF_FUNC_csum_update()</code> <br> <code>BPF_FUNC_l3_csum_replace()</code> <br> <code>BPF_FUNC_l4_csum_replace()</code> <br> <code>BPF_FUNC_set_hash_invalid()</code> <br> <code>LWT functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_SOCK_OPS</code></td><td><code>BPF_FUNC_setsockopt()</code> <br> <code>BPF_FUNC_getsockopt()</code> <br> <code>BPF_FUNC_sock_ops_cb_flags_set()</code> <br> <code>BPF_FUNC_sock_map_update()</code> <br> <code>BPF_FUNC_sock_hash_update()</code> <br> <code>BPF_FUNC_get_socket_cookie()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_SK_SKB</code></td><td><code>BPF_FUNC_skb_store_bytes()</code> <br> <code>BPF_FUNC_skb_load_bytes()</code> <br> <code>BPF_FUNC_skb_pull_data()</code> <br> <code>BPF_FUNC_skb_change_tail()</code> <br> <code>BPF_FUNC_skb_change_head()</code> <br> <code>BPF_FUNC_get_socket_cookie()</code> <br> <code>BPF_FUNC_get_socket_uid()</code> <br> <code>BPF_FUNC_sk_redirect_map()</code> <br> <code>BPF_FUNC_sk_redirect_hash()</code> <br> <code>BPF_FUNC_sk_lookup_tcp()</code> <br> <code>BPF_FUNC_sk_lookup_udp()</code> <br> <code>BPF_FUNC_sk_release()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_CGROUP_DEVICE</code></td><td><code>BPF_FUNC_map_lookup_elem()</code> <br> <code>BPF_FUNC_map_update_elem()</code> <br> <code>BPF_FUNC_map_delete_elem()</code> <br> <code>BPF_FUNC_get_current_uid_gid()</code> <br> <code>BPF_FUNC_trace_printk()</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_SK_MSG</code></td><td><code>BPF_FUNC_msg_redirect_map()</code> <br> <code>BPF_FUNC_msg_redirect_hash()</code> <br> <code>BPF_FUNC_msg_apply_bytes()</code> <br> <code>BPF_FUNC_msg_cork_bytes()</code> <br> <code>BPF_FUNC_msg_pull_data()</code> <br> <code>BPF_FUNC_msg_push_data()</code> <br> <code>BPF_FUNC_msg_pop_data()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_RAW_TRACEPOINT</code></td><td><code>BPF_FUNC_perf_event_output()</code> <br> <code>BPF_FUNC_get_stackid()</code> <br> <code>BPF_FUNC_get_stack()</code> <br> <code>BPF_FUNC_skb_output()</code> <br> <code>Tracing functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_CGROUP_SOCK_ADDR</code></td><td><code>BPF_FUNC_get_current_uid_gid()</code> <br> <code>BPF_FUNC_bind()</code> <br> <code>BPF_FUNC_get_socket_cookie()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_LWT_SEG6LOCAL</code></td><td><code>BPF_FUNC_lwt_seg6_store_bytes()</code> <br> <code>BPF_FUNC_lwt_seg6_action()</code> <br> <code>BPF_FUNC_lwt_seg6_adjust_srh()</code> <br> <code>LWT functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_LIRC_MODE2</code></td><td><code>BPF_FUNC_rc_repeat()</code> <br> <code>BPF_FUNC_rc_keydown()</code> <br> <code>BPF_FUNC_rc_pointer_rel()</code> <br> <code>BPF_FUNC_map_lookup_elem()</code> <br> <code>BPF_FUNC_map_update_elem()</code> <br> <code>BPF_FUNC_map_delete_elem()</code> <br> <code>BPF_FUNC_ktime_get_ns()</code> <br> <code>BPF_FUNC_tail_call()</code> <br> <code>BPF_FUNC_get_prandom_u32()</code> <br> <code>BPF_FUNC_trace_printk()</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_SK_REUSEPORT</code></td><td><code>BPF_FUNC_sk_select_reuseport()</code> <br> <code>BPF_FUNC_skb_load_bytes()</code> <br> <code>BPF_FUNC_load_bytes_relative()</code> <br> <code>Base functions</code></td></tr>
<tr><td><code>BPF_PROG_TYPE_FLOW_DISSECTOR</code></td><td><code>BPF_FUNC_skb_load_bytes()</code> <br> <code>Base functions</code></td></tr>
</tbody></table>
</div><div class="table-wrapper"><table><thead><tr><th>Function Group</th><th>Functions</th></tr></thead><tbody>
<tr><td><code>Base functions</code></td><td><code>BPF_FUNC_map_lookup_elem()</code> <br> <code>BPF_FUNC_map_update_elem()</code> <br> <code>BPF_FUNC_map_delete_elem()</code> <br> <code>BPF_FUNC_map_peek_elem()</code> <br> <code>BPF_FUNC_map_pop_elem()</code> <br> <code>BPF_FUNC_map_push_elem()</code> <br> <code>BPF_FUNC_get_prandom_u32()</code> <br> <code>BPF_FUNC_get_smp_processor_id()</code> <br> <code>BPF_FUNC_get_numa_node_id()</code> <br> <code>BPF_FUNC_tail_call()</code> <br> <code>BPF_FUNC_ktime_get_boot_ns()</code> <br> <code>BPF_FUNC_ktime_get_ns()</code> <br> <code>BPF_FUNC_trace_printk()</code> <br> <code>BPF_FUNC_spin_lock()</code> <br> <code>BPF_FUNC_spin_unlock()</code></td></tr>
<tr><td><code>Tracing functions</code></td><td><code>BPF_FUNC_map_lookup_elem()</code> <br> <code>BPF_FUNC_map_update_elem()</code> <br> <code>BPF_FUNC_map_delete_elem()</code> <br> <code>BPF_FUNC_probe_read()</code> <br> <code>BPF_FUNC_ktime_get_boot_ns()</code> <br> <code>BPF_FUNC_ktime_get_ns()</code> <br> <code>BPF_FUNC_tail_call()</code> <br> <code>BPF_FUNC_get_current_pid_tgid()</code> <br> <code>BPF_FUNC_get_current_task()</code> <br> <code>BPF_FUNC_get_current_uid_gid()</code> <br> <code>BPF_FUNC_get_current_comm()</code> <br> <code>BPF_FUNC_trace_printk()</code> <br> <code>BPF_FUNC_get_smp_processor_id()</code> <br> <code>BPF_FUNC_get_numa_node_id()</code> <br> <code>BPF_FUNC_perf_event_read()</code> <br> <code>BPF_FUNC_probe_write_user()</code> <br> <code>BPF_FUNC_current_task_under_cgroup()</code> <br> <code>BPF_FUNC_get_prandom_u32()</code> <br> <code>BPF_FUNC_probe_read_str()</code> <br> <code>BPF_FUNC_get_current_cgroup_id()</code> <br> <code>BPF_FUNC_send_signal()</code> <br> <code>BPF_FUNC_probe_read_kernel()</code> <br> <code>BPF_FUNC_probe_read_kernel_str()</code> <br> <code>BPF_FUNC_probe_read_user()</code> <br> <code>BPF_FUNC_probe_read_user_str()</code> <br> <code>BPF_FUNC_send_signal_thread()</code> <br> <code>BPF_FUNC_get_ns_current_pid_tgid()</code> <br> <code>BPF_FUNC_xdp_output()</code> <br> <code>BPF_FUNC_get_task_stack()</code></td></tr>
<tr><td><code>LWT functions</code></td><td><code>BPF_FUNC_skb_load_bytes()</code> <br> <code>BPF_FUNC_skb_pull_data()</code> <br> <code>BPF_FUNC_csum_diff()</code> <br> <code>BPF_FUNC_get_cgroup_classid()</code> <br> <code>BPF_FUNC_get_route_realm()</code> <br> <code>BPF_FUNC_get_hash_recalc()</code> <br> <code>BPF_FUNC_perf_event_output()</code> <br> <code>BPF_FUNC_get_smp_processor_id()</code> <br> <code>BPF_FUNC_skb_under_cgroup()</code></td></tr>
</tbody></table>
</div><div style="break-before: page; page-break-before: always;"></div><h1 id="kernel-configuration-for-bpf-features"><a class="header" href="#kernel-configuration-for-bpf-features">Kernel Configuration for BPF Features</a></h1>
<h2 id="bpf-related-kernel-configurations"><a class="header" href="#bpf-related-kernel-configurations">BPF Related Kernel Configurations</a></h2>
<div class="table-wrapper"><table><thead><tr><th style="text-align: left">Functionalities</th><th style="text-align: left">Kernel Configuration</th><th style="text-align: left">Description</th></tr></thead><tbody>
<tr><td style="text-align: left"><strong>Basic</strong></td><td style="text-align: left">CONFIG_BPF_SYSCALL</td><td style="text-align: left">Enable the bpf() system call</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_BPF_JIT</td><td style="text-align: left">BPF programs are normally handled by a BPF interpreter. This option allows the kernel to generate native code when a program is loaded into the kernel. This will significantly speed-up processing of BPF programs</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_HAVE_BPF_JIT</td><td style="text-align: left">Enable BPF Just In Time compiler</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_HAVE_EBPF_JIT</td><td style="text-align: left">Extended BPF JIT (eBPF)</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_HAVE_CBPF_JIT</td><td style="text-align: left">Classic BPF JIT (cBPF)</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_MODULES</td><td style="text-align: left">Enable to build loadable kernel modules</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_BPF</td><td style="text-align: left">BPF VM interpreter</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_BPF_EVENTS</td><td style="text-align: left">Allow the user to attach BPF programs to kprobe, uprobe, and tracepoint events</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_PERF_EVENTS</td><td style="text-align: left">Kernel performance events and counters</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_HAVE_PERF_EVENTS</td><td style="text-align: left">Enable perf events</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_PROFILING</td><td style="text-align: left">Enable the extended profiling support mechanisms used by profilers</td></tr>
<tr><td style="text-align: left"><strong>BTF</strong></td><td style="text-align: left">CONFIG_DEBUG_INFO_BTF</td><td style="text-align: left">Generate deduplicated BTF type information from DWARF debug info</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_PAHOLE_HAS_SPLIT_BTF</td><td style="text-align: left">Generate BTF for each selected kernel module</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_DEBUG_INFO_BTF_MODULES</td><td style="text-align: left">Generate compact split BTF type information for kernel modules</td></tr>
<tr><td style="text-align: left"><strong>Security</strong></td><td style="text-align: left">CONFIG_BPF_JIT_ALWAYS_ON</td><td style="text-align: left">Enable BPF JIT and removes BPF interpreter to avoid speculative execution</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_BPF_UNPRIV_DEFAULT_OFF</td><td style="text-align: left">Disable unprivileged BPF by default by setting</td></tr>
<tr><td style="text-align: left"><strong>Cgroup</strong></td><td style="text-align: left">CONFIG_CGROUP_BPF</td><td style="text-align: left">Support for BPF programs attached to cgroups</td></tr>
<tr><td style="text-align: left"><strong>Network</strong></td><td style="text-align: left">CONFIG_BPFILTER</td><td style="text-align: left">BPF based packet filtering framework (BPFILTER)</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_BPFILTER_UMH</td><td style="text-align: left">This builds bpfilter kernel module with embedded user mode helper</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_NET_CLS_BPF</td><td style="text-align: left">BPF-based classifier - to classify packets based on programmable BPF (JIT'ed) filters as an alternative to ematches</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_NET_ACT_BPF</td><td style="text-align: left">Execute BPF code on packets. The BPF code will decide if the packet should be dropped or not</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_BPF_STREAM_PARSER</td><td style="text-align: left">Enable this to allow a TCP stream parser to be used with BPF_MAP_TYPE_SOCKMAP</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_LWTUNNEL_BPF</td><td style="text-align: left">Allow to run BPF programs as a nexthop action following a route lookup for incoming and outgoing packets</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_NETFILTER_XT_MATCH_BPF</td><td style="text-align: left">BPF matching applies a linux socket filter to each packet and accepts those for which the filter returns non-zero</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_IPV6_SEG6_BPF</td><td style="text-align: left">To support BPF seg6local hook. bpf: Add IPv6 Segment Routing helpersy. <a href="https://github.com/torvalds/linux/commit/fe94cc290f535709d3c5ebd1e472dfd0aec7ee7">Reference</a></td></tr>
<tr><td style="text-align: left"><strong>kprobes</strong></td><td style="text-align: left">CONFIG_KPROBE_EVENTS</td><td style="text-align: left">This allows the user to add tracing events (similar to tracepoints) on the fly via the ftrace interface</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_KPROBES</td><td style="text-align: left">Enable kprobes-based dynamic events</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_HAVE_KPROBES</td><td style="text-align: left">Check if krpobes enabled</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_HAVE_REGS_AND_STACK_ACCESS_API</td><td style="text-align: left">This symbol should be selected by an architecture if it supports the API needed to access registers and stack entries from pt_regs. For example the kprobes-based event tracer needs this API.</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_KPROBES_ON_FTRACE</td><td style="text-align: left">Have kprobes on function tracer if arch supports full passing of pt_regs to function tracing</td></tr>
<tr><td style="text-align: left"><strong>kprobe multi</strong></td><td style="text-align: left">CONFIG_FPROBE</td><td style="text-align: left">Enable fprobe to attach the probe on multiple functions at once</td></tr>
<tr><td style="text-align: left"><strong>kprobe override</strong></td><td style="text-align: left">CONFIG_BPF_KPROBE_OVERRIDE</td><td style="text-align: left">Enable BPF programs to override a kprobed function</td></tr>
<tr><td style="text-align: left"><strong>uprobes</strong></td><td style="text-align: left">CONFIG_UPROBE_EVENTS</td><td style="text-align: left">Enable uprobes-based dynamic events</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_ARCH_SUPPORTS_UPROBES</td><td style="text-align: left">Arch specific uprobes support</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_UPROBES</td><td style="text-align: left">Uprobes is the user-space counterpart to kprobes: they enable instrumentation applications (such as 'perf probe') to establish unintrusive probes in user-space binaries and libraries, by executing handler functions when the probes are hit by user-space applications.</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_MMU</td><td style="text-align: left">MMU-based virtualised addressing space support by paged memory management</td></tr>
<tr><td style="text-align: left"><strong>Tracepoints</strong></td><td style="text-align: left">CONFIG_TRACEPOINTS</td><td style="text-align: left">Enable inserting tracepoints in the kernel and connect to proble functions</td></tr>
<tr><td style="text-align: left"></td><td style="text-align: left">CONFIG_HAVE_SYSCALL_TRACEPOINTS</td><td style="text-align: left">Enable syscall enter/exit tracing</td></tr>
<tr><td style="text-align: left"><strong>Raw Tracepoints</strong></td><td style="text-align: left">Same as Tracepoints</td><td style="text-align: left"></td></tr>
<tr><td style="text-align: left"><strong>LSM</strong></td><td style="text-align: left">CONFIG_BPF_LSM</td><td style="text-align: left">Enable instrumentation of the security hooks with BPF programs for implementing dynamic MAC and Audit Policies</td></tr>
<tr><td style="text-align: left"><strong>LIRC</strong></td><td style="text-align: left">CONFIG_BPF_LIRC_MODE2</td><td style="text-align: left">Allow attaching BPF programs to a lirc device</td></tr>
</tbody></table>
</div><div style="break-before: page; page-break-before: always;"></div><h1 id="bcc-reference-guide"><a class="header" href="#bcc-reference-guide">bcc Reference Guide</a></h1>
<p>Intended for search (Ctrl-F) and reference. For tutorials, start with <a href="bcc-documents/tutorial.html">tutorial.md</a>.</p>
<p>This guide is incomplete. If something feels missing, check the bcc and kernel source. And if you confirm we're missing something, please send a pull request to fix it, and help out everyone.</p>
<h2 id="contents"><a class="header" href="#contents">Contents</a></h2>
<ul>
<li>
<p><a href="bcc-documents/reference_guide.html#bpf-c">BPF C</a></p>
<ul>
<li><a href="bcc-documents/reference_guide.html#events--arguments">Events &amp; Arguments</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-kprobes">1. kprobes</a></li>
<li><a href="bcc-documents/reference_guide.html#2-kretprobes">2. kretprobes</a></li>
<li><a href="bcc-documents/reference_guide.html#3-tracepoints">3. Tracepoints</a></li>
<li><a href="bcc-documents/reference_guide.html#4-uprobes">4. uprobes</a></li>
<li><a href="bcc-documents/reference_guide.html#5-uretprobes">5. uretprobes</a></li>
<li><a href="bcc-documents/reference_guide.html#6-usdt-probes">6. USDT probes</a></li>
<li><a href="bcc-documents/reference_guide.html#7-raw-tracepoints">7. Raw Tracepoints</a></li>
<li><a href="bcc-documents/reference_guide.html#8-system-call-tracepoints">8. system call tracepoints</a></li>
<li><a href="bcc-documents/reference_guide.html#9-kfuncs">9. kfuncs</a></li>
<li><a href="bcc-documents/reference_guide.html#10-kretfuncs">10. kretfuncs</a></li>
<li><a href="bcc-documents/reference_guide.html#11-lsm-probes">11. lsm probes</a></li>
<li><a href="bcc-documents/reference_guide.html#12-bpf-iterators">12. bpf iterators</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#data">Data</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-bpf_probe_read_kernel">1. bpf_probe_read_kernel()</a></li>
<li><a href="bcc-documents/reference_guide.html#2-bpf_probe_read_kernel_str">2. bpf_probe_read_kernel_str()</a></li>
<li><a href="bcc-documents/reference_guide.html#3-bpf_ktime_get_ns">3. bpf_ktime_get_ns()</a></li>
<li><a href="bcc-documents/reference_guide.html#4-bpf_get_current_pid_tgid">4. bpf_get_current_pid_tgid()</a></li>
<li><a href="bcc-documents/reference_guide.html#5-bpf_get_current_uid_gid">5. bpf_get_current_uid_gid()</a></li>
<li><a href="bcc-documents/reference_guide.html#6-bpf_get_current_comm">6. bpf_get_current_comm()</a></li>
<li><a href="bcc-documents/reference_guide.html#7-bpf_get_current_task">7. bpf_get_current_task()</a></li>
<li><a href="bcc-documents/reference_guide.html#8-bpf_log2l">8. bpf_log2l()</a></li>
<li><a href="bcc-documents/reference_guide.html#9-bpf_get_prandom_u32">9. bpf_get_prandom_u32()</a></li>
<li><a href="bcc-documents/reference_guide.html#10-bpf_probe_read_user">10. bpf_probe_read_user()</a></li>
<li><a href="bcc-documents/reference_guide.html#11-bpf_probe_read_user_str">11. bpf_probe_read_user_str()</a></li>
<li><a href="bcc-documents/reference_guide.html#12-bpf_get_ns_current_pid_tgid">12. bpf_get_ns_current_pid_tgid()</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#debugging">Debugging</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-bpf_override_return">1. bpf_override_return()</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#output">Output</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-bpf_trace_printk">1. bpf_trace_printk()</a></li>
<li><a href="bcc-documents/reference_guide.html#2-bpf_perf_output">2. BPF_PERF_OUTPUT</a></li>
<li><a href="bcc-documents/reference_guide.html#3-perf_submit">3. perf_submit()</a></li>
<li><a href="bcc-documents/reference_guide.html#4-perf_submit_skb">4. perf_submit_skb()</a></li>
<li><a href="bcc-documents/reference_guide.html#5-bpf_ringbuf_output">5. BPF_RINGBUF_OUTPUT</a></li>
<li><a href="bcc-documents/reference_guide.html#6-ringbuf_output">6. ringbuf_output()</a></li>
<li><a href="bcc-documents/reference_guide.html#7-ringbuf_reserve">7. ringbuf_reserve()</a></li>
<li><a href="bcc-documents/reference_guide.html#8-ringbuf_submit">8. ringbuf_submit()</a></li>
<li><a href="bcc-documents/reference_guide.html#9-ringbuf_discard">9. ringbuf_discard()</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#maps">Maps</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-bpf_table">1. BPF_TABLE</a></li>
<li><a href="bcc-documents/reference_guide.html#2-bpf_hash">2. BPF_HASH</a></li>
<li><a href="bcc-documents/reference_guide.html#3-bpf_array">3. BPF_ARRAY</a></li>
<li><a href="bcc-documents/reference_guide.html#4-bpf_histogram">4. BPF_HISTOGRAM</a></li>
<li><a href="bcc-documents/reference_guide.html#5-bpf_stack_trace">5. BPF_STACK_TRACE</a></li>
<li><a href="bcc-documents/reference_guide.html#6-bpf_perf_array">6. BPF_PERF_ARRAY</a></li>
<li><a href="bcc-documents/reference_guide.html#7-bpf_percpu_hash">7. BPF_PERCPU_HASH</a></li>
<li><a href="bcc-documents/reference_guide.html#8-bpf_percpu_array">8. BPF_PERCPU_ARRAY</a></li>
<li><a href="bcc-documents/reference_guide.html#9-bpf_lpm_trie">9. BPF_LPM_TRIE</a></li>
<li><a href="bcc-documents/reference_guide.html#10-bpf_prog_array">10. BPF_PROG_ARRAY</a></li>
<li><a href="bcc-documents/reference_guide.html#11-bpf_devmap">11. BPF_DEVMAP</a></li>
<li><a href="bcc-documents/reference_guide.html#12-bpf_cpumap">12. BPF_CPUMAP</a></li>
<li><a href="bcc-documents/reference_guide.html#13-bpf_xskmap">13. BPF_XSKMAP</a></li>
<li><a href="bcc-documents/reference_guide.html#14-bpf_array_of_maps">14. BPF_ARRAY_OF_MAPS</a></li>
<li><a href="bcc-documents/reference_guide.html#15-bpf_hash_of_maps">15. BPF_HASH_OF_MAPS</a></li>
<li><a href="bcc-documents/reference_guide.html#16-bpf_stack">16. BPF_STACK</a></li>
<li><a href="bcc-documents/reference_guide.html#17-bpf_queue">17. BPF_QUEUE</a></li>
<li><a href="bcc-documents/reference_guide.html#18-bpf_sockhash">18. BPF_SOCKHASH</a></li>
<li><a href="bcc-documents/reference_guide.html#19-maplookup">19. map.lookup()</a></li>
<li><a href="bcc-documents/reference_guide.html#20-maplookup_or_try_init">20. map.lookup_or_try_init()</a></li>
<li><a href="bcc-documents/reference_guide.html#21-mapdelete">21. map.delete()</a></li>
<li><a href="bcc-documents/reference_guide.html#22-mapupdate">22. map.update()</a></li>
<li><a href="bcc-documents/reference_guide.html#23-mapinsert">23. map.insert()</a></li>
<li><a href="bcc-documents/reference_guide.html#24-mapincrement">24. map.increment()</a></li>
<li><a href="bcc-documents/reference_guide.html#25-mapget_stackid">25. map.get_stackid()</a></li>
<li><a href="bcc-documents/reference_guide.html#26-mapperf_read">26. map.perf_read()</a></li>
<li><a href="bcc-documents/reference_guide.html#27-mapcall">27. map.call()</a></li>
<li><a href="bcc-documents/reference_guide.html#28-mapredirect_map">28. map.redirect_map()</a></li>
<li><a href="bcc-documents/reference_guide.html#29-mappush">29. map.push()</a></li>
<li><a href="bcc-documents/reference_guide.html#30-mappop">30. map.pop()</a></li>
<li><a href="bcc-documents/reference_guide.html#31-mappeek">31. map.peek()</a></li>
<li><a href="bcc-documents/reference_guide.html#32-mapsock_hash_update">32. map.sock_hash_update()</a></li>
<li><a href="bcc-documents/reference_guide.html#33-mapmsg_redirect_hash">33. map.msg_redirect_hash()</a></li>
<li><a href="bcc-documents/reference_guide.html#34-mapsk_redirect_hash">34. map.sk_redirect_hash()</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#licensing">Licensing</a></li>
<li><a href="bcc-documents/reference_guide.html#rewriter">Rewriter</a></li>
</ul>
</li>
<li>
<p><a href="bcc-documents/reference_guide.html#bcc-python">bcc Python</a></p>
<ul>
<li><a href="bcc-documents/reference_guide.html#initialization">Initialization</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-bpf">1. BPF</a></li>
<li><a href="bcc-documents/reference_guide.html#2-usdt">2. USDT</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#events">Events</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-attach_kprobe">1. attach_kprobe()</a></li>
<li><a href="bcc-documents/reference_guide.html#2-attach_kretprobe">2. attach_kretprobe()</a></li>
<li><a href="bcc-documents/reference_guide.html#3-attach_tracepoint">3. attach_tracepoint()</a></li>
<li><a href="bcc-documents/reference_guide.html#4-attach_uprobe">4. attach_uprobe()</a></li>
<li><a href="bcc-documents/reference_guide.html#5-attach_uretprobe">5. attach_uretprobe()</a></li>
<li><a href="bcc-documents/reference_guide.html#6-usdtenable_probe">6. USDT.enable_probe()</a></li>
<li><a href="bcc-documents/reference_guide.html#7-attach_raw_tracepoint">7. attach_raw_tracepoint()</a></li>
<li><a href="bcc-documents/reference_guide.html#8-attach_raw_socket">8. attach_raw_socket()</a></li>
<li><a href="bcc-documents/reference_guide.html#9-attach_xdp">9. attach_xdp()</a></li>
<li><a href="bcc-documents/reference_guide.html#10-attach_func">10. attach_func()</a></li>
<li><a href="bcc-documents/reference_guide.html#11-detach_func">11. detach_func()</a></li>
<li><a href="bcc-documents/reference_guide.html#12-detach_kprobe">12. detach_kprobe()</a></li>
<li><a href="bcc-documents/reference_guide.html#13-detach_kretprobe">13. detach_kretprobe()</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#debug-output">Debug Output</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-trace_print">1. trace_print()</a></li>
<li><a href="bcc-documents/reference_guide.html#2-trace_fields">2. trace_fields()</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#output-apis">Output APIs</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-perf_buffer_poll">1. perf_buffer_poll()</a></li>
<li><a href="bcc-documents/reference_guide.html#2-ring_buffer_poll">2. ring_buffer_poll()</a></li>
<li><a href="bcc-documents/reference_guide.html#3-ring_buffer_consume">3. ring_buffer_consume()</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#map-apis">Map APIs</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-get_table">1. get_table()</a></li>
<li><a href="bcc-documents/reference_guide.html#2-open_perf_buffer">2. open_perf_buffer()</a></li>
<li><a href="bcc-documents/reference_guide.html#3-items">3. items()</a></li>
<li><a href="bcc-documents/reference_guide.html#4-values">4. values()</a></li>
<li><a href="bcc-documents/reference_guide.html#5-clear">5. clear()</a></li>
<li><a href="bcc-documents/reference_guide.html#6-items_lookup_and_delete_batch">6. items_lookup_and_delete_batch()</a></li>
<li><a href="bcc-documents/reference_guide.html#7-items_lookup_batch">7. items_lookup_batch()</a></li>
<li><a href="bcc-documents/reference_guide.html#8-items_delete_batch">8. items_delete_batch()</a></li>
<li><a href="bcc-documents/reference_guide.html#9-items_update_batch">9. items_update_batch()</a></li>
<li><a href="bcc-documents/reference_guide.html#10-print_log2_hist">10. print_log2_hist()</a></li>
<li><a href="bcc-documents/reference_guide.html#11-print_linear_hist">11. print_linear_hist()</a></li>
<li><a href="bcc-documents/reference_guide.html#12-open_ring_buffer">12. open_ring_buffer()</a></li>
<li><a href="bcc-documents/reference_guide.html#13-push">13. push()</a></li>
<li><a href="bcc-documents/reference_guide.html#14-pop">14. pop()</a></li>
<li><a href="bcc-documents/reference_guide.html#15-peek">15. peek()</a></li>
</ul>
</li>
<li><a href="bcc-documents/reference_guide.html#helpers">Helpers</a>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-ksym">1. ksym()</a></li>
<li><a href="bcc-documents/reference_guide.html#2-ksymname">2. ksymname()</a></li>
<li><a href="bcc-documents/reference_guide.html#3-sym">3. sym()</a></li>
<li><a href="bcc-documents/reference_guide.html#4-num_open_kprobes">4. num_open_kprobes()</a></li>
<li><a href="bcc-documents/reference_guide.html#5-get_syscall_fnname">5. get_syscall_fnname()</a></li>
</ul>
</li>
</ul>
</li>
<li>
<p><a href="bcc-documents/reference_guide.html#bpf-errors">BPF Errors</a></p>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-invalid-mem-access">1. Invalid mem access</a></li>
<li><a href="bcc-documents/reference_guide.html#2-cannot-call-gpl-only-function-from-proprietary-program">2. Cannot call GPL only function from proprietary program</a></li>
</ul>
</li>
<li>
<p><a href="bcc-documents/reference_guide.html#Environment-Variables">Environment Variables</a></p>
<ul>
<li><a href="bcc-documents/reference_guide.html#1-kernel-source-directory">1. kernel source directory</a></li>
<li><a href="bcc-documents/reference_guide.html#2-kernel-version-overriding">2. kernel version overriding</a></li>
</ul>
</li>
</ul>
<h1 id="bpf-c"><a class="header" href="#bpf-c">BPF C</a></h1>
<p>This section describes the C part of a bcc program.</p>
<h2 id="events--arguments"><a class="header" href="#events--arguments">Events &amp; Arguments</a></h2>
<h3 id="1-kprobes"><a class="header" href="#1-kprobes">1. kprobes</a></h3>
<p>Syntax: kprobe__<em>kernel_function_name</em></p>
<p><code>kprobe__</code> is a special prefix that creates a kprobe (dynamic tracing of a kernel function call) for the kernel function name provided as the remainder. You can also use kprobes by declaring a normal C function, then using the Python <code>BPF.attach_kprobe()</code> (covered later) to associate it with a kernel function.</p>
<p>Arguments are specified on the function declaration: kprobe__<em>kernel_function_name</em>(struct pt_regs *ctx [, <em>argument1</em> ...])</p>
<p>For example:</p>
<pre><code class="language-c">int kprobe__tcp_v4_connect(struct pt_regs *ctx, struct sock *sk) {
[...]
}
</code></pre>
<p>This instruments the tcp_v4_connect() kernel function using a kprobe, with the following arguments:</p>
<ul>
<li><code>struct pt_regs *ctx</code>: Registers and BPF context.</li>
<li><code>struct sock *sk</code>: First argument to tcp_v4_connect().</li>
</ul>
<p>The first argument is always <code>struct pt_regs *</code>, the remainder are the arguments to the function (they don't need to be specified, if you don't intend to use them).</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/blob/4afa96a71c5dbfc4c507c3355e20baa6c184a3a8/examples/tracing/tcpv4connect.py#L28">code</a> (<a href="https://github.com/iovisor/bcc/blob/5bd0eb21fd148927b078deb8ac29fff2fb044b66/examples/tracing/tcpv4connect_example.txt#L8">output</a>),
<a href="https://github.com/iovisor/bcc/commit/310ab53710cfd46095c1f6b3e44f1dbc8d1a41d8#diff-8cd1822359ffee26e7469f991ce0ef00R26">code</a> (<a href="https://github.com/iovisor/bcc/blob/3b9679a3bd9b922c736f6061dc65cb56de7e0250/examples/tracing/bitehist_example.txt#L6">output</a>)</p>
<!--- I can't add search links here, since github currently cannot handle partial-word searches needed for "kprobe__" --->
<h3 id="2-kretprobes"><a class="header" href="#2-kretprobes">2. kretprobes</a></h3>
<p>Syntax: kretprobe__<em>kernel_function_name</em></p>
<p><code>kretprobe__</code> is a special prefix that creates a kretprobe (dynamic tracing of a kernel function return) for the kernel function name provided as the remainder. You can also use kretprobes by declaring a normal C function, then using the Python <code>BPF.attach_kretprobe()</code> (covered later) to associate it with a kernel function.</p>
<p>Return value is available as <code>PT_REGS_RC(ctx)</code>, given a function declaration of: kretprobe__<em>kernel_function_name</em>(struct pt_regs *ctx)</p>
<p>For example:</p>
<pre><code class="language-C">int kretprobe__tcp_v4_connect(struct pt_regs *ctx)
{
int ret = PT_REGS_RC(ctx);
[...]
}
</code></pre>
<p>This instruments the return of the tcp_v4_connect() kernel function using a kretprobe, and stores the return value in <code>ret</code>.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/blob/4afa96a71c5dbfc4c507c3355e20baa6c184a3a8/examples/tracing/tcpv4connect.py#L38">code</a> (<a href="https://github.com/iovisor/bcc/blob/5bd0eb21fd148927b078deb8ac29fff2fb044b66/examples/tracing/tcpv4connect_example.txt#L8">output</a>)</p>
<h3 id="3-tracepoints"><a class="header" href="#3-tracepoints">3. Tracepoints</a></h3>
<p>Syntax: TRACEPOINT_PROBE(<em>category</em>, <em>event</em>)</p>
<p>This is a macro that instruments the tracepoint defined by <em>category</em>:<em>event</em>.</p>
<p>The tracepoint name is <code>&lt;category&gt;:&lt;event&gt;</code>.
The probe function name is <code>tracepoint__&lt;category&gt;__&lt;event&gt;</code>.</p>
<p>Arguments are available in an <code>args</code> struct, which are the tracepoint arguments. One way to list these is to cat the relevant format file under /sys/kernel/debug/tracing/events/<em>category</em>/<em>event</em>/format.</p>
<p>The <code>args</code> struct can be used in place of <code>ctx</code> in each functions requiring a context as an argument. This includes notably <a href="bcc-documents/reference_guide.html#3-perf_submit">perf_submit()</a>.</p>
<p>For example:</p>
<pre><code class="language-C">TRACEPOINT_PROBE(random, urandom_read) {
// args is from /sys/kernel/debug/tracing/events/random/urandom_read/format
bpf_trace_printk(&quot;%d\\n&quot;, args-&gt;got_bits);
return 0;
}
</code></pre>
<p>This instruments the tracepoint <code>random:urandom_read tracepoint</code>, and prints the tracepoint argument <code>got_bits</code>.
When using Python API, this probe is automatically attached to the right tracepoint target.
For C++, this tracepoint probe can be attached by specifying the tracepoint target and function name explicitly:
<code>BPF::attach_tracepoint(&quot;random:urandom_read&quot;, &quot;tracepoint__random__urandom_read&quot;)</code>
Note the name of the probe function defined above is <code>tracepoint__random__urandom_read</code>.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/blob/a4159da8c4ea8a05a3c6e402451f530d6e5a8b41/examples/tracing/urandomread.py#L19">code</a> (<a href="https://github.com/iovisor/bcc/commit/e422f5e50ecefb96579b6391a2ada7f6367b83c4#diff-41e5ecfae4a3b38de5f4e0887ed160e5R10">output</a>),
<a href="https://github.com/iovisor/bcc/search?q=TRACEPOINT_PROBE+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=TRACEPOINT_PROBE+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="4-uprobes"><a class="header" href="#4-uprobes">4. uprobes</a></h3>
<p>These are instrumented by declaring a normal function in C, then associating it as a uprobe probe in Python via <code>BPF.attach_uprobe()</code> (covered later).</p>
<p>Arguments can be examined using <code>PT_REGS_PARM</code> macros.</p>
<p>For example:</p>
<pre><code class="language-C">int count(struct pt_regs *ctx) {
char buf[64];
bpf_probe_read_user(&amp;buf, sizeof(buf), (void *)PT_REGS_PARM1(ctx));
bpf_trace_printk(&quot;%s %d&quot;, buf, PT_REGS_PARM2(ctx));
return(0);
}
</code></pre>
<p>This reads the first argument as a string, and then prints it with the second argument as an integer.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/blob/4afa96a71c5dbfc4c507c3355e20baa6c184a3a8/examples/tracing/strlen_count.py#L26">code</a></p>
<h3 id="5-uretprobes"><a class="header" href="#5-uretprobes">5. uretprobes</a></h3>
<p>These are instrumented by declaring a normal function in C, then associating it as a uretprobe probe in Python via <code>BPF.attach_uretprobe()</code> (covered later).</p>
<p>Return value is available as <code>PT_REGS_RC(ctx)</code>, given a function declaration of: <em>function_name</em>(struct pt_regs *ctx)</p>
<p>For example:</p>
<pre><code class="language-C">BPF_HISTOGRAM(dist);
int count(struct pt_regs *ctx) {
dist.increment(PT_REGS_RC(ctx));
return 0;
}
</code></pre>
<p>This increments the bucket in the <code>dist</code> histogram that is indexed by the return value.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/blob/4afa96a71c5dbfc4c507c3355e20baa6c184a3a8/examples/tracing/strlen_hist.py#L39">code</a> (<a href="https://github.com/iovisor/bcc/blob/4afa96a71c5dbfc4c507c3355e20baa6c184a3a8/examples/tracing/strlen_hist.py#L15">output</a>),
<a href="https://github.com/iovisor/bcc/blob/4afa96a71c5dbfc4c507c3355e20baa6c184a3a8/tools/bashreadline.py">code</a> (<a href="https://github.com/iovisor/bcc/commit/aa87997d21e5c1a6a20e2c96dd25eb92adc8e85d#diff-2fd162f9e594206f789246ce97d62cf0R7">output</a>)</p>
<h3 id="6-usdt-probes"><a class="header" href="#6-usdt-probes">6. USDT probes</a></h3>
<p>These are User Statically-Defined Tracing (USDT) probes, which may be placed in some applications or libraries to provide a user-level equivalent of tracepoints. The primary BPF method provided for USDT support method is <code>enable_probe()</code>. USDT probes are instrumented by declaring a normal function in C, then associating it as a USDT probe in Python via <code>USDT.enable_probe()</code>.</p>
<p>Arguments can be read via: bpf_usdt_readarg(<em>index</em>, ctx, &amp;addr)</p>
<p>For example:</p>
<pre><code class="language-C">int do_trace(struct pt_regs *ctx) {
uint64_t addr;
char path[128];
bpf_usdt_readarg(6, ctx, &amp;addr);
bpf_probe_read_user(&amp;path, sizeof(path), (void *)addr);
bpf_trace_printk(&quot;path:%s\\n&quot;, path);
return 0;
};
</code></pre>
<p>This reads the sixth USDT argument, and then pulls it in as a string to <code>path</code>.</p>
<p>When initializing USDTs via the third argument of <code>BPF::init</code> in the C API, if any USDT fails to <code>init</code>, entire <code>BPF::init</code> will fail. If you're OK with some USDTs failing to <code>init</code>, use <code>BPF::init_usdt</code> before calling <code>BPF::init</code>.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/commit/4f88a9401357d7b75e917abd994aa6ea97dda4d3#diff-04a7cad583be5646080970344c48c1f4R24">code</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_usdt_readarg+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_usdt_readarg+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="7-raw-tracepoints"><a class="header" href="#7-raw-tracepoints">7. Raw Tracepoints</a></h3>
<p>Syntax: RAW_TRACEPOINT_PROBE(<em>event</em>)</p>
<p>This is a macro that instruments the raw tracepoint defined by <em>event</em>.</p>
<p>The argument is a pointer to struct <code>bpf_raw_tracepoint_args</code>, which is defined in <a href="https://github.com/iovisor/bcc/blob/master/src/cc/compat/linux/virtual_bpf.h">bpf.h</a>. The struct field <code>args</code> contains all parameters of the raw tracepoint where you can found at linux tree <a href="https://github.com/torvalds/linux/tree/master/include/trace/events">include/trace/events</a>
directory.</p>
<p>For example:</p>
<pre><code class="language-C">RAW_TRACEPOINT_PROBE(sched_switch)
{
// TP_PROTO(bool preempt, struct task_struct *prev, struct task_struct *next)
struct task_struct *prev = (struct task_struct *)ctx-&gt;args[1];
struct task_struct *next= (struct task_struct *)ctx-&gt;args[2];
s32 prev_tgid, next_tgid;
bpf_probe_read_kernel(&amp;prev_tgid, sizeof(prev-&gt;tgid), &amp;prev-&gt;tgid);
bpf_probe_read_kernel(&amp;next_tgid, sizeof(next-&gt;tgid), &amp;next-&gt;tgid);
bpf_trace_printk(&quot;%d -&gt; %d\\n&quot;, prev_tgid, next_tgid);
}
</code></pre>
<p>This instruments the sched:sched_switch tracepoint, and prints the prev and next tgid.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=RAW_TRACEPOINT_PROBE+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="8-system-call-tracepoints"><a class="header" href="#8-system-call-tracepoints">8. system call tracepoints</a></h3>
<p>Syntax: <code>syscall__SYSCALLNAME</code></p>
<p><code>syscall__</code> is a special prefix that creates a kprobe for the system call name provided as the remainder. You can use it by declaring a normal C function, then using the Python <code>BPF.get_syscall_fnname(SYSCALLNAME)</code> and <code>BPF.attach_kprobe()</code> to associate it.</p>
<p>Arguments are specified on the function declaration: <code>syscall__SYSCALLNAME(struct pt_regs *ctx, [, argument1 ...])</code>.</p>
<p>For example:</p>
<pre><code class="language-C">int syscall__execve(struct pt_regs *ctx,
const char __user *filename,
const char __user *const __user *__argv,
const char __user *const __user *__envp)
{
[...]
}
</code></pre>
<p>This instruments the execve system call.</p>
<p>The first argument is always <code>struct pt_regs *</code>, the remainder are the arguments to the function (they don't need to be specified, if you don't intend to use them).</p>
<p>Corresponding Python code:</p>
<pre><code class="language-Python">b = BPF(text=bpf_text)
execve_fnname = b.get_syscall_fnname(&quot;execve&quot;)
b.attach_kprobe(event=execve_fnname, fn_name=&quot;syscall__execve&quot;)
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/blob/552658edda09298afdccc8a4b5e17311a2d8a771/tools/execsnoop.py#L101">code</a> (<a href="https://github.com/iovisor/bcc/blob/552658edda09298afdccc8a4b5e17311a2d8a771/tools/execsnoop_example.txt#L8">output</a>)</p>
<h3 id="9-kfuncs"><a class="header" href="#9-kfuncs">9. kfuncs</a></h3>
<p>Syntax: KFUNC_PROBE(<em>function</em>, typeof(arg1) arg1, typeof(arg2) arge ...)</p>
<p>This is a macro that instruments the kernel function via trampoline
<em>before</em> the function is executed. It's defined by <em>function</em> name and
the function arguments defined as <em>argX</em>.</p>
<p>For example:</p>
<pre><code class="language-C">KFUNC_PROBE(do_sys_open, int dfd, const char *filename, int flags, int mode)
{
...
</code></pre>
<p>This instruments the do_sys_open kernel function and make its arguments
accessible as standard argument values.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=KFUNC_PROBE+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="10-kretfuncs"><a class="header" href="#10-kretfuncs">10. kretfuncs</a></h3>
<p>Syntax: KRETFUNC_PROBE(<em>event</em>, typeof(arg1) arg1, typeof(arg2) arge ..., int ret)</p>
<p>This is a macro that instruments the kernel function via trampoline
<em>after</em> the function is executed. It's defined by <em>function</em> name and
the function arguments defined as <em>argX</em>.</p>
<p>The last argument of the probe is the return value of the instrumented function.</p>
<p>For example:</p>
<pre><code class="language-C">KRETFUNC_PROBE(do_sys_open, int dfd, const char *filename, int flags, int mode, int ret)
{
...
</code></pre>
<p>This instruments the do_sys_open kernel function and make its arguments
accessible as standard argument values together with its return value.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=KRETFUNC_PROBE+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="11-lsm-probes"><a class="header" href="#11-lsm-probes">11. LSM Probes</a></h3>
<p>Syntax: LSM_PROBE(<em>hook</em>, typeof(arg1) arg1, typeof(arg2) arg2 ...)</p>
<p>This is a macro that instruments an LSM hook as a BPF program. It can be
used to audit security events and implement MAC security policies in BPF.
It is defined by specifying the hook name followed by its arguments.</p>
<p>Hook names can be found in
<a href="https://github.com/torvalds/linux/blob/v5.15/include/linux/security.h#L260">include/linux/security.h</a>
by taking functions like <code>security_hookname</code> and taking just the <code>hookname</code> part.
For example, <code>security_bpf</code> would simply become <code>bpf</code>.</p>
<p>Unlike other BPF program types, the return value specified in an LSM probe
matters. A return value of 0 allows the hook to succeed, whereas
any non-zero return value will cause the hook to fail and deny the
security operation.</p>
<p>The following example instruments a hook that denies all future BPF operations:</p>
<pre><code class="language-C">LSM_PROBE(bpf, int cmd, union bpf_attr *attr, unsigned int size)
{
return -EPERM;
}
</code></pre>
<p>This instruments the <code>security_bpf</code> hook and causes it to return <code>-EPERM</code>.
Changing <code>return -EPERM</code> to <code>return 0</code> would cause the BPF program
to allow the operation instead.</p>
<p>LSM probes require at least a 5.7+ kernel with the following configuation options set:</p>
<ul>
<li><code>CONFIG_BPF_LSM=y</code></li>
<li><code>CONFIG_LSM</code> comma separated string must contain &quot;bpf&quot; (for example,
<code>CONFIG_LSM=&quot;lockdown,yama,bpf&quot;</code>)</li>
</ul>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=LSM_PROBE+path%3Atests&amp;type=Code">search /tests</a></p>
<h3 id="12-bpf-iterators"><a class="header" href="#12-bpf-iterators">12. BPF ITERATORS</a></h3>
<p>Syntax: BPF_ITER(target)</p>
<p>This is a macro to define a program signature for a bpf iterator program. The argument <em>target</em> specifies what to iterate for the program.</p>
<p>Currently, kernel does not have interface to discover what targets are supported. A good place to find what is supported is in <a href="https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/bpf_iter.c">tools/testing/selftests/bpf/prog_test/bpf_iter.c</a> and some sample bpf iter programs are in <a href="https://github.com/torvalds/linux/tree/master/tools/testing/selftests/bpf/progs">tools/testing/selftests/bpf/progs</a> with file name prefix <em>bpf_iter</em>.</p>
<p>The following example defines a program for target <em>task</em>, which traverses all tasks in the kernel.</p>
<pre><code class="language-C">BPF_ITER(task)
{
struct seq_file *seq = ctx-&gt;meta-&gt;seq;
struct task_struct *task = ctx-&gt;task;
if (task == (void *)0)
return 0;
... task-&gt;pid, task-&gt;tgid, task-&gt;comm, ...
return 0;
}
</code></pre>
<p>BPF iterators are introduced in 5.8 kernel for task, task_file, bpf_map, netlink_sock and ipv6_route . In 5.9, support is added to tcp/udp sockets and bpf map element (hashmap, arraymap and sk_local_storage_map) traversal.</p>
<h2 id="data"><a class="header" href="#data">Data</a></h2>
<h3 id="1-bpf_probe_read_kernel"><a class="header" href="#1-bpf_probe_read_kernel">1. bpf_probe_read_kernel()</a></h3>
<p>Syntax: <code>int bpf_probe_read_kernel(void *dst, int size, const void *src)</code></p>
<p>Return: 0 on success</p>
<p>This copies size bytes from kernel address space to the BPF stack, so that BPF can later operate on it. For safety, all kernel memory reads must pass through bpf_probe_read_kernel(). This happens automatically in some cases, such as dereferencing kernel variables, as bcc will rewrite the BPF program to include the necessary bpf_probe_read_kernel().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_probe_read_kernel+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_probe_read_kernel+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="2-bpf_probe_read_kernel_str"><a class="header" href="#2-bpf_probe_read_kernel_str">2. bpf_probe_read_kernel_str()</a></h3>
<p>Syntax: <code>int bpf_probe_read_kernel_str(void *dst, int size, const void *src)</code></p>
<p>Return:</p>
<ul>
<li>&gt; 0 length of the string including the trailing NULL on success</li>
<li>&lt; 0 error</li>
</ul>
<p>This copies a <code>NULL</code> terminated string from kernel address space to the BPF stack, so that BPF can later operate on it. In case the string length is smaller than size, the target is not padded with further <code>NULL</code> bytes. In case the string length is larger than size, just <code>size - 1</code> bytes are copied and the last byte is set to <code>NULL</code>.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_probe_read_kernel_str+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_probe_read_kernel_str+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="3-bpf_ktime_get_ns"><a class="header" href="#3-bpf_ktime_get_ns">3. bpf_ktime_get_ns()</a></h3>
<p>Syntax: <code>u64 bpf_ktime_get_ns(void)</code></p>
<p>Return: u64 number of nanoseconds. Starts at system boot time but stops during suspend.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_ktime_get_ns+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_ktime_get_ns+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="4-bpf_get_current_pid_tgid"><a class="header" href="#4-bpf_get_current_pid_tgid">4. bpf_get_current_pid_tgid()</a></h3>
<p>Syntax: <code>u64 bpf_get_current_pid_tgid(void)</code></p>
<p>Return: <code>current-&gt;tgid &lt;&lt; 32 | current-&gt;pid</code></p>
<p>Returns the process ID in the lower 32 bits (kernel's view of the PID, which in user space is usually presented as the thread ID), and the thread group ID in the upper 32 bits (what user space often thinks of as the PID). By directly setting this to a u32, we discard the upper 32 bits.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_current_pid_tgid+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_current_pid_tgid+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="5-bpf_get_current_uid_gid"><a class="header" href="#5-bpf_get_current_uid_gid">5. bpf_get_current_uid_gid()</a></h3>
<p>Syntax: <code>u64 bpf_get_current_uid_gid(void)</code></p>
<p>Return: <code>current_gid &lt;&lt; 32 | current_uid</code></p>
<p>Returns the user ID and group IDs.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_current_uid_gid+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_current_uid_gid+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="6-bpf_get_current_comm"><a class="header" href="#6-bpf_get_current_comm">6. bpf_get_current_comm()</a></h3>
<p>Syntax: <code>bpf_get_current_comm(char *buf, int size_of_buf)</code></p>
<p>Return: 0 on success</p>
<p>Populates the first argument address with the current process name. It should be a pointer to a char array of at least size TASK_COMM_LEN, which is defined in linux/sched.h. For example:</p>
<pre><code class="language-C">#include &lt;linux/sched.h&gt;
int do_trace(struct pt_regs *ctx) {
char comm[TASK_COMM_LEN];
bpf_get_current_comm(&amp;comm, sizeof(comm));
[...]
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_current_comm+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_current_comm+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="7-bpf_get_current_task"><a class="header" href="#7-bpf_get_current_task">7. bpf_get_current_task()</a></h3>
<p>Syntax: <code>bpf_get_current_task()</code></p>
<p>Return: current task as a pointer to struct task_struct.</p>
<p>Returns a pointer to the current task's task_struct object. This helper can be used to compute the on-CPU time for a process, identify kernel threads, get the current CPU's run queue, or retrieve many other pieces of information.</p>
<p>With Linux 4.13, due to issues with field randomization, you may need two #define directives before the includes:</p>
<pre><code class="language-C">#define randomized_struct_fields_start struct {
#define randomized_struct_fields_end };
#include &lt;linux/sched.h&gt;
int do_trace(void *ctx) {
struct task_struct *t = (struct task_struct *)bpf_get_current_task();
[...]
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_current_task+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_current_task+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="8-bpf_log2l"><a class="header" href="#8-bpf_log2l">8. bpf_log2l()</a></h3>
<p>Syntax: <code>unsigned int bpf_log2l(unsigned long v)</code></p>
<p>Returns the log-2 of the provided value. This is often used to create indexes for histograms, to construct power-of-2 histograms.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_log2l+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_log2l+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="9-bpf_get_prandom_u32"><a class="header" href="#9-bpf_get_prandom_u32">9. bpf_get_prandom_u32()</a></h3>
<p>Syntax: <code>u32 bpf_get_prandom_u32()</code></p>
<p>Returns a pseudo-random u32.</p>
<p>Example in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_prandom_u32+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_prandom_u32+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="10-bpf_probe_read_user"><a class="header" href="#10-bpf_probe_read_user">10. bpf_probe_read_user()</a></h3>
<p>Syntax: <code>int bpf_probe_read_user(void *dst, int size, const void *src)</code></p>
<p>Return: 0 on success</p>
<p>This attempts to safely read size bytes from user address space to the BPF stack, so that BPF can later operate on it. For safety, all user address space memory reads must pass through bpf_probe_read_user().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_probe_read_user+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_probe_read_user+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="11-bpf_probe_read_user_str"><a class="header" href="#11-bpf_probe_read_user_str">11. bpf_probe_read_user_str()</a></h3>
<p>Syntax: <code>int bpf_probe_read_user_str(void *dst, int size, const void *src)</code></p>
<p>Return:</p>
<ul>
<li>&gt; 0 length of the string including the trailing NULL on success</li>
<li>&lt; 0 error</li>
</ul>
<p>This copies a <code>NULL</code> terminated string from user address space to the BPF stack, so that BPF can later operate on it. In case the string length is smaller than size, the target is not padded with further <code>NULL</code> bytes. In case the string length is larger than size, just <code>size - 1</code> bytes are copied and the last byte is set to <code>NULL</code>.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_probe_read_user_str+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_probe_read_user_str+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="12-bpf_get_ns_current_pid_tgid"><a class="header" href="#12-bpf_get_ns_current_pid_tgid">12. bpf_get_ns_current_pid_tgid()</a></h3>
<p>Syntax: <code>u32 bpf_get_ns_current_pid_tgid(u64 dev, u64 ino, struct bpf_pidns_info* nsdata, u32 size)</code></p>
<p>Values for <em>pid</em> and <em>tgid</em> as seen from the current <em>namespace</em> will be returned in <em>nsdata</em>.</p>
<p>Return 0 on success, or one of the following in case of failure:</p>
<ul>
<li>
<p><strong>-EINVAL</strong> if dev and inum supplied don't match dev_t and inode number with nsfs of current task, or if dev conversion to dev_t lost high bits.</p>
</li>
<li>
<p><strong>-ENOENT</strong> if pidns does not exists for the current task.</p>
</li>
</ul>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_ns_current_pid_tgid+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_get_ns_current_pid_tgid+path%3Atools&amp;type=Code">search /tools</a></p>
<h2 id="debugging"><a class="header" href="#debugging">Debugging</a></h2>
<h3 id="1-bpf_override_return"><a class="header" href="#1-bpf_override_return">1. bpf_override_return()</a></h3>
<p>Syntax: <code>int bpf_override_return(struct pt_regs *, unsigned long rc)</code></p>
<p>Return: 0 on success</p>
<p>When used in a program attached to a function entry kprobe, causes the
execution of the function to be skipped, immediately returning <code>rc</code> instead.
This is used for targeted error injection.</p>
<p>bpf_override_return will only work when the kprobed function is whitelisted to
allow error injections. Whitelisting entails tagging a function with
<code>ALLOW_ERROR_INJECTION()</code> in the kernel source tree; see <code>io_ctl_init</code> for
an example. If the kprobed function is not whitelisted, the bpf program will
fail to attach with <code> ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument</code></p>
<pre><code class="language-C">int kprobe__io_ctl_init(void *ctx) {
bpf_override_return(ctx, -ENOMEM);
return 0;
}
</code></pre>
<h2 id="output"><a class="header" href="#output">Output</a></h2>
<h3 id="1-bpf_trace_printk"><a class="header" href="#1-bpf_trace_printk">1. bpf_trace_printk()</a></h3>
<p>Syntax: <code>int bpf_trace_printk(const char *fmt, ...)</code></p>
<p>Return: 0 on success</p>
<p>A simple kernel facility for printf() to the common trace_pipe (/sys/kernel/debug/tracing/trace_pipe). This is ok for some quick examples, but has limitations: 3 args max, 1 %s only, and trace_pipe is globally shared, so concurrent programs will have clashing output. A better interface is via BPF_PERF_OUTPUT(). Note that calling this helper is made simpler than the original kernel version, which has <code>fmt_size</code> as the second parameter.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=bpf_trace_printk+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=bpf_trace_printk+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="2-bpf_perf_output"><a class="header" href="#2-bpf_perf_output">2. BPF_PERF_OUTPUT</a></h3>
<p>Syntax: <code>BPF_PERF_OUTPUT(name)</code></p>
<p>Creates a BPF table for pushing out custom event data to user space via a perf ring buffer. This is the preferred method for pushing per-event data to user space.</p>
<p>For example:</p>
<pre><code class="language-C">struct data_t {
u32 pid;
u64 ts;
char comm[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(events);
int hello(struct pt_regs *ctx) {
struct data_t data = {};
data.pid = bpf_get_current_pid_tgid();
data.ts = bpf_ktime_get_ns();
bpf_get_current_comm(&amp;data.comm, sizeof(data.comm));
events.perf_submit(ctx, &amp;data, sizeof(data));
return 0;
}
</code></pre>
<p>The output table is named <code>events</code>, and data is pushed to it via <code>events.perf_submit()</code>.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_PERF_OUTPUT+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_PERF_OUTPUT+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="3-perf_submit"><a class="header" href="#3-perf_submit">3. perf_submit()</a></h3>
<p>Syntax: <code>int perf_submit((void *)ctx, (void *)data, u32 data_size)</code></p>
<p>Return: 0 on success</p>
<p>A method of a BPF_PERF_OUTPUT table, for submitting custom event data to user space. See the BPF_PERF_OUTPUT entry. (This ultimately calls bpf_perf_event_output().)</p>
<p>The <code>ctx</code> parameter is provided in <a href="bcc-documents/reference_guide.html#1-kprobes">kprobes</a> or <a href="bcc-documents/reference_guide.html#2-kretprobes">kretprobes</a>. For <code>SCHED_CLS</code> or <code>SOCKET_FILTER</code> programs, the <code>struct __sk_buff *skb</code> must be used instead.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=perf_submit+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=perf_submit+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="4-perf_submit_skb"><a class="header" href="#4-perf_submit_skb">4. perf_submit_skb()</a></h3>
<p>Syntax: <code>int perf_submit_skb((void *)ctx, u32 packet_size, (void *)data, u32 data_size)</code></p>
<p>Return: 0 on success</p>
<p>A method of a BPF_PERF_OUTPUT table available in networking program types, for submitting custom event data to user space, along with the first <code>packet_size</code> bytes of the packet buffer. See the BPF_PERF_OUTPUT entry. (This ultimately calls bpf_perf_event_output().)</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=perf_submit_skb+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=perf_submit_skb+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="5-bpf_ringbuf_output"><a class="header" href="#5-bpf_ringbuf_output">5. BPF_RINGBUF_OUTPUT</a></h3>
<p>Syntax: <code>BPF_RINGBUF_OUTPUT(name, page_cnt)</code></p>
<p>Creates a BPF table for pushing out custom event data to user space via a ringbuf ring buffer.
<code>BPF_RINGBUF_OUTPUT</code> has several advantages over <code>BPF_PERF_OUTPUT</code>, summarized as follows:</p>
<ul>
<li>Buffer is shared across all CPUs, meaning no per-CPU allocation</li>
<li>Supports two APIs for BPF programs
<ul>
<li><code>map.ringbuf_output()</code> works like <code>map.perf_submit()</code> (covered in <a href="bcc-documents/reference_guide.html#6-ringbuf_output">ringbuf_output</a>)</li>
<li><code>map.ringbuf_reserve()</code>/<code>map.ringbuf_submit()</code>/<code>map.ringbuf_discard()</code>
split the process of reserving buffer space and submitting events into two steps
(covered in <a href="bcc-documents/reference_guide.html#7-ringbuf_reserve">ringbuf_reserve</a>, <a href="bcc-documents/reference_guide.html#8-ringbuf_submit">ringbuf_submit</a>, <a href="bcc-documents/reference_guide.html#9-ringbuf_discard">ringbuf_discard</a>)</li>
</ul>
</li>
<li>BPF APIs do not require access to a CPU ctx argument</li>
<li>Superior performance and latency in userspace thanks to a shared ring buffer manager</li>
<li>Supports two ways of consuming data in userspace</li>
</ul>
<p>Starting in Linux 5.8, this should be the preferred method for pushing per-event data to user space.</p>
<p>Example of both APIs:</p>
<pre><code class="language-C">struct data_t {
u32 pid;
u64 ts;
char comm[TASK_COMM_LEN];
};
// Creates a ringbuf called events with 8 pages of space, shared across all CPUs
BPF_RINGBUF_OUTPUT(events, 8);
int first_api_example(struct pt_regs *ctx) {
struct data_t data = {};
data.pid = bpf_get_current_pid_tgid();
data.ts = bpf_ktime_get_ns();
bpf_get_current_comm(&amp;data.comm, sizeof(data.comm));
events.ringbuf_output(&amp;data, sizeof(data), 0 /* flags */);
return 0;
}
int second_api_example(struct pt_regs *ctx) {
struct data_t *data = events.ringbuf_reserve(sizeof(struct data_t));
if (!data) { // Failed to reserve space
return 1;
}
data-&gt;pid = bpf_get_current_pid_tgid();
data-&gt;ts = bpf_ktime_get_ns();
bpf_get_current_comm(&amp;data-&gt;comm, sizeof(data-&gt;comm));
events.ringbuf_submit(data, 0 /* flags */);
return 0;
}
</code></pre>
<p>The output table is named <code>events</code>. Data is allocated via <code>events.ringbuf_reserve()</code> and pushed to it via <code>events.ringbuf_submit()</code>.</p>
<p>Examples in situ: <!-- TODO -->
<a href="https://github.com/iovisor/bcc/search?q=BPF_RINGBUF_OUTPUT+path%3Aexamples&amp;type=Code">search /examples</a>,</p>
<h3 id="6-ringbuf_output"><a class="header" href="#6-ringbuf_output">6. ringbuf_output()</a></h3>
<p>Syntax: <code>int ringbuf_output((void *)data, u64 data_size, u64 flags)</code></p>
<p>Return: 0 on success</p>
<p>Flags:</p>
<ul>
<li><code>BPF_RB_NO_WAKEUP</code>: Do not sent notification of new data availability</li>
<li><code>BPF_RB_FORCE_WAKEUP</code>: Send notification of new data availability unconditionally</li>
</ul>
<p>A method of the BPF_RINGBUF_OUTPUT table, for submitting custom event data to user space. This method works like <code>perf_submit()</code>,
although it does not require a ctx argument.</p>
<p>Examples in situ: <!-- TODO -->
<a href="https://github.com/iovisor/bcc/search?q=ringbuf_output+path%3Aexamples&amp;type=Code">search /examples</a>,</p>
<h3 id="7-ringbuf_reserve"><a class="header" href="#7-ringbuf_reserve">7. ringbuf_reserve()</a></h3>
<p>Syntax: <code>void* ringbuf_reserve(u64 data_size)</code></p>
<p>Return: Pointer to data struct on success, NULL on failure</p>
<p>A method of the BPF_RINGBUF_OUTPUT table, for reserving space in the ring buffer and simultaenously
allocating a data struct for output. Must be used with one of <code>ringbuf_submit</code> or <code>ringbuf_discard</code>.</p>
<p>Examples in situ: <!-- TODO -->
<a href="https://github.com/iovisor/bcc/search?q=ringbuf_reserve+path%3Aexamples&amp;type=Code">search /examples</a>,</p>
<h3 id="8-ringbuf_submit"><a class="header" href="#8-ringbuf_submit">8. ringbuf_submit()</a></h3>
<p>Syntax: <code>void ringbuf_submit((void *)data, u64 flags)</code></p>
<p>Return: Nothing, always succeeds</p>
<p>Flags:</p>
<ul>
<li><code>BPF_RB_NO_WAKEUP</code>: Do not sent notification of new data availability</li>
<li><code>BPF_RB_FORCE_WAKEUP</code>: Send notification of new data availability unconditionally</li>
</ul>
<p>A method of the BPF_RINGBUF_OUTPUT table, for submitting custom event data to user space. Must be preceded by a call to
<code>ringbuf_reserve()</code> to reserve space for the data.</p>
<p>Examples in situ: <!-- TODO -->
<a href="https://github.com/iovisor/bcc/search?q=ringbuf_submit+path%3Aexamples&amp;type=Code">search /examples</a>,</p>
<h3 id="9-ringbuf_discard"><a class="header" href="#9-ringbuf_discard">9. ringbuf_discard()</a></h3>
<p>Syntax: <code>void ringbuf_discard((void *)data, u64 flags)</code></p>
<p>Return: Nothing, always succeeds</p>
<p>Flags:</p>
<ul>
<li><code>BPF_RB_NO_WAKEUP</code>: Do not sent notification of new data availability</li>
<li><code>BPF_RB_FORCE_WAKEUP</code>: Send notification of new data availability unconditionally</li>
</ul>
<p>A method of the BPF_RINGBUF_OUTPUT table, for discarding custom event data; userspace
ignores the data associated with the discarded event. Must be preceded by a call to
<code>ringbuf_reserve()</code> to reserve space for the data.</p>
<p>Examples in situ: <!-- TODO -->
<a href="https://github.com/iovisor/bcc/search?q=ringbuf_submit+path%3Aexamples&amp;type=Code">search /examples</a>,</p>
<h2 id="maps"><a class="header" href="#maps">Maps</a></h2>
<p>Maps are BPF data stores, and are the basis for higher level object types including tables, hashes, and histograms.</p>
<h3 id="1-bpf_table"><a class="header" href="#1-bpf_table">1. BPF_TABLE</a></h3>
<p>Syntax: <code>BPF_TABLE(_table_type, _key_type, _leaf_type, _name, _max_entries)</code></p>
<p>Creates a map named <code>_name</code>. Most of the time this will be used via higher-level macros, like BPF_HASH, BPF_ARRAY, BPF_HISTOGRAM, etc.</p>
<p><code>BPF_F_TABLE</code> is a variant that takes a flag in the last parameter. <code>BPF_TABLE(https://github.com/iovisor/bcc/tree/master.)</code> is actually a wrapper to <code>BPF_F_TABLE(https://github.com/iovisor/bcc/tree/master., 0 /* flag */)</code>.</p>
<p>Methods (covered later): map.lookup(), map.lookup_or_try_init(), map.delete(), map.update(), map.insert(), map.increment().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_TABLE+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_TABLE+path%3Atools&amp;type=Code">search /tools</a></p>
<h4 id="pinned-maps"><a class="header" href="#pinned-maps">Pinned Maps</a></h4>
<p>Syntax: <code>BPF_TABLE_PINNED(_table_type, _key_type, _leaf_type, _name, _max_entries, &quot;/sys/fs/bpf/xyz&quot;)</code></p>
<p>Create a new map if it doesn't exist and pin it to the bpffs as a FILE, otherwise use the map that was pinned to the bpffs. The type information is not enforced and the actual map type depends on the map that got pinned to the location.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_TABLE_PINNED(&quot;hash&quot;, u64, u64, ids, 1024, &quot;/sys/fs/bpf/ids&quot;);
</code></pre>
<h3 id="2-bpf_hash"><a class="header" href="#2-bpf_hash">2. BPF_HASH</a></h3>
<p>Syntax: <code>BPF_HASH(name [, key_type [, leaf_type [, size]]])</code></p>
<p>Creates a hash map (associative array) named <code>name</code>, with optional parameters.</p>
<p>Defaults: <code>BPF_HASH(name, key_type=u64, leaf_type=u64, size=10240)</code></p>
<p>For example:</p>
<pre><code class="language-C">BPF_HASH(start, struct request *);
</code></pre>
<p>This creates a hash named <code>start</code> where the key is a <code>struct request *</code>, and the value defaults to u64. This hash is used by the disksnoop.py example for saving timestamps for each I/O request, where the key is the pointer to struct request, and the value is the timestamp.</p>
<p>This is a wrapper macro for <code>BPF_TABLE(&quot;hash&quot;, ...)</code>.</p>
<p>Methods (covered later): map.lookup(), map.lookup_or_try_init(), map.delete(), map.update(), map.insert(), map.increment().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_HASH+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_HASH+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="3-bpf_array"><a class="header" href="#3-bpf_array">3. BPF_ARRAY</a></h3>
<p>Syntax: <code>BPF_ARRAY(name [, leaf_type [, size]])</code></p>
<p>Creates an int-indexed array which is optimized for fastest lookup and update, named <code>name</code>, with optional parameters.</p>
<p>Defaults: <code>BPF_ARRAY(name, leaf_type=u64, size=10240)</code></p>
<p>For example:</p>
<pre><code class="language-C">BPF_ARRAY(counts, u64, 32);
</code></pre>
<p>This creates an array named <code>counts</code> where with 32 buckets and 64-bit integer values. This array is used by the funccount.py example for saving call count of each function.</p>
<p>This is a wrapper macro for <code>BPF_TABLE(&quot;array&quot;, ...)</code>.</p>
<p>Methods (covered later): map.lookup(), map.update(), map.increment(). Note that all array elements are pre-allocated with zero values and can not be deleted.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_ARRAY+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_ARRAY+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="4-bpf_histogram"><a class="header" href="#4-bpf_histogram">4. BPF_HISTOGRAM</a></h3>
<p>Syntax: <code>BPF_HISTOGRAM(name [, key_type [, size ]])</code></p>
<p>Creates a histogram map named <code>name</code>, with optional parameters.</p>
<p>Defaults: <code>BPF_HISTOGRAM(name, key_type=int, size=64)</code></p>
<p>For example:</p>
<pre><code class="language-C">BPF_HISTOGRAM(dist);
</code></pre>
<p>This creates a histogram named <code>dist</code>, which defaults to 64 buckets indexed by keys of type int.</p>
<p>This is a wrapper macro for <code>BPF_TABLE(&quot;histgram&quot;, ...)</code>.</p>
<p>Methods (covered later): map.increment().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_HISTOGRAM+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_HISTOGRAM+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="5-bpf_stack_trace"><a class="header" href="#5-bpf_stack_trace">5. BPF_STACK_TRACE</a></h3>
<p>Syntax: <code>BPF_STACK_TRACE(name, max_entries)</code></p>
<p>Creates stack trace map named <code>name</code>, with a maximum entry count provided. These maps are used to store stack traces.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_STACK_TRACE(stack_traces, 1024);
</code></pre>
<p>This creates stack trace map named <code>stack_traces</code>, with a maximum number of stack trace entries of 1024.</p>
<p>This is a wrapper macro for <code>BPF_TABLE(&quot;stacktrace&quot;, ...)</code>.</p>
<p>Methods (covered later): map.get_stackid().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_STACK_TRACE+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_STACK_TRACE+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="6-bpf_perf_array"><a class="header" href="#6-bpf_perf_array">6. BPF_PERF_ARRAY</a></h3>
<p>Syntax: <code>BPF_PERF_ARRAY(name, max_entries)</code></p>
<p>Creates perf array named <code>name</code>, with a maximum entry count provided, which must be equal to the number of system cpus. These maps are used to fetch hardware performance counters.</p>
<p>For example:</p>
<pre><code class="language-C">text=&quot;&quot;&quot;
BPF_PERF_ARRAY(cpu_cycles, NUM_CPUS);
&quot;&quot;&quot;
b = bcc.BPF(text=text, cflags=[&quot;-DNUM_CPUS=%d&quot; % multiprocessing.cpu_count()])
b[&quot;cpu_cycles&quot;].open_perf_event(b[&quot;cpu_cycles&quot;].HW_CPU_CYCLES)
</code></pre>
<p>This creates a perf array named <code>cpu_cycles</code>, with number of entries equal to the number of cpus/cores. The array is configured so that later calling map.perf_read() will return a hardware-calculated counter of the number of cycles elapsed from some point in the past. Only one type of hardware counter may be configured per table at a time.</p>
<p>Methods (covered later): map.perf_read().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_PERF_ARRAY+path%3Atests&amp;type=Code">search /tests</a></p>
<h3 id="7-bpf_percpu_hash"><a class="header" href="#7-bpf_percpu_hash">7. BPF_PERCPU_HASH</a></h3>
<p>Syntax: <code>BPF_PERCPU_HASH(name [, key_type [, leaf_type [, size]]])</code></p>
<p>Creates NUM_CPU int-indexed hash maps (associative arrays) named <code>name</code>, with optional parameters. Each CPU will have a separate copy of this array. The copies are not kept synchronized in any way.</p>
<p>Note that due to limits defined in the kernel (in linux/mm/percpu.c), the <code>leaf_type</code> cannot have a size of more than 32KB.
In other words, <code>BPF_PERCPU_HASH</code> elements cannot be larger than 32KB in size.</p>
<p>Defaults: <code>BPF_PERCPU_HASH(name, key_type=u64, leaf_type=u64, size=10240)</code></p>
<p>For example:</p>
<pre><code class="language-C">BPF_PERCPU_HASH(start, struct request *);
</code></pre>
<p>This creates NUM_CPU hashes named <code>start</code> where the key is a <code>struct request *</code>, and the value defaults to u64.</p>
<p>This is a wrapper macro for <code>BPF_TABLE(&quot;percpu_hash&quot;, ...)</code>.</p>
<p>Methods (covered later): map.lookup(), map.lookup_or_try_init(), map.delete(), map.update(), map.insert(), map.increment().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_PERCPU_HASH+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_PERCPU_HASH+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="8-bpf_percpu_array"><a class="header" href="#8-bpf_percpu_array">8. BPF_PERCPU_ARRAY</a></h3>
<p>Syntax: <code>BPF_PERCPU_ARRAY(name [, leaf_type [, size]])</code></p>
<p>Creates NUM_CPU int-indexed arrays which are optimized for fastest lookup and update, named <code>name</code>, with optional parameters. Each CPU will have a separate copy of this array. The copies are not kept synchronized in any way.</p>
<p>Note that due to limits defined in the kernel (in linux/mm/percpu.c), the <code>leaf_type</code> cannot have a size of more than 32KB.
In other words, <code>BPF_PERCPU_ARRAY</code> elements cannot be larger than 32KB in size.</p>
<p>Defaults: <code>BPF_PERCPU_ARRAY(name, leaf_type=u64, size=10240)</code></p>
<p>For example:</p>
<pre><code class="language-C">BPF_PERCPU_ARRAY(counts, u64, 32);
</code></pre>
<p>This creates NUM_CPU arrays named <code>counts</code> where with 32 buckets and 64-bit integer values.</p>
<p>This is a wrapper macro for <code>BPF_TABLE(&quot;percpu_array&quot;, ...)</code>.</p>
<p>Methods (covered later): map.lookup(), map.update(), map.increment(). Note that all array elements are pre-allocated with zero values and can not be deleted.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_PERCPU_ARRAY+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_PERCPU_ARRAY+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="9-bpf_lpm_trie"><a class="header" href="#9-bpf_lpm_trie">9. BPF_LPM_TRIE</a></h3>
<p>Syntax: <code>BPF_LPM_TRIE(name [, key_type [, leaf_type [, size]]])</code></p>
<p>Creates a longest prefix match trie map named <code>name</code>, with optional parameters.</p>
<p>Defaults: <code>BPF_LPM_TRIE(name, key_type=u64, leaf_type=u64, size=10240)</code></p>
<p>For example:</p>
<pre><code class="language-c">BPF_LPM_TRIE(trie, struct key_v6);
</code></pre>
<p>This creates an LPM trie map named <code>trie</code> where the key is a <code>struct key_v6</code>, and the value defaults to u64.</p>
<p>This is a wrapper macro to <code>BPF_F_TABLE(&quot;lpm_trie&quot;, ..., BPF_F_NO_PREALLOC)</code>.</p>
<p>Methods (covered later): map.lookup(), map.lookup_or_try_init(), map.delete(), map.update(), map.insert(), map.increment().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_LPM_TRIE+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_LPM_TRIE+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="10-bpf_prog_array"><a class="header" href="#10-bpf_prog_array">10. BPF_PROG_ARRAY</a></h3>
<p>Syntax: <code>BPF_PROG_ARRAY(name, size)</code></p>
<p>This creates a program array named <code>name</code> with <code>size</code> entries. Each entry of the array is either a file descriptor to a bpf program or <code>NULL</code>. The array acts as a jump table so that bpf programs can &quot;tail-call&quot; other bpf programs.</p>
<p>This is a wrapper macro for <code>BPF_TABLE(&quot;prog&quot;, ...)</code>.</p>
<p>Methods (covered later): map.call().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_PROG_ARRAY+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF_PROG_ARRAY+path%3Atests&amp;type=Code">search /tests</a>,
<a href="https://github.com/iovisor/bcc/blob/master/examples/networking/tunnel_monitor/monitor.py#L24-L26">assign fd</a></p>
<h3 id="11-bpf_devmap"><a class="header" href="#11-bpf_devmap">11. BPF_DEVMAP</a></h3>
<p>Syntax: <code>BPF_DEVMAP(name, size)</code></p>
<p>This creates a device map named <code>name</code> with <code>size</code> entries. Each entry of the map is an <code>ifindex</code> to a network interface. This map is only used in XDP.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_DEVMAP(devmap, 10);
</code></pre>
<p>Methods (covered later): map.redirect_map().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_DEVMAP+path%3Aexamples&amp;type=Code">search /examples</a>,</p>
<h3 id="12-bpf_cpumap"><a class="header" href="#12-bpf_cpumap">12. BPF_CPUMAP</a></h3>
<p>Syntax: <code>BPF_CPUMAP(name, size)</code></p>
<p>This creates a cpu map named <code>name</code> with <code>size</code> entries. The index of the map represents the CPU id and each entry is the size of the ring buffer allocated for the CPU. This map is only used in XDP.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_CPUMAP(cpumap, 16);
</code></pre>
<p>Methods (covered later): map.redirect_map().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_CPUMAP+path%3Aexamples&amp;type=Code">search /examples</a>,</p>
<h3 id="13-bpf_xskmap"><a class="header" href="#13-bpf_xskmap">13. BPF_XSKMAP</a></h3>
<p>Syntax: <code>BPF_XSKMAP(name, size [, &quot;/sys/fs/bpf/xyz&quot;])</code></p>
<p>This creates a xsk map named <code>name</code> with <code>size</code> entries and pin it to the bpffs as a FILE. Each entry represents one NIC's queue id. This map is only used in XDP to redirect packet to an AF_XDP socket. If the AF_XDP socket is binded to a queue which is different than the current packet's queue id, the packet will be dropped. For kernel v5.3 and latter, <code>lookup</code> method is available and can be used to check whether and AF_XDP socket is available for the current packet's queue id. More details at <a href="https://www.kernel.org/doc/html/latest/networking/af_xdp.html">AF_XDP</a>.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_XSKMAP(xsks_map, 8);
</code></pre>
<p>Methods (covered later): map.redirect_map(). map.lookup()</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_XSKMAP+path%3Aexamples&amp;type=Code">search /examples</a>,</p>
<h3 id="14-bpf_array_of_maps"><a class="header" href="#14-bpf_array_of_maps">14. BPF_ARRAY_OF_MAPS</a></h3>
<p>Syntax: <code>BPF_ARRAY_OF_MAPS(name, inner_map_name, size)</code></p>
<p>This creates an array map with a map-in-map type (BPF_MAP_TYPE_HASH_OF_MAPS) map named <code>name</code> with <code>size</code> entries. The inner map meta data is provided by map <code>inner_map_name</code> and can be most of array or hash maps except <code>BPF_MAP_TYPE_PROG_ARRAY</code>, <code>BPF_MAP_TYPE_CGROUP_STORAGE</code> and <code>BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE</code>.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_TABLE(&quot;hash&quot;, int, int, ex1, 1024);
BPF_TABLE(&quot;hash&quot;, int, int, ex2, 1024);
BPF_ARRAY_OF_MAPS(maps_array, &quot;ex1&quot;, 10);
</code></pre>
<h3 id="15-bpf_hash_of_maps"><a class="header" href="#15-bpf_hash_of_maps">15. BPF_HASH_OF_MAPS</a></h3>
<p>Syntax: <code>BPF_HASH_OF_MAPS(name, key_type, inner_map_name, size)</code></p>
<p>This creates a hash map with a map-in-map type (BPF_MAP_TYPE_HASH_OF_MAPS) map named <code>name</code> with <code>size</code> entries. The inner map meta data is provided by map <code>inner_map_name</code> and can be most of array or hash maps except <code>BPF_MAP_TYPE_PROG_ARRAY</code>, <code>BPF_MAP_TYPE_CGROUP_STORAGE</code> and <code>BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE</code>.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_ARRAY(ex1, int, 1024);
BPF_ARRAY(ex2, int, 1024);
BPF_HASH_OF_MAPS(maps_hash, struct custom_key, &quot;ex1&quot;, 10);
</code></pre>
<h3 id="16-bpf_stack"><a class="header" href="#16-bpf_stack">16. BPF_STACK</a></h3>
<p>Syntax: <code>BPF_STACK(name, leaf_type, max_entries[, flags])</code></p>
<p>Creates a stack named <code>name</code> with value type <code>leaf_type</code> and max entries <code>max_entries</code>.
Stack and Queue maps are only available from Linux 4.20+.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_STACK(stack, struct event, 10240);
</code></pre>
<p>This creates a stack named <code>stack</code> where the value type is <code>struct event</code>, that holds up to 10240 entries.</p>
<p>Methods (covered later): map.push(), map.pop(), map.peek().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_STACK+path%3Atests&amp;type=Code">search /tests</a>,</p>
<h3 id="17-bpf_queue"><a class="header" href="#17-bpf_queue">17. BPF_QUEUE</a></h3>
<p>Syntax: <code>BPF_QUEUE(name, leaf_type, max_entries[, flags])</code></p>
<p>Creates a queue named <code>name</code> with value type <code>leaf_type</code> and max entries <code>max_entries</code>.
Stack and Queue maps are only available from Linux 4.20+.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_QUEUE(queue, struct event, 10240);
</code></pre>
<p>This creates a queue named <code>queue</code> where the value type is <code>struct event</code>, that holds up to 10240 entries.</p>
<p>Methods (covered later): map.push(), map.pop(), map.peek().</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF_QUEUE+path%3Atests&amp;type=Code">search /tests</a>,</p>
<h3 id="18-bpf_sockhash"><a class="header" href="#18-bpf_sockhash">18. BPF_SOCKHASH</a></h3>
<p>Syntax: <code>BPF_SOCKHASH(name[, key_type [, max_entries)</code></p>
<p>Creates a hash named <code>name</code>, with optional parameters. sockhash is only available from Linux 4.18+.</p>
<p>Default: <code>BPF_SOCKHASH(name, key_type=u32, max_entries=10240)</code></p>
<p>For example:</p>
<pre><code class="language-C">struct sock_key {
u32 remote_ip4;
u32 local_ip4;
u32 remote_port;
u32 local_port;
};
BPF_HASH(skh, struct sock_key, 65535);
</code></pre>
<p>This creates a hash named <code>skh</code> where the key is a <code>struct sock_key</code>.</p>
<p>A sockhash is a BPF map type that holds references to sock structs. Then with a new sk/msg redirect bpf helper BPF programs can use the map to redirect skbs/msgs between sockets (<code>map.sk_redirect_hash()/map.msg_redirect_hash()</code>).</p>
<p>The difference between <code>BPF_SOCKHASH</code> and <code>BPF_SOCKMAP</code> is that <code>BPF_SOCKMAP</code> is implemented based on an array, and enforces keys to be four bytes. While <code>BPF_SOCKHASH</code> is implemented based on hash table, and the type of key can be specified freely.</p>
<p>Methods (covered later): map.sock_hash_update(), map.msg_redirect_hash(), map.sk_redirect_hash().</p>
<p><a href="https://github.com/iovisor/bcc/search?q=BPF_SOCKHASH+path%3Atests&amp;type=Code">search /tests</a></p>
<h3 id="19-maplookup"><a class="header" href="#19-maplookup">19. map.lookup()</a></h3>
<p>Syntax: <code>*val map.lookup(&amp;key)</code></p>
<p>Lookup the key in the map, and return a pointer to its value if it exists, else NULL. We pass the key in as an address to a pointer.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=lookup+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=lookup+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="20-maplookup_or_try_init"><a class="header" href="#20-maplookup_or_try_init">20. map.lookup_or_try_init()</a></h3>
<p>Syntax: <code>*val map.lookup_or_try_init(&amp;key, &amp;zero)</code></p>
<p>Lookup the key in the map, and return a pointer to its value if it exists, else initialize the key's value to the second argument. This is often used to initialize values to zero. If the key cannot be inserted (e.g. the map is full) then NULL is returned.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=lookup_or_try_init+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=lookup_or_try_init+path%3Atools&amp;type=Code">search /tools</a></p>
<p>Note: The old map.lookup_or_init() may cause return from the function, so lookup_or_try_init() is recommended as it
does not have this side effect.</p>
<h3 id="21-mapdelete"><a class="header" href="#21-mapdelete">21. map.delete()</a></h3>
<p>Syntax: <code>map.delete(&amp;key)</code></p>
<p>Delete the key from the hash.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=delete+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=delete+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="22-mapupdate"><a class="header" href="#22-mapupdate">22. map.update()</a></h3>
<p>Syntax: <code>map.update(&amp;key, &amp;val)</code></p>
<p>Associate the value in the second argument to the key, overwriting any previous value.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=update+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=update+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="23-mapinsert"><a class="header" href="#23-mapinsert">23. map.insert()</a></h3>
<p>Syntax: <code>map.insert(&amp;key, &amp;val)</code></p>
<p>Associate the value in the second argument to the key, only if there was no previous value.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=insert+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=insert+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="24-mapincrement"><a class="header" href="#24-mapincrement">24. map.increment()</a></h3>
<p>Syntax: <code>map.increment(key[, increment_amount])</code></p>
<p>Increments the key's value by <code>increment_amount</code>, which defaults to 1. Used for histograms.</p>
<p><code>map.increment()</code> are not atomic. In the concurrency case. If you want more accurate results, use <code>map.atomic_increment()</code> instead of <code>map.increment()</code>. The overhead of <code>map.increment()</code> and <code>map.atomic_increment()</code> is similar.</p>
<p>Note. When using <code>map.atomic_increment()</code> to operate on a BPF map of type <code>BPF_MAP_TYPE_HASH</code>, <code>map.atomic_increment()</code> does not guarantee the atomicity of the operation when the specified key does not exist.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=increment+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=increment+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="25-mapget_stackid"><a class="header" href="#25-mapget_stackid">25. map.get_stackid()</a></h3>
<p>Syntax: <code>int map.get_stackid(void *ctx, u64 flags)</code></p>
<p>This walks the stack found via the struct pt_regs in <code>ctx</code>, saves it in the stack trace map, and returns a unique ID for the stack trace.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=get_stackid+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=get_stackid+path%3Atools&amp;type=Code">search /tools</a></p>
<h3 id="26-mapperf_read"><a class="header" href="#26-mapperf_read">26. map.perf_read()</a></h3>
<p>Syntax: <code>u64 map.perf_read(u32 cpu)</code></p>
<p>This returns the hardware performance counter as configured in <a href="bcc-documents/reference_guide.html#5-bpf_perf_array">5. BPF_PERF_ARRAY</a></p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=perf_read+path%3Atests&amp;type=Code">search /tests</a></p>
<h3 id="27-mapcall"><a class="header" href="#27-mapcall">27. map.call()</a></h3>
<p>Syntax: <code>void map.call(void *ctx, int index)</code></p>
<p>This invokes <code>bpf_tail_call()</code> to tail-call the bpf program which the <code>index</code> entry in <a href="bcc-documents/reference_guide.html#10-bpf_prog_array">BPF_PROG_ARRAY</a> points to. A tail-call is different from the normal call. It reuses the current stack frame after jumping to another bpf program and never goes back. If the <code>index</code> entry is empty, it won't jump anywhere and the program execution continues as normal.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_PROG_ARRAY(prog_array, 10);
int tail_call(void *ctx) {
bpf_trace_printk(&quot;Tail-call\n&quot;);
return 0;
}
int do_tail_call(void *ctx) {
bpf_trace_printk(&quot;Original program\n&quot;);
prog_array.call(ctx, 2);
return 0;
}
</code></pre>
<pre><code class="language-Python">b = BPF(src_file=&quot;example.c&quot;)
tail_fn = b.load_func(&quot;tail_call&quot;, BPF.KPROBE)
prog_array = b.get_table(&quot;prog_array&quot;)
prog_array[c_int(2)] = c_int(tail_fn.fd)
b.attach_kprobe(event=&quot;some_kprobe_event&quot;, fn_name=&quot;do_tail_call&quot;)
</code></pre>
<p>This assigns <code>tail_call()</code> to <code>prog_array[2]</code>. In the end of <code>do_tail_call()</code>, <code>prog_array.call(ctx, 2)</code> tail-calls <code>tail_call()</code> and executes it.</p>
<p><strong>NOTE:</strong> To prevent infinite loop, the maximum number of tail-calls is 32 (<a href="https://github.com/torvalds/linux/search?l=C&amp;q=MAX_TAIL_CALL_CNT+path%3Ainclude%2Flinux&amp;type=Code"><code>MAX_TAIL_CALL_CNT</code></a>).</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?l=C&amp;q=call+path%3Aexamples&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?l=C&amp;q=call+path%3Atests&amp;type=Code">search /tests</a></p>
<h3 id="28-mapredirect_map"><a class="header" href="#28-mapredirect_map">28. map.redirect_map()</a></h3>
<p>Syntax: <code>int map.redirect_map(int index, int flags)</code></p>
<p>This redirects the incoming packets based on the <code>index</code> entry. If the map is <a href="bcc-documents/reference_guide.html#11-bpf_devmap">BPF_DEVMAP</a>, the packet will be sent to the transmit queue of the network interface that the entry points to. If the map is <a href="bcc-documents/reference_guide.html#12-bpf_cpumap">BPF_CPUMAP</a>, the packet will be sent to the ring buffer of the <code>index</code> CPU and be processed by the CPU later. If the map is <a href="bcc-documents/reference_guide.html#13-bpf_xskmap">BPF_XSKMAP</a>, the packet will be sent to the AF_XDP socket attached to the queue.</p>
<p>If the packet is redirected successfully, the function will return XDP_REDIRECT. Otherwise, it will return XDP_ABORTED to discard the packet.</p>
<p>For example:</p>
<pre><code class="language-C">BPF_DEVMAP(devmap, 1);
int redirect_example(struct xdp_md *ctx) {
return devmap.redirect_map(0, 0);
}
int xdp_dummy(struct xdp_md *ctx) {
return XDP_PASS;
}
</code></pre>
<pre><code class="language-Python">ip = pyroute2.IPRoute()
idx = ip.link_lookup(ifname=&quot;eth1&quot;)[0]
b = bcc.BPF(src_file=&quot;example.c&quot;)
devmap = b.get_table(&quot;devmap&quot;)
devmap[c_uint32(0)] = c_int(idx)
in_fn = b.load_func(&quot;redirect_example&quot;, BPF.XDP)
out_fn = b.load_func(&quot;xdp_dummy&quot;, BPF.XDP)
b.attach_xdp(&quot;eth0&quot;, in_fn, 0)
b.attach_xdp(&quot;eth1&quot;, out_fn, 0)
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?l=C&amp;q=redirect_map+path%3Aexamples&amp;type=Code">search /examples</a>,</p>
<h3 id="29-mappush"><a class="header" href="#29-mappush">29. map.push()</a></h3>
<p>Syntax: <code>int map.push(&amp;val, int flags)</code></p>
<p>Push an element onto a Stack or Queue table.
Passing BPF_EXIST as a flag causes the Queue or Stack to discard the oldest element if it is full.
Returns 0 on success, negative error on failure.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=push+path%3Atests&amp;type=Code">search /tests</a>,</p>
<h3 id="30-mappop"><a class="header" href="#30-mappop">30. map.pop()</a></h3>
<p>Syntax: <code>int map.pop(&amp;val)</code></p>
<p>Pop an element from a Stack or Queue table. <code>*val</code> is populated with the result.
Unlike peeking, popping removes the element.
Returns 0 on success, negative error on failure.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=pop+path%3Atests&amp;type=Code">search /tests</a>,</p>
<h3 id="31-mappeek"><a class="header" href="#31-mappeek">31. map.peek()</a></h3>
<p>Syntax: <code>int map.peek(&amp;val)</code></p>
<p>Peek an element at the head of a Stack or Queue table. <code>*val</code> is populated with the result.
Unlike popping, peeking does not remove the element.
Returns 0 on success, negative error on failure.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=peek+path%3Atests&amp;type=Code">search /tests</a>,</p>
<h3 id="32-mapsock_hash_update"><a class="header" href="#32-mapsock_hash_update">32. map.sock_hash_update()</a></h3>
<p>Syntax: <code>int map.sock_hash_update(struct bpf_sock_ops *skops, &amp;key, int flags)</code></p>
<p>Add an entry to, or update a sockhash map referencing sockets. The skops is used as a new value for the entry associated to key. flags is one of:</p>
<pre><code class="language-sh">BPF_NOEXIST: The entry for key must not exist in the map.
BPF_EXIST: The entry for key must already exist in the map.
BPF_ANY: No condition on the existence of the entry for key.
</code></pre>
<p>If the map has eBPF programs (parser and verdict), those will be inherited by the socket being added. If the socket is already attached to eBPF programs, this results in an error.</p>
<p>Return 0 on success, or a negative error in case of failure.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=sock_hash_update+path%3Atests&amp;type=Code">search /tests</a>,</p>
<h3 id="33-mapmsg_redirect_hash"><a class="header" href="#33-mapmsg_redirect_hash">33. map.msg_redirect_hash()</a></h3>
<p>Syntax: <code>int map.msg_redirect_hash(struct sk_msg_buff *msg, void *key, u64 flags)</code></p>
<p>This helper is used in programs implementing policies at the socket level. If the message msg is allowed to pass (i.e. if the verdict eBPF program returns SK_PASS), redirect it to the socket referenced by map (of type BPF_MAP_TYPE_SOCKHASH) using hash key. Both ingress and egress interfaces can be used for redirection. The BPF_F_INGRESS value in flags is used to make the distinction (ingress path is selected if the flag is present, egress path otherwise). This is the only flag supported for now.</p>
<p>Return SK_PASS on success, or SK_DROP on error.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=msg_redirect_hash+path%3Atests&amp;type=Code">search /tests</a>,</p>
<h3 id="34-mapsk_redirect_hash"><a class="header" href="#34-mapsk_redirect_hash">34. map.sk_redirect_hash()</a></h3>
<p>Syntax: <code>int map.sk_redirect_hash(struct sk_buff *skb, void *key, u64 flags)</code></p>
<p>This helper is used in programs implementing policies at the skb socket level. If the sk_buff skb is allowed to pass (i.e. if the verdict eBPF program returns SK_PASS), redirect it to the socket referenced by map (of type BPF_MAP_TYPE_SOCKHASH) using hash key. Both ingress and egress interfaces can be used for redirection. The BPF_F_INGRESS value in flags is used to make the distinction (ingress path is selected if the flag is present, egress otherwise). This is the only flag supported for now.</p>
<p>Return SK_PASS on success, or SK_DROP on error.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=sk_redirect_hash+path%3Atests&amp;type=Code">search /tests</a>,</p>
<h2 id="licensing"><a class="header" href="#licensing">Licensing</a></h2>
<p>Depending on which <a href="bcc-documents/kernel-versions.html#helpers">BPF helpers</a> are used, a GPL-compatible license is required.</p>
<p>The special BCC macro <code>BPF_LICENSE</code> specifies the license of the BPF program. You can set the license as a comment in your source code, but the kernel has a special interface to specify it programmatically. If you need to use GPL-only helpers, it is recommended to specify the macro in your C code so that the kernel can understand it:</p>
<pre><code class="language-C">// SPDX-License-Identifier: GPL-2.0+
#define BPF_LICENSE GPL
</code></pre>
<p>Otherwise, the kernel may reject loading your program (see the <a href="bcc-documents/reference_guide.html#2-cannot-call-gpl-only-function-from-proprietary-program">error description</a> below). Note that it supports multiple words and quotes are not necessary:</p>
<pre><code class="language-C">// SPDX-License-Identifier: GPL-2.0+ OR BSD-2-Clause
#define BPF_LICENSE Dual BSD/GPL
</code></pre>
<p>Check the <a href="bcc-documents/kernel-versions.html#helpers">BPF helpers reference</a> to see which helpers are GPL-only and what the kernel understands as GPL-compatible.</p>
<p><strong>If the macro is not specified, BCC will automatically define the license of the program as GPL.</strong></p>
<h2 id="rewriter"><a class="header" href="#rewriter">Rewriter</a></h2>
<p>One of jobs for rewriter is to turn implicit memory accesses to explicit ones using kernel helpers. Recent kernel introduced a config option ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE which will be set for architectures who user address space and kernel address are disjoint. x86 and arm has this config option set while s390 does not. If ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE is not set, the bpf old helper <code>bpf_probe_read()</code> will not be available. Some existing users may have implicit memory accesses to access user memory, so using <code>bpf_probe_read_kernel()</code> will cause their application to fail. Therefore, for non-s390, the rewriter will use <code>bpf_probe_read()</code> for these implicit memory accesses. For s390, <code>bpf_probe_read_kernel()</code> is used as default and users should use <code>bpf_probe_read_user()</code> explicitly when accessing user memories.</p>
<h1 id="bcc-python"><a class="header" href="#bcc-python">bcc Python</a></h1>
<h2 id="initialization"><a class="header" href="#initialization">Initialization</a></h2>
<p>Constructors.</p>
<h3 id="1-bpf"><a class="header" href="#1-bpf">1. BPF</a></h3>
<p>Syntax: <code>BPF({text=BPF_program | src_file=filename} [, usdt_contexts=[USDT_object, ...]] [, cflags=[arg1, ...]] [, debug=int])</code></p>
<p>Creates a BPF object. This is the main object for defining a BPF program, and interacting with its output.</p>
<p>Exactly one of <code>text</code> or <code>src_file</code> must be supplied (not both).</p>
<p>The <code>cflags</code> specifies additional arguments to be passed to the compiler, for example <code>-DMACRO_NAME=value</code> or <code>-I/include/path</code>. The arguments are passed as an array, with each element being an additional argument. Note that strings are not split on whitespace, so each argument must be a different element of the array, e.g. <code>[&quot;-include&quot;, &quot;header.h&quot;]</code>.</p>
<p>The <code>debug</code> flags control debug output, and can be or'ed together:</p>
<ul>
<li><code>DEBUG_LLVM_IR = 0x1</code> compiled LLVM IR</li>
<li><code>DEBUG_BPF = 0x2</code> loaded BPF bytecode and register state on branches</li>
<li><code>DEBUG_PREPROCESSOR = 0x4</code> pre-processor result</li>
<li><code>DEBUG_SOURCE = 0x8</code> ASM instructions embedded with source</li>
<li><code>DEBUG_BPF_REGISTER_STATE = 0x10</code> register state on all instructions in addition to DEBUG_BPF</li>
<li><code>DEBUG_BTF = 0x20</code> print the messages from the <code>libbpf</code> library.</li>
</ul>
<p>Examples:</p>
<pre><code class="language-Python"># define entire BPF program in one line:
BPF(text='int do_trace(void *ctx) { bpf_trace_printk(&quot;hit!\\n&quot;); return 0; }');
# define program as a variable:
prog = &quot;&quot;&quot;
int hello(void *ctx) {
bpf_trace_printk(&quot;Hello, World!\\n&quot;);
return 0;
}
&quot;&quot;&quot;
b = BPF(text=prog)
# source a file:
b = BPF(src_file = &quot;vfsreadlat.c&quot;)
# include a USDT object:
u = USDT(pid=int(pid))
[...]
b = BPF(text=bpf_text, usdt_contexts=[u])
# add include paths:
u = BPF(text=prog, cflags=[&quot;-I/path/to/include&quot;])
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=BPF+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=BPF+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="2-usdt"><a class="header" href="#2-usdt">2. USDT</a></h3>
<p>Syntax: <code>USDT({pid=pid | path=path})</code></p>
<p>Creates an object to instrument User Statically-Defined Tracing (USDT) probes. Its primary method is <code>enable_probe()</code>.</p>
<p>Arguments:</p>
<ul>
<li>pid: attach to this process ID.</li>
<li>path: instrument USDT probes from this binary path.</li>
</ul>
<p>Examples:</p>
<pre><code class="language-Python"># include a USDT object:
u = USDT(pid=int(pid))
[...]
b = BPF(text=bpf_text, usdt_contexts=[u])
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=USDT+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=USDT+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h2 id="events"><a class="header" href="#events">Events</a></h2>
<h3 id="1-attach_kprobe"><a class="header" href="#1-attach_kprobe">1. attach_kprobe()</a></h3>
<p>Syntax: <code>BPF.attach_kprobe(event=&quot;event&quot;, fn_name=&quot;name&quot;)</code></p>
<p>Instruments the kernel function <code>event()</code> using kernel dynamic tracing of the function entry, and attaches our C defined function <code>name()</code> to be called when the kernel function is called.</p>
<p>For example:</p>
<pre><code class="language-Python">b.attach_kprobe(event=&quot;sys_clone&quot;, fn_name=&quot;do_trace&quot;)
</code></pre>
<p>This will instrument the kernel <code>sys_clone()</code> function, which will then run our BPF defined <code>do_trace()</code> function each time it is called.</p>
<p>You can call attach_kprobe() more than once, and attach your BPF function to multiple kernel functions.
You can also call attach_kprobe() more than once to attach multiple BPF functions to the same kernel function.</p>
<p>See the previous kprobes section for how to instrument arguments from BPF.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=attach_kprobe+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=attach_kprobe+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="2-attach_kretprobe"><a class="header" href="#2-attach_kretprobe">2. attach_kretprobe()</a></h3>
<p>Syntax: <code>BPF.attach_kretprobe(event=&quot;event&quot;, fn_name=&quot;name&quot; [, maxactive=int])</code></p>
<p>Instruments the return of the kernel function <code>event()</code> using kernel dynamic tracing of the function return, and attaches our C defined function <code>name()</code> to be called when the kernel function returns.</p>
<p>For example:</p>
<pre><code class="language-Python">b.attach_kretprobe(event=&quot;vfs_read&quot;, fn_name=&quot;do_return&quot;)
</code></pre>
<p>This will instrument the kernel <code>vfs_read()</code> function, which will then run our BPF defined <code>do_return()</code> function each time it is called.</p>
<p>You can call attach_kretprobe() more than once, and attach your BPF function to multiple kernel function returns.
You can also call attach_kretprobe() more than once to attach multiple BPF functions to the same kernel function return.</p>
<p>When a kretprobe is installed on a kernel function, there is a limit on how many parallel calls it can catch. You can change that limit with <code>maxactive</code>. See the kprobes documentation for its default value.</p>
<p>See the previous kretprobes section for how to instrument the return value from BPF.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=attach_kretprobe+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=attach_kretprobe+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="3-attach_tracepoint"><a class="header" href="#3-attach_tracepoint">3. attach_tracepoint()</a></h3>
<p>Syntax: <code>BPF.attach_tracepoint(tp=&quot;tracepoint&quot;, fn_name=&quot;name&quot;)</code></p>
<p>Instruments the kernel tracepoint described by <code>tracepoint</code>, and when hit, runs the BPF function <code>name()</code>.</p>
<p>This is an explicit way to instrument tracepoints. The <code>TRACEPOINT_PROBE</code> syntax, covered in the earlier tracepoints section, is an alternate method with the advantage of auto-declaring an <code>args</code> struct containing the tracepoint arguments. With <code>attach_tracepoint()</code>, the tracepoint arguments need to be declared in the BPF program.</p>
<p>For example:</p>
<pre><code class="language-Python"># define BPF program
bpf_text = &quot;&quot;&quot;
#include &lt;uapi/linux/ptrace.h&gt;
struct urandom_read_args {
// from /sys/kernel/debug/tracing/events/random/urandom_read/format
u64 __unused__;
u32 got_bits;
u32 pool_left;
u32 input_left;
};
int printarg(struct urandom_read_args *args) {
bpf_trace_printk(&quot;%d\\n&quot;, args-&gt;got_bits);
return 0;
};
&quot;&quot;&quot;
# load BPF program
b = BPF(text=bpf_text)
b.attach_tracepoint(&quot;random:urandom_read&quot;, &quot;printarg&quot;)
</code></pre>
<p>Notice how the first argument to <code>printarg()</code> is now our defined struct.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/blob/a4159da8c4ea8a05a3c6e402451f530d6e5a8b41/examples/tracing/urandomread-explicit.py#L41">code</a>,
<a href="https://github.com/iovisor/bcc/search?q=attach_tracepoint+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=attach_tracepoint+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="4-attach_uprobe"><a class="header" href="#4-attach_uprobe">4. attach_uprobe()</a></h3>
<p>Syntax: <code>BPF.attach_uprobe(name=&quot;location&quot;, sym=&quot;symbol&quot;, fn_name=&quot;name&quot; [, sym_off=int])</code>, <code>BPF.attach_uprobe(name=&quot;location&quot;, sym_re=&quot;regex&quot;, fn_name=&quot;name&quot;)</code>, <code>BPF.attach_uprobe(name=&quot;location&quot;, addr=int, fn_name=&quot;name&quot;)</code></p>
<p>Instruments the user-level function <code>symbol()</code> from either the library or binary named by <code>location</code> using user-level dynamic tracing of the function entry, and attach our C defined function <code>name()</code> to be called whenever the user-level function is called. If <code>sym_off</code> is given, the function is attached to the offset within the symbol.</p>
<p>The real address <code>addr</code> may be supplied in place of <code>sym</code>, in which case <code>sym</code> must be set to its default value. If the file is a non-PIE executable, <code>addr</code> must be a virtual address, otherwise it must be an offset relative to the file load address.</p>
<p>Instead of a symbol name, a regular expression can be provided in <code>sym_re</code>. The uprobe will then attach to symbols that match the provided regular expression.</p>
<p>Libraries can be given in the name argument without the lib prefix, or with the full path (/usr/lib/...). Binaries can be given only with the full path (/bin/sh).</p>
<p>For example:</p>
<pre><code class="language-Python">b.attach_uprobe(name=&quot;c&quot;, sym=&quot;strlen&quot;, fn_name=&quot;count&quot;)
</code></pre>
<p>This will instrument <code>strlen()</code> function from libc, and call our BPF function <code>count()</code> when it is called. Note how the &quot;lib&quot; in &quot;libc&quot; is not necessary to specify.</p>
<p>Other examples:</p>
<pre><code class="language-Python">b.attach_uprobe(name=&quot;c&quot;, sym=&quot;getaddrinfo&quot;, fn_name=&quot;do_entry&quot;)
b.attach_uprobe(name=&quot;/usr/bin/python&quot;, sym=&quot;main&quot;, fn_name=&quot;do_main&quot;)
</code></pre>
<p>You can call attach_uprobe() more than once, and attach your BPF function to multiple user-level functions.</p>
<p>See the previous uprobes section for how to instrument arguments from BPF.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=attach_uprobe+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=attach_uprobe+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="5-attach_uretprobe"><a class="header" href="#5-attach_uretprobe">5. attach_uretprobe()</a></h3>
<p>Syntax: <code>BPF.attach_uretprobe(name=&quot;location&quot;, sym=&quot;symbol&quot;, fn_name=&quot;name&quot;)</code></p>
<p>Instruments the return of the user-level function <code>symbol()</code> from either the library or binary named by <code>location</code> using user-level dynamic tracing of the function return, and attach our C defined function <code>name()</code> to be called whenever the user-level function returns.</p>
<p>For example:</p>
<pre><code class="language-Python">b.attach_uretprobe(name=&quot;c&quot;, sym=&quot;strlen&quot;, fn_name=&quot;count&quot;)
</code></pre>
<p>This will instrument <code>strlen()</code> function from libc, and call our BPF function <code>count()</code> when it returns.</p>
<p>Other examples:</p>
<pre><code class="language-Python">b.attach_uretprobe(name=&quot;c&quot;, sym=&quot;getaddrinfo&quot;, fn_name=&quot;do_return&quot;)
b.attach_uretprobe(name=&quot;/usr/bin/python&quot;, sym=&quot;main&quot;, fn_name=&quot;do_main&quot;)
</code></pre>
<p>You can call attach_uretprobe() more than once, and attach your BPF function to multiple user-level functions.</p>
<p>See the previous uretprobes section for how to instrument the return value from BPF.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=attach_uretprobe+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=attach_uretprobe+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="6-usdtenable_probe"><a class="header" href="#6-usdtenable_probe">6. USDT.enable_probe()</a></h3>
<p>Syntax: <code>USDT.enable_probe(probe=probe, fn_name=name)</code></p>
<p>Attaches a BPF C function <code>name</code> to the USDT probe <code>probe</code>.</p>
<p>Example:</p>
<pre><code class="language-Python"># enable USDT probe from given PID
u = USDT(pid=int(pid))
u.enable_probe(probe=&quot;http__server__request&quot;, fn_name=&quot;do_trace&quot;)
</code></pre>
<p>To check if your binary has USDT probes, and what they are, you can run <code>readelf -n binary</code> and check the stap debug section.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=enable_probe+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=enable_probe+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="7-attach_raw_tracepoint"><a class="header" href="#7-attach_raw_tracepoint">7. attach_raw_tracepoint()</a></h3>
<p>Syntax: <code>BPF.attach_raw_tracepoint(tp=&quot;tracepoint&quot;, fn_name=&quot;name&quot;)</code></p>
<p>Instruments the kernel raw tracepoint described by <code>tracepoint</code> (<code>event</code> only, no <code>category</code>), and when hit, runs the BPF function <code>name()</code>.</p>
<p>This is an explicit way to instrument tracepoints. The <code>RAW_TRACEPOINT_PROBE</code> syntax, covered in the earlier raw tracepoints section, is an alternate method.</p>
<p>For example:</p>
<pre><code class="language-Python">b.attach_raw_tracepoint(&quot;sched_switch&quot;, &quot;do_trace&quot;)
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=attach_raw_tracepoint+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="8-attach_raw_socket"><a class="header" href="#8-attach_raw_socket">8. attach_raw_socket()</a></h3>
<p>Syntax: <code>BPF.attach_raw_socket(fn, dev)</code></p>
<p>Attaches a BPF function to the specified network interface.</p>
<p>The <code>fn</code> must be the type of <code>BPF.function</code> and the bpf_prog type needs to be <code>BPF_PROG_TYPE_SOCKET_FILTER</code> (<code>fn=BPF.load_func(func_name, BPF.SOCKET_FILTER)</code>)</p>
<p><code>fn.sock</code> is a non-blocking raw socket that was created and bound to <code>dev</code>.</p>
<p>All network packets processed by <code>dev</code> are copied to the <code>recv-q</code> of <code>fn.sock</code> after being processed by bpf_prog. Try to recv packet form <code>fn.sock</code> with rev/recvfrom/recvmsg. Note that if the <code>recv-q</code> is not read in time after the <code>recv-q</code> is full, the copied packets will be discarded.</p>
<p>We can use this feature to capture network packets just like <code>tcpdump</code>.</p>
<p>We can use <code>ss --bpf --packet -p</code> to observe <code>fn.sock</code>.</p>
<p>Example:</p>
<pre><code class="language-Python">BPF.attach_raw_socket(bpf_func, ifname)
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=attach_raw_socket+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a></p>
<h3 id="9-attach_xdp"><a class="header" href="#9-attach_xdp">9. attach_xdp()</a></h3>
<p>Syntax: <code>BPF.attach_xdp(dev=&quot;device&quot;, fn=b.load_func(&quot;fn_name&quot;,BPF.XDP), flags)</code></p>
<p>Instruments the network driver described by <code>dev</code> , and then receives the packet, run the BPF function <code>fn_name()</code> with flags.</p>
<p>Here is a list of optional flags.</p>
<pre><code class="language-Python"># from xdp_flags uapi/linux/if_link.h
XDP_FLAGS_UPDATE_IF_NOEXIST = (1 &lt;&lt; 0)
XDP_FLAGS_SKB_MODE = (1 &lt;&lt; 1)
XDP_FLAGS_DRV_MODE = (1 &lt;&lt; 2)
XDP_FLAGS_HW_MODE = (1 &lt;&lt; 3)
XDP_FLAGS_REPLACE = (1 &lt;&lt; 4)
</code></pre>
<p>You can use flags like this <code>BPF.attach_xdp(dev=&quot;device&quot;, fn=b.load_func(&quot;fn_name&quot;,BPF.XDP), flags=BPF.XDP_FLAGS_UPDATE_IF_NOEXIST)</code></p>
<p>The default value of flags is 0. This means if there is no xdp program with <code>device</code>, the fn will run with that device. If there is an xdp program running with device, the old program will be replaced with new fn program.</p>
<p>Currently, bcc does not support XDP_FLAGS_REPLACE flag. The following are the descriptions of other flags.</p>
<h4 id="1-xdp_flags_update_if_noexist"><a class="header" href="#1-xdp_flags_update_if_noexist">1. XDP_FLAGS_UPDATE_IF_NOEXIST</a></h4>
<p>If an XDP program is already attached to the specified driver, attaching the XDP program again will fail.</p>
<h4 id="2-xdp_flags_skb_mode"><a class="header" href="#2-xdp_flags_skb_mode">2. XDP_FLAGS_SKB_MODE</a></h4>
<p>Driver doesn’t have support for XDP, but the kernel fakes it.
XDP program works, but there’s no real performance benefit because packets are handed to kernel stack anyways which then emulates XDP – this is usually supported with generic network drivers used in home computers, laptops, and virtualized HW.</p>
<h4 id="3-xdp_flags_drv_mode"><a class="header" href="#3-xdp_flags_drv_mode">3. XDP_FLAGS_DRV_MODE</a></h4>
<p>A driver has XDP support and can hand then to XDP without kernel stack interaction – Few drivers can support it and those are usually for enterprise HW.</p>
<h4 id="4-xdp_flags_hw_mode"><a class="header" href="#4-xdp_flags_hw_mode">4. XDP_FLAGS_HW_MODE</a></h4>
<p>XDP can be loaded and executed directly on the NIC – just a handful of NICs can do that.</p>
<p>For example:</p>
<pre><code class="language-Python">b.attach_xdp(dev=&quot;ens1&quot;, fn=b.load_func(&quot;do_xdp&quot;, BPF.XDP))
</code></pre>
<p>This will instrument the network device <code>ens1</code> , which will then run our BPF defined <code>do_xdp()</code> function each time it receives packets.</p>
<p>Don't forget to call <code>b.remove_xdp(&quot;ens1&quot;)</code> at the end!</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=attach_xdp+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=attach_xdp+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="10-attach_func"><a class="header" href="#10-attach_func">10. attach_func()</a></h3>
<p>Syntax: <code>BPF.attach_func(fn, attachable_fd, attach_type [, flags])</code></p>
<p>Attaches a BPF function of the specified type to a particular <code>attachable_fd</code>. if the <code>attach_type</code> is <code>BPF_FLOW_DISSECTOR</code>, the function is expected to attach to current net namespace and <code>attachable_fd</code> must be 0.</p>
<p>For example:</p>
<pre><code class="language-Python">b.attach_func(fn, cgroup_fd, BPFAttachType.CGROUP_SOCK_OPS)
b.attach_func(fn, map_fd, BPFAttachType.SK_MSG_VERDICT)
</code></pre>
<p>Note. When attached to &quot;global&quot; hooks (xdp, tc, lwt, cgroup). If the &quot;BPF function&quot; is no longer needed after the program terminates, be sure to call <code>detach_func</code> when the program exits.</p>
<p>Examples in situ:</p>
<p><a href="https://github.com/iovisor/bcc/search?q=attach_func+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,</p>
<h3 id="11-detach_func"><a class="header" href="#11-detach_func">11. detach_func()</a></h3>
<p>Syntax: <code>BPF.detach_func(fn, attachable_fd, attach_type)</code></p>
<p>Detaches a BPF function of the specified type.</p>
<p>For example:</p>
<pre><code class="language-Python">b.detach_func(fn, cgroup_fd, BPFAttachType.CGROUP_SOCK_OPS)
b.detach_func(fn, map_fd, BPFAttachType.SK_MSG_VERDICT)
</code></pre>
<p>Examples in situ:</p>
<p><a href="https://github.com/iovisor/bcc/search?q=detach_func+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,</p>
<h3 id="12-detach_kprobe"><a class="header" href="#12-detach_kprobe">12. detach_kprobe()</a></h3>
<p>Syntax: <code>BPF.detach_kprobe(event=&quot;event&quot;, fn_name=&quot;name&quot;)</code></p>
<p>Detach a kprobe handler function of the specified event.</p>
<p>For example:</p>
<pre><code class="language-Python">b.detach_kprobe(event=&quot;__page_cache_alloc&quot;, fn_name=&quot;trace_func_entry&quot;)
</code></pre>
<h3 id="13-detach_kretprobe"><a class="header" href="#13-detach_kretprobe">13. detach_kretprobe()</a></h3>
<p>Syntax: <code>BPF.detach_kretprobe(event=&quot;event&quot;, fn_name=&quot;name&quot;)</code></p>
<p>Detach a kretprobe handler function of the specified event.</p>
<p>For example:</p>
<pre><code class="language-Python">b.detach_kretprobe(event=&quot;__page_cache_alloc&quot;, fn_name=&quot;trace_func_return&quot;)
</code></pre>
<h2 id="debug-output"><a class="header" href="#debug-output">Debug Output</a></h2>
<h3 id="1-trace_print"><a class="header" href="#1-trace_print">1. trace_print()</a></h3>
<p>Syntax: <code>BPF.trace_print(fmt=&quot;fields&quot;)</code></p>
<p>This method continually reads the globally shared /sys/kernel/debug/tracing/trace_pipe file and prints its contents. This file can be written to via BPF and the bpf_trace_printk() function, however, that method has limitations, including a lack of concurrent tracing support. The BPF_PERF_OUTPUT mechanism, covered earlier, is preferred.</p>
<p>Arguments:</p>
<ul>
<li><code>fmt</code>: optional, and can contain a field formatting string. It defaults to <code>None</code>.</li>
</ul>
<p>Examples:</p>
<pre><code class="language-Python"># print trace_pipe output as-is:
b.trace_print()
# print PID and message:
b.trace_print(fmt=&quot;{1} {5}&quot;)
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=trace_print+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=trace_print+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="2-trace_fields"><a class="header" href="#2-trace_fields">2. trace_fields()</a></h3>
<p>Syntax: <code>BPF.trace_fields(nonblocking=False)</code></p>
<p>This method reads one line from the globally shared /sys/kernel/debug/tracing/trace_pipe file and returns it as fields. This file can be written to via BPF and the bpf_trace_printk() function, however, that method has limitations, including a lack of concurrent tracing support. The BPF_PERF_OUTPUT mechanism, covered earlier, is preferred.</p>
<p>Arguments:</p>
<ul>
<li><code>nonblocking</code>: optional, defaults to <code>False</code>. When set to <code>True</code>, the program will not block waiting for input.</li>
</ul>
<p>Examples:</p>
<pre><code class="language-Python">while 1:
try:
(task, pid, cpu, flags, ts, msg) = b.trace_fields()
except ValueError:
continue
[...]
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=trace_fields+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=trace_fields+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h2 id="output-apis"><a class="header" href="#output-apis">Output APIs</a></h2>
<p>Normal output from a BPF program is either:</p>
<ul>
<li>per-event: using PERF_EVENT_OUTPUT, open_perf_buffer(), and perf_buffer_poll().</li>
<li>map summary: using items(), or print_log2_hist(), covered in the Maps section.</li>
</ul>
<h3 id="1-perf_buffer_poll"><a class="header" href="#1-perf_buffer_poll">1. perf_buffer_poll()</a></h3>
<p>Syntax: <code>BPF.perf_buffer_poll(timeout=T)</code></p>
<p>This polls from all open perf ring buffers, calling the callback function that was provided when calling open_perf_buffer for each entry.</p>
<p>The timeout parameter is optional and measured in milliseconds. In its absence, polling continues indefinitely.</p>
<p>Example:</p>
<pre><code class="language-Python"># loop with callback to print_event
b[&quot;events&quot;].open_perf_buffer(print_event)
while 1:
try:
b.perf_buffer_poll()
except KeyboardInterrupt:
exit();
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/blob/v0.9.0/examples/tracing/hello_perf_output.py#L55">code</a>,
<a href="https://github.com/iovisor/bcc/search?q=perf_buffer_poll+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=perf_buffer_poll+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="2-ring_buffer_poll"><a class="header" href="#2-ring_buffer_poll">2. ring_buffer_poll()</a></h3>
<p>Syntax: <code>BPF.ring_buffer_poll(timeout=T)</code></p>
<p>This polls from all open ringbuf ring buffers, calling the callback function that was provided when calling open_ring_buffer for each entry.</p>
<p>The timeout parameter is optional and measured in milliseconds. In its absence, polling continues until
there is no more data or the callback returns a negative value.</p>
<p>Example:</p>
<pre><code class="language-Python"># loop with callback to print_event
b[&quot;events&quot;].open_ring_buffer(print_event)
while 1:
try:
b.ring_buffer_poll(30)
except KeyboardInterrupt:
exit();
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=ring_buffer_poll+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,</p>
<h3 id="3-ring_buffer_consume"><a class="header" href="#3-ring_buffer_consume">3. ring_buffer_consume()</a></h3>
<p>Syntax: <code>BPF.ring_buffer_consume()</code></p>
<p>This consumes from all open ringbuf ring buffers, calling the callback function that was provided when calling open_ring_buffer for each entry.</p>
<p>Unlike <code>ring_buffer_poll</code>, this method <strong>does not poll for data</strong> before attempting to consume.
This reduces latency at the expense of higher CPU consumption. If you are unsure which to use,
use <code>ring_buffer_poll</code>.</p>
<p>Example:</p>
<pre><code class="language-Python"># loop with callback to print_event
b[&quot;events&quot;].open_ring_buffer(print_event)
while 1:
try:
b.ring_buffer_consume()
except KeyboardInterrupt:
exit();
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=ring_buffer_consume+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,</p>
<h2 id="map-apis"><a class="header" href="#map-apis">Map APIs</a></h2>
<p>Maps are BPF data stores, and are used in bcc to implement a table, and then higher level objects on top of tables, including hashes and histograms.</p>
<h3 id="1-get_table"><a class="header" href="#1-get_table">1. get_table()</a></h3>
<p>Syntax: <code>BPF.get_table(name)</code></p>
<p>Returns a table object. This is no longer used, as tables can now be read as items from BPF. Eg: <code>BPF[name]</code>.</p>
<p>Examples:</p>
<pre><code class="language-Python">counts = b.get_table(&quot;counts&quot;)
counts = b[&quot;counts&quot;]
</code></pre>
<p>These are equivalent.</p>
<h3 id="2-open_perf_buffer"><a class="header" href="#2-open_perf_buffer">2. open_perf_buffer()</a></h3>
<p>Syntax: <code>table.open_perf_buffers(callback, page_cnt=N, lost_cb=None)</code></p>
<p>This operates on a table as defined in BPF as BPF_PERF_OUTPUT(), and associates the callback Python function <code>callback</code> to be called when data is available in the perf ring buffer. This is part of the recommended mechanism for transferring per-event data from kernel to user space. The size of the perf ring buffer can be specified via the <code>page_cnt</code> parameter, which must be a power of two number of pages and defaults to 8. If the callback is not processing data fast enough, some submitted data may be lost. <code>lost_cb</code> will be called to log / monitor the lost count. If <code>lost_cb</code> is the default <code>None</code> value, it will just print a line of message to <code>stderr</code>.</p>
<p>Example:</p>
<pre><code class="language-Python"># process event
def print_event(cpu, data, size):
event = ct.cast(data, ct.POINTER(Data)).contents
[...]
# loop with callback to print_event
b[&quot;events&quot;].open_perf_buffer(print_event)
while 1:
try:
b.perf_buffer_poll()
except KeyboardInterrupt:
exit()
</code></pre>
<p>Note that the data structure transferred will need to be declared in C in the BPF program. For example:</p>
<pre><code class="language-C">// define output data structure in C
struct data_t {
u32 pid;
u64 ts;
char comm[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(events);
[...]
</code></pre>
<p>In Python, you can either let bcc generate the data structure from C declaration automatically (recommended):</p>
<pre><code class="language-Python">def print_event(cpu, data, size):
event = b[&quot;events&quot;].event(data)
[...]
</code></pre>
<p>or define it manually:</p>
<pre><code class="language-Python"># define output data structure in Python
TASK_COMM_LEN = 16 # linux/sched.h
class Data(ct.Structure):
_fields_ = [(&quot;pid&quot;, ct.c_ulonglong),
(&quot;ts&quot;, ct.c_ulonglong),
(&quot;comm&quot;, ct.c_char * TASK_COMM_LEN)]
def print_event(cpu, data, size):
event = ct.cast(data, ct.POINTER(Data)).contents
[...]
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/blob/v0.9.0/examples/tracing/hello_perf_output.py#L52">code</a>,
<a href="https://github.com/iovisor/bcc/search?q=open_perf_buffer+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=open_perf_buffer+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="3-items"><a class="header" href="#3-items">3. items()</a></h3>
<p>Syntax: <code>table.items()</code></p>
<p>Returns an array of the keys in a table. This can be used with BPF_HASH maps to fetch, and iterate, over the keys.</p>
<p>Example:</p>
<pre><code class="language-Python"># print output
print(&quot;%10s %s&quot; % (&quot;COUNT&quot;, &quot;STRING&quot;))
counts = b.get_table(&quot;counts&quot;)
for k, v in sorted(counts.items(), key=lambda counts: counts[1].value):
print(&quot;%10d \&quot;%s\&quot;&quot; % (v.value, k.c.encode('string-escape')))
</code></pre>
<p>This example also uses the <code>sorted()</code> method to sort by value.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=items+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=items+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="4-values"><a class="header" href="#4-values">4. values()</a></h3>
<p>Syntax: <code>table.values()</code></p>
<p>Returns an array of the values in a table.</p>
<h3 id="5-clear"><a class="header" href="#5-clear">5. clear()</a></h3>
<p>Syntax: <code>table.clear()</code></p>
<p>Clears the table: deletes all entries.</p>
<p>Example:</p>
<pre><code class="language-Python"># print map summary every second:
while True:
time.sleep(1)
print(&quot;%-8s\n&quot; % time.strftime(&quot;%H:%M:%S&quot;), end=&quot;&quot;)
dist.print_log2_hist(sym + &quot; return:&quot;)
dist.clear()
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=clear+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=clear+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="6-items_lookup_and_delete_batch"><a class="header" href="#6-items_lookup_and_delete_batch">6. items_lookup_and_delete_batch()</a></h3>
<p>Syntax: <code>table.items_lookup_and_delete_batch()</code></p>
<p>Returns an array of the keys in a table with a single call to BPF syscall. This can be used with BPF_HASH maps to fetch, and iterate, over the keys. It also clears the table: deletes all entries.
You should rather use table.items_lookup_and_delete_batch() than table.items() followed by table.clear(). It requires kernel v5.6.</p>
<p>Example:</p>
<pre><code class="language-Python"># print call rate per second:
print(&quot;%9s-%9s-%8s-%9s&quot; % (&quot;PID&quot;, &quot;COMM&quot;, &quot;fname&quot;, &quot;counter&quot;))
while True:
for k, v in sorted(b['map'].items_lookup_and_delete_batch(), key=lambda kv: (kv[0]).pid):
print(&quot;%9s-%9s-%8s-%9d&quot; % (k.pid, k.comm, k.fname, v.counter))
sleep(1)
</code></pre>
<h3 id="7-items_lookup_batch"><a class="header" href="#7-items_lookup_batch">7. items_lookup_batch()</a></h3>
<p>Syntax: <code>table.items_lookup_batch()</code></p>
<p>Returns an array of the keys in a table with a single call to BPF syscall. This can be used with BPF_HASH maps to fetch, and iterate, over the keys.
You should rather use table.items_lookup_batch() than table.items(). It requires kernel v5.6.</p>
<p>Example:</p>
<pre><code class="language-Python"># print current value of map:
print(&quot;%9s-%9s-%8s-%9s&quot; % (&quot;PID&quot;, &quot;COMM&quot;, &quot;fname&quot;, &quot;counter&quot;))
while True:
for k, v in sorted(b['map'].items_lookup_batch(), key=lambda kv: (kv[0]).pid):
print(&quot;%9s-%9s-%8s-%9d&quot; % (k.pid, k.comm, k.fname, v.counter))
</code></pre>
<h3 id="8-items_delete_batch"><a class="header" href="#8-items_delete_batch">8. items_delete_batch()</a></h3>
<p>Syntax: <code>table.items_delete_batch(keys)</code></p>
<p>It clears all entries of a BPF_HASH map when keys is None. It is more efficient than table.clear() since it generates only one system call. You can delete a subset of a map by giving an array of keys as parameter. Those keys and their associated values will be deleted. It requires kernel v5.6.</p>
<p>Arguments:</p>
<ul>
<li>keys is optional and by default is None.</li>
</ul>
<h3 id="9-items_update_batch"><a class="header" href="#9-items_update_batch">9. items_update_batch()</a></h3>
<p>Syntax: <code>table.items_update_batch(keys, values)</code></p>
<p>Update all the provided keys with new values. The two arguments must be the same length and within the map limits (between 1 and the maximum entries). It requires kernel v5.6.</p>
<p>Arguments:</p>
<ul>
<li>keys is the list of keys to be updated</li>
<li>values is the list containing the new values.</li>
</ul>
<h3 id="10-print_log2_hist"><a class="header" href="#10-print_log2_hist">10. print_log2_hist()</a></h3>
<p>Syntax: <code>table.print_log2_hist(val_type=&quot;value&quot;, section_header=&quot;Bucket ptr&quot;, section_print_fn=None)</code></p>
<p>Prints a table as a log2 histogram in ASCII. The table must be stored as log2, which can be done using the BPF function <code>bpf_log2l()</code>.</p>
<p>Arguments:</p>
<ul>
<li>val_type: optional, column header.</li>
<li>section_header: if the histogram has a secondary key, multiple tables will print and section_header can be used as a header description for each.</li>
<li>section_print_fn: if section_print_fn is not None, it will be passed the bucket value.</li>
</ul>
<p>Example:</p>
<pre><code class="language-Python">b = BPF(text=&quot;&quot;&quot;
BPF_HISTOGRAM(dist);
int kprobe__blk_account_io_done(struct pt_regs *ctx, struct request *req)
{
dist.increment(bpf_log2l(req-&gt;__data_len / 1024));
return 0;
}
&quot;&quot;&quot;)
[...]
b[&quot;dist&quot;].print_log2_hist(&quot;kbytes&quot;)
</code></pre>
<p>Output:</p>
<pre><code class="language-sh"> kbytes : count distribution
0 -&gt; 1 : 3 | |
2 -&gt; 3 : 0 | |
4 -&gt; 7 : 211 |********** |
8 -&gt; 15 : 0 | |
16 -&gt; 31 : 0 | |
32 -&gt; 63 : 0 | |
64 -&gt; 127 : 1 | |
128 -&gt; 255 : 800 |**************************************|
</code></pre>
<p>This output shows a multi-modal distribution, with the largest mode of 128-&gt;255 kbytes and a count of 800.</p>
<p>This is an efficient way to summarize data, as the summarization is performed in-kernel, and only the count column is passed to user space.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=print_log2_hist+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=print_log2_hist+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="11-print_linear_hist"><a class="header" href="#11-print_linear_hist">11. print_linear_hist()</a></h3>
<p>Syntax: <code>table.print_linear_hist(val_type=&quot;value&quot;, section_header=&quot;Bucket ptr&quot;, section_print_fn=None)</code></p>
<p>Prints a table as a linear histogram in ASCII. This is intended to visualize small integer ranges, eg, 0 to 100.</p>
<p>Arguments:</p>
<ul>
<li>val_type: optional, column header.</li>
<li>section_header: if the histogram has a secondary key, multiple tables will print and section_header can be used as a header description for each.</li>
<li>section_print_fn: if section_print_fn is not None, it will be passed the bucket value.</li>
</ul>
<p>Example:</p>
<pre><code class="language-Python">b = BPF(text=&quot;&quot;&quot;
BPF_HISTOGRAM(dist);
int kprobe__blk_account_io_done(struct pt_regs *ctx, struct request *req)
{
dist.increment(req-&gt;__data_len / 1024);
return 0;
}
&quot;&quot;&quot;)
[...]
b[&quot;dist&quot;].print_linear_hist(&quot;kbytes&quot;)
</code></pre>
<p>Output:</p>
<pre><code class="language-sh"> kbytes : count distribution
0 : 3 |****** |
1 : 0 | |
2 : 0 | |
3 : 0 | |
4 : 19 |****************************************|
5 : 0 | |
6 : 0 | |
7 : 0 | |
8 : 4 |******** |
9 : 0 | |
10 : 0 | |
11 : 0 | |
12 : 0 | |
13 : 0 | |
14 : 0 | |
15 : 0 | |
16 : 2 |**** |
[...]
</code></pre>
<p>This is an efficient way to summarize data, as the summarization is performed in-kernel, and only the values in the count column are passed to user space.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=print_linear_hist+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=print_linear_hist+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="12-open_ring_buffer"><a class="header" href="#12-open_ring_buffer">12. open_ring_buffer()</a></h3>
<p>Syntax: <code>table.open_ring_buffer(callback, ctx=None)</code></p>
<p>This operates on a table as defined in BPF as BPF_RINGBUF_OUTPUT(), and associates the callback Python function <code>callback</code> to be called when data is available in the ringbuf ring buffer. This is part of the new (Linux 5.8+) recommended mechanism for transferring per-event data from kernel to user space. Unlike perf buffers, ringbuf sizes are specified within the BPF program, as part of the <code>BPF_RINGBUF_OUTPUT</code> macro. If the callback is not processing data fast enough, some submitted data may be lost. In this case, the events should be polled more frequently and/or the size of the ring buffer should be increased.</p>
<p>Example:</p>
<pre><code class="language-Python"># process event
def print_event(ctx, data, size):
event = ct.cast(data, ct.POINTER(Data)).contents
[...]
# loop with callback to print_event
b[&quot;events&quot;].open_ring_buffer(print_event)
while 1:
try:
b.ring_buffer_poll()
except KeyboardInterrupt:
exit()
</code></pre>
<p>Note that the data structure transferred will need to be declared in C in the BPF program. For example:</p>
<pre><code class="language-C">// define output data structure in C
struct data_t {
u32 pid;
u64 ts;
char comm[TASK_COMM_LEN];
};
BPF_RINGBUF_OUTPUT(events, 8);
[...]
</code></pre>
<p>In Python, you can either let bcc generate the data structure from C declaration automatically (recommended):</p>
<pre><code class="language-Python">def print_event(ctx, data, size):
event = b[&quot;events&quot;].event(data)
[...]
</code></pre>
<p>or define it manually:</p>
<pre><code class="language-Python"># define output data structure in Python
TASK_COMM_LEN = 16 # linux/sched.h
class Data(ct.Structure):
_fields_ = [(&quot;pid&quot;, ct.c_ulonglong),
(&quot;ts&quot;, ct.c_ulonglong),
(&quot;comm&quot;, ct.c_char * TASK_COMM_LEN)]
def print_event(ctx, data, size):
event = ct.cast(data, ct.POINTER(Data)).contents
[...]
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=open_ring_buffer+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,</p>
<h3 id="13-push"><a class="header" href="#13-push">13. push()</a></h3>
<p>Syntax: <code>table.push(leaf, flags=0)</code></p>
<p>Push an element onto a Stack or Queue table. Raises an exception if the operation does not succeed.
Passing QueueStack.BPF_EXIST as a flag causes the Queue or Stack to discard the oldest element if it is full.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=push+path%3Atests+language%3Apython&amp;type=Code">search /tests</a>,</p>
<h3 id="14-pop"><a class="header" href="#14-pop">14. pop()</a></h3>
<p>Syntax: <code>leaf = table.pop()</code></p>
<p>Pop an element from a Stack or Queue table. Unlike <code>peek()</code>, <code>pop()</code>
removes the element from the table before returning it.
Raises a KeyError exception if the operation does not succeed.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=pop+path%3Atests+language%3Apython&amp;type=Code">search /tests</a>,</p>
<h3 id="15-peek"><a class="header" href="#15-peek">15. peek()</a></h3>
<p>Syntax: <code>leaf = table.peek()</code></p>
<p>Peek the element at the head of a Stack or Queue table. Unlike <code>pop()</code>, <code>peek()</code>
does not remove the element from the table. Raises an exception if the operation does not succeed.</p>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=peek+path%3Atests+language%3Apython&amp;type=Code">search /tests</a>,</p>
<h2 id="helpers-1"><a class="header" href="#helpers-1">Helpers</a></h2>
<p>Some helper methods provided by bcc. Note that since we're in Python, we can import any Python library and their methods, including, for example, the libraries: argparse, collections, ctypes, datetime, re, socket, struct, subprocess, sys, and time.</p>
<h3 id="1-ksym"><a class="header" href="#1-ksym">1. ksym()</a></h3>
<p>Syntax: <code>BPF.ksym(addr)</code></p>
<p>Translate a kernel memory address into a kernel function name, which is returned.</p>
<p>Example:</p>
<pre><code class="language-Python">print(&quot;kernel function: &quot; + b.ksym(addr))
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=ksym+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=ksym+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="2-ksymname"><a class="header" href="#2-ksymname">2. ksymname()</a></h3>
<p>Syntax: <code>BPF.ksymname(name)</code></p>
<p>Translate a kernel name into an address. This is the reverse of ksym. Returns -1 when the function name is unknown.</p>
<p>Example:</p>
<pre><code class="language-Python">print(&quot;kernel address: %x&quot; % b.ksymname(&quot;vfs_read&quot;))
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=ksymname+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=ksymname+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="3-sym"><a class="header" href="#3-sym">3. sym()</a></h3>
<p>Syntax: <code>BPF.sym(addr, pid, show_module=False, show_offset=False)</code></p>
<p>Translate a memory address into a function name for a pid, which is returned. A pid of less than zero will access the kernel symbol cache. The <code>show_module</code> and <code>show_offset</code> parameters control whether the module in which the symbol lies should be displayed, and whether the instruction offset from the beginning of the symbol should be displayed. These extra parameters default to <code>False</code>.</p>
<p>Example:</p>
<pre><code class="language-python">print(&quot;function: &quot; + b.sym(addr, pid))
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=sym+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=sym+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="4-num_open_kprobes"><a class="header" href="#4-num_open_kprobes">4. num_open_kprobes()</a></h3>
<p>Syntax: <code>BPF.num_open_kprobes()</code></p>
<p>Returns the number of open k[ret]probes. Can be useful for scenarios where event_re is used while attaching and detaching probes. Excludes perf_events readers.</p>
<p>Example:</p>
<pre><code class="language-python">b.attach_kprobe(event_re=pattern, fn_name=&quot;trace_count&quot;)
matched = b.num_open_kprobes()
if matched == 0:
print(&quot;0 functions matched by \&quot;%s\&quot;. Exiting.&quot; % args.pattern)
exit()
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=num_open_kprobes+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=num_open_kprobes+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h3 id="5-get_syscall_fnname"><a class="header" href="#5-get_syscall_fnname">5. get_syscall_fnname()</a></h3>
<p>Syntax: <code>BPF.get_syscall_fnname(name : str)</code></p>
<p>Return the corresponding kernel function name of the syscall. This helper function will try different prefixes and use the right one to concatenate with the syscall name. Note that the return value may vary in different versions of linux kernel and sometimes it will causing trouble. (see <a href="https://github.com/iovisor/bcc/issues/2590">#2590</a>)</p>
<p>Example:</p>
<pre><code class="language-python">print(&quot;The function name of %s in kernel is %s&quot; % (&quot;clone&quot;, b.get_syscall_fnname(&quot;clone&quot;)))
# sys_clone or __x64_sys_clone or ...
</code></pre>
<p>Examples in situ:
<a href="https://github.com/iovisor/bcc/search?q=get_syscall_fnname+path%3Aexamples+language%3Apython&amp;type=Code">search /examples</a>,
<a href="https://github.com/iovisor/bcc/search?q=get_syscall_fnname+path%3Atools+language%3Apython&amp;type=Code">search /tools</a></p>
<h1 id="bpf-errors"><a class="header" href="#bpf-errors">BPF Errors</a></h1>
<p>See the &quot;Understanding eBPF verifier messages&quot; section in the kernel source under Documentation/networking/filter.txt.</p>
<h2 id="1-invalid-mem-access"><a class="header" href="#1-invalid-mem-access">1. Invalid mem access</a></h2>
<p>This can be due to trying to read memory directly, instead of operating on memory on the BPF stack. All kernel memory reads must be passed via bpf_probe_read_kernel() to copy kernel memory into the BPF stack, which can be automatic by the bcc rewriter in some cases of simple dereferencing. bpf_probe_read_kernel() does all the required checks.</p>
<p>Example:</p>
<pre><code class="language-sh">bpf: Permission denied
0: (bf) r6 = r1
1: (79) r7 = *(u64 *)(r6 +80)
2: (85) call 14
3: (bf) r8 = r0
[...]
23: (69) r1 = *(u16 *)(r7 +16)
R7 invalid mem access 'inv'
Traceback (most recent call last):
File &quot;./tcpaccept&quot;, line 179, in &lt;module&gt;
b = BPF(text=bpf_text)
File &quot;/usr/lib/python2.7/dist-packages/bcc/__init__.py&quot;, line 172, in __init__
self._trace_autoload()
File &quot;/usr/lib/python2.7/dist-packages/bcc/__init__.py&quot;, line 612, in _trace_autoload
fn = self.load_func(func_name, BPF.KPROBE)
File &quot;/usr/lib/python2.7/dist-packages/bcc/__init__.py&quot;, line 212, in load_func
raise Exception(&quot;Failed to load BPF program %s&quot; % func_name)
Exception: Failed to load BPF program kretprobe__inet_csk_accept
</code></pre>
<h2 id="2-cannot-call-gpl-only-function-from-proprietary-program"><a class="header" href="#2-cannot-call-gpl-only-function-from-proprietary-program">2. Cannot call GPL only function from proprietary program</a></h2>
<p>This error happens when a GPL-only helper is called from a non-GPL BPF program. To fix this error, do not use GPL-only helpers from a proprietary BPF program, or relicense the BPF program under a GPL-compatible license. Check which <a href="https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md#helpers">BPF helpers</a> are GPL-only, and what licenses are considered GPL-compatible.</p>
<p>Example calling <code>bpf_get_stackid()</code>, a GPL-only BPF helper, from a proprietary program (<code>#define BPF_LICENSE Proprietary</code>):</p>
<pre><code class="language-sh">bpf: Failed to load program: Invalid argument
[...]
8: (85) call bpf_get_stackid#27
cannot call GPL only function from proprietary program
</code></pre>
<h1 id="environment-variables"><a class="header" href="#environment-variables">Environment Variables</a></h1>
<h2 id="1-kernel-source-directory"><a class="header" href="#1-kernel-source-directory">1. Kernel source directory</a></h2>
<p>eBPF program compilation needs kernel sources or kernel headers with headers
compiled. In case your kernel sources are at a non-standard location where BCC
cannot find then, its possible to provide BCC the absolute path of the location
by setting <code>BCC_KERNEL_SOURCE</code> to it.</p>
<h2 id="2-kernel-version-overriding"><a class="header" href="#2-kernel-version-overriding">2. Kernel version overriding</a></h2>
<p>By default, BCC stores the <code>LINUX_VERSION_CODE</code> in the generated eBPF object
which is then passed along to the kernel when the eBPF program is loaded.
Sometimes this is quite inconvenient especially when the kernel is slightly
updated such as an LTS kernel release. Its extremely unlikely the slight
mismatch would cause any issues with the loaded eBPF program. By setting
<code>BCC_LINUX_VERSION_CODE</code> to the version of the kernel that's running, the check
for verifying the kernel version can be bypassed. This is needed for programs
that use kprobes. This needs to be encoded in the format: <code>(VERSION * 65536) + (PATCHLEVEL * 256) + SUBLEVEL</code>. For example, if the running kernel is <code>4.9.10</code>,
then can set <code>export BCC_LINUX_VERSION_CODE=264458</code> to override the kernel
version check successfully.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="special-filtering"><a class="header" href="#special-filtering">Special Filtering</a></h1>
<p>Some tools have special filtering capabitilies, the main use case is to trace
processes running in containers, but those mechanisms are generic and could
be used in other cases as well.</p>
<h2 id="filtering-by-cgroups"><a class="header" href="#filtering-by-cgroups">Filtering by cgroups</a></h2>
<p>Some tools have an option to filter by cgroup by referencing a pinned BPF hash
map managed externally.</p>
<p>Examples of commands:</p>
<pre><code class="language-sh"># ./opensnoop --cgroupmap /sys/fs/bpf/test01
# ./execsnoop --cgroupmap /sys/fs/bpf/test01
# ./tcpconnect --cgroupmap /sys/fs/bpf/test01
# ./tcpaccept --cgroupmap /sys/fs/bpf/test01
# ./tcptracer --cgroupmap /sys/fs/bpf/test01
</code></pre>
<p>The commands above will only display results from processes that belong to one
of the cgroups whose id, returned by <code>bpf_get_current_cgroup_id()</code>, is in the
pinned BPF hash map.</p>
<p>The BPF hash map can be created by:</p>
<pre><code class="language-sh"># bpftool map create /sys/fs/bpf/test01 type hash key 8 value 8 entries 128 \
name cgroupset flags 0
</code></pre>
<p>To get a shell in a new cgroup, you can use:</p>
<pre><code class="language-sh"># systemd-run --pty --unit test bash
</code></pre>
<p>The shell will be running in the cgroup
<code>/sys/fs/cgroup/unified/system.slice/test.service</code>.</p>
<p>The cgroup id can be discovered using the <code>name_to_handle_at()</code> system call. In
the examples/cgroupid, you will find an example of program to get the cgroup
id.</p>
<pre><code class="language-sh"># cd examples/cgroupid
# make
# ./cgroupid hex /sys/fs/cgroup/unified/system.slice/test.service
</code></pre>
<p>or, using Docker:</p>
<pre><code class="language-sh"># cd examples/cgroupid
# docker build -t cgroupid .
# docker run --rm --privileged -v /sys/fs/cgroup:/sys/fs/cgroup \
cgroupid cgroupid hex /sys/fs/cgroup/unified/system.slice/test.service
</code></pre>
<p>This prints the cgroup id as a hexadecimal string in the host endianness such
as <code>77 16 00 00 01 00 00 00</code>.</p>
<pre><code class="language-sh"># FILE=/sys/fs/bpf/test01
# CGROUPID_HEX=&quot;77 16 00 00 01 00 00 00&quot;
# bpftool map update pinned $FILE key hex $CGROUPID_HEX value hex 00 00 00 00 00 00 00 00 any
</code></pre>
<p>Now that the shell started by systemd-run has its cgroup id in the BPF hash
map, bcc tools will display results from this shell. Cgroups can be added and
removed from the BPF hash map without restarting the bcc tool.</p>
<p>This feature is useful for integrating bcc tools in external projects.</p>
<h2 id="filtering-by-mount-by-namespace"><a class="header" href="#filtering-by-mount-by-namespace">Filtering by mount by namespace</a></h2>
<p>The BPF hash map can be created by:</p>
<pre><code class="language-sh"># bpftool map create /sys/fs/bpf/mnt_ns_set type hash key 8 value 4 entries 128 \
name mnt_ns_set flags 0
</code></pre>
<p>Execute the <code>execsnoop</code> tool filtering only the mount namespaces
in <code>/sys/fs/bpf/mnt_ns_set</code>:</p>
<pre><code class="language-sh"># tools/execsnoop.py --mntnsmap /sys/fs/bpf/mnt_ns_set
</code></pre>
<p>Start a terminal in a new mount namespace:</p>
<pre><code class="language-sh"># unshare -m bash
</code></pre>
<p>Update the hash map with the mount namespace ID of the terminal above:</p>
<pre><code class="language-sh">FILE=/sys/fs/bpf/mnt_ns_set
if [ $(printf '\1' | od -dAn) -eq 1 ]; then
HOST_ENDIAN_CMD=tac
else
HOST_ENDIAN_CMD=cat
fi
NS_ID_HEX=&quot;$(printf '%016x' $(stat -Lc '%i' /proc/self/ns/mnt) | sed 's/.\{2\}/&amp;\n/g' | $HOST_ENDIAN_CMD)&quot;
bpftool map update pinned $FILE key hex $NS_ID_HEX value hex 00 00 00 00 any
</code></pre>
<p>Execute a command in this terminal:</p>
<pre><code class="language-sh"># ping kinvolk.io
</code></pre>
<p>You'll see how on the <code>execsnoop</code> terminal you started above the call is logged:</p>
<pre><code class="language-sh"># tools/execsnoop.py --mntnsmap /sys/fs/bpf/mnt_ns_set
[sudo] password for mvb:
PCOMM PID PPID RET ARGS
ping 8096 7970 0 /bin/ping kinvolk.io
</code></pre>
<div style="break-before: page; page-break-before: always;"></div><h1 id="bcc-tutorial"><a class="header" href="#bcc-tutorial">bcc Tutorial</a></h1>
<p>This tutorial covers how to use <a href="https://github.com/iovisor/bcc">bcc</a> tools to quickly solve performance, troubleshooting, and networking issues. If you want to develop new bcc tools, see <a href="bcc-documents/tutorial_bcc_python_developer.html">tutorial_bcc_python_developer.md</a> for that tutorial.</p>
<p>It is assumed for this tutorial that bcc is already installed, and you can run tools like execsnoop successfully. See <a href="https://github.com/iovisor/bcc/tree/master/INSTALL.md">INSTALL.md</a>. This uses enhancements added to the Linux 4.x series.</p>
<h2 id="observability"><a class="header" href="#observability">Observability</a></h2>
<p>Some quick wins.</p>
<h3 id="0-before-bcc"><a class="header" href="#0-before-bcc">0. Before bcc</a></h3>
<p>Before using bcc, you should start with the Linux basics. One reference is the <a href="https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55">Linux Performance Analysis in 60,000 Milliseconds</a> post, which covers these commands:</p>
<ol>
<li>uptime</li>
<li>dmesg | tail</li>
<li>vmstat 1</li>
<li>mpstat -P ALL 1</li>
<li>pidstat 1</li>
<li>iostat -xz 1</li>
<li>free -m</li>
<li>sar -n DEV 1</li>
<li>sar -n TCP,ETCP 1</li>
<li>top</li>
</ol>
<h3 id="1-general-performance"><a class="header" href="#1-general-performance">1. General Performance</a></h3>
<p>Here is a generic checklist for performance investigations with bcc, first as a list, then in detail:</p>
<ol>
<li>execsnoop</li>
<li>opensnoop</li>
<li>ext4slower (or btrfs*, xfs*, zfs*)</li>
<li>biolatency</li>
<li>biosnoop</li>
<li>cachestat</li>
<li>tcpconnect</li>
<li>tcpaccept</li>
<li>tcpretrans</li>
<li>runqlat</li>
<li>profile</li>
</ol>
<p>These tools may be installed on your system under /usr/share/bcc/tools, or you can run them from the bcc github repo under /tools where they have a .py extension. Browse the 50+ tools available for more analysis options.</p>
<h4 id="11-execsnoop"><a class="header" href="#11-execsnoop">1.1 execsnoop</a></h4>
<pre><code class="language-sh"># ./execsnoop
PCOMM PID RET ARGS
supervise 9660 0 ./run
supervise 9661 0 ./run
mkdir 9662 0 /bin/mkdir -p ./main
run 9663 0 ./run
[...]
</code></pre>
<p>execsnoop prints one line of output for each new process. Check for short-lived processes. These can consume CPU resources, but not show up in most monitoring tools that periodically take snapshots of which processes are running.</p>
<p>It works by tracing exec(), not the fork(), so it will catch many types of new processes but not all (eg, it won't see an application launching working processes, that doesn't exec() anything else).</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/execsnoop_example.txt">examples</a>.</p>
<h4 id="12-opensnoop"><a class="header" href="#12-opensnoop">1.2. opensnoop</a></h4>
<pre><code class="language-sh"># ./opensnoop
PID COMM FD ERR PATH
1565 redis-server 5 0 /proc/1565/stat
1565 redis-server 5 0 /proc/1565/stat
1565 redis-server 5 0 /proc/1565/stat
1603 snmpd 9 0 /proc/net/dev
1603 snmpd 11 0 /proc/net/if_inet6
1603 snmpd -1 2 /sys/class/net/eth0/device/vendor
1603 snmpd 11 0 /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms
1603 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/retrans_time_ms
1603 snmpd 11 0 /proc/sys/net/ipv6/conf/eth0/forwarding
[...]
</code></pre>
<p>opensnoop prints one line of output for each open() syscall, including details.</p>
<p>Files that are opened can tell you a lot about how applications work: identifying their data files, config files, and log files. Sometimes applications can misbehave, and perform poorly, when they are constantly attempting to read files that do not exist. opensnoop gives you a quick look.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/opensnoop_example.txt">examples</a>.</p>
<h4 id="13-ext4slower-or-btrfs-xfs-zfs"><a class="header" href="#13-ext4slower-or-btrfs-xfs-zfs">1.3. ext4slower (or btrfs*, xfs*, zfs*)</a></h4>
<pre><code class="language-sh"># ./ext4slower
Tracing ext4 operations slower than 10 ms
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
06:35:01 cron 16464 R 1249 0 16.05 common-auth
06:35:01 cron 16463 R 1249 0 16.04 common-auth
06:35:01 cron 16465 R 1249 0 16.03 common-auth
06:35:01 cron 16465 R 4096 0 10.62 login.defs
06:35:01 cron 16464 R 4096 0 10.61 login.defs
</code></pre>
<p>ext4slower traces the ext4 file system and times common operations, and then only prints those that exceed a threshold.</p>
<p>This is great for identifying or exonerating one type of performance issue: show individually slow disk i/O via the file system. Disks process I/O asynchronously, and it can be difficult to associate latency at that layer with the latency applications experience. Tracing higher up in the kernel stack, at the VFS -&gt; file system interface, will more closely match what an application suffers. Use this tool to identify if file system latency exceeds a given threshold.</p>
<p>Similar tools exist in bcc for other file systems: btrfsslower, xfsslower, and zfsslower. There is also fileslower, which works at the VFS layer and traces everything (although at some higher overhead).</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/ext4slower_example.txt">examples</a>.</p>
<h4 id="14-biolatency"><a class="header" href="#14-biolatency">1.4. biolatency</a></h4>
<pre><code class="language-sh"># ./biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
usecs : count distribution
0 -&gt; 1 : 0 | |
2 -&gt; 3 : 0 | |
4 -&gt; 7 : 0 | |
8 -&gt; 15 : 0 | |
16 -&gt; 31 : 0 | |
32 -&gt; 63 : 0 | |
64 -&gt; 127 : 1 | |
128 -&gt; 255 : 12 |******** |
256 -&gt; 511 : 15 |********** |
512 -&gt; 1023 : 43 |******************************* |
1024 -&gt; 2047 : 52 |**************************************|
2048 -&gt; 4095 : 47 |********************************** |
4096 -&gt; 8191 : 52 |**************************************|
8192 -&gt; 16383 : 36 |************************** |
16384 -&gt; 32767 : 15 |********** |
32768 -&gt; 65535 : 2 |* |
65536 -&gt; 131071 : 2 |* |
</code></pre>
<p>biolatency traces disk I/O latency (time from device issue to completion), and when the tool ends (Ctrl-C, or a given interval), it prints a histogram summary of the latency.</p>
<p>This is great for understanding disk I/O latency beyond the average times given by tools like iostat. I/O latency outliers will be visible at the end of the distribution, as well as multi-mode distributions.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/biolatency_example.txt">examples</a>.</p>
<h4 id="15-biosnoop"><a class="header" href="#15-biosnoop">1.5. biosnoop</a></h4>
<pre><code class="language-sh"># ./biosnoop
TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms)
0.000004001 supervise 1950 xvda1 W 13092560 4096 0.74
0.000178002 supervise 1950 xvda1 W 13092432 4096 0.61
0.001469001 supervise 1956 xvda1 W 13092440 4096 1.24
0.001588002 supervise 1956 xvda1 W 13115128 4096 1.09
1.022346001 supervise 1950 xvda1 W 13115272 4096 0.98
1.022568002 supervise 1950 xvda1 W 13188496 4096 0.93
[...]
</code></pre>
<p>biosnoop prints a line of output for each disk I/O, with details including latency (time from device issue to completion).</p>
<p>This allows you to examine disk I/O in more detail, and look for time-ordered patterns (eg, reads queueing behind writes). Note that the output will be verbose if your system performs disk I/O at a high rate.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/biosnoop_example.txt">examples</a>.</p>
<h4 id="16-cachestat"><a class="header" href="#16-cachestat">1.6. cachestat</a></h4>
<pre><code class="language-sh"># ./cachestat
HITS MISSES DIRTIES READ_HIT% WRITE_HIT% BUFFERS_MB CACHED_MB
1074 44 13 94.9% 2.9% 1 223
2195 170 8 92.5% 6.8% 1 143
182 53 56 53.6% 1.3% 1 143
62480 40960 20480 40.6% 19.8% 1 223
7 2 5 22.2% 22.2% 1 223
348 0 0 100.0% 0.0% 1 223
[...]
</code></pre>
<p>cachestat prints a one line summary every second (or every custom interval) showing statistics from the file system cache.</p>
<p>Use this to identify a low cache hit ratio, and a high rate of misses: which gives one lead for performance tuning.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/cachestat_example.txt">examples</a>.</p>
<h4 id="17-tcpconnect"><a class="header" href="#17-tcpconnect">1.7. tcpconnect</a></h4>
<pre><code class="language-sh"># ./tcpconnect
PID COMM IP SADDR DADDR DPORT
1479 telnet 4 127.0.0.1 127.0.0.1 23
1469 curl 4 10.201.219.236 54.245.105.25 80
1469 curl 4 10.201.219.236 54.67.101.145 80
1991 telnet 6 ::1 ::1 23
2015 ssh 6 fe80::2000:bff:fe82:3ac fe80::2000:bff:fe82:3ac 22
[...]
</code></pre>
<p>tcpconnect prints one line of output for every active TCP connection (eg, via connect()), with details including source and destination addresses.</p>
<p>Look for unexpected connections that may point to inefficiencies in application configuration, or an intruder.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/tcpconnect_example.txt">examples</a>.</p>
<h4 id="18-tcpaccept"><a class="header" href="#18-tcpaccept">1.8. tcpaccept</a></h4>
<pre><code class="language-sh"># ./tcpaccept
PID COMM IP RADDR LADDR LPORT
907 sshd 4 192.168.56.1 192.168.56.102 22
907 sshd 4 127.0.0.1 127.0.0.1 22
5389 perl 6 1234:ab12:2040:5020:2299:0:5:0 1234:ab12:2040:5020:2299:0:5:0 7001
[...]
</code></pre>
<p>tcpaccept prints one line of output for every passive TCP connection (eg, via accept()), with details including source and destination addresses.</p>
<p>Look for unexpected connections that may point to inefficiencies in application configuration, or an intruder.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/tcpaccept_example.txt">examples</a>.</p>
<h4 id="19-tcpretrans"><a class="header" href="#19-tcpretrans">1.9. tcpretrans</a></h4>
<pre><code class="language-sh"># ./tcpretrans
TIME PID IP LADDR:LPORT T&gt; RADDR:RPORT STATE
01:55:05 0 4 10.153.223.157:22 R&gt; 69.53.245.40:34619 ESTABLISHED
01:55:05 0 4 10.153.223.157:22 R&gt; 69.53.245.40:34619 ESTABLISHED
01:55:17 0 4 10.153.223.157:22 R&gt; 69.53.245.40:22957 ESTABLISHED
[...]
</code></pre>
<p>tcprerans prints one line of output for every TCP retransmit packet, with details including source and destination addresses, and kernel state of the TCP connection.</p>
<p>TCP retransmissions cause latency and throughput issues. For ESTABLISHED retransmits, look for patterns with networks. For SYN_SENT, this may point to target kernel CPU saturation and kernel packet drops.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/tcpretrans_example.txt">examples</a>.</p>
<h4 id="110-runqlat"><a class="header" href="#110-runqlat">1.10. runqlat</a></h4>
<pre><code class="language-sh"># ./runqlat
Tracing run queue latency... Hit Ctrl-C to end.
^C
usecs : count distribution
0 -&gt; 1 : 233 |*********** |
2 -&gt; 3 : 742 |************************************ |
4 -&gt; 7 : 203 |********** |
8 -&gt; 15 : 173 |******** |
16 -&gt; 31 : 24 |* |
32 -&gt; 63 : 0 | |
64 -&gt; 127 : 30 |* |
128 -&gt; 255 : 6 | |
256 -&gt; 511 : 3 | |
512 -&gt; 1023 : 5 | |
1024 -&gt; 2047 : 27 |* |
2048 -&gt; 4095 : 30 |* |
4096 -&gt; 8191 : 20 | |
8192 -&gt; 16383 : 29 |* |
16384 -&gt; 32767 : 809 |****************************************|
32768 -&gt; 65535 : 64 |*** |
</code></pre>
<p>runqlat times how long threads were waiting on the CPU run queues, and prints this as a histogram.</p>
<p>This can help quantify time lost waiting for a turn on CPU, during periods of CPU saturation.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/runqlat_example.txt">examples</a>.</p>
<h4 id="111-profile"><a class="header" href="#111-profile">1.11. profile</a></h4>
<pre><code class="language-sh"># ./profile
Sampling at 49 Hertz of all threads by user + kernel stack... Hit Ctrl-C to end.
^C
00007f31d76c3251 [unknown]
47a2c1e752bf47f7 [unknown]
- sign-file (8877)
1
ffffffff813d0af8 __clear_user
ffffffff813d5277 iov_iter_zero
ffffffff814ec5f2 read_iter_zero
ffffffff8120be9d __vfs_read
ffffffff8120c385 vfs_read
ffffffff8120d786 sys_read
ffffffff817cc076 entry_SYSCALL_64_fastpath
00007fc5652ad9b0 read
- dd (25036)
4
0000000000400542 func_a
0000000000400598 main
00007f12a133e830 __libc_start_main
083e258d4c544155 [unknown]
- func_ab (13549)
5
[...]
ffffffff8105eb66 native_safe_halt
ffffffff8103659e default_idle
ffffffff81036d1f arch_cpu_idle
ffffffff810bba5a default_idle_call
ffffffff810bbd07 cpu_startup_entry
ffffffff8104df55 start_secondary
- swapper/1 (0)
75
</code></pre>
<p>profile is a CPU profiler, which takes samples of stack traces at timed intervals, and prints a summary of unique stack traces and a count of their occurrence.</p>
<p>Use this tool to understand the code paths that are consuming CPU resources.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/profile_example.txt">examples</a>.</p>
<h3 id="2-observability-with-generic-tools"><a class="header" href="#2-observability-with-generic-tools">2. Observability with Generic Tools</a></h3>
<p>In addition to the above tools for performance tuning, below is a checklist for bcc generic tools, first as a list, and in detail:</p>
<ol>
<li>trace</li>
<li>argdist</li>
<li>funccount</li>
</ol>
<p>These generic tools may be useful to provide visibility to solve your specific problems.</p>
<h4 id="21-trace"><a class="header" href="#21-trace">2.1. trace</a></h4>
<h5 id="example-1"><a class="header" href="#example-1">Example 1</a></h5>
<p>Suppose you want to track file ownership change. There are three syscalls, <code>chown</code>, <code>fchown</code> and <code>lchown</code> which users can use to change file ownership. The corresponding syscall entry is <code>SyS_[f|l]chown</code>. The following command can be used to print out syscall parameters and the calling process user id. You can use <code>id</code> command to find the uid of a particular user.</p>
<pre><code class="language-sh">$ trace.py \
'p::SyS_chown &quot;file = %s, to_uid = %d, to_gid = %d, from_uid = %d&quot;, arg1, arg2, arg3, $uid' \
'p::SyS_fchown &quot;fd = %d, to_uid = %d, to_gid = %d, from_uid = %d&quot;, arg1, arg2, arg3, $uid' \
'p::SyS_lchown &quot;file = %s, to_uid = %d, to_gid = %d, from_uid = %d&quot;, arg1, arg2, arg3, $uid'
PID TID COMM FUNC -
1269255 1269255 python3.6 SyS_lchown file = /tmp/dotsync-usisgezu/tmp, to_uid = 128203, to_gid = 100, from_uid = 128203
1269441 1269441 zstd SyS_chown file = /tmp/dotsync-vic7ygj0/dotsync-package.zst, to_uid = 128203, to_gid = 100, from_uid = 128203
1269255 1269255 python3.6 SyS_lchown file = /tmp/dotsync-a40zd7ev/tmp, to_uid = 128203, to_gid = 100, from_uid = 128203
1269442 1269442 zstd SyS_chown file = /tmp/dotsync-gzp413o_/dotsync-package.zst, to_uid = 128203, to_gid = 100, from_uid = 128203
1269255 1269255 python3.6 SyS_lchown file = /tmp/dotsync-whx4fivm/tmp/.bash_profile, to_uid = 128203, to_gid = 100, from_uid = 128203
</code></pre>
<h5 id="example-2"><a class="header" href="#example-2">Example 2</a></h5>
<p>Suppose you want to count nonvoluntary context switches (<code>nvcsw</code>) in your bpf based performance monitoring tools and you do not know what is the proper method. <code>/proc/&lt;pid&gt;/status</code> already tells you the number (<code>nonvoluntary_ctxt_switches</code>) for a pid and you can use <code>trace.py</code> to do a quick experiment to verify your method. With kernel source code, the <code>nvcsw</code> is counted at file <code>linux/kernel/sched/core.c</code> function <code>__schedule</code> and under condition</p>
<pre><code class="language-c">!(!preempt &amp;&amp; prev-&gt;state) // i.e., preempt || !prev-&gt;state
</code></pre>
<p>The <code>__schedule</code> function is marked as <code>notrace</code>, and the best place to evaluate the above condition seems in <code>sched/sched_switch</code> tracepoint called inside function <code>__schedule</code> and defined in <code>linux/include/trace/events/sched.h</code>. <code>trace.py</code> already has <code>args</code> being the pointer to the tracepoint <code>TP_STRUCT__entry</code>. The above condition in function <code>__schedule</code> can be represented as</p>
<pre><code class="language-c">args-&gt;prev_state == TASK_STATE_MAX || args-&gt;prev_state == 0
</code></pre>
<p>The below command can be used to count the involuntary context switches (per process or per pid) and compare to <code>/proc/&lt;pid&gt;/status</code> or <code>/proc/&lt;pid&gt;/task/&lt;task_id&gt;/status</code> for correctness, as in typical cases, involuntary context switches are not very common.</p>
<pre><code class="language-sh">$ trace.py -p 1134138 't:sched:sched_switch (args-&gt;prev_state == TASK_STATE_MAX || args-&gt;prev_state == 0)'
PID TID COMM FUNC
1134138 1134140 contention_test sched_switch
1134138 1134142 contention_test sched_switch
...
$ trace.py -L 1134140 't:sched:sched_switch (args-&gt;prev_state == TASK_STATE_MAX || args-&gt;prev_state == 0)'
PID TID COMM FUNC
1134138 1134140 contention_test sched_switch
1134138 1134140 contention_test sched_switch
...
</code></pre>
<h5 id="example-3"><a class="header" href="#example-3">Example 3</a></h5>
<p>This example is related to issue <a href="https://github.com/iovisor/bcc/issues/1231">1231</a> and <a href="https://github.com/iovisor/bcc/issues/1516">1516</a> where uprobe does not work at all in certain cases. First, you can do a <code>strace</code> as below</p>
<pre><code class="language-sh">$ strace trace.py 'r:bash:readline &quot;%s&quot;, retval'
...
perf_event_open(0x7ffd968212f0, -1, 0, -1, 0x8 /* PERF_FLAG_??? */) = -1 EIO (Input/output error)
...
</code></pre>
<p>The <code>perf_event_open</code> syscall returns <code>-EIO</code>. Digging into kernel uprobe related codes in <code>/kernel/trace</code> and <code>/kernel/events</code> directories to search <code>EIO</code>, the function <code>uprobe_register</code> is the most suspicious. Let us find whether this function is called or not and what is the return value if it is called. In one terminal using the following command to print out the return value of uprobe_register,</p>
<pre><code class="language-sh">$ trace.py 'r::uprobe_register &quot;ret = %d&quot;, retval'
</code></pre>
<p>In another terminal run the same bash uretprobe tracing example, and you should get</p>
<pre><code class="language-sh">$ trace.py 'r::uprobe_register &quot;ret = %d&quot;, retval'
PID TID COMM FUNC -
1041401 1041401 python2.7 uprobe_register ret = -5
</code></pre>
<p>The <code>-5</code> error code is EIO. This confirms that the following code in function <code>uprobe_register</code> is the most suspicious culprit.</p>
<pre><code class="language-c"> if (!inode-&gt;i_mapping-&gt;a_ops-&gt;readpage &amp;&amp; !shmem_mapping(inode-&gt;i_mapping))
return -EIO;
</code></pre>
<p>The <code>shmem_mapping</code> function is defined as</p>
<pre><code class="language-c">bool shmem_mapping(struct address_space *mapping)
{
return mapping-&gt;a_ops == &amp;shmem_aops;
}
</code></pre>
<p>To confirm the theory, find what is <code>inode-&gt;i_mapping-&gt;a_ops</code> with the following command</p>
<pre><code class="language-sh">$ trace.py -I 'linux/fs.h' 'p::uprobe_register(struct inode *inode) &quot;a_ops = %llx&quot;, inode-&gt;i_mapping-&gt;a_ops'
PID TID COMM FUNC -
814288 814288 python2.7 uprobe_register a_ops = ffffffff81a2adc0
^C$ grep ffffffff81a2adc0 /proc/kallsyms
ffffffff81a2adc0 R empty_aops
</code></pre>
<p>The kernel symbol <code>empty_aops</code> does not have <code>readpage</code> defined and hence the above suspicious condition is true. Further examining the kernel source code shows that <code>overlayfs</code> does not provide its own <code>a_ops</code> while some other file systems (e.g., ext4) define their own <code>a_ops</code> (e.g., <code>ext4_da_aops</code>), and <code>ext4_da_aops</code> defines <code>readpage</code>. Hence, uprobe works fine on ext4 while not on overlayfs.</p>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/trace_example.txt">examples</a>.</p>
<h4 id="22-argdist"><a class="header" href="#22-argdist">2.2. argdist</a></h4>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/argdist_example.txt">examples</a>.</p>
<h4 id="23-funccount"><a class="header" href="#23-funccount">2.3. funccount</a></h4>
<p>More <a href="https://github.com/iovisor/bcc/tree/master/tools/funccount_example.txt">examples</a>.</p>
<h2 id="networking"><a class="header" href="#networking">Networking</a></h2>
<p>To do.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="bcc-python-developer-tutorial"><a class="header" href="#bcc-python-developer-tutorial">bcc Python Developer Tutorial</a></h1>
<p>This tutorial is about developing <a href="https://github.com/iovisor/bcc">bcc</a> tools and programs using the Python interface. There are two parts: observability then networking. Snippets are taken from various programs in bcc: see their files for licences.</p>
<p>Also see the bcc developer's <a href="bcc-documents/reference_guide.html">reference_guide.md</a>, and a tutorial for end-users of tools: <a href="bcc-documents/tutorial.html">tutorial.md</a>. There is also a lua interface for bcc.</p>
<h2 id="observability-1"><a class="header" href="#observability-1">Observability</a></h2>
<p>This observability tutorial contains 17 lessons, and 46 enumerated things to learn.</p>
<h3 id="lesson-1-hello-world"><a class="header" href="#lesson-1-hello-world">Lesson 1. Hello World</a></h3>
<p>Start by running <a href="https://github.com/iovisor/bcc/tree/master/examples/hello_world.py">examples/hello_world.py</a>, while running some commands (eg, &quot;ls&quot;) in another session. It should print &quot;Hello, World!&quot; for new processes. If not, start by fixing bcc: see <a href="https://github.com/iovisor/bcc/tree/master/INSTALL.md">INSTALL.md</a>.</p>
<pre><code class="language-sh"># ./examples/hello_world.py
bash-13364 [002] d... 24573433.052937: : Hello, World!
bash-13364 [003] d... 24573436.642808: : Hello, World!
[...]
</code></pre>
<p>Here's the code for hello_world.py:</p>
<pre><code class="language-Python">from bcc import BPF
BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk(&quot;Hello, World!\\n&quot;); return 0; }').trace_print()
</code></pre>
<p>There are six things to learn from this:</p>
<ol>
<li>
<p><code>text='...'</code>: This defines a BPF program inline. The program is written in C.</p>
</li>
<li>
<p><code>kprobe__sys_clone()</code>: This is a short-cut for kernel dynamic tracing via kprobes. If the C function begins with <code>kprobe__</code>, the rest is treated as a kernel function name to instrument, in this case, <code>sys_clone()</code>.</p>
</li>
<li>
<p><code>void *ctx</code>: ctx has arguments, but since we aren't using them here, we'll just cast it to <code>void *</code>.</p>
</li>
<li>
<p><code>bpf_trace_printk()</code>: A simple kernel facility for printf() to the common trace_pipe (/sys/kernel/debug/tracing/trace_pipe). This is ok for some quick examples, but has limitations: 3 args max, 1 %s only, and trace_pipe is globally shared, so concurrent programs will have clashing output. A better interface is via BPF_PERF_OUTPUT(), covered later.</p>
</li>
<li>
<p><code>return 0;</code>: Necessary formality (if you want to know why, see <a href="https://github.com/iovisor/bcc/issues/139">#139</a>).</p>
</li>
<li>
<p><code>.trace_print()</code>: A bcc routine that reads trace_pipe and prints the output.</p>
</li>
</ol>
<h3 id="lesson-2-sys_sync"><a class="header" href="#lesson-2-sys_sync">Lesson 2. sys_sync()</a></h3>
<p>Write a program that traces the sys_sync() kernel function. Print &quot;sys_sync() called&quot; when it runs. Test by running <code>sync</code> in another session while tracing. The hello_world.py program has everything you need for this.</p>
<p>Improve it by printing &quot;Tracing sys_sync()... Ctrl-C to end.&quot; when the program first starts. Hint: it's just Python.</p>
<h3 id="lesson-3-hello_fieldspy"><a class="header" href="#lesson-3-hello_fieldspy">Lesson 3. hello_fields.py</a></h3>
<p>This program is in <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/hello_fields.py">examples/tracing/hello_fields.py</a>. Sample output (run commands in another session):</p>
<pre><code class="language-sh"># examples/tracing/hello_fields.py
TIME(s) COMM PID MESSAGE
24585001.174885999 sshd 1432 Hello, World!
24585001.195710000 sshd 15780 Hello, World!
24585001.991976000 systemd-udevd 484 Hello, World!
24585002.276147000 bash 15787 Hello, World!
</code></pre>
<p>Code:</p>
<pre><code class="language-Python">from bcc import BPF
# define BPF program
prog = &quot;&quot;&quot;
int hello(void *ctx) {
bpf_trace_printk(&quot;Hello, World!\\n&quot;);
return 0;
}
&quot;&quot;&quot;
# load BPF program
b = BPF(text=prog)
b.attach_kprobe(event=b.get_syscall_fnname(&quot;clone&quot;), fn_name=&quot;hello&quot;)
# header
print(&quot;%-18s %-16s %-6s %s&quot; % (&quot;TIME(s)&quot;, &quot;COMM&quot;, &quot;PID&quot;, &quot;MESSAGE&quot;))
# format output
while 1:
try:
(task, pid, cpu, flags, ts, msg) = b.trace_fields()
except ValueError:
continue
print(&quot;%-18.9f %-16s %-6d %s&quot; % (ts, task, pid, msg))
</code></pre>
<p>This is similar to hello_world.py, and traces new processes via sys_clone() again, but has a few more things to learn:</p>
<ol>
<li>
<p><code>prog =</code>: This time we declare the C program as a variable, and later refer to it. This is useful if you want to add some string substitutions based on command line arguments.</p>
</li>
<li>
<p><code>hello()</code>: Now we're just declaring a C function, instead of the <code>kprobe__</code> shortcut. We'll refer to this later. All C functions declared in the BPF program are expected to be executed on a probe, hence they all need to take a <code>pt_reg* ctx</code> as first argument. If you need to define some helper function that will not be executed on a probe, they need to be defined as <code>static inline</code> in order to be inlined by the compiler. Sometimes you would also need to add <code>_always_inline</code> function attribute to it.</p>
</li>
<li>
<p><code>b.attach_kprobe(event=b.get_syscall_fnname(&quot;clone&quot;), fn_name=&quot;hello&quot;)</code>: Creates a kprobe for the kernel clone system call function, which will execute our defined hello() function. You can call attach_kprobe() more than once, and attach your C function to multiple kernel functions.</p>
</li>
<li>
<p><code>b.trace_fields()</code>: Returns a fixed set of fields from trace_pipe. Similar to trace_print(), this is handy for hacking, but for real tooling we should switch to BPF_PERF_OUTPUT().</p>
</li>
</ol>
<h3 id="lesson-4-sync_timingpy"><a class="header" href="#lesson-4-sync_timingpy">Lesson 4. sync_timing.py</a></h3>
<p>Remember the days of sysadmins typing <code>sync</code> three times on a slow console before <code>reboot</code>, to give the first asynchronous sync time to complete? Then someone thought <code>sync;sync;sync</code> was clever, to run them all on one line, which became industry practice despite defeating the original purpose! And then sync became synchronous, so more reasons it was silly. Anyway.</p>
<p>The following example times how quickly the <code>do_sync</code> function is called, and prints output if it has been called more recently than one second ago. A <code>sync;sync;sync</code> will print output for the 2nd and 3rd sync's:</p>
<pre><code class="language-sh"># examples/tracing/sync_timing.py
Tracing for quick sync's... Ctrl-C to end
At time 0.00 s: multiple syncs detected, last 95 ms ago
At time 0.10 s: multiple syncs detected, last 96 ms ago
</code></pre>
<p>This program is <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/sync_timing.py">examples/tracing/sync_timing.py</a>:</p>
<pre><code class="language-Python">from __future__ import print_function
from bcc import BPF
# load BPF program
b = BPF(text=&quot;&quot;&quot;
#include &lt;uapi/linux/ptrace.h&gt;
BPF_HASH(last);
int do_trace(struct pt_regs *ctx) {
u64 ts, *tsp, delta, key = 0;
// attempt to read stored timestamp
tsp = last.lookup(&amp;key);
if (tsp != NULL) {
delta = bpf_ktime_get_ns() - *tsp;
if (delta &lt; 1000000000) {
// output if time is less than 1 second
bpf_trace_printk(&quot;%d\\n&quot;, delta / 1000000);
}
last.delete(&amp;key);
}
// update stored timestamp
ts = bpf_ktime_get_ns();
last.update(&amp;key, &amp;ts);
return 0;
}
&quot;&quot;&quot;)
b.attach_kprobe(event=b.get_syscall_fnname(&quot;sync&quot;), fn_name=&quot;do_trace&quot;)
print(&quot;Tracing for quick sync's... Ctrl-C to end&quot;)
# format output
start = 0
while 1:
(task, pid, cpu, flags, ts, ms) = b.trace_fields()
if start == 0:
start = ts
ts = ts - start
print(&quot;At time %.2f s: multiple syncs detected, last %s ms ago&quot; % (ts, ms))
</code></pre>
<p>Things to learn:</p>
<ol>
<li><code>bpf_ktime_get_ns()</code>: Returns the time as nanoseconds.</li>
<li><code>BPF_HASH(last)</code>: Creates a BPF map object that is a hash (associative array), called &quot;last&quot;. We didn't specify any further arguments, so it defaults to key and value types of u64.</li>
<li><code>key = 0</code>: We'll only store one key/value pair in this hash, where the key is hardwired to zero.</li>
<li><code>last.lookup(&amp;key)</code>: Lookup the key in the hash, and return a pointer to its value if it exists, else NULL. We pass the key in as an address to a pointer.</li>
<li><code>if (tsp != NULL) {</code>: The verifier requires that pointer values derived from a map lookup must be checked for a null value before they can be dereferenced and used.</li>
<li><code>last.delete(&amp;key)</code>: Delete the key from the hash. This is currently required because of <a href="https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=a6ed3ea65d9868fdf9eff84e6fe4f666b8d14b02">a kernel bug in <code>.update()</code></a> (fixed in 4.8.10).</li>
<li><code>last.update(&amp;key, &amp;ts)</code>: Associate the value in the 2nd argument to the key, overwriting any previous value. This records the timestamp.</li>
</ol>
<h3 id="lesson-5-sync_countpy"><a class="header" href="#lesson-5-sync_countpy">Lesson 5. sync_count.py</a></h3>
<p>Modify the sync_timing.py program (prior lesson) to store the count of all kernel sync system calls (both fast and slow), and print it with the output. This count can be recorded in the BPF program by adding a new key index to the existing hash.</p>
<h3 id="lesson-6-disksnooppy"><a class="header" href="#lesson-6-disksnooppy">Lesson 6. disksnoop.py</a></h3>
<p>Browse the <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/disksnoop.py">examples/tracing/disksnoop.py</a> program to see what is new. Here is some sample output:</p>
<pre><code class="language-sh"># disksnoop.py
TIME(s) T BYTES LAT(ms)
16458043.436012 W 4096 3.13
16458043.437326 W 4096 4.44
16458044.126545 R 4096 42.82
16458044.129872 R 4096 3.24
[...]
</code></pre>
<p>And a code snippet:</p>
<pre><code class="language-Python">[...]
REQ_WRITE = 1 # from include/linux/blk_types.h
# load BPF program
b = BPF(text=&quot;&quot;&quot;
#include &lt;uapi/linux/ptrace.h&gt;
#include &lt;linux/blk-mq.h&gt;
BPF_HASH(start, struct request *);
void trace_start(struct pt_regs *ctx, struct request *req) {
// stash start timestamp by request ptr
u64 ts = bpf_ktime_get_ns();
start.update(&amp;req, &amp;ts);
}
void trace_completion(struct pt_regs *ctx, struct request *req) {
u64 *tsp, delta;
tsp = start.lookup(&amp;req);
if (tsp != 0) {
delta = bpf_ktime_get_ns() - *tsp;
bpf_trace_printk(&quot;%d %x %d\\n&quot;, req-&gt;__data_len,
req-&gt;cmd_flags, delta / 1000);
start.delete(&amp;req);
}
}
&quot;&quot;&quot;)
if BPF.get_kprobe_functions(b'blk_start_request'):
b.attach_kprobe(event=&quot;blk_start_request&quot;, fn_name=&quot;trace_start&quot;)
b.attach_kprobe(event=&quot;blk_mq_start_request&quot;, fn_name=&quot;trace_start&quot;)
if BPF.get_kprobe_functions(b'__blk_account_io_done'):
b.attach_kprobe(event=&quot;__blk_account_io_done&quot;, fn_name=&quot;trace_completion&quot;)
else:
b.attach_kprobe(event=&quot;blk_account_io_done&quot;, fn_name=&quot;trace_completion&quot;)
[...]
</code></pre>
<p>Things to learn:</p>
<ol>
<li><code>REQ_WRITE</code>: We're defining a kernel constant in the Python program because we'll use it there later. If we were using REQ_WRITE in the BPF program, it should just work (without needing to be defined) with the appropriate #includes.</li>
<li><code>trace_start(struct pt_regs *ctx, struct request *req)</code>: This function will later be attached to kprobes. The arguments to kprobe functions are <code>struct pt_regs *ctx</code>, for registers and BPF context, and then the actual arguments to the function. We'll attach this to blk_start_request(), where the first argument is <code>struct request *</code>.</li>
<li><code>start.update(&amp;req, &amp;ts)</code>: We're using the pointer to the request struct as a key in our hash. What? This is commonplace in tracing. Pointers to structs turn out to be great keys, as they are unique: two structs can't have the same pointer address. (Just be careful about when it gets free'd and reused.) So what we're really doing is tagging the request struct, which describes the disk I/O, with our own timestamp, so that we can time it. There's two common keys used for storing timestamps: pointers to structs, and, thread IDs (for timing function entry to return).</li>
<li><code>req-&gt;__data_len</code>: We're dereferencing members of <code>struct request</code>. See its definition in the kernel source for what members are there. bcc actually rewrites these expressions to be a series of <code>bpf_probe_read_kernel()</code> calls. Sometimes bcc can't handle a complex dereference, and you need to call <code>bpf_probe_read_kernel()</code> directly.</li>
</ol>
<p>This is a pretty interesting program, and if you can understand all the code, you'll understand many important basics. We're still using the bpf_trace_printk() hack, so let's fix that next.</p>
<h3 id="lesson-7-hello_perf_outputpy"><a class="header" href="#lesson-7-hello_perf_outputpy">Lesson 7. hello_perf_output.py</a></h3>
<p>Let's finally stop using bpf_trace_printk() and use the proper BPF_PERF_OUTPUT() interface. This will also mean we stop getting the free trace_field() members like PID and timestamp, and will need to fetch them directly. Sample output while commands are run in another session:</p>
<pre><code class="language-sh"># hello_perf_output.py
TIME(s) COMM PID MESSAGE
0.000000000 bash 22986 Hello, perf_output!
0.021080275 systemd-udevd 484 Hello, perf_output!
0.021359520 systemd-udevd 484 Hello, perf_output!
0.021590610 systemd-udevd 484 Hello, perf_output!
[...]
</code></pre>
<p>Code is <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/hello_perf_output.py">examples/tracing/hello_perf_output.py</a>:</p>
<pre><code class="language-Python">from bcc import BPF
# define BPF program
prog = &quot;&quot;&quot;
#include &lt;linux/sched.h&gt;
// define output data structure in C
struct data_t {
u32 pid;
u64 ts;
char comm[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(events);
int hello(struct pt_regs *ctx) {
struct data_t data = {};
data.pid = bpf_get_current_pid_tgid();
data.ts = bpf_ktime_get_ns();
bpf_get_current_comm(&amp;data.comm, sizeof(data.comm));
events.perf_submit(ctx, &amp;data, sizeof(data));
return 0;
}
&quot;&quot;&quot;
# load BPF program
b = BPF(text=prog)
b.attach_kprobe(event=b.get_syscall_fnname(&quot;clone&quot;), fn_name=&quot;hello&quot;)
# header
print(&quot;%-18s %-16s %-6s %s&quot; % (&quot;TIME(s)&quot;, &quot;COMM&quot;, &quot;PID&quot;, &quot;MESSAGE&quot;))
# process event
start = 0
def print_event(cpu, data, size):
global start
event = b[&quot;events&quot;].event(data)
if start == 0:
start = event.ts
time_s = (float(event.ts - start)) / 1000000000
print(&quot;%-18.9f %-16s %-6d %s&quot; % (time_s, event.comm, event.pid,
&quot;Hello, perf_output!&quot;))
# loop with callback to print_event
b[&quot;events&quot;].open_perf_buffer(print_event)
while 1:
b.perf_buffer_poll()
</code></pre>
<p>Things to learn:</p>
<ol>
<li><code>struct data_t</code>: This defines the C struct we'll use to pass data from kernel to user space.</li>
<li><code>BPF_PERF_OUTPUT(events)</code>: This names our output channel &quot;events&quot;.</li>
<li><code>struct data_t data = {};</code>: Create an empty data_t struct that we'll then populate.</li>
<li><code>bpf_get_current_pid_tgid()</code>: Returns the process ID in the lower 32 bits (kernel's view of the PID, which in user space is usually presented as the thread ID), and the thread group ID in the upper 32 bits (what user space often thinks of as the PID). By directly setting this to a u32, we discard the upper 32 bits. Should you be presenting the PID or the TGID? For a multi-threaded app, the TGID will be the same, so you need the PID to differentiate them, if that's what you want. It's also a question of expectations for the end user.</li>
<li><code>bpf_get_current_comm()</code>: Populates the first argument address with the current process name.</li>
<li><code>events.perf_submit()</code>: Submit the event for user space to read via a perf ring buffer.</li>
<li><code>def print_event()</code>: Define a Python function that will handle reading events from the <code>events</code> stream.</li>
<li><code>b[&quot;events&quot;].event(data)</code>: Now get the event as a Python object, auto-generated from the C declaration.</li>
<li><code>b[&quot;events&quot;].open_perf_buffer(print_event)</code>: Associate the Python <code>print_event</code> function with the <code>events</code> stream.</li>
<li><code>while 1: b.perf_buffer_poll()</code>: Block waiting for events.</li>
</ol>
<h3 id="lesson-8-sync_perf_outputpy"><a class="header" href="#lesson-8-sync_perf_outputpy">Lesson 8. sync_perf_output.py</a></h3>
<p>Rewrite sync_timing.py, from a prior lesson, to use <code>BPF_PERF_OUTPUT</code>.</p>
<h3 id="lesson-9-bitehistpy"><a class="header" href="#lesson-9-bitehistpy">Lesson 9. bitehist.py</a></h3>
<p>The following tool records a histogram of disk I/O sizes. Sample output:</p>
<pre><code class="language-sh"># bitehist.py
Tracing... Hit Ctrl-C to end.
^C
kbytes : count distribution
0 -&gt; 1 : 3 | |
2 -&gt; 3 : 0 | |
4 -&gt; 7 : 211 |********** |
8 -&gt; 15 : 0 | |
16 -&gt; 31 : 0 | |
32 -&gt; 63 : 0 | |
64 -&gt; 127 : 1 | |
128 -&gt; 255 : 800 |**************************************|
</code></pre>
<p>Code is <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/bitehist.py">examples/tracing/bitehist.py</a>:</p>
<pre><code class="language-Python">from __future__ import print_function
from bcc import BPF
from time import sleep
# load BPF program
b = BPF(text=&quot;&quot;&quot;
#include &lt;uapi/linux/ptrace.h&gt;
#include &lt;linux/blkdev.h&gt;
BPF_HISTOGRAM(dist);
int kprobe__blk_account_io_done(struct pt_regs *ctx, struct request *req)
{
dist.increment(bpf_log2l(req-&gt;__data_len / 1024));
return 0;
}
&quot;&quot;&quot;)
# header
print(&quot;Tracing... Hit Ctrl-C to end.&quot;)
# trace until Ctrl-C
try:
sleep(99999999)
except KeyboardInterrupt:
print()
# output
b[&quot;dist&quot;].print_log2_hist(&quot;kbytes&quot;)
</code></pre>
<p>A recap from earlier lessons:</p>
<ul>
<li><code>kprobe__</code>: This prefix means the rest will be treated as a kernel function name that will be instrumented using kprobe.</li>
<li><code>struct pt_regs *ctx, struct request *req</code>: Arguments to kprobe. The <code>ctx</code> is registers and BPF context, the <code>req</code> is the first argument to the instrumented function: <code>blk_account_io_done()</code>.</li>
<li><code>req-&gt;__data_len</code>: Dereferencing that member.</li>
</ul>
<p>New things to learn:</p>
<ol>
<li><code>BPF_HISTOGRAM(dist)</code>: Defines a BPF map object that is a histogram, and names it &quot;dist&quot;.</li>
<li><code>dist.increment()</code>: Increments the histogram bucket index provided as first argument by one by default. Optionally, custom increments can be passed as the second argument.</li>
<li><code>bpf_log2l()</code>: Returns the log-2 of the provided value. This becomes the index of our histogram, so that we're constructing a power-of-2 histogram.</li>
<li><code>b[&quot;dist&quot;].print_log2_hist(&quot;kbytes&quot;)</code>: Prints the &quot;dist&quot; histogram as power-of-2, with a column header of &quot;kbytes&quot;. The only data transferred from kernel to user space is the bucket counts, making this efficient.</li>
</ol>
<h3 id="lesson-10-disklatencypy"><a class="header" href="#lesson-10-disklatencypy">Lesson 10. disklatency.py</a></h3>
<p>Write a program that times disk I/O, and prints a histogram of their latency. Disk I/O instrumentation and timing can be found in the disksnoop.py program from a prior lesson, and histogram code can be found in bitehist.py from a prior lesson.</p>
<h3 id="lesson-11-vfsreadlatpy"><a class="header" href="#lesson-11-vfsreadlatpy">Lesson 11. vfsreadlat.py</a></h3>
<p>This example is split into separate Python and C files. Example output:</p>
<pre><code class="language-sh"># vfsreadlat.py 1
Tracing... Hit Ctrl-C to end.
usecs : count distribution
0 -&gt; 1 : 0 | |
2 -&gt; 3 : 2 |*********** |
4 -&gt; 7 : 7 |****************************************|
8 -&gt; 15 : 4 |********************** |
usecs : count distribution
0 -&gt; 1 : 29 |****************************************|
2 -&gt; 3 : 28 |************************************** |
4 -&gt; 7 : 4 |***** |
8 -&gt; 15 : 8 |*********** |
16 -&gt; 31 : 0 | |
32 -&gt; 63 : 0 | |
64 -&gt; 127 : 0 | |
128 -&gt; 255 : 0 | |
256 -&gt; 511 : 2 |** |
512 -&gt; 1023 : 0 | |
1024 -&gt; 2047 : 0 | |
2048 -&gt; 4095 : 0 | |
4096 -&gt; 8191 : 4 |***** |
8192 -&gt; 16383 : 6 |******** |
16384 -&gt; 32767 : 9 |************ |
32768 -&gt; 65535 : 6 |******** |
65536 -&gt; 131071 : 2 |** |
usecs : count distribution
0 -&gt; 1 : 11 |****************************************|
2 -&gt; 3 : 2 |******* |
4 -&gt; 7 : 10 |************************************ |
8 -&gt; 15 : 8 |***************************** |
16 -&gt; 31 : 1 |*** |
32 -&gt; 63 : 2 |******* |
[...]
</code></pre>
<p>Browse the code in <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/vfsreadlat.py">examples/tracing/vfsreadlat.py</a> and <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/vfsreadlat.c">examples/tracing/vfsreadlat.c</a>. Things to learn:</p>
<ol>
<li><code>b = BPF(src_file = &quot;vfsreadlat.c&quot;)</code>: Read the BPF C program from a separate source file.</li>
<li><code>b.attach_kretprobe(event=&quot;vfs_read&quot;, fn_name=&quot;do_return&quot;)</code>: Attaches the BPF C function <code>do_return()</code> to the return of the kernel function <code>vfs_read()</code>. This is a kretprobe: instrumenting the return from a function, rather than its entry.</li>
<li><code>b[&quot;dist&quot;].clear()</code>: Clears the histogram.</li>
</ol>
<h3 id="lesson-12-urandomreadpy"><a class="header" href="#lesson-12-urandomreadpy">Lesson 12. urandomread.py</a></h3>
<p>Tracing while a <code>dd if=/dev/urandom of=/dev/null bs=8k count=5</code> is run:</p>
<pre><code class="language-sh"># urandomread.py
TIME(s) COMM PID GOTBITS
24652832.956994001 smtp 24690 384
24652837.726500999 dd 24692 65536
24652837.727111001 dd 24692 65536
24652837.727703001 dd 24692 65536
24652837.728294998 dd 24692 65536
24652837.728888001 dd 24692 65536
</code></pre>
<p>Hah! I caught smtp by accident. Code is <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/urandomread.py">examples/tracing/urandomread.py</a>:</p>
<pre><code class="language-Python">from __future__ import print_function
from bcc import BPF
# load BPF program
b = BPF(text=&quot;&quot;&quot;
TRACEPOINT_PROBE(random, urandom_read) {
// args is from /sys/kernel/debug/tracing/events/random/urandom_read/format
bpf_trace_printk(&quot;%d\\n&quot;, args-&gt;got_bits);
return 0;
}
&quot;&quot;&quot;)
# header
print(&quot;%-18s %-16s %-6s %s&quot; % (&quot;TIME(s)&quot;, &quot;COMM&quot;, &quot;PID&quot;, &quot;GOTBITS&quot;))
# format output
while 1:
try:
(task, pid, cpu, flags, ts, msg) = b.trace_fields()
except ValueError:
continue
print(&quot;%-18.9f %-16s %-6d %s&quot; % (ts, task, pid, msg))
</code></pre>
<p>Things to learn:</p>
<ol>
<li><code>TRACEPOINT_PROBE(random, urandom_read)</code>: Instrument the kernel tracepoint <code>random:urandom_read</code>. These have a stable API, and thus are recommend to use instead of kprobes, wherever possible. You can run <code>perf list</code> for a list of tracepoints. Linux &gt;= 4.7 is required to attach BPF programs to tracepoints.</li>
<li><code>args-&gt;got_bits</code>: <code>args</code> is auto-populated to be a structure of the tracepoint arguments. The comment above says where you can see that structure. Eg:</li>
</ol>
<pre><code class="language-sh"># cat /sys/kernel/debug/tracing/events/random/urandom_read/format
name: urandom_read
ID: 972
format:
field:unsigned short common_type; offset:0; size:2; signed:0;
field:unsigned char common_flags; offset:2; size:1; signed:0;
field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
field:int common_pid; offset:4; size:4; signed:1;
field:int got_bits; offset:8; size:4; signed:1;
field:int pool_left; offset:12; size:4; signed:1;
field:int input_left; offset:16; size:4; signed:1;
print fmt: &quot;got_bits %d nonblocking_pool_entropy_left %d input_entropy_left %d&quot;, REC-&gt;got_bits, REC-&gt;pool_left, REC-&gt;input_left
</code></pre>
<p>In this case, we were printing the <code>got_bits</code> member.</p>
<h3 id="lesson-13-disksnooppy-fixed"><a class="header" href="#lesson-13-disksnooppy-fixed">Lesson 13. disksnoop.py fixed</a></h3>
<p>Convert disksnoop.py from a previous lesson to use the <code>block:block_rq_issue</code> and <code>block:block_rq_complete</code> tracepoints.</p>
<h3 id="lesson-14-strlen_countpy"><a class="header" href="#lesson-14-strlen_countpy">Lesson 14. strlen_count.py</a></h3>
<p>This program instruments a user-level function, the <code>strlen()</code> library function, and frequency counts its string argument. Example output:</p>
<pre><code class="language-sh"># strlen_count.py
Tracing strlen()... Hit Ctrl-C to end.
^C COUNT STRING
1 &quot; &quot;
1 &quot;/bin/ls&quot;
1 &quot;.&quot;
1 &quot;cpudist.py.1&quot;
1 &quot;.bashrc&quot;
1 &quot;ls --color=auto&quot;
1 &quot;key_t&quot;
[...]
10 &quot;a7:~# &quot;
10 &quot;/root&quot;
12 &quot;LC_ALL&quot;
12 &quot;en_US.UTF-8&quot;
13 &quot;en_US.UTF-8&quot;
20 &quot;~&quot;
70 &quot;#%^,~:-=?+/}&quot;
340 &quot;\x01\x1b]0;root@bgregg-test: ~\x07\x02root@bgregg-test:~# &quot;
</code></pre>
<p>These are various strings that are being processed by this library function while tracing, along with their frequency counts. <code>strlen()</code> was called on &quot;LC_ALL&quot; 12 times, for example.</p>
<p>Code is <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/strlen_count.py">examples/tracing/strlen_count.py</a>:</p>
<pre><code class="language-Python">from __future__ import print_function
from bcc import BPF
from time import sleep
# load BPF program
b = BPF(text=&quot;&quot;&quot;
#include &lt;uapi/linux/ptrace.h&gt;
struct key_t {
char c[80];
};
BPF_HASH(counts, struct key_t);
int count(struct pt_regs *ctx) {
if (!PT_REGS_PARM1(ctx))
return 0;
struct key_t key = {};
u64 zero = 0, *val;
bpf_probe_read_user(&amp;key.c, sizeof(key.c), (void *)PT_REGS_PARM1(ctx));
// could also use `counts.increment(key)`
val = counts.lookup_or_try_init(&amp;key, &amp;zero);
if (val) {
(*val)++;
}
return 0;
};
&quot;&quot;&quot;)
b.attach_uprobe(name=&quot;c&quot;, sym=&quot;strlen&quot;, fn_name=&quot;count&quot;)
# header
print(&quot;Tracing strlen()... Hit Ctrl-C to end.&quot;)
# sleep until Ctrl-C
try:
sleep(99999999)
except KeyboardInterrupt:
pass
# print output
print(&quot;%10s %s&quot; % (&quot;COUNT&quot;, &quot;STRING&quot;))
counts = b.get_table(&quot;counts&quot;)
for k, v in sorted(counts.items(), key=lambda counts: counts[1].value):
print(&quot;%10d \&quot;%s\&quot;&quot; % (v.value, k.c.encode('string-escape')))
</code></pre>
<p>Things to learn:</p>
<ol>
<li><code>PT_REGS_PARM1(ctx)</code>: This fetches the first argument to <code>strlen()</code>, which is the string.</li>
<li><code>b.attach_uprobe(name=&quot;c&quot;, sym=&quot;strlen&quot;, fn_name=&quot;count&quot;)</code>: Attach to library &quot;c&quot; (if this is the main program, use its pathname), instrument the user-level function <code>strlen()</code>, and on execution call our C function <code>count()</code>.</li>
</ol>
<h3 id="lesson-15-nodejs_http_serverpy"><a class="header" href="#lesson-15-nodejs_http_serverpy">Lesson 15. nodejs_http_server.py</a></h3>
<p>This program instruments a user statically-defined tracing (USDT) probe, which is the user-level version of a kernel tracepoint. Sample output:</p>
<pre><code class="language-sh"># nodejs_http_server.py 24728
TIME(s) COMM PID ARGS
24653324.561322998 node 24728 path:/index.html
24653335.343401998 node 24728 path:/images/welcome.png
24653340.510164998 node 24728 path:/images/favicon.png
</code></pre>
<p>Relevant code from <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/nodejs_http_server.py">examples/tracing/nodejs_http_server.py</a>:</p>
<pre><code class="language-Python">from __future__ import print_function
from bcc import BPF, USDT
import sys
if len(sys.argv) &lt; 2:
print(&quot;USAGE: nodejs_http_server PID&quot;)
exit()
pid = sys.argv[1]
debug = 0
# load BPF program
bpf_text = &quot;&quot;&quot;
#include &lt;uapi/linux/ptrace.h&gt;
int do_trace(struct pt_regs *ctx) {
uint64_t addr;
char path[128]={0};
bpf_usdt_readarg(6, ctx, &amp;addr);
bpf_probe_read_user(&amp;path, sizeof(path), (void *)addr);
bpf_trace_printk(&quot;path:%s\\n&quot;, path);
return 0;
};
&quot;&quot;&quot;
# enable USDT probe from given PID
u = USDT(pid=int(pid))
u.enable_probe(probe=&quot;http__server__request&quot;, fn_name=&quot;do_trace&quot;)
if debug:
print(u.get_text())
print(bpf_text)
# initialize BPF
b = BPF(text=bpf_text, usdt_contexts=[u])
</code></pre>
<p>Things to learn:</p>
<ol>
<li><code>bpf_usdt_readarg(6, ctx, &amp;addr)</code>: Read the address of argument 6 from the USDT probe into <code>addr</code>.</li>
<li><code>bpf_probe_read_user(&amp;path, sizeof(path), (void *)addr)</code>: Now the string <code>addr</code> points to into our <code>path</code> variable.</li>
<li><code>u = USDT(pid=int(pid))</code>: Initialize USDT tracing for the given PID.</li>
<li><code>u.enable_probe(probe=&quot;http__server__request&quot;, fn_name=&quot;do_trace&quot;)</code>: Attach our <code>do_trace()</code> BPF C function to the Node.js <code>http__server__request</code> USDT probe.</li>
<li><code>b = BPF(text=bpf_text, usdt_contexts=[u])</code>: Need to pass in our USDT object, <code>u</code>, to BPF object creation.</li>
</ol>
<h3 id="lesson-16-task_switchc"><a class="header" href="#lesson-16-task_switchc">Lesson 16. task_switch.c</a></h3>
<p>This is an older tutorial included as a bonus lesson. Use this for recap and to reinforce what you've already learned.</p>
<p>This is a slightly more complex tracing example than Hello World. This program
will be invoked for every task change in the kernel, and record in a BPF map
the new and old pids.</p>
<p>The C program below introduces a new concept: the prev argument. This
argument is treated specially by the BCC frontend, such that accesses
to this variable are read from the saved context that is passed by the
kprobe infrastructure. The prototype of the args starting from
position 1 should match the prototype of the kernel function being
kprobed. If done so, the program will have seamless access to the
function parameters.</p>
<pre><code class="language-c">#include &lt;uapi/linux/ptrace.h&gt;
#include &lt;linux/sched.h&gt;
struct key_t {
u32 prev_pid;
u32 curr_pid;
};
BPF_HASH(stats, struct key_t, u64, 1024);
int count_sched(struct pt_regs *ctx, struct task_struct *prev) {
struct key_t key = {};
u64 zero = 0, *val;
key.curr_pid = bpf_get_current_pid_tgid();
key.prev_pid = prev-&gt;pid;
// could also use `stats.increment(key);`
val = stats.lookup_or_try_init(&amp;key, &amp;zero);
if (val) {
(*val)++;
}
return 0;
}
</code></pre>
<p>The userspace component loads the file shown above, and attaches it to the
<code>finish_task_switch</code> kernel function.
The <code>[]</code> operator of the BPF object gives access to each BPF_HASH in the
program, allowing pass-through access to the values residing in the kernel. Use
the object as you would any other python dict object: read, update, and deletes
are all allowed.</p>
<pre><code class="language-python">from bcc import BPF
from time import sleep
b = BPF(src_file=&quot;task_switch.c&quot;)
b.attach_kprobe(event=&quot;finish_task_switch&quot;, fn_name=&quot;count_sched&quot;)
# generate many schedule events
for i in range(0, 100): sleep(0.01)
for k, v in b[&quot;stats&quot;].items():
print(&quot;task_switch[%5d-&gt;%5d]=%u&quot; % (k.prev_pid, k.curr_pid, v.value))
</code></pre>
<p>These programs can be found in the files <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/task_switch.c">examples/tracing/task_switch.c</a> and <a href="https://github.com/iovisor/bcc/tree/master/examples/tracing/task_switch.py">examples/tracing/task_switch.py</a> respectively.</p>
<h3 id="lesson-17-further-study"><a class="header" href="#lesson-17-further-study">Lesson 17. Further Study</a></h3>
<p>For further study, see Sasha Goldshtein's <a href="https://github.com/goldshtn/linux-tracing-workshop">linux-tracing-workshop</a>, which contains additional labs. There are also many tools in bcc /tools to study.</p>
<p>Please read <a href="https://github.com/iovisor/bcc/tree/master/CONTRIBUTING-SCRIPTS.md">CONTRIBUTING-SCRIPTS.md</a> if you wish to contribute tools to bcc. At the bottom of the main <a href="https://github.com/iovisor/bcc/tree/master/README.md">README.md</a>, you'll also find methods for contacting us. Good luck, and happy tracing!</p>
<h2 id="networking-1"><a class="header" href="#networking-1">Networking</a></h2>
<p>To do.</p>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<!-- Mobile navigation buttons -->
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
</nav>
</div>
<script>
window.playground_copyable = true;
</script>
<script src="elasticlunr.min.js"></script>
<script src="mark.min.js"></script>
<script src="searcher.js"></script>
<script src="clipboard.min.js"></script>
<script src="highlight.js"></script>
<script src="book.js"></script>
<!-- Custom JS scripts -->
<script>
window.addEventListener('load', function() {
window.setTimeout(window.print, 100);
});
</script>
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink