kernel/bpf-developer-tutorial
1
0
Fork 0
mirror of https://github.com/eunomia-bpf/bpf-developer-tutorial.git synced 2026-10-30 12:15:40 +08:00
Code Issues Packages Projects Releases Wiki Activity

Enhance GPU and NPU kernel driver documentation and monitoring scripts

Browse Source 
- Updated README.zh.md for GPU kernel driver to improve clarity and formatting.
- Added nvidia_driver.bt script for monitoring NVIDIA proprietary GPU driver activity using kernel probes.
- Revised README.md for NPU kernel driver to enhance explanations and correct minor grammatical issues.
This commit is contained in:
yunwei37
2025-10-13 07:18:48 -07:00
parent d4ec997ab2
commit b8cc834d7f
4 changed files with 668 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

231
src/xpu/gpu-kernel-driver/README.md
View File

@@ -12,7 +12,7 @@ GPU tracepoints are instrumentation points built into the kernel's Direct Render
The key insight: kernel tracepoints activate only when events occur, adding nanoseconds of overhead per event. They capture 100% of activity including microsecond-duration jobs. Polling-based monitoring checks GPU state every 100ms and misses short-lived operations entirely.
GPU tracepoints span three layers. **DRM scheduler tracepoints** (`gpu_scheduler` event group) are stable uAPI - their format never changes. They work identically across Intel, AMD, and Nouveau drivers for vendor-neutral monitoring. **Vendor-specific tracepoints** expose driver internals - Intel i915 tracks GEM object creation and VMA binding, AMD AMDGPU monitors buffer objects and command submission. **Generic DRM tracepoints** handle display synchronization through vblank events for diagnosing frame drops.
GPU tracepoints span three layers. DRM scheduler tracepoints (`gpu_scheduler` event group) are stable uAPI; their format never changes. They work identically across Intel, AMD, and Nouveau drivers for vendor-neutral monitoring. Vendor-specific tracepoints expose driver internals. Intel i915 tracks GEM object creation and VMA binding, while AMD AMDGPU monitors buffer objects and command submission. Generic DRM tracepoints handle display synchronization through vblank events for diagnosing frame drops.
## DRM Scheduler Monitor: Universal GPU Tracking
@@ -109,7 +109,7 @@ END
### Understanding the Script
The script attaches to three stable DRM scheduler tracepoints. When `drm_run_job` fires, a job transitions from "queued in software" to "running on silicon." The tracepoint captures `args->id` (job ID for correlation), `args->name` (ring name - which execution engine like graphics, compute, or video decode), `args->job_count` (queue depth - how many jobs are waiting), and `args->hw_job_count` (jobs currently executing on GPU hardware).
The script attaches to three stable DRM scheduler tracepoints. When `drm_run_job` fires, a job transitions from "queued in software" to "running on silicon." The tracepoint captures `args->id` (job ID for correlation), `args->name` (ring name indicating which execution engine like graphics, compute, or video decode), `args->job_count` (queue depth indicating how many jobs are waiting), and `args->hw_job_count` (jobs currently executing on GPU hardware).
The format `entity=0xffff888... id=12345 fence=0xffff888... ring=gfx job count:5 hw job count:2` tells you job 12345 on the graphics ring started executing with 5 jobs queued behind it and 2 jobs currently running on hardware. Multi-engine GPUs can run jobs in parallel across different rings.
@@ -125,45 +125,242 @@ At program end, the `END` block prints statistics. `@jobs_per_ring` shows job co
## Intel i915 Tracepoints: Memory Management Deep Dive
Intel's i915 driver exposes detailed tracepoints for memory operations. These require `CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS=y` in your kernel config - check with `grep CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS /boot/config-$(uname -r)`.
Intel's i915 driver exposes detailed tracepoints for memory operations. These require `CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS=y` in your kernel config; check with `grep CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS /boot/config-$(uname -r)`.
**i915_gem_object_create** fires when the driver allocates a GEM (Graphics Execution Manager) object - the fundamental unit of GPU-accessible memory. Format: `obj=0xffff888... size=0x100000` indicates allocating a 1MB object. Track total allocated memory over time to detect leaks. Sudden allocation spikes before performance drops suggest memory pressure. Correlate object pointers with subsequent bind/fault events to understand object lifecycle.
i915_gem_object_create fires when the driver allocates a GEM (Graphics Execution Manager) object, the fundamental unit of GPU-accessible memory. Format: `obj=0xffff888... size=0x100000` indicates allocating a 1MB object. Track total allocated memory over time to detect leaks. Sudden allocation spikes before performance drops suggest memory pressure. Correlate object pointers with subsequent bind/fault events to understand object lifecycle.
**i915_vma_bind** tracks mapping memory into GPU address space. Allocating memory isn't enough - it must be bound into GPU virtual address space. Format: `obj=0xffff888... offset=0x0000100000 size=0x10000 mappable vm=0xffff888...` shows 64KB bound at GPU virtual address 0x100000. Frequent rebinding indicates memory thrashing - the driver evicting and rebinding objects under pressure. GPU page faults often correlate with bind operations.
i915_vma_bind tracks mapping memory into GPU address space. Allocating memory isn't enough; it must be bound into GPU virtual address space. Format: `obj=0xffff888... offset=0x0000100000 size=0x10000 mappable vm=0xffff888...` shows 64KB bound at GPU virtual address 0x100000. Frequent rebinding indicates memory thrashing, where the driver evicts and rebinds objects under pressure. GPU page faults often correlate with bind operations.
**i915_gem_shrink** captures memory pressure response. Under memory pressure, the driver reclaims GPU memory. Format: `dev=0 target=0x1000000 flags=0x3` means the driver tries to reclaim 16MB. High shrink activity indicates undersized GPU memory for the workload. Correlate with performance drops - if shrinking happens during frame rendering, it causes stutters.
i915_gem_shrink captures memory pressure response. Under memory pressure, the driver reclaims GPU memory. Format: `dev=0 target=0x1000000 flags=0x3` means the driver tries to reclaim 16MB. High shrink activity indicates undersized GPU memory for the workload. Correlate with performance drops; if shrinking happens during frame rendering, it causes stutters.
**i915_gem_object_fault** tracks page faults when CPU or GPU accesses unmapped memory. Format: `obj=0xffff888... GTT index=128 writable` indicates a write fault on Graphics Translation Table page 128. Faults are expensive - they stall execution while the kernel resolves the missing mapping. Write faults are more expensive than reads (require invalidating caches). GTT faults indicate incomplete resource binding before job submission.
i915_gem_object_fault tracks page faults when CPU or GPU accesses unmapped memory. Format: `obj=0xffff888... GTT index=128 writable` indicates a write fault on Graphics Translation Table page 128. Faults are expensive because they stall execution while the kernel resolves the missing mapping. Write faults are more expensive than reads since they require invalidating caches. GTT faults indicate incomplete resource binding before job submission.
## AMD AMDGPU Tracepoints: Command Submission Pipeline
AMD's AMDGPU driver provides comprehensive tracing of command submission and hardware interrupts.
**amdgpu_cs_ioctl** captures userspace command submission. When an application submits GPU work via ioctl, this tracepoint fires. Format: `sched_job=12345 timeline=gfx context=1000 seqno=567 ring_name=gfx_0.0.0 num_ibs=2` shows job 12345 submitted to graphics ring with 2 indirect buffers. This marks when userspace hands off work to kernel. Record timestamp to measure submission-to-execution latency when combined with `amdgpu_sched_run_job`. High frequency indicates small batches - potential for better batching.
amdgpu_cs_ioctl captures userspace command submission. When an application submits GPU work via ioctl, this tracepoint fires. Format: `sched_job=12345 timeline=gfx context=1000 seqno=567 ring_name=gfx_0.0.0 num_ibs=2` shows job 12345 submitted to graphics ring with 2 indirect buffers. This marks when userspace hands off work to kernel. Record timestamp to measure submission to execution latency when combined with `amdgpu_sched_run_job`. High frequency indicates small batches and potential for better batching.
**amdgpu_sched_run_job** fires when the kernel scheduler starts executing a previously submitted job. Comparing timestamps with `amdgpu_cs_ioctl` reveals submission latency. Submission latencies over 100μs indicate kernel scheduling delays. Per-ring latencies show if specific engines are scheduling-bound.
amdgpu_sched_run_job fires when the kernel scheduler starts executing a previously submitted job. Comparing timestamps with `amdgpu_cs_ioctl` reveals submission latency. Submission latencies over 100μs indicate kernel scheduling delays. Per-ring latencies show if specific engines are scheduling-bound.
**amdgpu_bo_create** tracks buffer object allocation - AMD's equivalent to i915 GEM objects. Format: `bo=0xffff888... pages=256 type=2 preferred=4 allowed=7 visible=1` allocates 1MB (256 pages). Type indicates VRAM vs GTT (system memory accessible by GPU). Preferred/allowed domains show placement policy. Type mismatches (requesting VRAM but falling back to GTT) indicate VRAM exhaustion. Visible flag indicates CPU-accessible memory - expensive, use sparingly.
amdgpu_bo_create tracks buffer object allocation, AMD's equivalent to i915 GEM objects. Format: `bo=0xffff888... pages=256 type=2 preferred=4 allowed=7 visible=1` allocates 1MB (256 pages). Type indicates VRAM vs GTT (system memory accessible by GPU). Preferred/allowed domains show placement policy. Type mismatches where VRAM is requested but GTT is used indicate VRAM exhaustion. Visible flag indicates CPU-accessible memory, which is expensive and should be used sparingly.
**amdgpu_bo_move** fires when buffer objects migrate between VRAM and GTT. Migrations are expensive (require copying data over PCIe). Excessive moves indicate memory thrashing - working set exceeds VRAM capacity. Measure move frequency and size to quantify PCIe bandwidth consumption. Correlate with performance drops - migrations stall GPU execution.
amdgpu_bo_move fires when buffer objects migrate between VRAM and GTT. Migrations are expensive because they require copying data over PCIe. Excessive moves indicate memory thrashing where the working set exceeds VRAM capacity. Measure move frequency and size to quantify PCIe bandwidth consumption. Correlate with performance drops since migrations stall GPU execution.
**amdgpu_iv** captures GPU interrupts. The GPU signals interrupts for completed work, errors, and events. Format: `ih:0 client_id:1 src_id:42 ring:0 vmid:5 timestamp:1234567890 pasid:100 src_data: 00000001...` captures interrupt details. Source ID indicates interrupt type (completion, fault, thermal). High interrupt rates impact CPU performance. VMID and PASID identify which process/VM triggered the interrupt - critical for multi-tenant debugging.
amdgpu_iv captures GPU interrupts. The GPU signals interrupts for completed work, errors, and events. Format: `ih:0 client_id:1 src_id:42 ring:0 vmid:5 timestamp:1234567890 pasid:100 src_data: 00000001...` captures interrupt details. Source ID indicates interrupt type (completion, fault, thermal). High interrupt rates impact CPU performance. VMID and PASID identify which process/VM triggered the interrupt, which is critical for multi-tenant debugging.
## DRM Vblank Tracepoints: Display Synchronization
Vblank (vertical blanking) events synchronize rendering with display refresh. Missing vblanks causes dropped frames and stutter.
**drm_vblank_event** fires when the display enters vertical blanking period. Format: `crtc=0 seq=12345 time=1234567890 high-prec=true` indicates vblank on display controller 0, sequence number 12345. Track vblank frequency to verify refresh rate (60Hz = 60 vblanks/second). Missed sequences indicate frame drops. High-precision timestamps enable sub-millisecond frame timing analysis.
drm_vblank_event fires when the display enters vertical blanking period. Format: `crtc=0 seq=12345 time=1234567890 high-prec=true` indicates vblank on display controller 0, sequence number 12345. Track vblank frequency to verify refresh rate (60Hz = 60 vblanks/second). Missed sequences indicate frame drops. High-precision timestamps enable sub-millisecond frame timing analysis.
**drm_vblank_event_queued** and **drm_vblank_event_delivered** track vblank event delivery to userspace. Queuing latency (queue to delivery) measures kernel scheduling delay. Total latency (vblank to delivery) includes both kernel and driver processing. Latencies over 1ms indicate compositor problems. Correlate with frame drops visible to users - events delivered late mean missed frames.
drm_vblank_event_queued and drm_vblank_event_delivered track vblank event delivery to userspace. Queuing latency (queue to delivery) measures kernel scheduling delay. Total latency (vblank to delivery) includes both kernel and driver processing. Latencies over 1ms indicate compositor problems. Correlate with frame drops visible to users since events delivered late mean missed frames.
## NVIDIA Proprietary Driver: Different Architecture
Unlike Intel, AMD, and Nouveau which use the kernel's Direct Rendering Manager (DRM) subsystem, NVIDIA's proprietary driver (nvidia.ko) operates outside DRM. It implements its own kernel module interface with vendor-specific functions and a single tracepoint. This architectural difference means NVIDIA GPUs require different monitoring approaches; we attach to kernel probes on nvidia.ko functions instead of DRM tracepoints.
The key distinction: DRM drivers expose standardized `gpu_scheduler` tracepoints that work identically across vendors. NVIDIA's closed-source driver provides only one tracepoint (`nvidia:nvidia_dev_xid` for hardware errors) and requires monitoring internal kernel functions like `nvidia_open`, `nvidia_unlocked_ioctl`, and `nvidia_isr`. This makes NVIDIA monitoring more fragile since function names can change between driver versions, but it still provides valuable insights into GPU activity.
### NVIDIA Driver Monitoring: nvidia_driver.bt
The `nvidia_driver.bt` script tracks NVIDIA GPU operations through kernel probes on the proprietary driver. Unlike DRM scheduler monitoring which is vendor-neutral, this script is NVIDIA-specific and requires the proprietary nvidia.ko module loaded.
The script monitors six key areas:
- **Device operations**: Tracks when processes open/close GPU devices and issue ioctl commands
- **Memory management**: Records mmap operations, page faults, and VMA lifecycle
- **Interrupt handling**: Measures ISR latency from hardware interrupt to processing
- **P2P communication**: Captures GPU-to-GPU page requests and DMA mapping
- **Power management**: Times suspend/resume cycles
- **Error reporting**: Reports Xid hardware/driver errors immediately
### Complete Bpftrace Script: scripts/nvidia_driver.bt
```c
#!/usr/bin/env bpftrace
/* nvidia_driver.bt - Monitor NVIDIA proprietary GPU driver activity */
BEGIN
{
printf("Tracing NVIDIA GPU driver activity... Hit Ctrl-C to end.\n");
printf("%-12s %-18s %-16s %-8s %-8s %-20s\n",
"TIME(ms)", "EVENT", "COMM", "PID", "GPU_ID", "DETAILS");
}
kprobe:nvidia_open
{
printf("%-12llu %-18s %-16s %-8d %-8s %s\n",
elapsed / 1000000, "OPEN", comm, pid, "-", "GPU device opened");
@opens[comm] = count();
@open_pids[pid] = 1;
}
kprobe:nvidia_unlocked_ioctl
{
@ioctl_count = count();
@ioctls_per_process[comm] = count();
if (rand % 100 == 0) { /* Sample 1% */
printf("%-12llu %-18s %-16s %-8d %-8s cmd=0x%lx\n",
elapsed / 1000000, "IOCTL", comm, pid, "-", arg1);
}
}
kprobe:nvidia_mmap
{
@mmap_count = count();
@total_mmap_bytes = sum(arg2);
printf("%-12llu %-18s %-16s %-8d %-8s offset=0x%lx size=%lu\n",
elapsed / 1000000, "MMAP", comm, pid, "-", arg1, arg2);
}
kprobe:nvidia_isr
{
@isr_count = count();
@last_isr_time = nsecs;
}
kprobe:nvidia_isr_kthread_bh
{
@isr_bh_count = count();
if (@last_isr_time > 0) {
@isr_latency_us = hist((nsecs - @last_isr_time) / 1000);
}
}
tracepoint:nvidia:nvidia_dev_xid
{
printf("\n!!! GPU ERROR !!!\n");
printf(" └─ Xid: %u - %s\n\n", args->error_code, str(args->msg));
@xid_errors = count();
@xid_codes[args->error_code] = count();
}
END
{
printf("\n=== NVIDIA GPU Driver Statistics ===\n");
printf("Opens by process:\n"); print(@opens);
printf("Total ioctls:\n"); print(@ioctl_count);
printf("Top ioctl callers:\n"); print(@ioctls_per_process);
printf("Total mmaps:\n"); print(@mmap_count);
printf("Poll calls:\n"); print(@poll_count);
}
```
### Understanding NVIDIA Driver Operations
Device Operations: `nvidia_open` fires when a process opens `/dev/nvidia0` (or other GPU device nodes). This is the entry point for GPU access. CUDA applications, OpenGL contexts, and compute workloads all start here. Track `@opens[comm]` to see which applications use the GPU. Each open usually corresponds to a CUDA context or graphics context creation.
IOCTL Commands: `nvidia_unlocked_ioctl` is the highest-frequency operation. Every GPU command submission, memory allocation, synchronization, and query goes through ioctls. A single frame of graphics rendering may issue hundreds of ioctls. The script samples 1% of ioctls to reduce overhead while maintaining visibility. High ioctl rates (>100k/sec) indicate fine-grained GPU interactions and potential for better batching. The `arg1` parameter contains the ioctl command code identifying the operation type.
Memory Mapping: `nvidia_mmap` maps GPU memory into process virtual address space, enabling CPU access to GPU buffers. Format `offset=0x100000 size=1048576` maps 1MB of GPU memory. Track `@total_mmap_bytes` to understand GPU memory usage. Frequent large mmaps may indicate CPU-GPU data transfer patterns. Unified memory (CUDA managed memory) triggers extensive mmap activity as the driver migrates pages between CPU and GPU.
Page Faults: `nvidia_fault` captures expensive events when CPU or GPU accesses unmapped memory. Page faults stall execution while the driver resolves the mapping. High fault counts indicate unified memory page migration under memory pressure, incomplete memory binding before kernel launch, or CPU accessing GPU memory without proper mapping. Correlate faults with performance drops. Faults during critical sections (kernel execution) directly impact throughput.
Interrupt Handling: `nvidia_isr` fires when the GPU signals an interrupt, typically for completed work, errors, or synchronization events. Modern GPUs use MSI-X interrupts for lower latency. The bottom-half handler (`nvidia_isr_kthread_bh`) performs the actual work processing. ISR latency (time from hardware interrupt to bottom-half processing) indicates kernel scheduling efficiency. High ISR rates (>10k/sec) may impact CPU performance since each interrupt costs CPU cycles.
P2P Transfers: `nvidia_p2p_get_pages` and `nvidia_p2p_dma_map_pages` enable direct GPU-to-GPU transfers over NVLink or PCIe without CPU involvement. Multi-GPU workloads (distributed training, GPU clusters) rely on P2P for high bandwidth. Track P2P operations to verify GPU-GPU communication is working. Missing P2P support (older PCIe configurations) forces slower CPU-mediated transfers.
Xid Errors: The `nvidia:nvidia_dev_xid` tracepoint is NVIDIA's only exposed tracepoint. Xid errors indicate hardware problems (GPU faults, memory errors, thermal issues) or driver bugs. Common Xids include Xid 31 (GPU memory page fault), Xid 43 (GPU stopped responding/hang), Xid 45 (GPU memory ECC error), and Xid 79 (GPU fell off the bus/PCIe error). Any Xid error requires investigation since they often precede crashes or data corruption.
### Running NVIDIA Driver Monitor
Verify NVIDIA driver is loaded and check available probes:
```bash
# Check NVIDIA driver module
lsmod | grep nvidia
# List available NVIDIA probes
sudo bpftrace -l 'kprobe:nvidia_*' | head -20
sudo bpftrace -l 'tracepoint:nvidia:*'
```
Run the monitor during GPU workloads:
```bash
cd bpf-developer-tutorial/src/xpu/gpu-kernel-driver
sudo bpftrace scripts/nvidia_driver.bt
```
**Real execution output** capturing llama-server (LLM inference), nvtop (GPU monitoring), and CUDA application cleanup:
```
Attaching 18 probes...
Tracing NVIDIA GPU driver activity... Hit Ctrl-C to end.
TIME(ms) EVENT COMM PID GPU_ID DETAILS
2627 IOCTL nvtop 759434 - cmd=0xc020462a
38984 CLOSE python 783815 - GPU device closed
70693 CLOSE cuda00001400006 781802 - GPU device closed
72427 OPEN llama-server 800150 - GPU device opened
72427 CLOSE llama-server 800150 - GPU device closed
72427 OPEN llama-server 800150 - GPU device opened
72428 OPEN llama-server 800150 - GPU device opened
72431 MMAP llama-server 800150 - offset=0xffff968357d37140 size=...
72448 OPEN llama-server 800150 - GPU device opened
72458 OPEN llama-server 800150 - GPU device opened
... (39 opens, 26 mmaps from llama-server during initialization)
========================================
NVIDIA GPU Driver Statistics
========================================
--- Device Operations ---
Opens by process:
@opens[llama-server]: 39
Closes by process:
@closes[llama-server]: 1
@closes[python]: 8
@closes[cuda00001400006]: 38
Total ioctls:
@ioctl_count: 2779
Top ioctl callers:
@ioctls_per_process[llama-server]: 422
@ioctls_per_process[nvtop]: 2357
Total mmaps:
@mmap_count: 26
Total mmap bytes:
@total_mmap_bytes: 18446744046197555104
--- Memory Management ---
Total page faults:
@fault_count: 0
VMA releases:
@vma_release_count: 29
--- Interrupt Handling ---
Total ISR calls:
@isr_count: 0
--- Async Operations ---
Poll calls:
@poll_count: 24254
Currently open PIDs:
@open_pids[800150]: 1
```
**Analysis**: This real-world trace reveals several patterns. The llama-server process opened the GPU device 39 times during initialization - typical for LLM inference engines that initialize multiple CUDA contexts for different model layers or batching strategies. The 422 ioctls from llama-server indicate active inference work. The nvtop monitoring tool issued 2,357 ioctls polling GPU state. The script captured 38 device closes from a terminating CUDA application (cuda00001400006) and 8 from a Python process - showing cleanup patterns. The 24,254 poll calls indicate high async I/O activity from monitoring tools. Zero page faults suggests all memory was properly pre-allocated. Zero ISR events during this capture window indicates the GPU was between computation batches - ISRs fire when GPU work completes. No Xid errors means healthy hardware operation. The currently-open PID 800150 (llama-server) remained active after the trace ended.
## Running the Monitor Scripts
Navigate to the scripts directory and run the DRM scheduler monitor. It works on all GPUs:
Navigate to the tutorial directory and run the appropriate monitor for your GPU.
**For DRM-based GPUs (Intel, AMD, Nouveau)** - Universal monitoring:
```bash
cd bpf-developer-tutorial/src/xpu/gpu-kernel-driver/scripts
sudo bpftrace drm_scheduler.bt
cd bpf-developer-tutorial/src/xpu/gpu-kernel-driver
sudo bpftrace scripts/drm_scheduler.bt
```
**For NVIDIA Proprietary Driver**:
```bash
cd bpf-developer-tutorial/src/xpu/gpu-kernel-driver
sudo bpftrace scripts/nvidia_driver.bt
```
Expected output: