mirror of
https://github.com/eunomia-bpf/bpf-developer-tutorial.git
synced 2026-03-19 19:35:40 +08:00
Fix curl hanging in XDP load balancer: backend servers reject mismatched Host headers (#198)
* Initial plan * Add simple HTTP server and update docs to fix curl hanging issue Co-authored-by: yunwei37 <34985212+yunwei37@users.noreply.github.com> * Fix code review issues: remove unnecessary shutdown() and fix spacing Co-authored-by: yunwei37 <34985212+yunwei37@users.noreply.github.com> * Add test script to verify HTTP servers handle mismatched Host headers Co-authored-by: yunwei37 <34985212+yunwei37@users.noreply.github.com> * Add comprehensive solution documentation for curl hanging issue Co-authored-by: yunwei37 <34985212+yunwei37@users.noreply.github.com> * Co-authored-by: yunwei37 <34985212+yunwei37@users.noreply.github.com> * Update src/42-xdp-loadbalancer/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: 云微 <1067852565@qq.com> * Update src/42-xdp-loadbalancer/README.zh.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: 云微 <1067852565@qq.com> * Update src/42-xdp-loadbalancer/SOLUTION.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: 云微 <1067852565@qq.com> * Delete src/42-xdp-loadbalancer/__pycache__/simple_http_server.cpython-312.pyc Signed-off-by: 云微 <1067852565@qq.com> * Delete src/42-xdp-loadbalancer/SOLUTION.md Signed-off-by: 云微 <1067852565@qq.com> --------- Signed-off-by: 云微 <1067852565@qq.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: yunwei37 <34985212+yunwei37@users.noreply.github.com> Co-authored-by: 云微 <1067852565@qq.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Copilot
@@ -470,9 +470,20 @@ You can test the setup by starting HTTP servers on the two backend namespaces (`
|
||||
|
||||
Start servers on `h2` and `h3`:
|
||||
|
||||
**Important**: The HTTP servers must bind to `0.0.0.0` so they listen on all local addresses in the namespace and can accept connections arriving via the load balancer. The forwarded HTTP requests will use the load balancer's virtual IP and port in the `Host` header (for example, `Host: 10.0.0.10:8000`), which may differ from the backend's own IP address; for this tutorial we use simple HTTP servers that accept such requests without enforcing strict `Host` header checks.
|
||||
|
||||
**Option 1**: Using the provided simple HTTP server (recommended):
|
||||
|
||||
```sh
|
||||
sudo ip netns exec h2 python3 -m http.server
|
||||
sudo ip netns exec h3 python3 -m http.server
|
||||
sudo ip netns exec h2 python3 simple_http_server.py &
|
||||
sudo ip netns exec h3 python3 simple_http_server.py &
|
||||
```
|
||||
|
||||
**Option 2**: Using Python's built-in http.server with explicit binding:
|
||||
|
||||
```sh
|
||||
sudo ip netns exec h2 python3 -m http.server --bind 0.0.0.0 &
|
||||
sudo ip netns exec h3 python3 -m http.server --bind 0.0.0.0 &
|
||||
```
|
||||
|
||||
Then, send a request to the load balancer IP:
|
||||
@@ -483,6 +494,8 @@ curl 10.0.0.10:8000
|
||||
|
||||
The load balancer will distribute traffic to the backends (`h2` and `h3`) based on the hashing function.
|
||||
|
||||
> **Note**: If you experience hanging requests with `curl`, ensure the backend HTTP servers are bound to `0.0.0.0` and accept requests with any Host header. The XDP load balancer operates at Layer 3/4 (IP/TCP) and does not modify HTTP headers, so the Host header in requests will still show `10.0.0.10:8000` even though packets are forwarded to the backend IPs (10.0.0.2 or 10.0.0.3).
|
||||
|
||||
### Monitoring with `bpf_printk`
|
||||
|
||||
You can monitor the load balancer's activity by checking the `bpf_printk` logs. The BPF program prints diagnostic messages whenever a packet is processed. You can view these logs using:
|
||||
@@ -507,6 +520,24 @@ Example output:
|
||||
|
||||
### Debugging Issues
|
||||
|
||||
#### Curl Requests Hanging
|
||||
|
||||
If `curl` requests to the load balancer hang and never complete, the most likely cause is that the backend HTTP servers are rejecting requests with mismatched Host headers.
|
||||
|
||||
**Problem**: When you run `curl 10.0.0.10:8000`, the HTTP request includes a Host header set to `10.0.0.10:8000`. The XDP load balancer forwards the packet at Layer 3/4 (IP/TCP) to a backend server (10.0.0.2 or 10.0.0.3), but the HTTP headers remain unchanged. If the backend HTTP server validates the Host header and expects it to match its own IP address, it may reject or drop the request.
|
||||
|
||||
**Solution**: Ensure backend HTTP servers bind to `0.0.0.0` and accept requests with any Host header:
|
||||
- Use `python3 simple_http_server.py` (provided in this directory), or
|
||||
- Use `python3 -m http.server --bind 0.0.0.0`
|
||||
|
||||
**Verification**: Check the backend server logs to see if requests are being received. You can also use `tcpdump` in the backend namespace to verify packets are arriving:
|
||||
|
||||
```sh
|
||||
sudo ip netns exec h2 tcpdump -i veth2 -n port 8000
|
||||
```
|
||||
|
||||
#### XDP Packet Forwarding Issues
|
||||
|
||||
Some systems may experience packet loss or failure to forward packets due to issues similar to those described in this [blog post](https://fedepaol.github.io/blog/2023/09/11/xdp-ate-my-packets-and-how-i-debugged-it/). You can debug these issues using `bpftrace` to trace XDP errors:
|
||||
|
||||
```sh
|
||||
|
||||
@@ -463,13 +463,25 @@ Press Ctrl+C to exit...
|
||||
|
||||
### 测试设置
|
||||
|
||||
您可以通过在两个后端命名空间（`h2` 和 `h3`）启动 HTTP 服务器，并从本地机器向负载均衡器发送请求来测试设置：
|
||||
您可以通过在两个后端命名空间（`h2` 和 `h3`）启动 HTTP 服务器, 并从本地机器向负载均衡器发送请求来测试设置：
|
||||
|
||||
在 `h2` 和 `h3` 上启动服务器：
|
||||
|
||||
**重要提示（监听地址）**：在本实验环境中，HTTP 服务器应绑定到 `0.0.0.0`，这样它会在该网络命名空间中的所有本地地址上监听，从而能够接受通过后端 IP 转发过来的连接。
|
||||
|
||||
**重要提示（Host 头）**：负载均衡器对外暴露的虚拟 IP 地址是 `10.0.0.10`，并以该地址作为目标 IP 转发 HTTP 请求，因此请求中的 Host 头将类似于 `Host: 10.0.0.10:8000`，这与后端命名空间内实际分配给服务器进程的本地 IP 不同。是否接受这样的 Host 头由具体的 HTTP 服务器实现决定，与是否绑定到 `0.0.0.0` 无关；本教程中使用的 `simple_http_server.py` 和 `python -m http.server` 默认都会接受这种 Host 头，无需额外配置。
|
||||
**选项 1**：使用提供的简单 HTTP 服务器（推荐）：
|
||||
|
||||
```sh
|
||||
sudo ip netns exec h2 python3 -m http.server
|
||||
sudo ip netns exec h3 python3 -m http.server
|
||||
sudo ip netns exec h2 python3 simple_http_server.py &
|
||||
sudo ip netns exec h3 python3 simple_http_server.py &
|
||||
```
|
||||
|
||||
**选项 2**：使用 Python 内置的 http.server 并显式绑定：
|
||||
|
||||
```sh
|
||||
sudo ip netns exec h2 python3 -m http.server --bind 0.0.0.0 &
|
||||
sudo ip netns exec h3 python3 -m http.server --bind 0.0.0.0 &
|
||||
```
|
||||
|
||||
然后，向负载均衡器 IP 发送请求：
|
||||
@@ -480,6 +492,8 @@ curl 10.0.0.10:8000
|
||||
|
||||
负载均衡器将根据哈希函数将流量分配到后端服务器（`h2` 和 `h3`）。
|
||||
|
||||
> **注意**：如果您使用 `curl` 时遇到请求挂起的问题，请确保后端 HTTP 服务器绑定到 `0.0.0.0` 并接受任何 Host 头的请求。XDP 负载均衡器在第 3/4 层（IP/TCP）运行，不会修改 HTTP 头，因此请求中的 Host 头仍将显示 `10.0.0.10:8000`，即使数据包被转发到后端 IP（10.0.0.2 或 10.0.0.3）。
|
||||
|
||||
### 使用 `bpf_printk` 进行监控
|
||||
|
||||
您可以通过查看 `bpf_printk` 日志来监控负载均衡器的活动。BPF 程序在处理每个数据包时会打印诊断消息。您可以使用以下命令查看这些日志：
|
||||
@@ -504,6 +518,24 @@ sudo cat /sys/kernel/debug/tracing/trace_pipe
|
||||
|
||||
### 调试问题
|
||||
|
||||
#### Curl 请求挂起
|
||||
|
||||
如果对负载均衡器的 `curl` 请求挂起且永不完成，最可能的原因是后端 HTTP 服务器拒绝了具有不匹配 Host 头的请求。
|
||||
|
||||
**问题**：当您运行 `curl 10.0.0.10:8000` 时，HTTP 请求包含设置为 `10.0.0.10:8000` 的 Host 头。XDP 负载均衡器在第 3/4 层（IP/TCP）将数据包转发到后端服务器（10.0.0.2 或 10.0.0.3），但 HTTP 头保持不变。如果后端 HTTP 服务器验证 Host 头并期望它与自己的 IP 地址匹配，它可能会拒绝或丢弃该请求。
|
||||
|
||||
**解决方案**：确保后端 HTTP 服务器绑定到 `0.0.0.0` 并接受任何 Host 头的请求：
|
||||
- 使用 `python3 simple_http_server.py`（在此目录中提供），或
|
||||
- 使用 `python3 -m http.server --bind 0.0.0.0`
|
||||
|
||||
**验证**：检查后端服务器日志以查看是否收到请求。您还可以在后端命名空间中使用 `tcpdump` 验证数据包是否到达：
|
||||
|
||||
```sh
|
||||
sudo ip netns exec h2 tcpdump -i veth2 -n port 8000
|
||||
```
|
||||
|
||||
#### XDP 数据包转发问题
|
||||
|
||||
某些系统可能会因为类似于此[博客文章](https://fedepaol.github.io/blog/2023/09/11/xdp-ate-my-packets-and-how-i-debugged-it/)中描述的问题而导致数据包丢失或转发失败。您可以使用 `bpftrace` 跟踪 XDP 错误进行调试：
|
||||
|
||||
```sh
|
||||
|
||||
47
src/42-xdp-loadbalancer/simple_http_server.py
Executable file
47
src/42-xdp-loadbalancer/simple_http_server.py
Executable file
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple HTTP server that doesn't validate the Host header.
|
||||
This server is designed to work with load balancers that forward requests
|
||||
with mismatched Host headers.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import http.server
|
||||
import socketserver
|
||||
|
||||
class SimpleHTTPRequestHandler(http.server.SimpleHTTPRequestHandler):
|
||||
"""
|
||||
HTTP request handler that doesn't validate the Host header.
|
||||
This allows the server to work behind a load balancer.
|
||||
"""
|
||||
|
||||
def version_string(self):
|
||||
"""Return the server software version string."""
|
||||
return f'SimpleHTTP/{sys.version.split()[0]}'
|
||||
|
||||
def log_message(self, format, *args):
|
||||
"""Log an arbitrary message."""
|
||||
# Include the Host header in the log for debugging
|
||||
host_header = self.headers.get('Host', 'unknown')
|
||||
sys.stderr.write(f"[{self.log_date_time_string()}] Host: {host_header} - {format % args}\n")
|
||||
|
||||
|
||||
def main():
|
||||
PORT = 8000
|
||||
|
||||
# Allow reuse of address to avoid "Address already in use" errors
|
||||
socketserver.TCPServer.allow_reuse_address = True
|
||||
|
||||
# Bind to 0.0.0.0 to accept connections from any interface
|
||||
with socketserver.TCPServer(("0.0.0.0", PORT), SimpleHTTPRequestHandler) as httpd:
|
||||
print(f"Server listening on 0.0.0.0:{PORT}")
|
||||
print("This server accepts requests with any Host header.")
|
||||
print("Press Ctrl+C to stop the server.")
|
||||
try:
|
||||
httpd.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
print("\nShutting down server...")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
69
src/42-xdp-loadbalancer/test_http_server.sh
Executable file
69
src/42-xdp-loadbalancer/test_http_server.sh
Executable file
@@ -0,0 +1,69 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Test script to verify that the HTTP servers correctly handle
|
||||
# requests with mismatched Host headers, which is essential for
|
||||
# the XDP load balancer to work correctly.
|
||||
#
|
||||
# This test demonstrates that the fix resolves the curl hanging issue.
|
||||
|
||||
set -e
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
echo "=== Testing HTTP Server with Mismatched Host Headers ==="
|
||||
echo ""
|
||||
|
||||
# Test 1: simple_http_server.py (uses port 8000 by default)
|
||||
echo "Test 1: Testing simple_http_server.py"
|
||||
echo "Starting simple_http_server.py on default port 8000..."
|
||||
python3 simple_http_server.py > /tmp/simple_http_test.log 2>&1 &
|
||||
SERVER_PID=$!
|
||||
sleep 3
|
||||
|
||||
# Test with mismatched Host header
|
||||
echo "Sending request with Host: 10.0.0.10:8000 to 127.0.0.1:8000..."
|
||||
RESPONSE=$(curl -s -H "Host: 10.0.0.10:8000" http://127.0.0.1:8000/ 2>&1 | head -5)
|
||||
if [ -n "$RESPONSE" ]; then
|
||||
echo "✓ simple_http_server.py successfully handled request with mismatched Host header"
|
||||
else
|
||||
echo "✗ simple_http_server.py failed to handle request"
|
||||
kill $SERVER_PID 2>/dev/null
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check server logs
|
||||
echo "Server log snippet:"
|
||||
tail -n 1 /tmp/simple_http_test.log
|
||||
kill $SERVER_PID 2>/dev/null
|
||||
sleep 1
|
||||
echo ""
|
||||
|
||||
# Test 2: python -m http.server --bind 0.0.0.0
|
||||
echo "Test 2: Testing python3 -m http.server --bind 0.0.0.0"
|
||||
echo "Starting http.server on port 8001..."
|
||||
python3 -m http.server --bind 0.0.0.0 8001 > /tmp/builtin_http_test.log 2>&1 &
|
||||
SERVER_PID=$!
|
||||
sleep 3
|
||||
|
||||
# Test with mismatched Host header
|
||||
echo "Sending request with Host: 10.0.0.10:8001 to 127.0.0.1:8001..."
|
||||
RESPONSE=$(curl -s -H "Host: 10.0.0.10:8001" http://127.0.0.1:8001/ 2>&1 | head -5)
|
||||
if [ -n "$RESPONSE" ]; then
|
||||
echo "✓ http.server --bind 0.0.0.0 successfully handled request with mismatched Host header"
|
||||
else
|
||||
echo "✗ http.server --bind 0.0.0.0 failed to handle request"
|
||||
kill $SERVER_PID 2>/dev/null
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check server logs
|
||||
echo "Server log snippet:"
|
||||
tail -n 1 /tmp/builtin_http_test.log
|
||||
kill $SERVER_PID 2>/dev/null
|
||||
sleep 1
|
||||
echo ""
|
||||
|
||||
echo "=== All Tests Passed ==="
|
||||
echo ""
|
||||
echo "Both HTTP server options correctly handle requests with mismatched Host headers,"
|
||||
echo "which fixes the curl hanging issue in the XDP load balancer setup."
|
||||
Reference in New Issue
Block a user