mirror of
https://gitea.com/gitea/act_runner.git
synced 2026-04-24 12:50:31 +08:00
f2d545565f
## Summary Many teams self-host Gitea + Act Runner at scale. The current runner design causes excessive HTTP requests to the Gitea server, leading to high server load. This PR addresses three root causes: aggressive fixed-interval polling, per-task status reporting every 1 second regardless of activity, and unoptimized HTTP client configuration. ## Problem The original architecture has these issues: **1. Fixed 1-second reporting interval (RunDaemon)** - Every running task calls ReportLog + ReportState every 1 second (2 HTTP requests/sec/task) - These requests are sent even when there are no new log rows or state changes - With 200 runners × 3 tasks each = **1,200 req/sec just for status reporting** **2. Fixed 2-second polling interval (no backoff)** - Idle runners poll FetchTask every 2 seconds forever, even when no jobs are queued - No exponential backoff or jitter — all runners can synchronize after network recovery (thundering herd) - 200 idle runners = **100 req/sec doing nothing useful** **3. HTTP client not tuned** - Uses http.DefaultClient with MaxIdleConnsPerHost=2, causing frequent TCP/TLS reconnects - Creates two separate http.Client instances (one for Ping, one for Runner service) instead of sharing **Total: ~1,300 req/sec for 200 runners with 3 tasks each** ## Solution ### Adaptive Event-Driven Log Reporting Replace the recursive `time.AfterFunc(1s)` pattern in RunDaemon with a goroutine-based select event loop using three trigger mechanisms: | Trigger | Default | Purpose | |---------|---------|---------| | `log_report_max_latency` | 3s | Guarantee even a single log line is delivered within this time | | `log_report_interval` | 5s | Periodic sweep — steady-state cadence | | `log_report_batch_size` | 100 rows | Immediate flush during bursty output (e.g., npm install) | **Key design**: `log_report_max_latency` (3s) must be less than `log_report_interval` (5s) so the max-latency timer fires before the periodic ticker for single-line scenarios. State reporting is decoupled to its own `state_report_interval` (default 5s), with immediate flush on step transitions (start/stop) via a stateNotify channel for responsive frontend UX. Additionally: - Skip ReportLog when `len(rows) == 0` (no pending log rows) - Skip ReportState when `stateChanged == false && len(outputs) == 0` (nothing changed) - Move expensive `proto.Clone` after the early-return check to avoid deep copies on no-op paths ### Polling Backoff with Jitter Replace fixed `rate.Limiter` with adaptive exponential backoff: - Track `consecutiveEmpty` and `consecutiveErrors` counters - Interval doubles with each empty/error response: `base × 2^(n-1)`, capped at `fetch_interval_max` (default 60s) - Add ±20% random jitter to prevent thundering herd - Fetch first, sleep after ��� preserves burst=1 behavior for immediate first fetch on startup and after task completion ### HTTP Client Tuning - Configure custom `http.Transport` with `MaxIdleConnsPerHost=10` (was 2) - Share a single `http.Client` between PingService and RunnerService - Add `IdleConnTimeout=90s` for clean connection lifecycle ## Load Reduction For 200 runners × 3 tasks (70% with active log output): | Component | Before | After | Reduction | |-----------|--------|-------|-----------| | Polling (idle) | 100 req/s | ~3.4 req/s | 97% | | Log reporting | 420 req/s | ~84 req/s | 80% | | State reporting | 126 req/s | ~25 req/s | 80% | | **Total** | **~1,300 req/s** | **~113 req/s** | **~91%** | ## Frontend UX Impact | Scenario | Before | After | Notes | |----------|--------|-------|-------| | Continuous output (npm install) | ~1s | ~5s | Periodic ticker sweep | | Single line then silence | ~1s | ≤3s | maxLatencyTimer guarantee | | Bursty output (100+ lines) | ~1s | <1s | Batch size immediate flush | | Step start/stop | ~1s | <1s | stateNotify immediate flush | | Job completion | ~1s | ~1s | Close() retry unchanged | ## New Configuration Options All have safe defaults — existing config files need no changes: ```yaml runner: fetch_interval_max: 60s # Max backoff interval when idle log_report_interval: 5s # Periodic log flush interval log_report_max_latency: 3s # Max time a log row waits (must be < log_report_interval) log_report_batch_size: 100 # Immediate flush threshold state_report_interval: 5s # State flush interval (step transitions are always immediate) ``` Config validation warns on invalid combinations: - `fetch_interval_max < fetch_interval` → auto-corrected - `log_report_max_latency >= log_report_interval` → warning (timer would be redundant) ## Test Plan - [x] `go build ./...` passes - [x] `go test ./...` passes (all existing + 3 new tests) - [x] `golangci-lint run` — 0 issues - [x] TestReporter_MaxLatencyTimer — verifies single log line flushed by maxLatencyTimer before logTicker - [x] TestReporter_BatchSizeFlush — verifies batch size threshold triggers immediate flush - [x] TestReporter_StateNotifyFlush — verifies step transition triggers immediate state flush - [x] TestReporter_EphemeralRunnerDeletion — verifies Close/RunDaemon race safety - [x] TestReporter_RunDaemonClose_Race — verifies concurrent Close safety Reviewed-on: https://gitea.com/gitea/act_runner/pulls/819 Reviewed-by: Nicolas <173651+bircni@noreply.gitea.com> Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com> Co-committed-by: Bo-Yi Wu <appleboy.tw@gmail.com>
135 lines
6.9 KiB
YAML
135 lines
6.9 KiB
YAML
# Example configuration file, it's safe to copy this as the default config file without any modification.
|
|
|
|
# You don't have to copy this file to your instance,
|
|
# just run `./act_runner generate-config > config.yaml` to generate a config file.
|
|
|
|
log:
|
|
# The level of logging, can be trace, debug, info, warn, error, fatal
|
|
level: info
|
|
|
|
runner:
|
|
# Where to store the registration result.
|
|
file: .runner
|
|
# Execute how many tasks concurrently at the same time.
|
|
capacity: 1
|
|
# Extra environment variables to run jobs.
|
|
envs:
|
|
A_TEST_ENV_NAME_1: a_test_env_value_1
|
|
A_TEST_ENV_NAME_2: a_test_env_value_2
|
|
# Extra environment variables to run jobs from a file.
|
|
# It will be ignored if it's empty or the file doesn't exist.
|
|
env_file: .env
|
|
# The timeout for a job to be finished.
|
|
# Please note that the Gitea instance also has a timeout (3h by default) for the job.
|
|
# So the job could be stopped by the Gitea instance if its timeout is shorter than this.
|
|
timeout: 3h
|
|
# The timeout for the runner to wait for running jobs to finish when shutting down.
|
|
# Any running jobs that haven't finished after this timeout will be cancelled.
|
|
shutdown_timeout: 0s
|
|
# Whether skip verifying the TLS certificate of the Gitea instance.
|
|
insecure: false
|
|
# The timeout for fetching the job from the Gitea instance.
|
|
fetch_timeout: 5s
|
|
# The interval for fetching the job from the Gitea instance.
|
|
fetch_interval: 2s
|
|
# The maximum interval for fetching the job from the Gitea instance.
|
|
# The runner uses exponential backoff when idle, increasing the interval up to this maximum.
|
|
# Set to 0 or same as fetch_interval to disable backoff.
|
|
fetch_interval_max: 60s
|
|
# The base interval for periodic log flush to the Gitea instance.
|
|
# Logs may be sent earlier if the buffer reaches log_report_batch_size
|
|
# or if log_report_max_latency expires after the first buffered row.
|
|
log_report_interval: 5s
|
|
# The maximum time a log row can wait before being sent.
|
|
# This ensures even a single log line appears on the frontend within this duration.
|
|
# Must be less than log_report_interval to have any effect.
|
|
log_report_max_latency: 3s
|
|
# Flush logs immediately when the buffer reaches this many rows.
|
|
# This ensures bursty output (e.g., npm install) is delivered promptly.
|
|
log_report_batch_size: 100
|
|
# The interval for reporting task state (step status, timing) to the Gitea instance.
|
|
# State is also reported immediately on step transitions (start/stop).
|
|
state_report_interval: 5s
|
|
# The github_mirror of a runner is used to specify the mirror address of the github that pulls the action repository.
|
|
# It works when something like `uses: actions/checkout@v4` is used and DEFAULT_ACTIONS_URL is set to github,
|
|
# and github_mirror is not empty. In this case,
|
|
# it replaces https://github.com with the value here, which is useful for some special network environments.
|
|
github_mirror: ''
|
|
# The labels of a runner are used to determine which jobs the runner can run, and how to run them.
|
|
# Like: "macos-arm64:host" or "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
|
# Find more images provided by Gitea at https://gitea.com/gitea/runner-images .
|
|
# If it's empty when registering, it will ask for inputting labels.
|
|
# If it's empty when execute `daemon`, will use labels in `.runner` file.
|
|
labels:
|
|
- "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
|
- "ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04"
|
|
- "ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04"
|
|
|
|
cache:
|
|
# Enable cache server to use actions/cache.
|
|
enabled: true
|
|
# The directory to store the cache data.
|
|
# If it's empty, the cache data will be stored in $HOME/.cache/actcache.
|
|
dir: ""
|
|
# The host of the cache server.
|
|
# It's not for the address to listen, but the address to connect from job containers.
|
|
# So 0.0.0.0 is a bad choice, leave it empty to detect automatically.
|
|
host: ""
|
|
# The port of the cache server.
|
|
# 0 means to use a random available port.
|
|
port: 0
|
|
# The external cache server URL. Valid only when enable is true.
|
|
# If it's specified, act_runner will use this URL as the ACTIONS_CACHE_URL rather than start a server by itself.
|
|
# The URL should generally end with "/".
|
|
external_server: ""
|
|
|
|
container:
|
|
# Specifies the network to which the container will connect.
|
|
# Could be host, bridge or the name of a custom network.
|
|
# If it's empty, act_runner will create a network automatically.
|
|
network: ""
|
|
# Whether to use privileged mode or not when launching task containers (privileged mode is required for Docker-in-Docker).
|
|
privileged: false
|
|
# Any other options to be used when the container is started (e.g., --add-host=my.gitea.url:host-gateway).
|
|
options:
|
|
# The parent directory of a job's working directory.
|
|
# NOTE: There is no need to add the first '/' of the path as act_runner will add it automatically.
|
|
# If the path starts with '/', the '/' will be trimmed.
|
|
# For example, if the parent directory is /path/to/my/dir, workdir_parent should be path/to/my/dir
|
|
# If it's empty, /workspace will be used.
|
|
workdir_parent:
|
|
# Volumes (including bind mounts) can be mounted to containers. Glob syntax is supported, see https://github.com/gobwas/glob
|
|
# You can specify multiple volumes. If the sequence is empty, no volumes can be mounted.
|
|
# For example, if you only allow containers to mount the `data` volume and all the json files in `/src`, you should change the config to:
|
|
# valid_volumes:
|
|
# - data
|
|
# - /src/*.json
|
|
# If you want to allow any volume, please use the following configuration:
|
|
# valid_volumes:
|
|
# - '**'
|
|
valid_volumes: []
|
|
# Overrides the docker client host with the specified one.
|
|
# If it's empty, act_runner will find an available docker host automatically.
|
|
# If it's "-", act_runner will find an available docker host automatically, but the docker host won't be mounted to the job containers and service containers.
|
|
# If it's not empty or "-", the specified docker host will be used. An error will be returned if it doesn't work.
|
|
docker_host: ""
|
|
# Pull docker image(s) even if already present
|
|
force_pull: true
|
|
# Rebuild docker image(s) even if already present
|
|
force_rebuild: false
|
|
# Always require a reachable docker daemon, even if not required by act_runner
|
|
require_docker: false
|
|
# Timeout to wait for the docker daemon to be reachable, if docker is required by require_docker or act_runner
|
|
docker_timeout: 0s
|
|
# Bind the workspace to the host filesystem instead of using Docker volumes.
|
|
# This is required for Docker-in-Docker (DinD) setups when jobs use docker compose
|
|
# with bind mounts (e.g., ".:/app"), as volume-based workspaces are not accessible
|
|
# from the DinD daemon's filesystem. When enabled, ensure the workspace parent
|
|
# directory is also mounted into the runner container and listed in valid_volumes.
|
|
bind_workdir: false
|
|
|
|
host:
|
|
# The parent directory of a job's working directory.
|
|
# If it's empty, $HOME/.cache/act/ will be used.
|
|
workdir_parent:
|