mirror of
https://gitea.com/gitea/act_runner.git
synced 2026-04-24 21:00:27 +08:00
f33e5a6245
## What
Add an optional Prometheus `/metrics` HTTP endpoint to `act_runner` so operators can observe runner health, polling behavior, job outcomes, and RPC latency without scraping logs.
New surface:
- `internal/pkg/metrics/metrics.go` — metric definitions, custom `Registry`, static Go/process collectors, label constants, `ResultToStatusLabel` helper.
- `internal/pkg/metrics/server.go` — hardened `http.Server` serving `/metrics` and `/healthz` with Slowloris-safe timeouts (`ReadHeaderTimeout` 5s, `ReadTimeout`/`WriteTimeout` 10s, `IdleTimeout` 60s) and a 5s graceful shutdown.
- `daemon.go` wires it up behind `cfg.Metrics.Enabled` (disabled by default).
- `poller.go` / `reporter.go` / `runner.go` instrument their existing hot paths with counters/histograms/gauges — no behavior change.
Metrics exported (namespace `act_runner_`):
| Subsystem | Metric | Type | Labels |
|---|---|---|---|
| — | `info` | Gauge | `version`, `name` |
| — | `capacity`, `uptime_seconds` | Gauge | — |
| `poll` | `fetch_total`, `client_errors_total` | Counter | `result` / `method` |
| `poll` | `fetch_duration_seconds`, `backoff_seconds` | Histogram / Gauge | — |
| `job` | `total` | Counter | `status` |
| `job` | `duration_seconds`, `running`, `capacity_utilization_ratio` | Histogram / GaugeFunc | — |
| `report` | `log_total`, `state_total` | Counter | `result` |
| `report` | `log_duration_seconds`, `state_duration_seconds` | Histogram | — |
| `report` | `log_buffer_rows` | Gauge | — |
| — | `go_*`, `process_*` | standard collectors | — |
All label values are predefined constants — **no high-cardinality labels** (no task IDs, repo URLs, branches, tokens, or secrets) so scraping is safe and bounded.
## Why
Teams self-hosting Gitea + `act_runner` at scale need to answer basic SRE questions that are currently invisible:
- How often are RPCs failing? Which RPC? (`act_runner_client_errors_total`)
- Are runners saturated? (`act_runner_job_capacity_utilization_ratio`, `act_runner_job_running`)
- How long do jobs take? (`act_runner_job_duration_seconds`)
- Is polling backing off? (`act_runner_poll_backoff_seconds`, `act_runner_poll_fetch_total{result=\"error\"}`)
- Are log/state reports slow? (`act_runner_report_{log,state}_duration_seconds`)
- Is the log buffer draining? (`act_runner_report_log_buffer_rows`)
Today operators have to grep logs. This PR makes all of the above first-class metrics so they can feed dashboards and alerts (`rate(act_runner_client_errors_total[5m]) > 0.1`, capacity saturation alerts, etc.).
The endpoint is **disabled by default** and binds to `127.0.0.1:9101` when enabled, so it's opt-in and safe for existing deployments.
## How
### Config
```yaml
metrics:
enabled: false # opt-in
addr: 127.0.0.1:9101 # change to 0.0.0.0:9101 only behind a reverse proxy
```
`config.example.yaml` documents both fields plus a security note about binding externally without auth.
### Wiring
1. `daemon.go` calls `metrics.Init()` (guarded by `sync.Once`), sets `act_runner_info`, `act_runner_capacity`, registers uptime + running-jobs GaugeFuncs, then starts the server goroutine with the daemon context — it shuts down cleanly on `ctx.Done()`.
2. `poller.fetchTask` observes RPC latency / result / error counters. `DeadlineExceeded` (long-poll idle) is treated as an empty result and **not** observed into the histogram so the 5s timeout doesn't swamp the buckets.
3. `poller.pollOnce` reports `poll_backoff_seconds` using the pre-jitter base interval (the true backoff level), and only when it changes — prevents noisy no-op gauge updates at the `FetchIntervalMax` plateau.
4. `reporter.ReportLog` / `ReportState` record duration histograms and success/error counters; `log_buffer_rows` is updated only when the value changes, guarded by the already-held `clientM`.
5. `runner.Run` observes `job_duration_seconds` and increments `job_total` by outcome via `metrics.ResultToStatusLabel`.
### Safety / security review
- All timeouts set; Slowloris-safe.
- Custom `prometheus.NewRegistry()` — no global registration side-effects.
- No sensitive data in labels (reviewed every instrumentation site).
- Single new dependency: `github.com/prometheus/client_golang v1.23.2`.
- Endpoint is unauthenticated by design and documented as such; default localhost bind mitigates exposure. Operators exposing externally should front it with a reverse proxy.
## Verification
### Unit tests
\`\`\`bash
go build ./...
go vet ./...
go test ./...
\`\`\`
### Manual smoke test
1. Enable metrics in `config.yaml`:
\`\`\`yaml
metrics:
enabled: true
addr: 127.0.0.1:9101
\`\`\`
2. Start the runner against a Gitea instance: \`./act_runner daemon\`.
3. Scrape the endpoint:
\`\`\`bash
curl -s http://127.0.0.1:9101/metrics | grep '^act_runner_'
curl -s http://127.0.0.1:9101/healthz # → ok
\`\`\`
4. Confirm the static series appear immediately: \`act_runner_info\`, \`act_runner_capacity\`, \`act_runner_uptime_seconds\`, \`act_runner_job_running\`, \`act_runner_job_capacity_utilization_ratio\`.
5. Trigger a workflow and confirm counters increment: \`act_runner_poll_fetch_total{result=\"task\"}\`, \`act_runner_job_total{status=\"success\"}\`, \`act_runner_report_log_total{result=\"success\"}\`.
6. Leave the runner idle and confirm \`act_runner_poll_backoff_seconds\` settles (and does **not** churn on every poll).
7. Ctrl-C and confirm a clean \"metrics server shutdown\" log line (no port-in-use error on restart within 5s).
### Prometheus integration
Add to \`prometheus.yml\`:
\`\`\`yaml
scrape_configs:
- job_name: act_runner
static_configs:
- targets: ['127.0.0.1:9101']
\`\`\`
Sample alert to try:
\`\`\`
sum(rate(act_runner_client_errors_total[5m])) by (method) > 0.1
\`\`\`
## Out of scope (follow-ups)
- TLS and auth on the metrics endpoint (mitigated today by localhost default; add when operators need external scraping).
- Per-task labels (intentionally avoided for cardinality safety).
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed-on: https://gitea.com/gitea/act_runner/pulls/820
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Bo-Yi Wu <appleboy.tw@gmail.com>
Co-committed-by: Bo-Yi Wu <appleboy.tw@gmail.com>
125 lines
5.6 KiB
Modula-2
125 lines
5.6 KiB
Modula-2
module gitea.com/gitea/act_runner
|
|
|
|
go 1.26.0
|
|
|
|
require (
|
|
code.gitea.io/actions-proto-go v0.4.1
|
|
code.gitea.io/gitea-vet v0.2.3
|
|
connectrpc.com/connect v1.19.1
|
|
github.com/avast/retry-go/v4 v4.7.0
|
|
github.com/docker/docker v25.0.13+incompatible
|
|
github.com/joho/godotenv v1.5.1
|
|
github.com/mattn/go-isatty v0.0.20
|
|
github.com/nektos/act v0.0.0 // will be replaced
|
|
github.com/sirupsen/logrus v1.9.4
|
|
github.com/spf13/cobra v1.10.2
|
|
github.com/stretchr/testify v1.11.1
|
|
go.yaml.in/yaml/v4 v4.0.0-rc.3
|
|
golang.org/x/term v0.40.0
|
|
golang.org/x/time v0.14.0 // indirect
|
|
google.golang.org/protobuf v1.36.11
|
|
gopkg.in/yaml.v3 v3.0.1
|
|
gotest.tools/v3 v3.5.2
|
|
)
|
|
|
|
require github.com/prometheus/client_golang v1.23.2
|
|
|
|
require (
|
|
cyphar.com/go-pathrs v0.2.3 // indirect
|
|
dario.cat/mergo v1.0.2 // indirect
|
|
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 // indirect
|
|
github.com/Masterminds/semver v1.5.0 // indirect
|
|
github.com/Microsoft/go-winio v0.6.2 // indirect
|
|
github.com/ProtonMail/go-crypto v1.3.0 // indirect
|
|
github.com/beorn7/perks v1.0.1 // indirect
|
|
github.com/bmatcuk/doublestar/v4 v4.10.0 // indirect
|
|
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
|
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
|
|
github.com/cloudflare/circl v1.6.3 // indirect
|
|
github.com/containerd/containerd v1.7.29 // indirect
|
|
github.com/containerd/log v0.1.0 // indirect
|
|
github.com/creack/pty v1.1.24 // indirect
|
|
github.com/cyphar/filepath-securejoin v0.6.1 // indirect
|
|
github.com/davecgh/go-spew v1.1.1 // indirect
|
|
github.com/distribution/reference v0.6.0 // indirect
|
|
github.com/docker/cli v25.0.3+incompatible // indirect
|
|
github.com/docker/distribution v2.8.3+incompatible // indirect
|
|
github.com/docker/docker-credential-helpers v0.9.5 // indirect
|
|
github.com/docker/go-connections v0.6.0 // indirect
|
|
github.com/docker/go-units v0.5.0 // indirect
|
|
github.com/emirpasic/gods v1.18.1 // indirect
|
|
github.com/fatih/color v1.18.0 // indirect
|
|
github.com/felixge/httpsnoop v1.0.4 // indirect
|
|
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
|
|
github.com/go-git/go-billy/v5 v5.7.0 // indirect
|
|
github.com/go-git/go-git/v5 v5.16.5 // indirect
|
|
github.com/go-logr/logr v1.4.3 // indirect
|
|
github.com/go-logr/stdr v1.2.2 // indirect
|
|
github.com/gobwas/glob v0.2.3 // indirect
|
|
github.com/gogo/protobuf v1.3.2 // indirect
|
|
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
|
|
github.com/google/go-cmp v0.7.0 // indirect
|
|
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
|
|
github.com/imdario/mergo v0.3.16 // indirect
|
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
|
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
|
|
github.com/julienschmidt/httprouter v1.3.0 // indirect
|
|
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
|
|
github.com/kevinburke/ssh_config v1.6.0 // indirect
|
|
github.com/klauspost/compress v1.18.4 // indirect
|
|
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
|
|
github.com/mattn/go-colorable v0.1.14 // indirect
|
|
github.com/mattn/go-runewidth v0.0.20 // indirect
|
|
github.com/mattn/go-shellwords v1.0.12 // indirect
|
|
github.com/mitchellh/mapstructure v1.1.2 // indirect
|
|
github.com/moby/buildkit v0.13.2 // indirect
|
|
github.com/moby/patternmatcher v0.6.0 // indirect
|
|
github.com/moby/sys/sequential v0.6.0 // indirect
|
|
github.com/moby/sys/user v0.4.0 // indirect
|
|
github.com/moby/sys/userns v0.1.0 // indirect
|
|
github.com/moby/term v0.5.2 // indirect
|
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
|
github.com/opencontainers/go-digest v1.0.0 // indirect
|
|
github.com/opencontainers/image-spec v1.1.1 // indirect
|
|
github.com/opencontainers/selinux v1.13.1 // indirect
|
|
github.com/pjbgf/sha1cd v0.5.0 // indirect
|
|
github.com/pkg/errors v0.9.1 // indirect
|
|
github.com/pmezard/go-difflib v1.0.0 // indirect
|
|
github.com/prometheus/client_model v0.6.2 // indirect
|
|
github.com/prometheus/common v0.66.1 // indirect
|
|
github.com/prometheus/procfs v0.16.1 // indirect
|
|
github.com/rhysd/actionlint v1.7.11 // indirect
|
|
github.com/robfig/cron/v3 v3.0.1 // indirect
|
|
github.com/sergi/go-diff v1.4.0 // indirect
|
|
github.com/skeema/knownhosts v1.3.2 // indirect
|
|
github.com/spf13/pflag v1.0.10 // indirect
|
|
github.com/stretchr/objx v0.5.3 // indirect
|
|
github.com/timshannon/bolthold v0.0.0-20240314194003-30aac6950928 // indirect
|
|
github.com/xanzy/ssh-agent v0.3.3 // indirect
|
|
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
|
|
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
|
|
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
|
|
go.etcd.io/bbolt v1.4.3 // indirect
|
|
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
|
|
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0 // indirect
|
|
go.opentelemetry.io/otel v1.40.0 // indirect
|
|
go.opentelemetry.io/otel/metric v1.40.0 // indirect
|
|
go.opentelemetry.io/otel/trace v1.40.0 // indirect
|
|
go.yaml.in/yaml/v2 v2.4.2 // indirect
|
|
golang.org/x/crypto v0.48.0 // indirect
|
|
golang.org/x/net v0.50.0 // indirect
|
|
golang.org/x/sync v0.19.0 // indirect
|
|
golang.org/x/sys v0.41.0 // indirect
|
|
golang.org/x/tools v0.42.0 // indirect
|
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
|
|
google.golang.org/grpc v1.67.0 // indirect
|
|
gopkg.in/warnings.v0 v0.1.2 // indirect
|
|
gopkg.in/yaml.v2 v2.4.0 // indirect
|
|
)
|
|
|
|
replace github.com/nektos/act => gitea.com/gitea/act v0.261.10
|
|
|
|
// Remove after github.com/docker/distribution is updated to support distribution/reference v0.6.0
|
|
// (pulled in via moby/buildkit, breaks on undefined: reference.SplitHostname)
|
|
replace github.com/distribution/reference v0.6.0 => github.com/distribution/reference v0.5.0
|