1952e5cdb6
Adds OpenTelemetry tracing, configured through the standard `OTEL_*` environment variables and off until an OTLP endpoint is set.  Each job becomes a trace with a span per step, so operators can see where CI time goes in their existing observability backend. Jobs receive only `TRACEPARENT`, so instrumented tools inside them join the trace without seeing the collector's credentials. Metrics stay on the Prometheus endpoint, which a collector can scrape. See [docs/telemetry.md](https://gitea.com/wmTJc9IK0Q/runner/src/branch/otel/docs/telemetry.md) for setup and security notes. Also keeps `GITHUB_SHA` and `GITHUB_REF` fixed for a whole run when HEAD moves during `exec` runs and the act tests. [image.png](https://gitea.com/attachments/c4b49cdf-a32d-403c-94e9-30cd06ee6477) [image.png](https://gitea.com/attachments/c4b49cdf-a32d-403c-94e9-30cd06ee6477) --------- Co-authored-by: silverwind <2021+silverwind@noreply.gitea.com> Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: bircni <bircni@icloud.com> Reviewed-on: https://gitea.com/gitea/runner/pulls/1206 Reviewed-by: bircni <bircni@icloud.com> Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com> Co-authored-by: wmTJc9IK0Q <231276+wmtjc9ik0q@noreply.gitea.com>
3.2 KiB
OpenTelemetry tracing
The runner can send a trace of every job it runs to an OpenTelemetry collector or any backend that accepts OTLP. Each job becomes a trace with a span per step, so you can see where CI time goes and which steps fail. Tracing stays off until you set an endpoint.
Setup
Set the endpoint in the runner's environment, for example in its systemd unit, with docker run -e or in the Kubernetes pod spec:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
| Variable | Purpose |
|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
Collector base URL, traces go to /v1/traces under it |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT |
Full traces URL, takes precedence over the base URL |
OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERS |
Request headers such as an API key, as comma-separated key=value pairs with URL-encoded values |
OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES |
Resource attributes, the service name defaults to gitea-runner |
OTEL_TRACES_SAMPLER, OTEL_BSP_* |
Sampling and batching, see the SDK variables |
OTEL_SDK_DISABLED=true, OTEL_TRACES_EXPORTER=none |
Turn tracing off |
Only the http/protobuf protocol is supported.
What is sent
- A span per job, named
RUN {workflow} {job}, with the repository, ref, commit, run URL, run number, attempt and result. - A span per step, named as in Gitea's step list, including
PreandPoststages and the steps of composite actions, with the step's result. - The runner's name, UUID and version.
Logs, step output and secrets are never sent. Step names are taken from the workflow file as written, without expanding ${{ }} expressions.
Tracing inside jobs
Jobs receive TRACEPARENT, so tools in a job that use OpenTelemetry can add their spans to the job's trace. To let them export, set their OTEL_* variables in runner.envs, see Security.
Metrics
Metrics stay on the Prometheus endpoint enabled by metrics.enabled. To get them into OpenTelemetry, scrape /metrics with the collector's prometheus receiver.
Security
- OTLP has no authentication of its own. Send to a collector on the runner's host that holds the backend credentials, as the OpenTelemetry security guide recommends, or use
httpswith a token inOTEL_EXPORTER_OTLP_HEADERS. Over plainhttp, headers travel unencrypted. - TLS certificates are verified against the system CA bundle, set
SSL_CERT_FILEto use your own. Client certificates are not supported, use a local collector for mTLS. Redirects are not followed, so headers only reach the configured endpoint. - Jobs in containers never see the runner's
OTEL_*variables. Jobs on the host inherit the runner's whole environment, headers included, and can read its files anyway since they run as the same user. - Every workflow can read
runner.envs, so never put credentials there. Point jobs at a collector that holds them instead, keeping in mind that any workflow can then send data through it. - Traces carry the repository and step names of every job, private repositories included, so give the collector the same access controls as the runner's logs.