Compare commits

...

25 Commits

Author SHA1 Message Date
bircni
3dbed63f55 fix: close the action header group before the action runs (#1345)
`uses:` steps and job hooks kept their "Run …" group open until they finished. Gitea doesn't nest groups, so the action's own first `::group::` closed the header and everything before it stayed hidden there, e.g. all of goreleaser's output in https://gitea.com/gitea/cli/actions/runs/1121069/jobs/1519275. The header group now closes right after its details, as in actions/runner.

The action header also matches actions/runner's otherwise now: it is titled by the `uses:` reference, lists the resolved inputs including `action.yml` defaults without empty ones, is shown for pre stages too, and post stages start with `Post job cleanup.`.

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1345
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: bircni <bircni@icloud.com>
2026-10-26 17:42:16 +00:00
silverwind
dafe8d8d0c fix: keep dry runs off the docker daemon and speed up the tests (#1349)
Without a docker daemon six `act/runner` tests failed instead of passing. A dry run still created and removed job networks, the job teardown dropped the dry-run flag along with every other context value, and the container mocks asked the daemon for the runner arch.

This also speeds up the tests and CI without dropping assertions. Most of the time went to git and docker round trips, which are now pinned, batched or coalesced, and the dind and e2e steps cache or overlap more of their setup. On CI, `act/runner` went from 77s to 55s and `test-dind` from 108s to 55s. Locally, `act/runner` without docker went from 53s to 7s.

Split out of https://gitea.com/gitea/runner/pulls/1348, which adds Windows CI on top. The go cache changes moved to https://gitea.com/gitea/runner/pulls/1350.

Written by Claude Code.

---------

Co-authored-by: bircni <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1349
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-26 17:25:17 +00:00
Zettat123
8ae48ae033 fix: keep the shared action cache as extracted, re-fetch damaged trees (#1352)
The toolkit patch rewrote bundles in place in the shared action cache. Since https://gitea.com/gitea/runner/pulls/1316 trees are never refreshed, so a bundle left truncated failed every later job using it.

Patched bundles now only overwrite the job container's copy, so the runner no longer writes to an extracted tree.

Extraction records each file's size beside the tree, and a tree that no longer matches is fetched again, offline mode included, as a missing one always was. This also catches damage from outside the runner, such as storage losing a write, for a stat per file, a fraction of the copy each stage already makes. Trees from older runners have no record and download once more, except in offline mode, which keeps using them as before.

Generated with Claude Code.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1352
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: Zettat123 <zettat123@gmail.com>
2026-10-26 17:20:53 +00:00
Renovate Bot
8dd3776014 chore(deps): update dependencies (#1351)
Co-authored-by: Renovate Bot <renovate-bot@gitea.com>
2026-10-26 07:06:33 +00:00
silverwind
c6ca020281 ci: implement gitea's cache-seeder (#1350)
Like Gitea's CI, only a `cache-seeder` workflow on main saves the go caches, all other jobs only restore them. Caches are scoped per repository, so until now any pull request, forks included, could write a cache that other runs restore. The seeder compiles the tests without running them, so no test result is ever cached.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1350
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-24 17:04:51 +00:00
silverwind
fe2a66dc25 fix: align runner behaviours (#1338)
Contains a number of small bug fixes where jobs behaved differently from GitHub or failed silently.

1. Hooks fail on a refused `::set-env::` or `::add-path::` and no longer inherit the last step's identity
1. Docker build, exec and wait errors are returned instead of swallowed, token-only registry credentials work, logs of `--tty` services decode
1. Env files split lines like GitHub on each OS, so a carriage return is kept on Linux and CRLF ends a line on Windows, empty heredoc names or delimiters are rejected
1. Post steps run after their pre step even when main was skipped, composites see undeclared inputs, `runs.args` wins without `with.args` being parsed
1. Docker actions skip evaluating defaults of inputs given in `with` and read `entryPoint` and `args` in any case, a lowercase `dockerfile` is found, undeclared inputs get GitHub's warning
1. A user-set `CI` is kept, `GITHUB_REPOSITORY_ID` and the other missing `GITHUB_*` variables are exported
1. `timeout-minutes` can read the step's own `env`, a step-level `ACTIONS_ALLOW_UNSECURE_COMMANDS: false` beats a job-level `true` while the runner's own opt-in still wins, `pwsh` quotes the script path
1. A step `env:` given as one expression such as `${{ fromJSON(matrix.env) }}` is applied, `hashFiles` fails with GitHub's messages on script errors and after 120 seconds, a job image without node still gets an empty hash
1. A step whose command files cannot be reset fails instead of importing stale ones, a timeout between steps fails the job instead of reporting success
1. One named volume can be mounted at several paths, host-mode PTY steps keep their last output when they fail, host scratch directories are removed when job setup fails
1. Cancelling a step right after it starts kills its whole process tree
1. Cache prefix restores pick the newest entry created within the same second, also on S3, cache commits with a wrong size are rejected
1. Artifact chunks are written at their offset, download URLs are escaped, listing a run without artifacts no longer crashes
1. Poll backoff reaches `fetch_interval_max`, proxy passwords are masked in more forms, secrets are also masked in step names and in their quoted form in the runner log, a failed final report is logged
1. On Windows hosts a differently cased unsecure-commands opt-in or proxy variable counts, and an early cancel no longer leaks a job object handle
1. Interactive `register` prompts for a missing token and stops on SIGTERM, `exec --list --detect-event` no longer panics without events, `config set` works on `cache.s3` keys

Reviewed-on: https://gitea.com/gitea/runner/pulls/1338
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-23 09:18:24 +00:00
Zettat123
c719a88f36 fix(k8s): retry the startup exec until the kubelet accepts it (#1337)
A new node can run pods before its kubelet has a serving certificate, so the first exec into the job container fails with `error dialing backend: remote error: tls: internal error` and the job fails during setup.

That exec only reads the arch and PATH, so it now retries for up to a minute unless the API server rejects it (4xx) or the command ran.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1337
Reviewed-by: bircni <bircni@icloud.com>
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Zettat123 <39446+zettat123@noreply.gitea.com>
2026-10-22 20:59:48 +00:00
silverwind
d882311ec6 refactor: remove dead code and consolidate tests (#1335)
Removes dead code and redundant logic and consolidates tests, so there is less to read and maintain. No behavior change.

1. **act/runner**: inline single-use wrappers, compile regexes once instead of per call, embed the trampoline as a string, drop the unused legacy result of `parseActionCommand`
2. **act/container**: `sanitizeConfig` and `mergeContainerConfigs` stop returning the configs they already mutate in place, drop never-set option fields and the unused `user` parameter of `ExecWithCmdLine`
3. **act/common and cache server**: drop the unused `Trace`/`Tracef` loggers and the short SHA of `FindGitRevision`, `RegisterJob` releases through `RevokeJob`
4. **daemon, poller and reporter**: one shutdown wait for both poll modes, drop an unreachable fetch timeout branch, guard the reporter's last report time with its existing lock instead of an atomic
5. **build**: drop unused Makefile and release workflow variables, collapse the goreleaser ignore list to the same build targets
6. **tests**: fold near-duplicate tests into tables with shared setup

Throughout, hand-rolled logic gives way to `cmp.Or`, `strings.Cut`, `slices` and `min`/`max`.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1335
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-22 19:41:10 +00:00
silverwind
923a0bfb59 refactor: remove bolthold and gofakes3 deps (#1336)
1. Replace bolthold with direct bbolt access in the cache server. bolthold has not changed since March 2024 and kept `encoding/gob` in the binary although the store was already JSON. The on-disk layout stays the same, so existing caches keep working, and the index buckets bolthold left behind are dropped on the first sweep. Queries now run against an in-memory copy of the entries that is only re-read after another process wrote to the store, so lookups and sweeps no longer decode the whole store.

2. Replace gofakes3 with a small in-memory S3 fake in the one test that used it, which removes aws-sdk-go-v2, afero, mgo and x/tools from the module graph.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1336
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-22 16:28:49 +00:00
silverwind
6f0d4f1b48 refactor: migrate to connect-go v2 (#1325)
Migrates the runner to `connectrpc.com/connect/v2` by pinning actionslib v1.5.0, which carries https://gitea.com/gitea/actionslib/pulls/28. The wire behaviour towards Gitea is unchanged and works against Gitea servers on connect v1 and v2. The client keeps v1's unlimited response size, and the error codes and messages the runner checks stay the same.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1325
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-21 16:10:58 +00:00
silverwind
430f7bb2cf fix: handle workflow commands like GitHub's runner (#1333)
Commands the runner applies itself, like the `::set-output::` older actions still print, showed up raw in the job log. The reporter also re-parsed every line with one job-wide stop-commands and echo state, while act keeps one command handler per step, stage, composite child and hook, as GitHub does. A `::stop-commands::` left active in one of them kept the reporter suppressed for the others, so their `::add-mask::` values were never registered, and post steps logged command lines twice.

- act's command handler decides what reaches the log, as GitHub's runner does: applied commands only under `ACTIONS_STEP_DEBUG` or `::echo::on`, `::stop-commands::` and its resume line shown with long tokens masked
- Masks go straight to the reporter, which no longer parses command lines
- One parser follows GitHub's rules: leading whitespace, `##[...]` anywhere in a line, registered commands only
- Service logs honour only `::add-mask::`

---------

Co-authored-by: bircni <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1333
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-21 16:03:40 +00:00
silverwind
befb1b4bb9 fix: improve the actions toolkit patching (#1328)
Patch the scripts beside each node action entrypoint too, so code-split bundles like rust-cache v2.9.2, setup-java v6 and setup-deno v2 reach the v2 cache API. Entrypoints may point anywhere in the action's repository, as `gradle/actions/setup-gradle` does with `../dist`.

The `.LOCALHOST` match is now anchored on the `.GHE.COM` test before it, so another library's uppercase localhost check is never rewritten.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1328
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
2026-10-21 05:26:32 +00:00
silverwind
1a7f09224f perf: speed up job setup, action preparation and the cache server (#1327)
- Look up containers by name instead of listing all of them
- On Linux, cache the docker proxy discovery instead of starting a probe container per job
- Download a job's actions concurrently and share identical fetches between concurrent jobs
- Cache `runner.arch` per run and poll service health every 250ms
- Cache server: queue on an in-process lock instead of polling the database file lock, upload S3 objects up to 64MiB in one request and larger ones in 4 concurrent parts

| `exec`, median of 5 | Before | After |
|---|---:|---:|
| 1 step | 1.70s | 0.87s |
| 10 steps | 2.84s | 1.97s |
| Service + step | 3.78s | 2.18s |
| checkout + Node action | 3.64s | 2.52s |

Reviewed-on: https://gitea.com/gitea/runner/pulls/1327
Reviewed-by: Zettat123 <39446+zettat123@noreply.gitea.com>
2026-10-21 05:12:54 +00:00
bircni
af2b3f6df5 fix(deps): update go toolchain directive to v1.27.3 (#1332)
Go 1.27.3 includes a crypto/x509 security fix that distrusts a set of intermediate certificates (https://github.com/golang/go/issues/82345), so release builds need the new toolchain.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1332
Reviewed-by: TheFox0x7 <95654+thefox0x7@noreply.gitea.com>
2026-10-20 19:31:44 +00:00
silverwind
785313cc21 build: drop the xgo release targets (#1329)
Releases are built by goreleaser with `CGO_ENABLED=0`, so the xgo `release*` Makefile targets and their gxz compression are unused. Gitea dropped xgo the same way in https://github.com/go-gitea/gitea/pull/37828.

---------

Co-authored-by: bircni <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1329
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-20 16:21:47 +00:00
silverwind
41f1e08bec chore(deps): bump the pins Renovate does not update (#1330)
Renovate gets no release timestamps for Docker Hub images with more than 1000 tags (https://github.com/renovatebot/renovate/issues/46925) or for quay.io, so the 5-day minimum release age holds them back forever. The e2e workflows, the Kubernetes examples and the toolkit test pins are outside its managers, so these are bumped by hand.

The builder follows the current Alpine instead of the frozen 3.23.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1330
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-20 16:14:16 +00:00
silverwind
4158e05dae refactor: drop logrus and other direct dependencies (#1319)
After https://gitea.com/gitea/runner/pulls/1304, logrus stayed only to forward logs from docker/cli and moby/go-archive. The runner used a small part of those: volume parsing, the build context tar, the `ssh://` connection and registry credentials. Own code that matches them replaces them, so logrus leaves the module graph. docker/cli stays for its logrus-free `opts` and `connhelper/ssh` packages.

Other dependencies that the standard library or existing ones cover go too: mergo, httprouter (replaced by `http.ServeMux`), retry-go, go-isatty, go-selinux, container-device-interface, gotest.tools and go-cmp (replaced by testify). The binary shrinks by 5%.

Behaviour differences:

1. An unreadable file in a build context is skipped with a warning, where go-archive left a truncated tar.
2. Build contexts no longer carry `security.capability` xattrs.
3. The cache and artifact servers answer HEAD on GET routes, and no longer redirect trailing slashes or answer OPTIONS.

*Written by Claude.*

---------

Co-authored-by: bircni <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1319
Reviewed-by: bircni <bircni@icloud.com>
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: silverwind <2021+silverwind@noreply.gitea.com>
2026-10-18 18:57:08 +00:00
silverwind
e356c2ed85 chore(config): update stale go-yaml wrapping comment (#1324)
https://github.com/yaml/go-yaml/pull/441 fixed the wrapping bug this comment waited for, so reword it. `WithLineWidth(-1)` stays, `config edit` re-encodes the whole file and wrapping would reflow every long value.

Gitea: https://github.com/go-gitea/gitea/pull/39844

This PR was written with Claude.
Reviewed-on: https://gitea.com/gitea/runner/pulls/1324
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-18 13:15:28 +00:00
silverwind
ed940db4f0 fix: don't fail host steps over a leftover holding their output (#1323)
A successful host step whose background child kept its output open failed with `exec: WaitDelay expired before I/O complete`. Like GitHub's runner (`ProcessExitedHandler` in https://github.com/actions/runner/blob/main/src/Runner.Sdk/ProcessInvoker.cs), the step now succeeds after the drain timeout.

The leftover is killed only on Windows, as GitHub does there. A step whose timeout or cancellation hits during the drain still fails.

Fixes: https://gitea.com/gitea/runner/issues/1290
Co-authored-by: bircni <bircni@icloud.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1323
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-17 09:03:48 +00:00
Zettat123
755af87095 enhance(k8s): reach Kubernetes services by their id when it is a DNS subdomain (#1320)
On Kubernetes a service is reached by its id made a DNS label, so some service ids might become unreachable. For example, the `devstoreaccount1.azurite.local` service of Gitea's [MSSQL tests](2fe74d3cdd/.github/workflows/pull-db-tests.yml (L229)) is only in the pod's hosts as `devstoreaccount1-azurite-local`, while the tests connect to the id as written. Container names must be DNS labels, but `hostAliases` accepts DNS subdomains, so an id that is one is now added there too, and to `no_proxy`, as on Docker where a service is reached by its id.

The warning for a service not reachable by its id now also covers ids that differ only in case, as Kubernetes hostnames are lowercase.

---------

Co-authored-by: silverwind <me@silverwind.io>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1320
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Reviewed-by: bircni <bircni@icloud.com>
Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com>
Co-authored-by: Zettat123 <39446+zettat123@noreply.gitea.com>
2026-10-17 04:09:17 +00:00
silverwind
b9aa47d1a5 fix(deps): bump go-yaml to fix block and folded scalar round-trips (#1321)
Bumps go-yaml to its main branch and drops the workaround it makes obsolete, matching https://github.com/go-gitea/gitea/pull/39787.

- go-yaml main keeps block scalar indentation indicators (https://github.com/yaml/go-yaml/pull/414), so values with a leading newline no longer need quoting
- `config edit` no longer wraps lines, because go-yaml still wraps more-indented lines of folded scalars and turns a space into a newline

Reviewed-on: https://gitea.com/gitea/runner/pulls/1321
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-16 21:34:11 +00:00
silverwind
0d84ec6dab fix: remove the job network even when container removal overruns (#1318)
Job teardown ran every step on one shared minute. When removing the job container took longer, the network removal ran on an expired context and the network stayed behind until the orphan reclaim, which only takes networks older than `workdir_cleanup_age`, 24h by default. All that time it held a subnet of the daemon's address pool.

The job and service containers are now removed in parallel, and the docker proxy sweep, the network and the volumes get a fresh budget after them. The idle reclaim takes unused networks and volumes once they are 10 minutes old, and network removal treats a network that is already gone as removed.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1318
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <2021+silverwind@noreply.gitea.com>
2026-10-15 14:15:41 +00:00
silverwind
b769a9765f perf: download actions as archives once per job (#1316)
Each step refreshed its own clone of an action, so a composite using one action repository ten times fetched it ten times per job, and a single network error failed the job.

Like actions/runner, a job now downloads each action repository and ref once, as an archive extracted into a directory per commit. Each action still gets its own copy in the job container. This restores the per-job dedupe of https://gitea.com/gitea/runner/pulls/1178, which only reached a since-deleted branch.

How a download works:

1. The action URL is taken after `insteadOf` rewriting, as git would use it.
2. The archive comes from `<url>/archive/<ref>.tar.gz`, or from codeload.github.com for github.com, as for actions/runner.
3. A cached tag or branch is revalidated with its ETag, which GitHub and Gitea 28+ answer with a 304. A cached commit pin needs no request.
4. Transient failures (5xx, 408, 429 honouring `Retry-After`, dropped transfers) are retried three times with a 10 to 30 second backoff.
5. Git takes over, cloning and exporting the commit into the same directory layout, when the URL is not HTTP(S), the ref is a short commit, the host serves no usable archive, the host cannot be connected to, or the retries run out.

Median of 3 runs, git (shallow clone, then fetch) against archives, on a cold and a warm cache:

| Action | git cold | git warm | archive cold | archive warm |
|---|---|---|---|---|
| github.com actions/checkout@v4 | 937ms | 726ms | 286ms | 171ms |
| github.com actions/cache@v4 | 1006ms | 801ms | 516ms | 161ms |
| github.com actions/setup-node@v4 | 1046ms | 786ms | 463ms | 156ms |
| gitea.com actions/checkout@v4 | 2007ms | 1165ms | 667ms | 222ms |

The action cache layout changes, so existing caches download once more.

*Written by Claude.*

Fixes: https://gitea.com/gitea/runner/issues/1220
Reviewed-on: https://gitea.com/gitea/runner/pulls/1316
Reviewed-by: bircni <bircni@icloud.com>
Co-authored-by: silverwind <me@silverwind.io>
2026-10-15 12:49:37 +00:00
Lunny Xiao
9c4c66e70f enhance(report): add runner.log_report_level for the uploaded runner messages (#1310)
The runner's own messages between steps (not step output) were uploaded to the job log at `log.level`, so raising `log.level` for local debugging also changed what workflow authors see in Gitea, and lowering it hid messages from them.

Add `runner.log_report_level` to set that independently. Unset, it still follows `log.level`, so existing setups behave as before. An invalid value is rejected when the config loads.

Assisted-by: Codet
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: silverwind <2021+silverwind@noreply.gitea.com>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1310
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2026-10-14 14:15:26 +00:00
silverwind
42858c61df fix: stop pruning the cached action ref on refresh (#1312)
The action cache refresh fetched with `--prune` and the unqualified refspec `+<ref>:refs/runner/<hash>`. Git cannot map that ref back to a remote ref, so it deleted it on every other refresh. The first job after a branch or tag moved then failed with `cannot lock ref`, and a moved tag in a full clone could silently run its old commit. `--no-prune` fixes this and also overrides an inherited `fetch.prune`.

Fixes: https://gitea.com/gitea/runner/issues/1311
Reviewed-on: https://gitea.com/gitea/runner/pulls/1312
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Reviewed-by: Zettat123 <39446+zettat123@noreply.gitea.com>
2026-10-14 13:15:01 +00:00
200 changed files with 8965 additions and 9478 deletions

View File

@@ -45,8 +45,13 @@ cpu.out
/bin
/dist
/.tmp
/.git/hooks
/.git/logs
/.git/rr-cache
/.env
/.runner
/.runner.lock
/config.yaml
/Dockerfile
.DS_Store

View File

@@ -0,0 +1,29 @@
name: go-setup
description: Set up go and restore its caches, which only the cache-seeder workflow saves
runs:
using: composite
steps:
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
id: setup-go
with:
go-version-file: go.mod
check-latest: true
cache: false
- if: ${{ github.workflow == 'cache-seeder' }}
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/.cache/golangci-lint
key: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum', '.golangci.yml', 'Makefile', '.gitea/actions/go-setup/action.yml', '.gitea/workflows/cache-seeder.yml') }}
- if: ${{ github.workflow != 'cache-seeder' }}
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/.cache/golangci-lint
key: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum', '.golangci.yml', 'Makefile', '.gitea/actions/go-setup/action.yml', '.gitea/workflows/cache-seeder.yml') }}
restore-keys: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-

View File

@@ -0,0 +1,26 @@
name: cache-seeder
on:
workflow_dispatch:
push:
branches:
- main
paths:
- go.mod
- go.sum
- .golangci.yml
- Makefile
- .gitea/actions/go-setup/action.yml
- .gitea/workflows/cache-seeder.yml
jobs:
seed:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: ./.gitea/actions/go-setup
- run: make deps-tools lint build
- name: compile the test binaries without running them
run: |
make test GOTEST_FLAGS='-race -exec true'
go test -tags e2e -exec true ./e2e/...

View File

@@ -11,7 +11,6 @@ on:
env:
DOCKER_ORG: gitea
DOCKER_LATEST: nightly
jobs:
goreleaser:
@@ -36,7 +35,7 @@ jobs:
go-version-file: "go.mod"
check-latest: true
- name: goreleaser
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7
uses: goreleaser/goreleaser-action@a4f02ab5355a589de60d917a659ce617ed19b3de # v7
with:
distribution: goreleaser-pro
args: release --nightly
@@ -80,7 +79,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}

View File

@@ -35,7 +35,7 @@ jobs:
passphrase: ${{ secrets.PASSPHRASE }}
fingerprint: CC64B1DB67ABBEECAB24B6455FC346329753F4B0
- name: goreleaser
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7
uses: goreleaser/goreleaser-action@a4f02ab5355a589de60d917a659ce617ed19b3de # v7
with:
distribution: goreleaser-pro
args: release
@@ -69,7 +69,6 @@ jobs:
image: catthehacker/ubuntu:act-latest
env:
DOCKER_ORG: gitea
DOCKER_LATEST: latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
@@ -83,7 +82,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}

View File

@@ -17,25 +17,24 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: 'go.mod'
check-latest: true
- uses: ./.gitea/actions/go-setup
- name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
# Pre-pull act/runner's two largest base images so a slow pull can't dominate `make test`;
# the rest (alpine/ubuntu) pull on demand, absorbed by the make-test -timeout. The host
# daemon retains them between runs, so this is usually a fast manifest re-check.
# daemon retains them between runs, and their digest pins skip the registry when it has them.
- name: pre-pull test images
env:
TEST_JOB_IMAGE: node:24-bookworm-slim@sha256:d6aa754f16b3197301076f047b5def2f02ea1dbbc2ca920407d46d7ec7f87b20 # renovate: datasource=docker
TEST_SERVICE_IMAGE: nginx:alpine@sha256:df221db836e1754089190208cee7eeda94f233197056426eda74a43ab1abeac2 # renovate: datasource=docker
TEST_JOB_IMAGE: node:24-bookworm-slim@sha256:55fb5f5a527ea2c5c4a95f963e48ef90aa38c7b346054fdc8ee9957a2b55d205 # renovate: datasource=docker
TEST_SERVICE_IMAGE: nginx:alpine@sha256:a3554c0d0b48177b1413c26e01f695334428020d9df01243fbae3e11cfd17417 # renovate: datasource=docker
run: |
for image in "$TEST_JOB_IMAGE" "$TEST_SERVICE_IMAGE"; do
for attempt in 1 2 3; do
docker pull "$image" && break
[ "$attempt" = 3 ] || sleep 5
done
if [ "$(docker image inspect --format '{{.Os}}/{{.Architecture}}' "$image" 2>/dev/null)" != linux/amd64 ]; then
for attempt in 1 2 3; do
docker pull "$image" && break
[ "$attempt" = 3 ] || sleep 5
done
fi
docker tag "$image" "${image%@*}"
done
- name: lint
@@ -68,10 +67,7 @@ jobs:
- gitea/gitea:main-nightly
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: 'go.mod'
check-latest: true
- uses: ./.gitea/actions/go-setup
- name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
- name: e2e compatibility

View File

@@ -44,8 +44,6 @@ linters:
- $all
- "!**/act/common/logging/**"
deny:
- pkg: github.com/sirupsen/logrus
desc: log through gitea.com/gitea/runner/act/common/logging
- pkg: log/slog
desc: log through gitea.com/gitea/runner/act/common/logging
nolintlint:

View File

@@ -28,38 +28,18 @@ builds:
ignore:
- goos: darwin
goarch: arm
- goos: darwin
goarch: ppc64le
- goos: darwin
goarch: s390x
- goos: windows
goarch: ppc64le
- goos: windows
goarch: s390x
- goos: windows
goarch: arm
goarm: "5"
- goos: windows
goarch: arm
goarm: "6"
- goos: windows
goarch: arm
goarm: "7"
- goos: windows
goarch: arm64
- goos: freebsd
goarch: ppc64le
- goos: freebsd
goarch: s390x
- goos: freebsd
goarch: arm
goarm: "5"
- goos: freebsd
goarch: arm
goarm: "6"
- goos: freebsd
goarch: arm
goarm: "7"
- goos: freebsd
goarch: arm64
flags:

View File

@@ -28,23 +28,23 @@ depending on the prefix:
- **No prefix?** Do not escape, and split multi-line values into one row each.
- **Interpolating a secret?** Masking runs after escaping, so `AppendSecretMasker` registers the
encoded forms too.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the reporter
decodes exactly those two when folding a location into an annotation.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the command
handler folds an annotation's location into its decoded message and escapes the result again.
## Logging
Log through `act/common/logging`; the linter rejects importing `log/slog` or logrus elsewhere.
Log through `act/common/logging`; the linter rejects importing `log/slog` elsewhere.
A job's logger comes from `common.Logger(ctx)`; code with no job uses the package functions
(`logging.Infof`, `logging.With`). Both take printf-style messages, and `With("key", value)` adds
fields.
- The process-wide logger masks the secrets of every live task, the job logger masks its own.
Going around either leaks them.
- Libraries that log through logrus's or slog's global logger are forwarded to the process-wide
one, so they follow its level and masking too.
- That forwarding only works while logrus's global logger is left as `logging` configures it. A
dependency that calls `logrus.SetOutput` or `SetLevel` breaks it, so check
`logging.LogrusBridgeIntact()` when adding or upgrading one.
- The job logger's output can reach the job log workflow authors see, the process-wide logger's
stays on the runner host, so keep operator-only detail (host paths, internal addresses) off the
job logger.
- Libraries that log through slog's global logger are forwarded to the process-wide one, so they
follow its level and masking too. Avoid dependencies that log through logrus, nothing forwards it.
- Tests read what was logged with `logtest.NewLogger()`, or `logtest.SetDefault(t)` for the
package functions.

View File

@@ -1,7 +1,7 @@
### BUILDER STAGE
#
#
FROM golang:1.27-alpine3.23 AS builder
FROM golang:1.27-alpine AS builder
# Do not remove `git` here, it is required for getting runner version when executing `make build`
RUN apk add --no-cache make git
@@ -12,12 +12,14 @@ ENV GOPROXY=${GOPROXY:-}
COPY . /opt/src/runner
WORKDIR /opt/src/runner
RUN make clean && make build
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
make clean && make build
### DIND VARIANT
#
#
FROM docker:29.7.2-dind AS dind
FROM docker:29.9.1-dind AS dind
ARG VERSION=dev
@@ -39,7 +41,7 @@ ENTRYPOINT ["s6-svscan","/etc/s6"]
### DIND-ROOTLESS VARIANT
#
#
FROM docker:29.7.2-dind-rootless AS dind-rootless
FROM docker:29.9.1-dind-rootless AS dind-rootless
ARG VERSION=dev

View File

@@ -1,31 +1,22 @@
DIST := dist
EXECUTABLE := gitea-runner
DIST_DIRS := $(DIST)/binaries $(DIST)/release
GO ?= go
SHASUM ?= shasum -a 256
HAS_GO = $(shell hash $(GO) > /dev/null 2>&1 && echo "GO" || echo "NOGO" )
XGO_PACKAGE ?= src.techknowlogick.com/xgo@v1.9.0 # renovate: datasource=go
XGO_VERSION := go-1.27.x
GXZ_PACKAGE ?= github.com/ulikunitz/xz/cmd/gxz@v0.5.17 # renovate: datasource=go
PODMAN_TEST_IMAGE ?= quay.io/containers/podman:v5.8.4-immutable@sha256:90405e3c6a86f33c5a6f8dbdfa8e02a98143a8ba4337b84bdaec5336d2ef008a # renovate: datasource=docker
E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:0e62e56b382ebf485bff1e51a05cbe27c708545a2e91425467fd31eb3e249521 # renovate: datasource=docker
SERVICE_IMAGE ?= nginx:1.31.4-alpine@sha256:db35bfc6b2951e7f8a72db5db120288c127ffaeeb4a6d4b95a26fead017d5913 # renovate: datasource=docker
PODMAN_TEST_IMAGE ?= quay.io/containers/podman:v5.8.9-immutable@sha256:87f42f37fa85b421664c30b8f5f9cfc23920057f5e5fcadf0b6f68d9edc6d415 # renovate: datasource=docker
E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:8856b200466e9a5af9cc0d4de4ca96f73547833bcdddf9a567189ff772c4d372 # renovate: datasource=docker
SERVICE_IMAGE ?= nginx:1.31.7-alpine@sha256:892bb2152464f0c5d8972593a5f5afaf8054552580b699d95f8ff4af08c2d33d # renovate: datasource=docker
E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly
E2E_K3S_IMAGE ?= rancher/k3s:v1.36.4-k3s1@sha256:edad48e12bf81c3a09ac1c05c0c0ffaaa22145980b989d6fae84543a76b83657 # renovate: datasource=docker
E2E_K3S_IMAGE ?= rancher/k3s:v1.37.1-k3s1@sha256:ca7f37d993d82ef0dcdcfecb2e0e2618ea541dbaffc620c8cedebe01a82acd0d # renovate: datasource=docker
LINUX_ARCHS ?= linux/amd64,linux/arm64
DARWIN_ARCHS ?= darwin-12/amd64,darwin-12/arm64
WINDOWS_ARCHS ?= windows/amd64
GOFILES := $(shell find . -type f -name "*.go" -o -name "go.mod" ! -name "generated.*")
DOCKER_IMAGE ?= gitea/runner
DOCKER_TAG ?= nightly
DOCKER_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)
DOCKER_ROOTLESS_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)-dind-rootless
GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.14.0 # renovate: datasource=go
GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.8.0 # renovate: datasource=go
GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.15.0 # renovate: datasource=go
GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.9.0 # renovate: datasource=go
GOTEST_FLAGS ?= -race -timeout 20m -parallel 8
@@ -38,22 +29,16 @@ endif
ifeq ($(HAS_GO), GO)
GOPATH ?= $(shell $(GO) env GOPATH)
export PATH := $(GOPATH)/bin:$(PATH)
CGO_EXTRA_CFLAGS := -DSQLITE_MAX_VARIABLE_NUMBER=32766
CGO_CFLAGS ?= $(shell $(GO) env CGO_CFLAGS) $(CGO_EXTRA_CFLAGS)
endif
ifeq ($(OS), Windows_NT)
GOFLAGS := -v -buildmode=exe
EXECUTABLE ?= $(EXECUTABLE).exe
GO_ENV_WINDOWS := set GOOS=windows&&
else ifeq ($(OS), Windows)
GOFLAGS := -v -buildmode=exe
EXECUTABLE ?= $(EXECUTABLE).exe
GO_ENV_WINDOWS := set GOOS=windows&&
else
GOFLAGS := -v
EXECUTABLE ?= $(EXECUTABLE)
GO_ENV_WINDOWS := GOOS=windows
endif
@@ -108,8 +93,6 @@ fmt-check: ## check the Go code formatting
.PHONY: deps-tools
deps-tools: ## install tool dependencies
$(GO) install $(GOLANGCI_LINT_PACKAGE) & \
$(GO) install $(GXZ_PACKAGE) & \
$(GO) install $(XGO_PACKAGE) & \
$(GO) install $(GOVULNCHECK_PACKAGE) & \
wait
@@ -187,45 +170,6 @@ $(EXECUTABLE): $(GOFILES)
deps-backend: ## install backend dependencies
$(GO) mod download
.PHONY: release
release: release-windows release-linux release-darwin release-copy release-compress release-check ## build release artifacts
$(DIST_DIRS):
mkdir -p $(DIST_DIRS)
.PHONY: release-windows
release-windows: | $(DIST_DIRS)
CGO_CFLAGS="$(CGO_CFLAGS)" $(GO) run $(XGO_PACKAGE) -go $(XGO_VERSION) -buildmode exe -dest $(DIST)/binaries -tags 'netgo osusergo $(TAGS)' -ldflags '-s -w -linkmode external -extldflags "-static" $(LDFLAGS)' -targets '$(WINDOWS_ARCHS)' -out $(EXECUTABLE)-$(VERSION) .
ifeq ($(CI),true)
cp -r /build/* $(DIST)/binaries/
endif
.PHONY: release-linux
release-linux: | $(DIST_DIRS)
CGO_CFLAGS="$(CGO_CFLAGS)" $(GO) run $(XGO_PACKAGE) -go $(XGO_VERSION) -dest $(DIST)/binaries -tags 'netgo osusergo $(TAGS)' -ldflags '-s -w -linkmode external -extldflags "-static" $(LDFLAGS)' -targets '$(LINUX_ARCHS)' -out $(EXECUTABLE)-$(VERSION) .
ifeq ($(CI),true)
cp -r /build/* $(DIST)/binaries/
endif
.PHONY: release-darwin
release-darwin: | $(DIST_DIRS)
CGO_CFLAGS="$(CGO_CFLAGS)" $(GO) run $(XGO_PACKAGE) -go $(XGO_VERSION) -dest $(DIST)/binaries -tags 'netgo osusergo $(TAGS)' -ldflags '-s -w $(LDFLAGS)' -targets '$(DARWIN_ARCHS)' -out $(EXECUTABLE)-$(VERSION) .
ifeq ($(CI),true)
cp -r /build/* $(DIST)/binaries/
endif
.PHONY: release-copy
release-copy: | $(DIST_DIRS)
cd $(DIST); for file in `find . -type f -name "*"`; do cp $${file} ./release/; done;
.PHONY: release-check
release-check: | $(DIST_DIRS)
cd $(DIST)/release/; for file in `find . -type f -name "*"`; do echo "checksumming $${file}" && $(SHASUM) `echo $${file} | sed 's/^..//'` > $${file}.sha256; done;
.PHONY: release-compress
release-compress: | $(DIST_DIRS)
cd $(DIST)/release/; for file in `find . -type f -name "*"`; do echo "compressing $${file}" && $(GO) run $(GXZ_PACKAGE) -k -9 $${file}; done;
.PHONY: docker
docker: ## build the docker image
if ! docker buildx version >/dev/null 2>&1; then \

View File

@@ -6,7 +6,7 @@
On the runner:
- `git` 2.34.1 or newer on `PATH`, to download actions and reusable workflows.
- `git` 2.34.1 or newer on `PATH`, to clone reusable workflows, and actions whose host serves no archive of them.
- `docker` or `podman` with `podman.socket` enabled, for docker mode, Docker actions and `docker://` steps. To install Docker CE, follow the official [install instructions](https://docs.docker.com/engine/install/).
Where steps run, in the job image or on the runner in host mode:
@@ -242,7 +242,7 @@ A job's `services` are started before its steps run. When a service's image or i
```yaml
services:
postgres:
image: postgres:17
image: postgres:18
options: >-
--health-cmd pg_isready
--health-interval 5s
@@ -368,7 +368,7 @@ While the runner is idle it cleans up after earlier jobs:
- when `container.bind_workdir` is enabled, stale task workspace directories older than `runner.workdir_cleanup_age` are removed (default: `24h`; set `0` to disable)
- only purely numeric subdirectories under `container.workdir_parent` are treated as task workspaces and may be removed
- cleanup assumes `container.workdir_parent` is not shared across multiple runners
- on runners that use docker, per-job networks left behind by jobs the runner did not live to tear down are removed, identified by the `com.gitea.runner.uuid` label carrying this runner's uuid
- on runners that use docker, per-job networks and volumes left behind by jobs the runner did not live to tear down are removed once unused and 10 minutes old, identified by the `com.gitea.runner.uuid` label carrying this runner's uuid
- on runners with `kubernetes` labels, job pods with that label older than `runner.workdir_cleanup_age` are removed the same way, together with their Secrets
- cleanup runs every `runner.idle_cleanup_interval` (default: `10m`; set `0` to disable), and setting either knob to `0` disables all of the above

View File

@@ -15,6 +15,7 @@ import (
"encoding/json/v2"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/http/httputil"
@@ -32,9 +33,8 @@ import (
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/internal/pkg/disk"
"github.com/julienschmidt/httprouter"
"github.com/timshannon/bolthold"
"go.etcd.io/bbolt"
berrors "go.etcd.io/bbolt/errors"
)
const (
@@ -45,6 +45,8 @@ const (
// Short enough that a leaked URL is near-worthless; long enough to let the
// @actions/cache client download a big blob that was returned from /cache.
artifactURLTTL = 10 * time.Minute
dbOpenTimeout = 5 * time.Second
)
type credKey struct{}
@@ -80,12 +82,14 @@ type Handler struct {
dir string
storage cacheStorage
s3 *s3Storage
router *httprouter.Router
listener net.Listener
port int
server *http.Server
logger *logging.Logger
dbLock chan struct{} // queues store openers without polling bolt's file lock
entries cacheEntries // guarded by dbLock
gcing atomic.Bool
gcAt time.Time
@@ -143,6 +147,7 @@ func StartHandler(opts Options) (*Handler, error) {
internalSecret: opts.InternalSecret,
policy: opts.Policy.withDefaults(),
freeDisk: disk.FreeBytes,
dbLock: make(chan struct{}, 1),
}
if logger == nil {
@@ -195,30 +200,26 @@ func StartHandler(opts Options) (*Handler, error) {
h.storage = h.s3
}
secret, err := loadOrCreateSecret(dir)
if err != nil {
if h.secret, err = loadOrCreateSecret(dir); err != nil {
return nil, err
}
h.secret = secret
router := httprouter.New()
router.GET(apiPath+"/cache", h.bearerAuth(h.find))
router.POST(apiPath+"/caches", h.bearerAuth(h.reserve))
router.PATCH(apiPath+"/caches/:id", h.bearerAuth(h.upload))
router.POST(apiPath+"/caches/:id", h.bearerAuth(h.commit))
router.POST(apiPath+"/clean", h.bearerAuth(h.clean))
router := http.NewServeMux()
router.HandleFunc("GET "+apiPath+"/cache", h.bearerAuth(h.find))
router.HandleFunc("POST "+apiPath+"/caches", h.bearerAuth(h.reserve))
router.HandleFunc("PATCH "+apiPath+"/caches/{id}", h.bearerAuth(h.upload))
router.HandleFunc("POST "+apiPath+"/caches/{id}", h.bearerAuth(h.commit))
router.HandleFunc("POST "+apiPath+"/clean", h.bearerAuth(h.clean))
// Artifact GET is signed via query-string HMAC because @actions/cache
// does not attach Authorization when downloading archiveLocation.
router.GET(apiPath+"/artifacts/:id", h.signedAuth("", h.get))
router.HandleFunc("GET "+apiPath+"/artifacts/{id}", h.signedAuth("", h.get))
// Control-plane: a remote runner registers/revokes per-job tokens so the
// cache API can authenticate them. Always wired so the routes exist; the
// handlers themselves 401 when internalSecret is unset.
router.POST(internalPath+"/register", h.internalAuth(h.internalRegister))
router.POST(internalPath+"/revoke", h.internalAuth(h.internalRevoke))
router.HandleFunc("POST "+internalPath+"/register", h.internalAuth(h.internalRegister))
router.HandleFunc("POST "+internalPath+"/revoke", h.internalAuth(h.internalRevoke))
h.registerV2Routes(router)
router.NotFound = http.HandlerFunc(h.forwardOrNotFound)
h.router = router
router.HandleFunc(artifactServicePath, h.forwardOrNotFound)
h.gcCache()
@@ -328,16 +329,7 @@ func (h *Handler) RegisterJob(token string, cred JobCredential) func() {
}
}
h.credMu.Unlock()
return func() {
h.credMu.Lock()
if entry, ok := h.creds[token]; ok {
entry.refs--
if entry.refs <= 0 {
delete(h.creds, token)
}
}
h.credMu.Unlock()
}
return func() { h.RevokeJob(token) }
}
// RevokeJob explicitly revokes one registration of token, mirroring one call
@@ -400,10 +392,7 @@ func (h *Handler) Close() error {
}
var retErr error
if h.server != nil {
err := h.server.Close()
if err != nil {
retErr = err
}
retErr = h.server.Close()
h.server = nil
}
if h.listener != nil {
@@ -419,16 +408,170 @@ func (h *Handler) Close() error {
return retErr
}
func (h *Handler) openDB() (*bolthold.Store, error) {
return bolthold.Open(filepath.Join(h.dir, "bolt.db"), 0o644, &bolthold.Options{
Encoder: func(value any) ([]byte, error) { return json.Marshal(value) },
Decoder: func(data []byte, value any) error { return json.Unmarshal(data, value) },
Options: &bbolt.Options{
Timeout: 5 * time.Second,
NoGrowSync: bbolt.DefaultOptions.NoGrowSync,
FreelistType: bbolt.DefaultOptions.FreelistType,
},
// cacheBucket holds each entry as JSON under its decimal ID, the layout bolthold wrote, so older stores stay readable.
var cacheBucket = []byte("Cache")
var errCacheNotFound = errors.New("cache not found")
type cacheDB struct {
bolt *bbolt.DB
close func() error
entries *cacheEntries
}
// cacheEntries mirrors the bucket as of transaction tx, so queries decode it again only after another process wrote.
type cacheEntries struct {
tx int
byID map[uint64]*Cache
}
func (db *cacheDB) Close() error {
return db.close()
}
func cacheKey(id uint64) []byte {
return strconv.AppendUint(nil, id, 10)
}
func (db *cacheDB) get(id uint64) (*Cache, error) {
if err := db.bolt.View(db.load); err != nil {
return nil, err
}
cache, ok := db.entries.byID[id]
if !ok {
return nil, errCacheNotFound
}
copied := *cache
return &copied, nil
}
// find returns copies of the matching entries in ID order, so ties break the same way every time.
func (db *cacheDB) find(match func(*Cache) bool) ([]*Cache, error) {
if err := db.bolt.View(db.load); err != nil {
return nil, err
}
var caches []*Cache
for _, cache := range db.entries.byID {
if match(cache) {
copied := *cache
caches = append(caches, &copied)
}
}
slices.SortFunc(caches, func(left, right *Cache) int { return cmp.Compare(left.ID, right.ID) })
return caches, nil
}
func (db *cacheDB) load(tx *bbolt.Tx) error {
if db.entries.byID != nil && db.entries.tx == tx.ID() {
return nil
}
byID := make(map[uint64]*Cache)
if bucket := tx.Bucket(cacheBucket); bucket != nil {
if err := bucket.ForEach(func(key, data []byte) (err error) {
cache := &Cache{}
if err := json.Unmarshal(data, cache); err != nil {
return err
}
cache.ID, err = strconv.ParseUint(string(key), 10, 64) // an interrupted legacy insert left "id":0 in the value
byID[cache.ID] = cache
return err
}); err != nil {
return err
}
}
db.entries.byID, db.entries.tx = byID, tx.ID()
return nil
}
// update keeps the mirror in step with its own write, and drops it when another process wrote in between or the write rolled back.
func (db *cacheDB) update(fn func(*bbolt.Bucket) error) error {
err := db.bolt.Update(func(tx *bbolt.Tx) error {
if db.entries.tx != tx.ID()-1 {
db.entries.byID = nil
}
db.entries.tx = tx.ID()
bucket, err := tx.CreateBucketIfNotExists(cacheBucket)
if err != nil {
return err
}
return fn(bucket)
})
if err != nil {
db.entries.byID = nil
}
return err
}
func (db *cacheDB) put(bucket *bbolt.Bucket, cache *Cache) error {
data, err := json.Marshal(cache)
if err != nil {
return err
}
if db.entries.byID != nil {
copied := *cache
db.entries.byID[cache.ID] = &copied
}
return bucket.Put(cacheKey(cache.ID), data)
}
func (db *cacheDB) delete(bucket *bbolt.Bucket, id uint64) error {
delete(db.entries.byID, id)
return bucket.Delete(cacheKey(id))
}
// dropLegacyIndexes deletes the index buckets bolthold kept, all created together and never recreated, checking first so later runs do not write.
func (db *cacheDB) dropLegacyIndexes() error {
var found bool
_ = db.bolt.View(func(tx *bbolt.Tx) error {
found = tx.Bucket([]byte("_index:Cache:Key")) != nil
return nil
})
if !found {
return nil
}
return db.bolt.Update(func(tx *bbolt.Tx) error {
for _, field := range []string{"Repo", "Key", "Version", "Complete", "UsedAt", "CreatedAt"} {
if err := tx.DeleteBucket([]byte("_index:Cache:" + field)); err != nil && !errors.Is(err, berrors.ErrBucketNotFound) {
return err
}
}
return nil
})
}
// save rewrites a stored entry, failing once a sweep has removed it so a late commit cannot resurrect it.
func (db *cacheDB) save(cache *Cache) error {
return db.update(func(bucket *bbolt.Bucket) error {
if bucket.Get(cacheKey(cache.ID)) == nil {
return errCacheNotFound
}
return db.put(bucket, cache)
})
}
func (h *Handler) openDB() (store *cacheDB, err error) {
select {
case h.dbLock <- struct{}{}:
case <-time.After(dbOpenTimeout):
return nil, berrors.ErrTimeout
}
defer func() {
if store == nil { // only an opened store hands the lock to Close, even when Open panics
<-h.dbLock
}
}()
db, err := bbolt.Open(filepath.Join(h.dir, "bolt.db"), 0o644, &bbolt.Options{
Timeout: dbOpenTimeout,
NoGrowSync: bbolt.DefaultOptions.NoGrowSync,
FreelistType: bbolt.DefaultOptions.FreelistType,
})
if err != nil {
return nil, err
}
return &cacheDB{bolt: db, entries: &h.entries, close: sync.OnceValue(func() error {
defer func() { <-h.dbLock }()
return db.Close()
})}, nil
}
// syncMetadata mirrors the bucket into the store until ctx ends, so no request waits on S3 for it.
@@ -453,7 +596,7 @@ func (h *Handler) importMetadata(ctx context.Context) error {
if err != nil {
return err
}
local := h.caches(db, bolthold.Where("Complete").Eq(true).Index("Complete"))
local := h.caches(db, func(cache *Cache) bool { return cache.Complete })
db.Close()
published, err := h.s3.published(ctx)
@@ -481,15 +624,15 @@ func (h *Handler) importMetadata(ctx context.Context) error {
return err
}
defer db.Close()
return db.Bolt().Update(func(tx *bbolt.Tx) error {
return db.update(func(bucket *bbolt.Bucket) error {
for _, cache := range added {
if err := db.TxUpsert(tx, cache.ID, cache); err != nil {
if err := db.put(bucket, cache); err != nil {
return err
}
}
for _, cache := range gone {
_ = h.s3.Storage.Remove(cache.ID) // a blob kept on disk before the runner used S3
if err := db.TxDelete(tx, cache.ID, cache); err != nil && !errors.Is(err, bolthold.ErrNotFound) {
if err := db.delete(bucket, cache.ID); err != nil {
return err
}
}
@@ -498,7 +641,7 @@ func (h *Handler) importMetadata(ctx context.Context) error {
}
// GET /_apis/artifactcache/cache
func (h *Handler) find(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
func (h *Handler) find(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context())
keys := strings.Split(r.URL.Query().Get("keys"), ",")
version := r.URL.Query().Get("version")
@@ -529,7 +672,7 @@ func (h *Handler) find(w http.ResponseWriter, r *http.Request, _ httprouter.Para
// lookupCache returns the entry to restore for these keys, or (nil, nil) when there is none:
// either nothing matched, or the match had lost its blob to a prune, in which case the dangling
// entry is dropped on the way out.
func (h *Handler) lookupCache(db *bolthold.Store, repo string, keys []string, version string) (*Cache, error) {
func (h *Handler) lookupCache(db *cacheDB, repo string, keys []string, version string) (*Cache, error) {
cache, err := findCache(db, repo, keys, version)
if err != nil || cache == nil {
return nil, err
@@ -549,7 +692,7 @@ func (h *Handler) lookupCache(db *bolthold.Store, repo string, keys []string, ve
}
// POST /_apis/artifactcache/caches
func (h *Handler) reserve(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
func (h *Handler) reserve(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context())
api := &Request{}
if err := json.UnmarshalRead(r.Body, api); err != nil {
@@ -568,9 +711,10 @@ func (h *Handler) reserve(w http.ResponseWriter, r *http.Request, _ httprouter.P
}
defer db.Close()
now := time.Now().Unix()
cache.CreatedAt = now
cache.UsedAt = now
now := time.Now()
cache.CreatedAt = now.Unix()
cache.CreatedAtNanos = now.Nanosecond()
cache.UsedAt = now.Unix()
if err := h.insertCache(db, cache); err != nil {
h.responseJSON(w, r, 500, err)
return
@@ -581,23 +725,23 @@ func (h *Handler) reserve(w http.ResponseWriter, r *http.Request, _ httprouter.P
}
// PATCH /_apis/artifactcache/caches/:id
func (h *Handler) upload(w http.ResponseWriter, r *http.Request, params httprouter.Params) {
func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context())
id, err := strconv.ParseInt(params.ByName("id"), 10, 64)
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
h.responseJSON(w, r, 400, err)
return
}
cache := &Cache{}
db, err := h.openDB()
if err != nil {
h.responseJSON(w, r, 500, err)
return
}
defer db.Close()
if err := db.Get(id, cache); err != nil {
if errors.Is(err, bolthold.ErrNotFound) {
cache, err := db.get(uint64(id))
if err != nil {
if errors.Is(err, errCacheNotFound) {
h.responseJSON(w, r, 400, fmt.Errorf("cache %d: not reserved", id))
return
}
@@ -615,7 +759,7 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request, params httprout
return
}
db.Close()
start, _, err := parseContentRange(r.Header.Get("Content-Range"))
start, err := parseContentRange(r.Header.Get("Content-Range"))
if err != nil {
h.responseJSON(w, r, 400, err)
return
@@ -629,23 +773,34 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request, params httprout
}
// POST /_apis/artifactcache/caches/:id
func (h *Handler) commit(w http.ResponseWriter, r *http.Request, params httprouter.Params) {
func (h *Handler) commit(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context())
id, err := strconv.ParseInt(params.ByName("id"), 10, 64)
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
h.responseJSON(w, r, 400, err)
return
}
var request struct {
Size *int64 `json:"size"`
}
body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
if err == nil && len(body) > 0 {
err = json.Unmarshal(body, &request)
}
if err != nil {
h.responseJSON(w, r, http.StatusBadRequest, err)
return
}
cache := &Cache{}
db, err := h.openDB()
if err != nil {
h.responseJSON(w, r, 500, err)
return
}
defer db.Close()
if err := db.Get(id, cache); err != nil {
if errors.Is(err, bolthold.ErrNotFound) {
cache, err := db.get(uint64(id))
if err != nil {
if errors.Is(err, errCacheNotFound) {
h.responseJSON(w, r, 400, fmt.Errorf("cache %d: not reserved", id))
return
}
@@ -663,6 +818,14 @@ func (h *Handler) commit(w http.ResponseWriter, r *http.Request, params httprout
return
}
if request.Size != nil {
if *request.Size < 0 || cache.Size >= 0 && cache.Size != *request.Size {
h.responseJSON(w, r, http.StatusBadRequest, errors.New("invalid cache size"))
return
}
cache.Size = *request.Size
}
db.Close()
if err := h.commitCache(cache); err != nil {
@@ -697,7 +860,7 @@ func (h *Handler) commitCache(cache *Cache) error {
return err
}
defer db.Close()
if err := db.Update(cache.ID, cache); err != nil {
if err := db.save(cache); err != nil {
return err
}
if h.s3 != nil {
@@ -716,8 +879,8 @@ func (h *Handler) commitCache(cache *Cache) error {
// @actions/cache toolkit downloads archiveLocation without Authorization.
// Repository scoping is already enforced at find() time; the signature binds
// the URL to the specific cache ID and an expiry.
func (h *Handler) get(w http.ResponseWriter, r *http.Request, params httprouter.Params) {
id, err := strconv.ParseInt(params.ByName("id"), 10, 64)
func (h *Handler) get(w http.ResponseWriter, r *http.Request) {
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
h.responseJSON(w, r, 400, err)
return
@@ -727,7 +890,7 @@ func (h *Handler) get(w http.ResponseWriter, r *http.Request, params httprouter.
}
// POST /_apis/artifactcache/clean
func (h *Handler) clean(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
func (h *Handler) clean(w http.ResponseWriter, r *http.Request) {
// TODO: don't support force deleting cache entries
// see: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#force-deleting-cache-entries
@@ -737,8 +900,8 @@ func (h *Handler) clean(w http.ResponseWriter, r *http.Request, _ httprouter.Par
// bearerAuth resolves ACTIONS_RUNTIME_TOKEN against the set of currently
// registered jobs. A match attaches the job's JobCredential to the request
// context; a miss returns 401 before the handler body runs.
func (h *Handler) bearerAuth(handler httprouter.Handle) httprouter.Handle {
return func(w http.ResponseWriter, r *http.Request, params httprouter.Params) {
func (h *Handler) bearerAuth(handler http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
h.logger.Debugf("%s %s", r.Method, r.URL.Path)
token := bearerToken(r)
if token == "" {
@@ -751,7 +914,7 @@ func (h *Handler) bearerAuth(handler httprouter.Handle) httprouter.Handle {
return
}
ctx := context.WithValue(r.Context(), credKey{}, cred)
handler(w, r.WithContext(ctx), params)
handler(w, r.WithContext(ctx))
go h.gcCache()
}
}
@@ -766,10 +929,10 @@ func (h *Handler) unauthorized(w http.ResponseWriter, r *http.Request, err error
// signedAuth authenticates a signed URL. purpose separates the flavours of URL the
// handler hands out, so one cannot be replayed as another; see computeSignature.
func (h *Handler) signedAuth(purpose string, handler httprouter.Handle) httprouter.Handle {
return func(w http.ResponseWriter, r *http.Request, params httprouter.Params) {
func (h *Handler) signedAuth(purpose string, handler http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
h.logger.Debugf("%s %s", r.Method, r.URL.Path)
id, err := strconv.ParseInt(params.ByName("id"), 10, 64)
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
h.responseJSON(w, r, 400, err)
return
@@ -794,7 +957,7 @@ func (h *Handler) signedAuth(purpose string, handler httprouter.Handle) httprout
h.responseJSON(w, r, http.StatusUnauthorized, errors.New("bad signature"))
return
}
handler(w, r, params)
handler(w, r)
go h.gcCache()
}
}
@@ -803,8 +966,8 @@ func (h *Handler) signedAuth(purpose string, handler httprouter.Handle) httprout
// constant-time-equal the configured internalSecret. If the secret is empty,
// the control-plane is disabled and every request gets 404 — which matches
// the upstream nektos/act behavior of "the route does not exist".
func (h *Handler) internalAuth(handler httprouter.Handle) httprouter.Handle {
return func(w http.ResponseWriter, r *http.Request, params httprouter.Params) {
func (h *Handler) internalAuth(handler http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if h.internalSecret == "" {
http.NotFound(w, r)
return
@@ -814,7 +977,7 @@ func (h *Handler) internalAuth(handler httprouter.Handle) httprouter.Handle {
h.responseJSON(w, r, http.StatusUnauthorized, errors.New("internal: bad secret"))
return
}
handler(w, r, params)
handler(w, r)
}
}
@@ -837,7 +1000,7 @@ func (h *Handler) ResultsURL(cred JobCredential) string {
return h.baseURL(cred)
}
func (h *Handler) internalRegister(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
func (h *Handler) internalRegister(w http.ResponseWriter, r *http.Request) {
var body internalRegisterBody
if err := json.UnmarshalRead(r.Body, &body); err != nil {
h.responseJSON(w, r, http.StatusBadRequest, err)
@@ -852,7 +1015,7 @@ func (h *Handler) internalRegister(w http.ResponseWriter, r *http.Request, _ htt
}
// POST /_internal/revoke
func (h *Handler) internalRevoke(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
func (h *Handler) internalRevoke(w http.ResponseWriter, r *http.Request) {
var body internalRevokeBody
if err := json.UnmarshalRead(r.Body, &body); err != nil {
h.responseJSON(w, r, http.StatusBadRequest, err)
@@ -882,10 +1045,8 @@ func bearerToken(r *http.Request) string {
}
func credFromContext(ctx context.Context) JobCredential {
if cred, ok := ctx.Value(credKey{}).(JobCredential); ok {
return cred
}
return JobCredential{}
cred, _ := ctx.Value(credKey{}).(JobCredential)
return cred
}
// computeSignature signs a URL for one cache entry and expiry. purpose is mixed into the
@@ -911,8 +1072,7 @@ func (h *Handler) signedArtifactURL(cred JobCredential, cacheID uint64, exp time
}
// if not found, return (nil, nil) instead of an error.
func findCache(db *bolthold.Store, repo string, keys []string, version string) (*Cache, error) {
cache := &Cache{}
func findCache(db *cacheDB, repo string, keys []string, version string) (*Cache, error) {
for _, prefix := range keys {
// if a key in the list matches exactly, don't return partial matches
exact, err := findExactCache(db, repo, prefix, version, true)
@@ -922,22 +1082,28 @@ func findCache(db *bolthold.Store, repo string, keys []string, version string) (
if exact != nil {
return exact, nil
}
if err := db.FindOne(cache,
bolthold.Where("Repo").Eq(repo).Index("Repo").
And("Key").MatchFunc(func(key string) (bool, error) { return strings.HasPrefix(key, prefix), nil }).
And("Version").Eq(version).
And("Complete").Eq(true).
SortBy("CreatedAt").Reverse()); err != nil {
if errors.Is(err, bolthold.ErrNotFound) {
continue
}
caches, err := db.find(func(cache *Cache) bool {
return cache.Repo == repo && strings.HasPrefix(cache.Key, prefix) && cache.Version == version && cache.Complete
})
if err != nil {
return nil, fmt.Errorf("find cache: %w", err)
}
return cache, nil
if cache := latest(caches, (*Cache).createdAtNano); cache != nil {
return cache, nil
}
}
return nil, nil //nolint:nilnil // pre-existing issue from nektos/act
}
func latest(caches []*Cache, stamp func(*Cache) int64) *Cache {
if len(caches) == 0 {
return nil
}
return slices.MaxFunc(caches, func(left, right *Cache) int {
return cmp.Or(cmp.Compare(stamp(left), stamp(right)), cmp.Compare(left.ID, right.ID))
})
}
// findExactCache returns the entry for exactly this key and version, or (nil, nil) if there is
// none. Unlike findCache it never falls back to a prefix (restore-key) match, which is what both
// its callers need: a new key that is only a prefix of an existing key is not the same entry.
@@ -945,43 +1111,33 @@ func findCache(db *bolthold.Store, repo string, keys []string, version string) (
// A completed entry is the one to restore, sorted by when it was written. An incomplete one is a
// reservation being uploaded to, sorted by when it was last written to, because the upload route
// touches UsedAt on every part.
func findExactCache(db *bolthold.Store, repo, key, version string, complete bool) (*Cache, error) {
sortBy := "UsedAt"
if complete {
sortBy = "CreatedAt"
}
cache := &Cache{}
err := db.FindOne(cache,
bolthold.Where("Key").Eq(key).Index("Key").
And("Repo").Eq(repo).
And("Version").Eq(version).
And("Complete").Eq(complete).
SortBy(sortBy).Reverse())
if errors.Is(err, bolthold.ErrNotFound) {
return nil, nil //nolint:nilnil // absence is not an error here
}
func findExactCache(db *cacheDB, repo, key, version string, complete bool) (*Cache, error) {
caches, err := db.find(func(cache *Cache) bool {
return cache.Key == key && cache.Repo == repo && cache.Version == version && cache.Complete == complete
})
if err != nil {
return nil, fmt.Errorf("find cache: %w", err)
}
return cache, nil
stamp := func(cache *Cache) int64 { return cache.UsedAt }
if complete {
stamp = (*Cache).createdAtNano
}
return latest(caches, stamp), nil
}
func (h *Handler) insertCache(db *bolthold.Store, cache *Cache) error {
if h.s3 != nil {
var id [8]byte
_, _ = rand.Read(id[:])
cache.ID = binary.BigEndian.Uint64(id[:]) >> 11 // IDs must be unique across runners, and v1 clients parse an ID as a JS number
return db.Insert(cache.ID, cache)
}
return db.Bolt().Update(func(tx *bbolt.Tx) error {
if err := db.TxInsert(tx, bolthold.NextSequence(), cache); err != nil {
func (h *Handler) insertCache(db *cacheDB, cache *Cache) error {
return db.update(func(bucket *bbolt.Bucket) (err error) {
if h.s3 != nil {
var id [8]byte
_, _ = rand.Read(id[:])
cache.ID = binary.BigEndian.Uint64(id[:]) >> 11 // IDs must be unique across runners, and v1 clients parse an ID as a JS number
} else if cache.ID, err = bucket.NextSequence(); err != nil {
return fmt.Errorf("insert cache: %w", err)
}
// write back id to db
if err := db.TxUpdate(tx, cache.ID, cache); err != nil {
return fmt.Errorf("write back id to db: %w", err)
if bucket.Get(cacheKey(cache.ID)) != nil {
return fmt.Errorf("insert cache: id %d is taken", cache.ID)
}
return nil
return db.put(bucket, cache)
})
}
@@ -997,11 +1153,11 @@ func (h *Handler) touchCache(id uint64, requireIncomplete bool) error {
}
defer db.Close()
cache := &Cache{}
if err := db.Get(id, cache); err != nil {
if errors.Is(err, bolthold.ErrNotFound) {
return nil
}
cache, err := db.get(id)
if errors.Is(err, errCacheNotFound) {
return nil
}
if err != nil {
return err
}
if requireIncomplete && cache.Complete {
@@ -1011,7 +1167,7 @@ func (h *Handler) touchCache(id uint64, requireIncomplete bool) error {
return nil
}
cache.UsedAt = time.Now().Unix()
return db.Update(cache.ID, cache)
return db.save(cache)
}
const (
@@ -1057,9 +1213,6 @@ func (p Policy) withDefaults() Policy {
}
func (h *Handler) gcCache() {
if h.gcing.Load() {
return
}
if !h.gcing.CompareAndSwap(false, true) {
return
}
@@ -1078,6 +1231,9 @@ func (h *Handler) gcCache() {
}
defer db.Close()
if err := db.dropLegacyIndexes(); err != nil {
h.logger.Warnf("drop legacy indexes: %v", err)
}
h.evictIncomplete(db)
if h.s3 != nil {
return // lifecycle rules on the bucket expire its entries
@@ -1090,7 +1246,7 @@ func (h *Handler) gcCache() {
// evictForFreeSpace bounds the volume itself, so it also covers bytes the cache never
// accounted for.
func (h *Handler) evictForFreeSpace(db *bolthold.Store) {
func (h *Handler) evictForFreeSpace(db *cacheDB) {
free, err := h.freeDisk(h.dir)
if err != nil {
h.logger.Debugf("free disk check: %v", err) // unsupported platform, treat as unavailable rather than full
@@ -1110,38 +1266,33 @@ func (h *Handler) evictForFreeSpace(db *bolthold.Store) {
}
// evictIncomplete removes uploads that stopped part way, which are most likely broken.
func (h *Handler) evictIncomplete(db *bolthold.Store) {
h.sweep(db, bolthold.
Where("UsedAt").Lt(time.Now().Add(-uploadStallTimeout).Unix()).
And("Complete").Eq(false).
Index("UsedAt"))
func (h *Handler) evictIncomplete(db *cacheDB) {
stalled := time.Now().Add(-uploadStallTimeout).Unix()
h.sweep(db, func(cache *Cache) bool { return cache.UsedAt < stalled && !cache.Complete })
}
func (h *Handler) evictExpired(db *bolthold.Store) {
func (h *Handler) evictExpired(db *cacheDB) {
if h.policy.Retention <= 0 {
return
}
// Never below inUseGrace, or a short retention would outrun a signed URL already issued.
window := max(h.policy.Retention+touchStale, inUseGrace)
h.sweep(db, bolthold.Where("UsedAt").Lt(time.Now().Add(-window).Unix()).Index("UsedAt"))
expired := time.Now().Add(-max(h.policy.Retention+touchStale, inUseGrace)).Unix()
h.sweep(db, func(cache *Cache) bool { return cache.UsedAt < expired })
}
// evictSuperseded removes entries a newer one with the same key and version replaced. The
// aggregation includes Repo so two repos sharing a (key, version) do not evict each other.
func (h *Handler) evictSuperseded(db *bolthold.Store) {
results, err := db.FindAggregate(&Cache{}, bolthold.Where("Complete").Eq(true).Index("Complete"), "Repo", "Key", "Version")
if err != nil {
h.logger.Warnf("find aggregate caches: %v", err)
return
// grouping includes Repo so two repos sharing a (key, version) do not evict each other.
func (h *Handler) evictSuperseded(db *cacheDB) {
type entry struct{ repo, key, version string }
groups := make(map[entry][]*Cache)
for _, cache := range h.caches(db, func(cache *Cache) bool { return cache.Complete }) {
group := entry{cache.Repo, cache.Key, cache.Version}
groups[group] = append(groups[group], cache)
}
var caches []*Cache
for _, result := range results {
if result.Count() <= 1 {
continue
}
result.Sort("CreatedAt")
caches = caches[:0]
result.Reduction(&caches)
for _, caches := range groups {
slices.SortFunc(caches, func(left, right *Cache) int {
return cmp.Or(cmp.Compare(left.createdAtNano(), right.createdAtNano()), cmp.Compare(left.ID, right.ID))
})
for _, cache := range caches[:len(caches)-1] {
if inUse(cache) {
continue
@@ -1153,7 +1304,7 @@ func (h *Handler) evictSuperseded(db *bolthold.Store) {
// evictOversized applies the per-repository limit, then the whole-store one. Only completed
// entries count, since only those carry a size measured at commit rather than claimed.
func (h *Handler) evictOversized(db *bolthold.Store) {
func (h *Handler) evictOversized(db *cacheDB) {
if h.policy.RepoSizeLimit > 0 {
byRepo := make(map[string][]*Cache)
for _, cache := range h.completedByUse(db) {
@@ -1168,7 +1319,7 @@ func (h *Handler) evictOversized(db *bolthold.Store) {
// evictTotal caps the store as a whole. It re-queries because the per-repo pass may have
// deleted rows an earlier result still holds.
func (h *Handler) evictTotal(db *bolthold.Store) {
func (h *Handler) evictTotal(db *cacheDB) {
if h.policy.SizeLimit <= 0 {
return
}
@@ -1176,15 +1327,15 @@ func (h *Handler) evictTotal(db *bolthold.Store) {
}
// evictRepo reclaims space when a commit pushes a repo over, rather than at the next sweep.
func (h *Handler) evictRepo(db *bolthold.Store, repo string) {
func (h *Handler) evictRepo(db *cacheDB, repo string) {
if h.policy.RepoSizeLimit <= 0 {
return
}
h.evictTo(db, h.cachesByUse(db, bolthold.Where("Repo").Eq(repo).And("Complete").Eq(true).Index("Repo")), h.policy.RepoSizeLimit, "repository "+repo)
h.evictTo(db, h.cachesByUse(db, func(cache *Cache) bool { return cache.Repo == repo && cache.Complete }), h.policy.RepoSizeLimit, "repository "+repo)
}
// evictTo deletes until caches fit limit. caches must be ordered by UsedAt ascending.
func (h *Handler) evictTo(db *bolthold.Store, caches []*Cache, limit int64, scope string) {
func (h *Handler) evictTo(db *cacheDB, caches []*Cache, limit int64, scope string) {
// An entry bigger than the limit never fits, so it goes on its own account instead of
// dragging every neighbour out first and then following them next sweep.
fits := caches[:0]
@@ -1231,41 +1382,39 @@ func touchNeeded(cache *Cache) bool {
// touch stamps UsedAt through the caller's store, a bolt write on the read path. It cannot
// go through touchCache, which opens its own store and would block on the exclusive lock
// for as long as the caller holds one.
func (h *Handler) touch(db *bolthold.Store, cache *Cache) {
func (h *Handler) touch(db *cacheDB, cache *Cache) {
if !touchNeeded(cache) {
return
}
cache.UsedAt = time.Now().Unix()
if err := db.Update(cache.ID, cache); err != nil {
if err := db.save(cache); err != nil {
h.logger.Warnf("touch cache: %v", err)
}
}
func (h *Handler) sweep(db *bolthold.Store, query *bolthold.Query) {
for _, cache := range h.caches(db, query) {
func (h *Handler) sweep(db *cacheDB, match func(*Cache) bool) {
for _, cache := range h.caches(db, match) {
h.deleteCache(db, cache)
}
}
func (h *Handler) caches(db *bolthold.Store, query *bolthold.Query) []*Cache {
var caches []*Cache
if err := db.Find(&caches, query); err != nil {
func (h *Handler) caches(db *cacheDB, match func(*Cache) bool) []*Cache {
caches, err := db.find(match)
if err != nil {
h.logger.Warnf("find caches: %v", err)
}
return caches
}
// cachesByUse returns matches least recently accessed first, sorting here rather than with
// bolthold's SortBy, which reflects over every field it compares.
func (h *Handler) cachesByUse(db *bolthold.Store, query *bolthold.Query) []*Cache {
caches := h.caches(db, query)
func (h *Handler) cachesByUse(db *cacheDB, match func(*Cache) bool) []*Cache {
caches := h.caches(db, match)
slices.SortFunc(caches, func(a, b *Cache) int { return cmp.Compare(a.UsedAt, b.UsedAt) })
return caches
}
// completedByUse returns every entry the size limits count, least recently accessed first.
func (h *Handler) completedByUse(db *bolthold.Store) []*Cache {
return h.cachesByUse(db, bolthold.Where("Complete").Eq(true).Index("Complete"))
func (h *Handler) completedByUse(db *cacheDB) []*Cache {
return h.cachesByUse(db, func(cache *Cache) bool { return cache.Complete })
}
func totalSize(caches []*Cache) int64 {
@@ -1278,7 +1427,7 @@ func totalSize(caches []*Cache) int64 {
// deleteCache drops an entry and its bytes, reporting whether it went fully. The blob goes
// first, so a failed unlink leaves the row for the next sweep instead of orphaning bytes.
func (h *Handler) deleteCache(db *bolthold.Store, cache *Cache) bool {
func (h *Handler) deleteCache(db *cacheDB, cache *Cache) bool {
storage := h.storage
if h.s3 != nil && !cache.Complete {
storage = h.s3.Storage // an upload not yet committed has nothing in the bucket
@@ -1287,7 +1436,7 @@ func (h *Handler) deleteCache(db *bolthold.Store, cache *Cache) bool {
h.logger.Warnf("remove cache blob: %v", err)
return false
}
if err := db.Delete(cache.ID, cache); err != nil {
if err := db.update(func(bucket *bbolt.Bucket) error { return db.delete(bucket, cache.ID) }); err != nil {
h.logger.Warnf("delete cache: %v", err)
return false
}
@@ -1312,18 +1461,17 @@ func (h *Handler) responseJSON(w http.ResponseWriter, r *http.Request, code int,
_, _ = w.Write(data)
}
func parseContentRange(s string) (int64, int64, error) {
func parseContentRange(s string) (int64, error) {
// support the format like "bytes 11-22/*" only
s, _, _ = strings.Cut(strings.TrimPrefix(s, "bytes "), "/")
s1, s2, _ := strings.Cut(s, "-")
start, err := strconv.ParseInt(s1, 10, 64)
if err != nil {
return 0, 0, fmt.Errorf("parse %q: %w", s, err)
return 0, fmt.Errorf("parse %q: %w", s, err)
}
stop, err := strconv.ParseInt(s2, 10, 64)
if err != nil {
return 0, 0, fmt.Errorf("parse %q: %w", s, err)
if _, err := strconv.ParseInt(s2, 10, 64); err != nil {
return 0, fmt.Errorf("parse %q: %w", s, err)
}
return start, stop, nil
return start, nil
}

File diff suppressed because it is too large Load Diff

View File

@@ -15,8 +15,6 @@ import (
"strconv"
"strings"
"time"
"github.com/julienschmidt/httprouter"
)
// The cache service v2 API. A client on this version talks twirp to
@@ -44,16 +42,16 @@ const (
twirpUnauthenticated = "unauthenticated"
)
func (h *Handler) registerV2Routes(router *httprouter.Router) {
router.POST(cacheServiceV2Path+"/CreateCacheEntry", h.bearerAuth(h.v2CreateCacheEntry))
router.POST(cacheServiceV2Path+"/FinalizeCacheEntryUpload", h.bearerAuth(h.v2FinalizeCacheEntryUpload))
router.POST(cacheServiceV2Path+"/GetCacheEntryDownloadURL", h.bearerAuth(h.v2GetCacheEntryDownloadURL))
router.PUT(blobPath+"/:id", h.signedAuth(blobUploadPurpose, h.v2UploadBlob))
func (h *Handler) registerV2Routes(router *http.ServeMux) {
router.HandleFunc("POST "+cacheServiceV2Path+"/CreateCacheEntry", h.bearerAuth(h.v2CreateCacheEntry))
router.HandleFunc("POST "+cacheServiceV2Path+"/FinalizeCacheEntryUpload", h.bearerAuth(h.v2FinalizeCacheEntryUpload))
router.HandleFunc("POST "+cacheServiceV2Path+"/GetCacheEntryDownloadURL", h.bearerAuth(h.v2GetCacheEntryDownloadURL))
router.HandleFunc("PUT "+blobPath+"/{id}", h.signedAuth(blobUploadPurpose, h.v2UploadBlob))
}
// An entry that already exists is reported as not ok, which is how the client learns to skip
// the upload.
func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context())
req, err := decodeTwirpRequest[v2CreateRequest](r)
if err != nil {
@@ -102,15 +100,16 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request, _ h
return
}
now := time.Now().Unix()
now := time.Now()
cache := &Cache{
Repo: cred.Repo,
Key: req.Key,
Version: req.Version,
Owner: owner,
Size: -1, // the size is only known at finalize time
CreatedAt: now,
UsedAt: now,
Repo: cred.Repo,
Key: req.Key,
Version: req.Version,
Owner: owner,
Size: -1, // the size is only known at finalize time
CreatedAt: now.Unix(),
CreatedAtNanos: now.Nanosecond(),
UsedAt: now.Unix(),
}
if err := h.insertCache(db, cache); err != nil {
h.twirpError(w, r, twirpInternal, err)
@@ -123,7 +122,7 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request, _ h
})
}
func (h *Handler) v2FinalizeCacheEntryUpload(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
func (h *Handler) v2FinalizeCacheEntryUpload(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context())
req, err := decodeTwirpRequest[v2FinalizeRequest](r)
if err != nil {
@@ -172,7 +171,7 @@ func (h *Handler) v2FinalizeCacheEntryUpload(w http.ResponseWriter, r *http.Requ
})
}
func (h *Handler) v2GetCacheEntryDownloadURL(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
func (h *Handler) v2GetCacheEntryDownloadURL(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context())
req, err := decodeTwirpRequest[v2DownloadRequest](r)
if err != nil {
@@ -207,8 +206,8 @@ func (h *Handler) v2GetCacheEntryDownloadURL(w http.ResponseWriter, r *http.Requ
// The archive arrives over the subset of the Azure blob API the toolkit uses: a small
// cache is a single PUT, a large one is staged as blocks that a final block list puts
// in order.
func (h *Handler) v2UploadBlob(w http.ResponseWriter, r *http.Request, params httprouter.Params) {
id, err := strconv.ParseUint(params.ByName("id"), 10, 64)
func (h *Handler) v2UploadBlob(w http.ResponseWriter, r *http.Request) {
id, err := strconv.ParseUint(r.PathValue("id"), 10, 64)
if err != nil {
h.responseJSON(w, r, http.StatusBadRequest, err)
return

View File

@@ -19,8 +19,6 @@ import (
"github.com/stretchr/testify/require"
)
// v2Call posts a twirp request to the cache service and returns the decoded response.
// Field names are the proto ones, which is what the toolkit's client sends.
func v2Call(t *testing.T, handler *Handler, client *http.Client, method string, request any) map[string]any {
t.Helper()
@@ -37,7 +35,6 @@ func v2Call(t *testing.T, handler *Handler, client *http.Client, method string,
return got
}
// putBlob uploads to a signed URL and returns the status, so a test can assert a refusal.
func putBlob(t *testing.T, url string, content []byte) int {
t.Helper()
@@ -47,8 +44,6 @@ func putBlob(t *testing.T, url string, content []byte) int {
require.NoError(t, err)
defer resp.Body.Close()
if resp.StatusCode == http.StatusCreated {
// The Azure SDK client dereferences this header without checking, so a blob upload that
// omits it panics the caller rather than failing it.
require.NotEmpty(t, resp.Header.Get("x-ms-request-id"))
}
return resp.StatusCode
@@ -67,8 +62,6 @@ func getURL(t *testing.T, url string) []byte {
return body
}
// saveV2 runs the reserve/upload/finalize sequence and returns the finalize response along
// with the upload URL it used.
func saveV2(t *testing.T, handler *Handler, key, version string, content []byte) (finalized map[string]any, uploadURL string) {
t.Helper()
@@ -84,10 +77,8 @@ func saveV2(t *testing.T, handler *Handler, key, version string, content []byte)
}), uploadURL
}
// The whole round trip an actions/cache v2 client makes, plus the guarantees on the signed
// URLs it is handed: unsigned requests are refused, an upload URL cannot be replayed to read
// or to replace a finalized entry.
func TestCacheServiceV2RoundTrip(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
content := []byte("the cached archive")
@@ -98,10 +89,8 @@ func TestCacheServiceV2RoundTrip(t *testing.T) {
require.Equal(t, true, finalized["ok"])
assert.NotEmpty(t, finalized["entry_id"])
// The upload URL outlives the finalize call, so replaying it must not poison the entry,
// and it is an upload URL only: nothing reads a blob back through it.
assert.Equal(t, http.StatusBadRequest, putBlob(t, uploadURL, []byte("poisoned")))
resp, err := http.Get(uploadURL) //nolint:noctx // the URL is the server under test
resp, err := http.Get(uploadURL)
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode)
@@ -114,9 +103,8 @@ func TestCacheServiceV2RoundTrip(t *testing.T) {
assert.Equal(t, content, getURL(t, downloadURL))
}
// A large archive is staged as blocks and only put in order by the final block list, so
// blocks that arrive out of order must still be assembled the way the client asked.
func TestCacheServiceV2BlockUpload(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "blocks", "version": "v1"})
@@ -130,7 +118,6 @@ func TestCacheServiceV2BlockUpload(t *testing.T) {
blocks[blockID] = []byte(part)
order = append(order, blockID)
}
// Upload in an order that is not the block list order.
for _, blockID := range []string{order[2], order[0], order[1]} {
require.Equal(t, http.StatusCreated, putBlob(t, uploadURL+"&comp=block&blockid="+blockID, blocks[blockID]))
}
@@ -165,6 +152,7 @@ func TestCacheServiceV2BlockUpload(t *testing.T) {
}
func TestCacheServiceV2Lookups(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
saved, _ := saveV2(t, handler, "deps-abc", "v1", []byte("x"))
require.Equal(t, true, saved["ok"])
@@ -174,8 +162,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, got["ok"])
})
// The toolkit serialises with the proto field names; the camelCase spellings of the same
// proto JSON mapping are accepted alongside them.
for _, field := range []string{"restore_keys", "restoreKeys"} {
t.Run("restore keys match by prefix, spelled "+field, func(t *testing.T) {
got := v2Call(t, handler, testClient, "GetCacheEntryDownloadURL", map[string]any{
@@ -191,9 +177,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, again["ok"])
})
// A key that is only a prefix of an existing one is a different entry, so the
// reservation check must be exact and not a restore-key prefix match, or the shorter
// key would be reported as existing and silently never saved.
t.Run("a prefix of an existing key is still reserved", func(t *testing.T) {
reserved := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "deps", "version": "v1"})
require.Equal(t, true, reserved["ok"])
@@ -219,7 +202,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, got["ok"])
})
// The size the client declares is what Commit validates the assembled archive against.
t.Run("finalizing with the wrong size is not ok", func(t *testing.T) {
created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "wrong-size", "version": "v1"})
require.Equal(t, http.StatusCreated, putBlob(t, created["signed_upload_url"].(string), []byte("four")))
@@ -230,8 +212,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, got["ok"])
})
// Both API versions are served from one store, so an entry written through v2 is a hit for
// a v1 client asking for the same key and version.
t.Run("a v1 client sees an entry written through v2", func(t *testing.T) {
resp, err := testClient.Get(fmt.Sprintf("%s%s/cache?keys=deps-abc&version=v1", handler.ExternalURL(), apiPath))
require.NoError(t, err)
@@ -244,7 +224,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.NotEmpty(t, got["archiveLocation"])
})
// The cache of one repository must stay invisible to another, as it does for the v1 API.
t.Run("another repository sees nothing", func(t *testing.T) {
handler.RegisterJob("other-runtime-token", JobCredential{Repo: "other/repo"})
otherClient := &http.Client{Transport: &bearerTransport{token: "other-runtime-token"}}
@@ -255,6 +234,7 @@ func TestCacheServiceV2Lookups(t *testing.T) {
}
func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
tests := []struct {
@@ -291,6 +271,7 @@ func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) {
}
func TestCacheServiceV2ConcurrentSaveAndRetries(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
content := []byte("the cached archive")
@@ -321,20 +302,12 @@ func TestCacheServiceV2ConcurrentSaveAndRetries(t *testing.T) {
"another job's upload is not this job's to finalize")
base := handler.ExternalURL() + apiPath
reserveV1, err := json.Marshal(&Request{Key: "v1-made", Version: "v1", Size: int64(len(content))})
require.NoError(t, err)
resp, err := testClient.Post(base+"/caches", "application/json", bytes.NewReader(reserveV1))
require.NoError(t, err)
var madeByV1 struct {
CacheID uint64 `json:"cacheId"`
}
require.NoError(t, json.UnmarshalRead(resp.Body, &madeByV1))
resp.Body.Close()
madeByV1 := reserveCache(t, testClient, base, "v1-made", "v1", int64(len(content)))
upload, err := http.NewRequestWithContext(t.Context(), http.MethodPatch,
fmt.Sprintf("%s/caches/%d", base, madeByV1.CacheID), bytes.NewReader(content))
fmt.Sprintf("%s/caches/%d", base, madeByV1), bytes.NewReader(content))
require.NoError(t, err)
upload.Header.Set("Content-Range", fmt.Sprintf("bytes 0-%d/*", len(content)-1))
resp, err = testClient.Do(upload)
resp, err := testClient.Do(upload)
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)

View File

@@ -4,6 +4,8 @@
package artifactcache
import "time"
type Request struct {
Key string `json:"key" `
Version string `json:"version"`
@@ -11,13 +13,18 @@ type Request struct {
}
type Cache struct {
ID uint64 `json:"id" boltholdKey:"ID"`
Repo string `json:"repo" boltholdIndex:"Repo"`
Key string `json:"key" boltholdIndex:"Key"`
Version string `json:"version" boltholdIndex:"Version"`
Size int64 `json:"cacheSize"`
Owner string `json:"owner"` // hashed bearer, so a job's own retry is not another job's reservation
Complete bool `json:"complete" boltholdIndex:"Complete"`
UsedAt int64 `json:"usedAt" boltholdIndex:"UsedAt"`
CreatedAt int64 `json:"createdAt" boltholdIndex:"CreatedAt"`
ID uint64 `json:"id"`
Repo string `json:"repo"`
Key string `json:"key"`
Version string `json:"version"`
Size int64 `json:"cacheSize"`
Owner string `json:"owner"` // hashed bearer, so a job's own retry is not another job's reservation
Complete bool `json:"complete"`
UsedAt int64 `json:"usedAt"`
CreatedAt int64 `json:"createdAt"`
CreatedAtNanos int `json:"createdAtNanos,omitzero"`
}
func (cache *Cache) createdAtNano() int64 {
return cache.CreatedAt*int64(time.Second) + int64(cache.CreatedAtNanos)
}

View File

@@ -4,6 +4,7 @@
package artifactcache
import (
"cmp"
"crypto/tls"
"errors"
"fmt"
@@ -74,14 +75,7 @@ func (h *Handler) forwardOrNotFound(w http.ResponseWriter, r *http.Request) {
// A protocol switch loses the bearer or the agent, and an untrusted instance needs skipped verification.
func (h *Handler) mustProxy(r *http.Request, cred JobCredential, target *url.URL) bool {
if cred.InsecureTLS && target.Scheme == "https" {
return true
}
scheme := r.Header.Get("X-Forwarded-Proto")
if scheme == "" {
scheme = "http"
}
return scheme != target.Scheme
return cred.InsecureTLS && target.Scheme == "https" || cmp.Or(r.Header.Get("X-Forwarded-Proto"), "http") != target.Scheme
}
func mergeQuery(target, request string) string {

View File

@@ -17,6 +17,7 @@ import (
)
func TestFrontResultsService(t *testing.T) {
t.Parallel()
var gotHost, gotPath, gotProto string
gitea := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotHost, gotPath, gotProto = r.Host, r.URL.Path, r.Header.Get("X-Forwarded-Proto")

View File

@@ -24,6 +24,10 @@ import (
"strconv"
"strings"
"time"
"gitea.com/gitea/runner/act/common"
"golang.org/x/sync/errgroup"
)
// S3Options configures the bucket that stores completed cache entries.
@@ -47,7 +51,7 @@ type cacheStorage interface {
Remove(uint64) error
}
const s3PartSize = 5 << 30 // the largest part S3 accepts
const s3PartSize = 64 << 20
// s3Storage stages uploads on disk and keeps completed entries and their metadata in a bucket.
type s3Storage struct {
@@ -69,6 +73,7 @@ func newS3Storage(staging *Storage, opts S3Options) (*s3Storage, error) {
}
transport, _ := http.DefaultTransport.(*http.Transport)
transport = transport.Clone()
transport.MaxIdleConnsPerHost = 32 // 4 parts each for the 8 jobs a runner commonly runs
transport.ResponseHeaderTimeout = time.Minute // a stalled endpoint must not hold the store forever
pathStyle := !strings.HasSuffix(endpoint.Hostname(), ".amazonaws.com")
if opts.PathStyle != nil {
@@ -209,26 +214,43 @@ func (s *s3Storage) Commit(id uint64, size int64) (int64, error) {
return written, nil
}
// upload always goes in parts, the only way S3 takes an object over 5 GiB.
func (s *s3Storage) upload(ctx context.Context, key string, file io.ReaderAt, size int64) error {
func (s *s3Storage) upload(ctx context.Context, key string, file io.ReaderAt, size int64) (uploadErr error) {
if size <= s3PartSize {
_, err := s.call(ctx, http.MethodPut, key, nil, io.NewSectionReader(file, 0, size), nil)
return err
}
var created struct {
UploadID string `xml:"UploadId"`
}
if _, err := s.call(ctx, http.MethodPost, key, url.Values{"uploads": {""}}, nil, &created); err != nil {
return err
}
defer func() {
if uploadErr != nil && created.UploadID != "" {
abortCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
defer cancel()
_, _ = s.call(abortCtx, http.MethodDelete, key, url.Values{"uploadId": {created.UploadID}}, nil, nil)
}
}()
var completion struct {
XMLName xml.Name `xml:"CompleteMultipartUpload"`
Parts []s3Part `xml:"Part"`
}
for offset := int64(0); offset == 0 || offset < size; offset += s3PartSize {
number, length := len(completion.Parts)+1, min(s3PartSize, size-offset)
query := url.Values{"partNumber": {strconv.Itoa(number)}, "uploadId": {created.UploadID}}
header, err := s.call(ctx, http.MethodPut, key, query, io.NewSectionReader(file, offset, length), nil)
if err != nil {
partSize := max(int64(s3PartSize), (size-1)/10000+1)
completion.Parts = make([]s3Part, (size-1)/partSize+1)
workers, workerCtx := errgroup.WithContext(ctx)
workers.SetLimit(4)
for part := range completion.Parts {
workers.Go(func() (err error) {
defer common.RecoverPanic(&err)
offset := int64(part) * partSize
header, err := s.call(workerCtx, http.MethodPut, key, url.Values{"partNumber": {strconv.Itoa(part + 1)}, "uploadId": {created.UploadID}}, io.NewSectionReader(file, offset, min(partSize, size-offset)), nil)
completion.Parts[part] = s3Part{part + 1, header.Get("ETag")}
return err
}
completion.Parts = append(completion.Parts, s3Part{number, header.Get("ETag")})
})
}
if err := workers.Wait(); err != nil {
return err
}
body, err := xml.Marshal(completion)
if err != nil {

View File

@@ -4,40 +4,135 @@
package artifactcache
import (
"encoding/xml"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3mem"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) {
backend := s3mem.New()
require.NoError(t, backend.CreateBucket("cache"))
server := httptest.NewServer(gofakes3.New(backend).Server())
t.Parallel()
var objects sync.Map
var uploads atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
key := strings.TrimPrefix(request.URL.Path, "/cache/")
switch {
case request.Method == http.MethodPut:
if strings.Contains(key, "/blobs/") {
uploads.Add(1)
}
body, err := io.ReadAll(request.Body)
assert.NoError(t, err)
objects.Store(key, body)
case request.URL.Query().Has("list-type"):
_, _ = io.WriteString(writer, "<ListBucketResult>")
objects.Range(func(stored, _ any) bool {
_, _ = io.WriteString(writer, "<Contents><Key>"+stored.(string)+"</Key></Contents>")
return true
})
_, _ = io.WriteString(writer, "</ListBucketResult>")
default:
if body, ok := objects.Load(key); ok {
_, _ = writer.Write(body.([]byte))
} else {
writer.WriteHeader(http.StatusNotFound)
}
}
}))
t.Cleanup(server.Close)
opts := Options{S3: &S3Options{Endpoint: server.URL, Bucket: "cache", Prefix: "runners", AccessKey: "key", SecretKey: "secret"}}
saver, restorer := startTestHandler(t, opts), startTestHandler(t, opts)
entry := map[string]any{"key": "deps", "version": "v1"}
started := time.Now()
finalized, _ := saveV2(t, saver, "deps", "v1", []byte("archive"))
require.Equal(t, true, finalized["ok"])
assert.EqualValues(t, 1, uploads.Load())
id, err := strconv.ParseUint(finalized["entry_id"].(string), 10, 64)
require.NoError(t, err)
cache, err := saver.s3.fetch(t.Context(), id)
require.NoError(t, err)
assert.WithinRange(t, time.Unix(cache.CreatedAt, int64(cache.CreatedAtNanos)), started, time.Now())
require.NoError(t, restorer.importMetadata(t.Context()))
found := v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", entry)
require.Equal(t, true, found["ok"])
assert.Equal(t, []byte("archive"), getURL(t, found["signed_download_url"].(string)))
require.NoError(t, backend.ForceDeleteBucket("cache"))
require.NoError(t, backend.CreateBucket("cache"))
for _, cache := range []*Cache{
{ID: 2, Repo: testRepo, Key: "deps-old", Version: "v1", Complete: true, CreatedAt: cache.CreatedAt},
{ID: 1, Repo: testRepo, Key: "deps-new", Version: "v1", Complete: true, CreatedAt: cache.CreatedAt, CreatedAtNanos: 1},
} {
require.NoError(t, saver.s3.publish(cache))
objects.Store(saver.s3.blobKey(cache.ID), []byte(cache.Key))
}
require.NoError(t, restorer.importMetadata(t.Context()))
found = v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-", "version": "v1"})
require.Equal(t, true, found["ok"])
assert.Equal(t, "deps-new", found["matched_key"])
assert.Equal(t, []byte("deps-new"), getURL(t, found["signed_download_url"].(string)))
objects.Clear()
require.NoError(t, restorer.importMetadata(t.Context()))
assert.Equal(t, true, v2Call(t, restorer, testClient, "CreateCacheEntry", entry)["ok"])
}
type s3TestTransport func(*http.Request) (*http.Response, error)
func (transport s3TestTransport) RoundTrip(req *http.Request) (*http.Response, error) {
return transport(req)
}
func TestS3StorageUploadsLargeObjectsInPartsAndAbortsOnFailure(t *testing.T) {
t.Parallel()
for _, failingPart := range []string{"", "2"} {
var parts, aborts atomic.Int32
var completion struct {
Parts []s3Part `xml:"Part"`
}
storage := &s3Storage{bucketURL: httptest.NewRequest(http.MethodGet, "http://cache.local/bucket", nil).URL}
storage.client = &http.Client{Transport: s3TestTransport(func(req *http.Request) (*http.Response, error) {
query, response := req.URL.Query(), &http.Response{StatusCode: http.StatusOK, Header: make(http.Header), Body: http.NoBody}
switch {
case query.Has("uploads"):
response.Body = io.NopCloser(strings.NewReader("<InitiateMultipartUploadResult><UploadId>upload</UploadId></InitiateMultipartUploadResult>"))
case req.Method == http.MethodPut:
parts.Add(1)
response.Header.Set("ETag", "etag-"+query.Get("partNumber"))
if query.Get("partNumber") == failingPart {
response.StatusCode, response.Body = http.StatusBadRequest, io.NopCloser(strings.NewReader("<Error><Code>InvalidPart</Code></Error>"))
}
case req.Method == http.MethodPost:
require.NoError(t, xml.NewDecoder(req.Body).Decode(&completion))
response.Body = io.NopCloser(strings.NewReader("<CompleteMultipartUploadResult/>"))
case req.Method == http.MethodDelete:
aborts.Add(1)
}
return response, nil
})}
err := storage.upload(t.Context(), "blobs/1", strings.NewReader(""), 2*int64(s3PartSize)+1)
assert.EqualValues(t, 3, parts.Load())
if failingPart == "" {
require.NoError(t, err)
assert.Equal(t, []s3Part{{1, "etag-1"}, {2, "etag-2"}, {3, "etag-3"}}, completion.Parts)
assert.Zero(t, aborts.Load())
} else {
require.ErrorContains(t, err, "InvalidPart")
assert.EqualValues(t, 1, aborts.Load())
}
}
}
func TestS3StorageSignsLikeAWS(t *testing.T) {
t.Parallel()
storage := &s3Storage{opts: S3Options{Region: "us-east-1", AccessKey: "AKIAIOSFODNN7EXAMPLE", SecretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"}}
req := httptest.NewRequest(http.MethodGet, "https://examplebucket.s3.amazonaws.com/?max-keys=2&prefix=J", nil)
req.Header.Set("X-Amz-Content-Sha256", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")

View File

@@ -32,13 +32,11 @@ func NewStorage(rootDir string) (*Storage, error) {
}
func (s *Storage) Exist(id uint64) (bool, error) {
name := s.filename(id)
if _, err := os.Stat(name); os.IsNotExist(err) {
_, err := os.Stat(s.filename(id))
if os.IsNotExist(err) {
return false, nil
} else if err != nil {
return false, err
}
return true, nil
return err == nil, err
}
func (s *Storage) Write(id uint64, offset int64, reader io.Reader) error {
@@ -92,9 +90,7 @@ func (s *Storage) OrderBlocks(id uint64, blockIDs []string) error {
}
func (s *Storage) Commit(id uint64, size int64) (int64, error) {
defer func() {
_ = os.RemoveAll(s.tempDir(id))
}()
defer func() { _ = os.RemoveAll(s.tempDir(id)) }()
name := s.filename(id)
tempNames, err := s.tempNames(id)
@@ -153,8 +149,7 @@ func assemble(name string, tempNames []string) (int64, error) {
}
func (s *Storage) Serve(w http.ResponseWriter, r *http.Request, id uint64) {
name := s.filename(id)
http.ServeFile(w, r, name)
http.ServeFile(w, r, s.filename(id))
}
// Remove deletes an entry's blob and any staged parts. It reports failure so the caller can

View File

@@ -13,14 +13,13 @@ import (
"io/fs"
"net"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"time"
"gitea.com/gitea/runner/act/common"
"github.com/julienschmidt/httprouter"
)
type FileContainerResourceURL struct {
@@ -67,39 +66,44 @@ func writeJSON(w http.ResponseWriter, value any) {
}
}
func uploads(router *httprouter.Router, baseDir string) {
router.POST("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
runID := params.ByName("runId")
func uploads(router *http.ServeMux, baseDir string) {
router.HandleFunc("POST /_apis/pipelines/workflows/{runId}/artifacts", func(w http.ResponseWriter, req *http.Request) {
writeJSON(w, FileContainerResourceURL{
FileContainerResourceURL: fmt.Sprintf("http://%s/upload/%s", req.Host, runID),
FileContainerResourceURL: fmt.Sprintf("http://%s/upload/%s", req.Host, req.PathValue("runId")),
})
})
router.PUT("/upload/:runId", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
router.HandleFunc("PUT /upload/{runId}", func(w http.ResponseWriter, req *http.Request) {
itemPath := req.URL.Query().Get("itemPath")
runID := params.ByName("runId")
if req.Header.Get("Content-Encoding") == "gzip" {
itemPath += gzipExtension
}
safeRunPath := safeResolve(baseDir, runID)
safePath := safeResolve(safeRunPath, itemPath)
safePath := safeResolve(safeResolve(baseDir, req.PathValue("runId")), itemPath)
if err := os.MkdirAll(filepath.Dir(safePath), os.ModePerm); err != nil {
panic(err)
}
flags := os.O_CREATE | os.O_WRONLY | os.O_TRUNC
appendUpload := req.Header.Get("Content-Range")
if appendUpload != "" && !strings.HasPrefix(appendUpload, "bytes 0-") {
flags = os.O_CREATE | os.O_WRONLY | os.O_APPEND
var start int64
if contentRange := req.Header.Get("Content-Range"); contentRange != "" {
if _, err := fmt.Sscanf(contentRange, "bytes %d-", &start); err != nil || start < 0 {
http.Error(w, "invalid content range", http.StatusBadRequest)
return
}
}
flags := os.O_CREATE | os.O_WRONLY
if start == 0 {
flags |= os.O_TRUNC
}
file, err := os.OpenFile(safePath, flags, 0o644)
if err != nil {
panic(err)
}
defer file.Close()
if _, err := file.Seek(start, io.SeekStart); err != nil {
panic(err)
}
if req.Body == nil {
panic(errors.New("no body given"))
}
@@ -114,21 +118,19 @@ func uploads(router *httprouter.Router, baseDir string) {
})
})
router.PATCH("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
router.HandleFunc("PATCH /_apis/pipelines/workflows/{runId}/artifacts", func(w http.ResponseWriter, _req *http.Request) {
writeJSON(w, ResponseMessage{
Message: "success",
})
})
}
func downloads(router *httprouter.Router, baseDir string) {
router.GET("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
runID := params.ByName("runId")
func downloads(router *http.ServeMux, baseDir string) {
router.HandleFunc("GET /_apis/pipelines/workflows/{runId}/artifacts", func(w http.ResponseWriter, req *http.Request) {
runID := req.PathValue("runId")
safePath := safeResolve(baseDir, runID)
entries, err := os.ReadDir(safePath)
if err != nil {
entries, err := os.ReadDir(safeResolve(baseDir, runID))
if err != nil && !os.IsNotExist(err) {
panic(err)
}
@@ -146,13 +148,16 @@ func downloads(router *httprouter.Router, baseDir string) {
})
})
router.GET("/download/:container", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
container := params.ByName("container")
router.HandleFunc("GET /download/{container}", func(w http.ResponseWriter, req *http.Request) {
container := req.PathValue("container")
itemPath := req.URL.Query().Get("itemPath")
safePath := safeResolve(baseDir, filepath.Join(container, itemPath))
var files []ContainerItem
err := filepath.WalkDir(safePath, func(path string, entry fs.DirEntry, err error) error {
if err != nil {
return err
}
if !entry.IsDir() {
rel, err := filepath.Rel(safePath, path)
if err != nil {
@@ -161,20 +166,16 @@ func downloads(router *httprouter.Router, baseDir string) {
// if it was upload as gzip
rel = strings.TrimSuffix(rel, gzipExtension)
path := filepath.Join(itemPath, rel)
rel = filepath.ToSlash(rel)
path = filepath.ToSlash(path)
files = append(files, ContainerItem{
Path: path,
Path: filepath.ToSlash(filepath.Join(itemPath, rel)),
ItemType: "file",
ContentLocation: fmt.Sprintf("http://%s/artifact/%s/%s/%s", req.Host, container, itemPath, rel),
ContentLocation: (&url.URL{Scheme: "http", Host: req.Host, Path: "/artifact/" + container + "/" + itemPath + "/" + filepath.ToSlash(rel)}).String(),
})
}
return nil
})
if err != nil {
if err != nil && !os.IsNotExist(err) {
panic(err)
}
@@ -183,10 +184,8 @@ func downloads(router *httprouter.Router, baseDir string) {
})
})
router.GET("/artifact/*path", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
path := params.ByName("path")[1:]
safePath := safeResolve(baseDir, path)
router.HandleFunc("GET /artifact/{path...}", func(w http.ResponseWriter, req *http.Request) {
safePath := safeResolve(baseDir, req.PathValue("path"))
file, err := os.Open(safePath)
if err != nil {
@@ -214,7 +213,7 @@ func Serve(ctx context.Context, artifactPath, addr, port string) context.CancelF
return cancel
}
router := httprouter.New()
router := http.NewServeMux()
logger.Debugf("Artifacts base path '%s'", artifactPath)
uploads(router, artifactPath)

View File

@@ -19,14 +19,13 @@ import (
"testing"
"time"
"github.com/julienschmidt/httprouter"
"github.com/stretchr/testify/require"
)
func TestArtifactFlow(t *testing.T) {
artifactPath := t.TempDir()
router := httprouter.New()
router := http.NewServeMux()
uploads(router, artifactPath)
downloads(router, artifactPath)
server := httptest.NewServer(router)
@@ -49,8 +48,17 @@ func TestArtifactFlow(t *testing.T) {
return resp.StatusCode, data
}
t.Run("no-artifacts", func(t *testing.T) {
status, data := request(t, http.MethodGet, baseURL+"/_apis/pipelines/workflows/0/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data))
var list NamedFileContainerResourceURLResponse
require.NoError(t, json.Unmarshal(data, &list))
require.Zero(t, list.Count)
require.Empty(t, list.Value)
})
t.Run("upload-and-download", func(t *testing.T) {
const runID, item, content = "1", "my-artifact/data.txt", "hello artifact\n"
const runID, item, content = "1", "my-artifact/data#1%25.txt", "hello artifact\n"
status, data := request(t, http.MethodPost, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data))
@@ -82,13 +90,13 @@ func TestArtifactFlow(t *testing.T) {
require.NoError(t, json.Unmarshal(data, &items))
require.Len(t, items.Value, 1)
require.Equal(t, "file", items.Value[0].ItemType)
require.Equal(t, "my-artifact/data.txt", items.Value[0].Path)
require.Equal(t, item, items.Value[0].Path)
status, data = request(t, http.MethodGet, items.Value[0].ContentLocation, nil, nil)
require.Equal(t, http.StatusOK, status)
require.Equal(t, content, string(data))
stored, err := os.ReadFile(filepath.Join(artifactPath, runID, "my-artifact", "data.txt"))
stored, err := os.ReadFile(filepath.Join(artifactPath, runID, filepath.FromSlash(item)))
require.NoError(t, err)
require.Equal(t, content, string(stored))
})
@@ -96,16 +104,25 @@ func TestArtifactFlow(t *testing.T) {
t.Run("content-range", func(t *testing.T) {
const rawURL = "/upload/4?itemPath=chunks.txt"
status, data := request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("first"),
http.Header{"Content-Range": []string{"bytes 0-4/11"}})
http.Header{"Content-Range": []string{"bytes 0-4/12"}})
require.Equal(t, http.StatusOK, status, string(data))
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("-second"),
http.Header{"Content-Range": []string{"bytes 5-11/11"}})
require.Equal(t, http.StatusOK, status, string(data))
for range 2 {
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("-second"),
http.Header{"Content-Range": []string{"bytes 5-11/12"}})
require.Equal(t, http.StatusOK, status, string(data))
}
stored, err := os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt"))
require.NoError(t, err)
require.Equal(t, "first-second", string(stored))
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader(""),
http.Header{"Content-Range": []string{"bytes 0--1/0"}})
require.Equal(t, http.StatusOK, status, string(data))
stored, err = os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt"))
require.NoError(t, err)
require.Empty(t, stored)
})
t.Run("gzip-roundtrip", func(t *testing.T) {

View File

@@ -14,13 +14,8 @@ const dryrunContextKeyVal = dryrunContextKey("dryrun")
// Dryrun returns true if the current context is dryrun
func Dryrun(ctx context.Context) bool {
val := ctx.Value(dryrunContextKeyVal)
if val != nil {
if dryrun, ok := val.(bool); ok {
return dryrun
}
}
return false
dryrun, _ := ctx.Value(dryrunContextKeyVal).(bool)
return dryrun
}
// WithDryrun adds a value to the context for dryrun

View File

@@ -21,8 +21,7 @@ type Conditional func(ctx context.Context) bool
// NewInfoExecutor is an executor that logs messages
func NewInfoExecutor(format string, args ...any) Executor {
return func(ctx context.Context) error {
logger := Logger(ctx)
logger.Infof(format, args...)
Logger(ctx).Infof(format, args...)
return nil
}
}
@@ -30,8 +29,7 @@ func NewInfoExecutor(format string, args ...any) Executor {
// NewDebugExecutor is an executor that logs messages
func NewDebugExecutor(format string, args ...any) Executor {
return func(ctx context.Context) error {
logger := Logger(ctx)
logger.Debugf(format, args...)
Logger(ctx).Debugf(format, args...)
return nil
}
}
@@ -39,9 +37,7 @@ func NewDebugExecutor(format string, args ...any) Executor {
// NewPipelineExecutor creates a new executor from a series of other executors
func NewPipelineExecutor(executors ...Executor) Executor {
if len(executors) == 0 {
return func(ctx context.Context) error {
return nil
}
return NewErrorExecutor(nil)
}
var rtn Executor
for _, executor := range executors {
@@ -61,6 +57,13 @@ func NewErrorExecutor(err error) Executor {
}
}
func RecoverPanic(err *error) { // defer directly in workers to keep stacks in host logs
if recovered := recover(); recovered != nil {
logging.Errorf("panic: %v\n%s", recovered, debug.Stack())
*err = fmt.Errorf("panic: %v", recovered)
}
}
// NewParallelExecutor creates a new executor from a parallel of other executors
func NewParallelExecutor(parallel int, executors ...Executor) Executor {
if len(executors) == 0 {
@@ -80,8 +83,8 @@ func NewParallelExecutor(parallel int, executors ...Executor) Executor {
logging.Infof("NewParallelExecutor: Creating %d workers for %d executors", parallel, len(executors))
for i := 0; i < parallel; i++ {
go func(workerID int, work <-chan Executor, errs chan<- error) {
for workerID := range parallel {
go func() {
logging.Debugf("Worker %d started", workerID)
taskCount := 0
for executor := range work {
@@ -91,17 +94,12 @@ func NewParallelExecutor(parallel int, executors ...Executor) Executor {
// goroutine which would leave the runner process hung.
// https://gitea.com/gitea/runner/issues/371
errs <- func() (err error) {
defer func() {
if r := recover(); r != nil {
logging.Errorf("panic in executor: %v\n%s", r, debug.Stack())
err = fmt.Errorf("panic: %v", r)
}
}()
defer RecoverPanic(&err)
return executor(ctx)
}()
}
logging.Debugf("Worker %d finished (%d tasks executed)", workerID, taskCount)
}(i, work, errs)
}()
}
for i := range executors {
@@ -150,12 +148,7 @@ func (e Executor) If(conditional Conditional) Executor {
// IfNot only runs this executor if conditional is true
func (e Executor) IfNot(conditional Conditional) Executor {
return func(ctx context.Context) error {
if !conditional(ctx) {
return e(ctx)
}
return nil
}
return e.If(func(ctx context.Context) bool { return !conditional(ctx) })
}
// IfBool only runs this executor if conditional is true
@@ -171,7 +164,10 @@ func (e Executor) Finally(finally Executor) Executor {
err := e(ctx)
err2 := finally(ctx)
if err2 != nil {
return fmt.Errorf("error occurred running finally: %v (original error: %v)", err2, err)
if err != nil {
return fmt.Errorf("error occurred running finally: %w (original error: %w)", err2, err)
}
return fmt.Errorf("error occurred running finally: %w (original error: %v)", err2, err)
}
return err
}

View File

@@ -7,8 +7,7 @@ package common
import (
"context"
"errors"
"reflect"
"strings"
"fmt"
"sync"
"sync/atomic"
"testing"
@@ -18,42 +17,25 @@ import (
)
func TestNewWorkflow(t *testing.T) {
assert := assert.New(t)
ctx := t.Context()
require.NoError(t, NewPipelineExecutor()(ctx))
require.EqualError(t, NewErrorExecutor(errors.New("test error"))(ctx), "test error")
// empty
emptyWorkflow := NewPipelineExecutor()
assert.NoError(emptyWorkflow(ctx)) //nolint:testifylint // pre-existing issue from nektos/act
// error case
errorWorkflow := NewErrorExecutor(errors.New("test error"))
assert.Error(errorWorkflow(ctx)) //nolint:testifylint // pre-existing issue from nektos/act
// multiple success case
runcount := 0
successWorkflow := NewPipelineExecutor(
func(ctx context.Context) error {
runcount++
return nil
},
func(ctx context.Context) error {
runcount++
return nil
})
assert.NoError(successWorkflow(ctx)) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(2, runcount)
increment := func(_ctx context.Context) error {
runcount++
return nil
}
require.NoError(t, NewPipelineExecutor(increment, increment)(ctx))
assert.Equal(t, 2, runcount)
}
// concurrencyProbe returns an executor recording the peak number of concurrent copies. Copies
// block until wantActive are in flight so the peak is exact without sleeping, and later copies
// find the gate already open so the last one still finishes with no partner left.
func concurrencyProbe(wantActive int32) (exec Executor, count, maxActive *atomic.Int32) {
var counted, active, peak atomic.Int32
var once sync.Once
reached := make(chan struct{})
return func(ctx context.Context) error {
return func(_ctx context.Context) error {
counted.Add(1)
running := active.Add(1)
for {
@@ -72,18 +54,14 @@ func concurrencyProbe(wantActive int32) (exec Executor, count, maxActive *atomic
}
func TestNewParallelExecutor(t *testing.T) {
ctx := t.Context()
exec, count, maxActive := concurrencyProbe(2)
require.NoError(t, NewParallelExecutor(2, exec, exec, exec)(ctx))
assert.Equal(t, int32(3), count.Load(), "should run all 3 executors")
assert.Equal(t, int32(2), maxActive.Load(), "should run at most 2 executors in parallel")
// parallelism below 1 falls back to a single worker
exec, count, maxActive = concurrencyProbe(1)
require.NoError(t, NewParallelExecutor(0, exec, exec, exec)(ctx))
assert.Equal(t, int32(3), count.Load(), "should run all 3 executors")
assert.Equal(t, int32(1), maxActive.Load(), "should run at most 1 executor in parallel")
for name, parallel := range map[string]int{"parallel": 2, "below minimum": 0} {
t.Run(name, func(t *testing.T) {
exec, count, maxActive := concurrencyProbe(int32(max(1, parallel)))
require.NoError(t, NewParallelExecutor(parallel, exec, exec, exec)(t.Context()))
assert.Equal(t, int32(3), count.Load())
assert.Equal(t, int32(max(1, parallel)), maxActive.Load())
})
}
}
func TestNewParallelExecutorEmpty(t *testing.T) {
@@ -99,66 +77,49 @@ func TestNewParallelExecutorEmpty(t *testing.T) {
assert.ErrorIs(err, context.Canceled)
}
func TestNewParallelExecutorFailed(t *testing.T) {
assert := assert.New(t)
ctx, cancel := context.WithCancel(t.Context())
cancel()
count := 0
errorWorkflow := NewPipelineExecutor(func(ctx context.Context) error {
count++
return errors.New("fake error")
})
err := NewParallelExecutor(1, errorWorkflow)(ctx)
assert.Equal(1, count)
assert.ErrorIs(context.Canceled, err)
}
func TestNewParallelExecutorCanceled(t *testing.T) {
assert := assert.New(t)
ctx, cancel := context.WithCancel(t.Context())
cancel()
errExpected := errors.New("fake error")
var count atomic.Int32
successWorkflow := NewPipelineExecutor(func(ctx context.Context) error {
count.Add(1)
return nil
})
errorWorkflow := NewPipelineExecutor(func(ctx context.Context) error {
count.Add(1)
return errExpected
})
err := NewParallelExecutor(3, errorWorkflow, successWorkflow, successWorkflow)(ctx)
assert.Equal(int32(3), count.Load())
assert.Error(errExpected, err) //nolint:testifylint // pre-existing issue from nektos/act
}
func TestNewParallelExecutorRunsRemainingAfterFailure(t *testing.T) {
var successCount atomic.Int32
executors := make([]Executor, 5)
for i := range executors {
executors[i] = func(ctx context.Context) error {
if i == 2 {
return errors.New("fake error")
func TestNewParallelExecutorRunsAllTasksAfterFailureOrCancellation(t *testing.T) {
for name, tc := range map[string]struct {
canceled bool
parallel, tasks int
}{
"failure": {false, 2, 5},
"canceled": {true, 2, 5},
"canceled single task": {true, 1, 1},
} {
t.Run(name, func(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
if tc.canceled {
cancel()
}
var calls atomic.Int32
executors := make([]Executor, tc.tasks)
for i := range executors {
executors[i] = func(_ctx context.Context) error {
calls.Add(1)
if i == tc.tasks/2 {
return errors.New("fake error")
}
return nil
}
}
successCount.Add(1)
return nil
}
}
require.Error(t, NewParallelExecutor(2, executors...)(t.Context()))
assert.Equal(t, int32(4), successCount.Load(), "a failing executor must not stop the others")
err := NewParallelExecutor(tc.parallel, executors...)(ctx)
if tc.canceled {
require.ErrorIs(t, err, context.Canceled)
} else {
require.EqualError(t, err, "fake error")
}
assert.Equal(t, int32(tc.tasks), calls.Load())
})
}
}
func TestExecutorConditionalsAndFinally(t *testing.T) {
ctx := t.Context()
var calls []string
record := func(name string) Executor {
return func(ctx context.Context) error {
return func(_ctx context.Context) error {
calls = append(calls, name)
return nil
}
@@ -170,19 +131,17 @@ func TestExecutorConditionalsAndFinally(t *testing.T) {
require.NoError(t, record("if-bool").IfBool(true)(ctx))
require.NoError(t, record("main").Finally(record("finally"))(ctx))
want := []string{"if-true", "if-not", "if-bool", "main", "finally"}
if !reflect.DeepEqual(calls, want) {
t.Fatalf("calls = %v, want %v", calls, want)
}
assert.Equal(t, []string{"if-true", "if-not", "if-bool", "main", "finally"}, calls)
}
func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) {
mainErr := errors.New("main failed")
finalErr := errors.New("cleanup failed")
err := NewErrorExecutor(mainErr).Finally(NewErrorExecutor(finalErr))(t.Context())
require.Error(t, err)
if !strings.Contains(err.Error(), "cleanup failed") || !strings.Contains(err.Error(), "main failed") {
t.Fatalf("finally error = %q, want both cleanup and original error", err)
for _, mainErr := range []error{context.Canceled, nil} {
err := NewErrorExecutor(mainErr).Finally(NewErrorExecutor(finalErr))(t.Context())
require.EqualError(t, err, fmt.Sprintf("error occurred running finally: cleanup failed (original error: %v)", mainErr))
require.ErrorIs(t, err, finalErr)
if mainErr != nil {
require.ErrorIs(t, err, mainErr)
}
}
}

View File

@@ -0,0 +1,349 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package git
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"encoding/json"
"errors"
"io"
"math/rand/v2"
"net"
"net/http"
"net/url"
"os"
"path/filepath"
"reflect"
"runtime"
"strconv"
"strings"
"sync"
"time"
"gitea.com/gitea/runner/act/common"
)
var actionFetches sync.Map
const treeSizes = ".sizes" // beside the tree rather than in it, which is copied into job containers as it is
type treeFile struct {
Name []byte // not a string, which JSON would turn into UTF-8, as git allows any bytes but NUL
Size int64
}
type actionFetch struct {
done chan struct{}
dir, sha string
err error
retry bool // the leader was cancelled or panicked, so followers fetch themselves
}
var archiveRetryDelay = func(retryAfter string) time.Duration { // as actions/runner backs off
if seconds, err := strconv.Atoi(retryAfter); err == nil {
return min(max(time.Duration(seconds)*time.Second, 10*time.Second), 10*time.Minute)
}
return time.Duration(10+rand.IntN(21)) * time.Second
}
func FetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, string, error) { // input.Dir is one clone URL and ref's, holding a directory per commit
if !reflect.ValueOf(input).Comparable() {
return fetchAction(ctx, input)
}
for ctx.Err() == nil {
call := &actionFetch{done: make(chan struct{}), retry: true}
previous, loaded := actionFetches.LoadOrStore(input, call)
if !loaded {
func() {
defer func() { actionFetches.Delete(input); close(call.done) }()
call.dir, call.sha, call.err = fetchAction(ctx, input)
call.retry = ctx.Err() != nil
}()
if call.retry {
continue
}
return call.dir, call.sha, call.err
}
call, _ = previous.(*actionFetch)
select {
case <-ctx.Done():
case <-call.done:
if !call.retry && ctx.Err() == nil {
return call.dir, call.sha, call.err
}
}
}
return "", "", ctx.Err()
}
func fetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, string, error) {
unlock, err := AcquireCloneLock(ctx, input.Dir)
if err != nil {
return "", "", err
}
defer unlock()
if err := os.MkdirAll(input.Dir, 0o755); err != nil {
return "", "", err
}
tree := func(sha string) string { return filepath.Join(input.Dir, sha) }
record := filepath.Join(input.Dir, "ref")
data, _ := os.ReadFile(record)
sha, etag, _ := strings.Cut(string(data), "\n")
switch {
case !isFullObjectID(sha):
sha, etag = "", ""
case input.OfflineMode && exists(tree(sha)) && !exists(tree(sha)+treeSizes):
return tree(sha), sha, nil // from an older runner, with no sizes to check, so offline mode uses it as before
case !treeIntact(input.Dir, sha):
sha, etag = "", ""
case input.OfflineMode || isFullObjectID(input.Ref):
return tree(sha), sha, nil
}
newSha, newEtag, err := downloadArchive(ctx, input, sha, etag)
if err != nil && ctx.Err() == nil {
common.Logger(ctx).Debugf("Cloning %s@%s: %v", input.URL, input.Ref, err)
newEtag = ""
newSha, err = exportClone(ctx, input)
}
if err != nil {
return "", "", err
}
if newSha != sha || newEtag != etag {
if err := os.WriteFile(record, []byte(newSha+"\n"+newEtag), 0o600); err != nil {
return "", "", err
}
}
return tree(newSha), newSha, nil
}
func downloadArchive(ctx context.Context, input NewGitCloneExecutorInput, sha, etag string) (string, string, error) {
rewritten, err := gitOutput(ctx, "", "ls-remote", "--get-url", input.URL)
if err != nil {
return "", "", err
}
source, err := url.Parse(rewritten)
if err != nil || source.Scheme != "http" && source.Scheme != "https" || hexRefRegex.MatchString(input.Ref) && !isFullObjectID(input.Ref) { // only git tells a short commit from a branch
return "", "", errors.New("no archive")
}
archive := source.JoinPath("archive", input.Ref+".tar.gz")
if source.Host == "github.com" {
source.Host = "codeload.github.com"
archive = source.JoinPath("tar.gz", input.Ref)
}
client := &http.Client{Transport: input.Transport, Timeout: 5 * time.Minute, CheckRedirect: func(request *http.Request, via []*http.Request) error {
if len(via) >= 10 {
return errors.New("stopped after 10 redirects")
}
if request.URL.Host != via[0].URL.Host || request.URL.Scheme != via[0].URL.Scheme { // the token stays on its origin
request.Header.Del("Authorization")
}
return nil
}}
for attempt := 1; ; attempt++ {
newSha, newEtag, retryAfter, err := downloadArchiveOnce(ctx, client, archive.String(), input.Token, sha, etag, input.Dir)
if retryAfter == 0 || attempt == 3 || ctx.Err() != nil {
return newSha, newEtag, err
}
common.Logger(ctx).Warnf("Failed to download action '%s'. Error: %v. Back off %s before retry.", archive.Redacted(), err, retryAfter)
select {
case <-ctx.Done():
return "", "", ctx.Err()
case <-time.After(retryAfter):
}
}
}
func downloadArchiveOnce(ctx context.Context, client *http.Client, archive, token, sha, etag, dir string) (string, string, time.Duration, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, archive, nil)
if err != nil {
return "", "", 0, err
}
if etag != "" {
request.Header.Set("If-None-Match", etag)
}
if token != "" {
request.SetBasicAuth("token", token)
}
response, err := client.Do(request)
if err != nil {
var opErr *net.OpError
if errors.As(err, &opErr) && opErr.Op == "dial" {
return "", "", 0, err
}
return "", "", archiveRetryDelay(""), err
}
defer response.Body.Close()
switch status := response.StatusCode; {
case status == http.StatusNotModified && sha != "":
return sha, etag, 0, nil
case status == http.StatusOK:
newEtag := response.Header.Get("ETag")
gz, err := gzip.NewReader(response.Body)
if err == nil {
sha, err = extractTree(ctx, gz, dir, !strings.HasSuffix(newEtag, `-false"`)) // Gitea's mark of an unprefixed archive
}
var netErr net.Error
if errors.Is(err, io.ErrUnexpectedEOF) || errors.As(err, &netErr) {
return "", "", archiveRetryDelay(""), err
}
return sha, newEtag, 0, err
case status == http.StatusRequestTimeout || status == http.StatusTooManyRequests || status >= http.StatusInternalServerError:
return "", "", archiveRetryDelay(response.Header.Get("Retry-After")), errors.New(response.Status)
default:
return "", "", 0, errors.New(response.Status)
}
}
func exportClone(ctx context.Context, input NewGitCloneExecutorInput) (string, error) {
clone := input
clone.Dir = filepath.Join(input.Dir, "git")
sha, _, err := resolveClone(ctx, clone)
if err != nil || treeIntact(input.Dir, sha) {
return sha, err
}
archive, err := runGit(ctx, clone.Dir, nil, "archive", "--format=tar", sha)
if err != nil {
return "", err
}
_, err = extractTree(ctx, bytes.NewReader(archive), input.Dir, false)
return sha, err
}
func extractTree(ctx context.Context, archive io.Reader, dir string, strip bool) (string, error) { // into dir/<commit of the pax header>, strip dropping the directory wrapping the files
temp, err := os.MkdirTemp(dir, "tmp-")
if err != nil {
return "", err
}
defer os.RemoveAll(temp)
root, err := os.OpenRoot(temp)
if err != nil {
return "", err
}
defer root.Close()
sha := ""
sizes := map[string]int64{}
reader := tar.NewReader(archive)
for {
header, err := reader.Next()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return "", err
}
if header.Typeflag == tar.TypeXGlobalHeader {
if sha = header.PAXRecords["comment"]; isFullObjectID(sha) && treeIntact(dir, sha) {
return sha, nil
}
continue
}
name := header.Name
if strip {
_, name, _ = strings.Cut(name, "/")
}
if name = strings.TrimSuffix(name, "/"); name == "" {
continue
}
switch header.Typeflag {
case tar.TypeDir:
err = root.MkdirAll(name, 0o755)
case tar.TypeReg:
sizes[name], err = writeTreeFile(root, name, header.Mode, reader)
case tar.TypeSymlink:
if err = root.Symlink(header.Linkname, name); err != nil && runtime.GOOS == "windows" {
sizes[name], err = writeTreeFile(root, name, 0, strings.NewReader(header.Linkname))
}
}
if err != nil {
return "", err
}
}
if !isFullObjectID(sha) {
return "", errors.New("the archive names no commit")
}
if err := root.Close(); err != nil {
return "", err
}
return sha, installTree(ctx, dir, sha, temp, sizes)
}
var lockTree = AcquireCloneLock // replaced in tests to see installTree wait for a job holding the tree
// installTree moves temp into place as dir/sha, replacing a tree that failed treeIntact.
func installTree(ctx context.Context, dir, sha, temp string, sizes map[string]int64) error {
files := make([]treeFile, 0, len(sizes))
for name, size := range sizes {
files = append(files, treeFile{[]byte(name), size})
}
data, err := json.Marshal(files)
if err != nil {
return err
}
// Written before the swap: it describes the commit, not a copy of it, so it holds for whichever tree is left.
if err := os.WriteFile(filepath.Join(dir, sha+treeSizes), data, 0o600); err != nil { // a torn record fails treeIntact, so it needs no rename
return err
}
tree := filepath.Join(dir, sha)
if !exists(tree) {
return os.Rename(temp, tree)
}
stale, err := os.MkdirTemp(dir, "tmp-")
if err != nil {
return err
}
defer os.RemoveAll(stale) // after the unlock, as no job reads the old tree once it is moved
unlock, err := lockTree(ctx, tree) // jobs copying the old tree hold it
if err != nil {
return err
}
defer unlock()
if err := os.Rename(tree, filepath.Join(stale, sha)); err != nil {
return err
}
return os.Rename(temp, tree)
}
// treeIntact reports whether dir/sha still holds each file extracted into it at the size it was written.
// Only sizes are compared, not content: a stat per file is cheap enough for every job, hashing is not.
func treeIntact(dir, sha string) bool {
data, err := os.ReadFile(filepath.Join(dir, sha+treeSizes))
if err != nil || !exists(filepath.Join(dir, sha)) {
return false
}
var files []treeFile
if json.Unmarshal(data, &files) != nil {
return false
}
for _, file := range files {
info, err := os.Lstat(filepath.Join(dir, sha, filepath.FromSlash(string(file.Name))))
if err != nil || !info.Mode().IsRegular() || info.Size() != file.Size {
return false
}
}
return true
}
func writeTreeFile(root *os.Root, name string, mode int64, content io.Reader) (int64, error) {
perm := os.FileMode(0o644)
if mode&0o111 != 0 {
perm = 0o755
}
file, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm)
if err != nil {
return 0, err
}
n, err := io.Copy(file, content)
return n, errors.Join(err, file.Close())
}
func exists(name string) bool {
_, err := os.Stat(name)
return err == nil
}

View File

@@ -0,0 +1,412 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package git
import (
"archive/tar"
"bytes"
"context"
"encoding/base64"
"fmt"
"net/http"
"net/http/cgi"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"gitea.com/gitea/runner/act/common"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestFetchAction(t *testing.T) {
original := archiveRetryDelay
archiveRetryDelay = func(_retryAfter string) time.Duration { return time.Nanosecond }
t.Cleanup(func() { archiveRetryDelay = original })
root := t.TempDir()
remote := filepath.Join(root, "org", "action")
require.NoError(t, gitCmd("init", "--initial-branch=main", remote))
require.NoError(t, os.WriteFile(filepath.Join(remote, "action.yml"), []byte("name: action\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(remote, "run.sh"), []byte("#!/bin/sh\n"), 0o755))
require.NoError(t, os.Symlink("action.yml", filepath.Join(remote, "link")))
require.NoError(t, gitCmd("-C", remote, "add", "."))
require.NoError(t, gitCmd("-C", remote, "commit", "-m", "initial"))
require.NoError(t, gitCmd("-C", remote, "tag", "v1"))
expectedSha := gitRevParse(t, remote, "main")
gitPath, err := exec.LookPath("git")
require.NoError(t, err)
backend := &cgi.Handler{Path: gitPath, Args: []string{"http-backend"}, Env: []string{"GIT_PROJECT_ROOT=" + root, "GIT_HTTP_EXPORT_ALL=1"}}
var mu sync.Mutex
var archiveRequests []http.Header
var mirror *httptest.Server
archiveCache := map[string][]byte{}
requests, archives, failures := 0, "", 0
handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
mu.Lock()
defer mu.Unlock()
requests++
ref, ok := strings.CutPrefix(request.URL.Path, "/org/action/archive/")
if !ok {
backend.ServeHTTP(writer, request)
return
}
archiveRequests = append(archiveRequests, request.Header.Clone())
switch {
case failures > 0:
failures--
writer.WriteHeader(http.StatusServiceUnavailable)
return
case archives == "none":
http.NotFound(writer, request)
return
case archives == "unreachable":
http.Redirect(writer, request, "http://127.0.0.1:1"+request.URL.Path, http.StatusFound)
return
case archives == "redirect" && request.Host != mirror.Listener.Addr().String():
http.Redirect(writer, request, mirror.URL+request.URL.Path, http.StatusFound)
return
}
sha, _ := gitOutput(request.Context(), remote, "rev-parse", strings.TrimSuffix(ref, ".tar.gz"))
etag := fmt.Sprintf(`W/"%s-tar.gz-%t"`, sha, archives != "unprefixed")
if request.Header.Get("If-None-Match") == etag {
writer.WriteHeader(http.StatusNotModified)
return
}
args := []string{"archive", "--format=tar.gz", sha}
if archives != "unprefixed" {
args = append(args, "--prefix=action/")
}
archiveKey := strings.Join(args, "\x00")
archive, ok := archiveCache[archiveKey]
if !ok {
archive, _ = runGit(request.Context(), remote, nil, args...)
archiveCache[archiveKey] = archive
}
writer.Header().Set("ETag", etag)
_, _ = writer.Write(archive)
})
server := httptest.NewServer(handler)
mirror = httptest.NewServer(handler)
t.Cleanup(server.Close)
t.Cleanup(mirror.Close)
serve := func(mode string, failing int) {
mu.Lock()
defer mu.Unlock()
requests, archives, failures, archiveRequests = 0, mode, failing, nil
}
served := func() (int, []http.Header) {
mu.Lock()
defer mu.Unlock()
return requests, append([]http.Header(nil), archiveRequests...)
}
newInput := func(ref string) NewGitCloneExecutorInput {
return NewGitCloneExecutorInput{URL: server.URL + "/org/action", Ref: ref, Dir: t.TempDir(), Token: "secret"}
}
t.Run("shares identical fetches and isolates cancellation, panics and differing inputs", func(t *testing.T) {
for _, scenario := range []string{"shared", "leader cancelled", "leader panics", "different token"} {
t.Run(scenario, func(t *testing.T) {
serve("prefixed", 0)
input := newInput("main")
started, release := make(chan struct{}), make(chan struct{})
releaseFetches := sync.OnceFunc(func() { close(release) })
t.Cleanup(releaseFetches)
var once sync.Once
var panicked atomic.Bool
transport := actionFetchTransport(func(request *http.Request) (*http.Response, error) {
once.Do(func() { close(started) })
select {
case <-request.Context().Done():
return nil, request.Context().Err()
case <-release:
}
if scenario == "leader panics" && panicked.CompareAndSwap(false, true) {
panic("broken archive")
}
return http.DefaultTransport.RoundTrip(request)
})
input.Transport = &transport
leaderCtx, cancelLeader := context.WithCancel(t.Context())
defer cancelLeader()
waiting := &actionFetchContext{Context: t.Context(), waiting: make(chan struct{})}
followerInput := input
expectedRequests := 1
if scenario == "different token" {
followerInput.Token, expectedRequests = "other", 2
}
leaderDone, followerDone := make(chan error, 1), make(chan error, 1)
fetch := func(ctx context.Context, input NewGitCloneExecutorInput, done chan<- error) {
var err error
defer func() { done <- err }()
defer common.RecoverPanic(&err)
var tree, sha string
if tree, sha, err = FetchAction(ctx, input); err == nil {
assert.Equal(t, expectedSha, sha)
assert.Equal(t, filepath.Join(input.Dir, sha), tree)
}
}
go fetch(leaderCtx, input, leaderDone)
<-started
go fetch(waiting, followerInput, followerDone)
<-waiting.waiting
if scenario == "leader cancelled" {
cancelLeader()
require.ErrorIs(t, <-leaderDone, context.Canceled)
}
releaseFetches()
switch scenario {
case "leader cancelled":
case "leader panics":
require.ErrorContains(t, <-leaderDone, "panic: broken archive")
default:
require.NoError(t, <-leaderDone)
}
require.NoError(t, <-followerDone)
_, headers := served()
assert.Len(t, headers, expectedRequests)
if scenario == "different token" {
assert.NotEqual(t, headers[0].Get("Authorization"), headers[1].Get("Authorization"))
}
})
}
})
t.Run("downloads, revalidates and follows a moved tag", func(t *testing.T) {
serve("prefixed", 0)
input := newInput("v1")
tree, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, expectedSha, sha)
assert.Equal(t, filepath.Join(input.Dir, sha), tree)
assert.NoDirExists(t, filepath.Join(tree, ".git"))
info, err := os.Stat(filepath.Join(tree, "run.sh"))
require.NoError(t, err)
assert.Equal(t, os.FileMode(0o755), info.Mode().Perm())
link, err := os.Readlink(filepath.Join(tree, "link"))
require.NoError(t, err)
assert.Equal(t, "action.yml", link)
again, _, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, tree, again)
require.NoError(t, gitCmd("-C", remote, "commit", "--allow-empty", "-m", "moved"))
require.NoError(t, gitCmd("-C", remote, "tag", "--force", "v1"))
expectedSha = gitRevParse(t, remote, "v1")
moved, movedSha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, expectedSha, movedSha)
assert.NotEqual(t, tree, moved)
assert.FileExists(t, filepath.Join(tree, "action.yml"))
input.OfflineMode = true
_, offlineSha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, movedSha, offlineSha)
require.NoError(t, os.Truncate(filepath.Join(moved, "action.yml"), 4)) // fetched again, offline or not
_, _, err = FetchAction(t.Context(), input)
require.NoError(t, err)
require.NoError(t, os.Remove(moved+treeSizes)) // from an older runner, used as it is offline
reused, _, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, moved, reused)
requests, headers := served()
assert.Equal(t, 4, requests)
require.Len(t, headers, 4)
assert.Equal(t, "Basic "+base64.StdEncoding.EncodeToString([]byte("token:secret")), headers[0].Get("Authorization"))
assert.Empty(t, headers[0].Get("If-None-Match"))
assert.Equal(t, fmt.Sprintf(`W/"%s-tar.gz-true"`, sha), headers[1].Get("If-None-Match"))
assert.Equal(t, headers[1].Get("If-None-Match"), headers[2].Get("If-None-Match"))
})
t.Run("downloads a pinned commit once and rejects a short one", func(t *testing.T) {
serve("prefixed", 0)
sha := expectedSha
input := newInput(sha)
for range 2 {
tree, got, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, sha, got)
assert.FileExists(t, filepath.Join(tree, "action.yml"))
}
_, _, err := FetchAction(t.Context(), newInput(sha[:7]))
require.ErrorIs(t, err, ErrShortRef)
_, headers := served()
assert.Len(t, headers, 1)
assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
})
t.Run("downloads again over a tree whose files changed size or that has no record, not under a job copying it", func(t *testing.T) {
sha := gitRevParse(t, remote, "main")
for _, mode := range []string{"prefixed", "none"} {
serve(mode, 0)
input := newInput(sha)
tree, _, err := FetchAction(t.Context(), input)
require.NoError(t, err, mode)
yml := filepath.Join(tree, "action.yml")
require.NoError(t, os.WriteFile(filepath.Join(tree, "trampoline.js"), nil, 0o644)) // the runner's own, no damage
_, _, err = FetchAction(t.Context(), input)
require.NoError(t, err, mode)
for _, damage := range []func() error{
func() error { return os.Truncate(yml, 4) },
func() error { return os.Remove(tree + treeSizes) },
} {
require.NoError(t, damage(), mode)
_, _, err := FetchAction(t.Context(), input)
require.NoError(t, err, mode)
body, err := os.ReadFile(yml)
require.NoError(t, err, mode)
assert.Equal(t, "name: action\n", string(body), mode)
}
_, headers := served()
assert.Len(t, headers, 3, mode)
}
serve("prefixed", 0)
input := newInput(sha)
tree, _, err := FetchAction(t.Context(), input)
require.NoError(t, err)
yml := filepath.Join(tree, "action.yml")
require.NoError(t, os.Truncate(yml, 4))
unlock, err := AcquireCloneLock(t.Context(), tree)
require.NoError(t, err)
release := sync.OnceFunc(unlock)
defer release()
waiting := make(chan struct{})
lockTree = func(ctx context.Context, dir string) (func(), error) {
close(waiting)
return AcquireCloneLock(ctx, dir)
}
t.Cleanup(func() { lockTree = AcquireCloneLock })
done := make(chan error, 1)
go func() {
_, _, err := FetchAction(t.Context(), input)
done <- err
}()
select {
case err := <-done:
t.Fatalf("FetchAction replaced the tree without waiting for the job holding it: %v", err)
case <-waiting:
}
info, err := os.Stat(yml)
require.NoError(t, err)
assert.Equal(t, int64(4), info.Size(), "the tree a job holds is left alone")
release()
require.NoError(t, <-done)
body, err := os.ReadFile(yml)
require.NoError(t, err)
assert.Equal(t, "name: action\n", string(body))
})
t.Run("retries a failing download and drops the token on a redirect to another origin", func(t *testing.T) {
serve("redirect", 2)
input := newInput("main")
_, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, expectedSha, sha)
assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
_, headers := served()
require.Len(t, headers, 4)
assert.NotEmpty(t, headers[2].Get("Authorization"))
assert.Empty(t, headers[3].Get("Authorization"))
})
t.Run("keeps an unprefixed archive whole, downloads where insteadOf points and exports a clone without an archive or from an unreachable host", func(t *testing.T) {
for _, mode := range []string{"unprefixed", "none", "unreachable"} {
serve(mode, 0)
tree, sha, err := FetchAction(t.Context(), newInput("main"))
require.NoError(t, err, mode)
assert.Equal(t, expectedSha, sha, mode)
assert.FileExists(t, filepath.Join(tree, "action.yml"), mode)
assert.NoDirExists(t, filepath.Join(tree, ".git"), mode)
_, headers := served()
assert.Len(t, headers, 1, mode)
}
serve("prefixed", 0)
config := filepath.Join(t.TempDir(), "gitconfig")
require.NoError(t, os.WriteFile(config, []byte("[url \""+server.URL+"/org/\"]\n\tinsteadOf = https://example.invalid/org/\n"), 0o644))
t.Setenv("GIT_CONFIG_GLOBAL", config)
input := newInput("main")
input.URL = "https://example.invalid/org/action"
_, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, expectedSha, sha)
_, headers := served()
assert.Len(t, headers, 1)
assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
})
}
type actionFetchTransport func(*http.Request) (*http.Response, error)
func (transport actionFetchTransport) RoundTrip(request *http.Request) (*http.Response, error) {
return transport(request)
}
type actionFetchContext struct {
context.Context
waiting chan struct{}
once sync.Once
}
func (ctx *actionFetchContext) Done() <-chan struct{} {
ctx.once.Do(func() { close(ctx.waiting) })
return ctx.Context.Done()
}
func TestExtractTreeConfinesEntries(t *testing.T) {
outside := t.TempDir()
for name, entries := range map[string][]*tar.Header{
"parent path": {{Name: "../escape", Typeflag: tar.TypeReg}},
"through a symlink": {{Name: "out", Typeflag: tar.TypeSymlink, Linkname: outside}, {Name: "out/escape", Typeflag: tar.TypeReg}},
} {
t.Run(name, func(t *testing.T) {
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
require.NoError(t, writer.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, PAXRecords: map[string]string{"comment": strings.Repeat("a", 40)}}))
for _, entry := range entries {
require.NoError(t, writer.WriteHeader(entry))
}
require.NoError(t, writer.Close())
dir := t.TempDir()
_, err := extractTree(t.Context(), &archive, dir, false)
require.Error(t, err)
assert.NoFileExists(t, filepath.Join(dir, "escape"))
assert.NoFileExists(t, filepath.Join(outside, "escape"))
})
}
}
func TestTreeIntactFindsNamesThatAreNotUTF8(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("macOS and Windows store no name that is not UTF-8")
}
sha := strings.Repeat("a", 40)
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
require.NoError(t, writer.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, PAXRecords: map[string]string{"comment": sha}}))
require.NoError(t, writer.WriteHeader(&tar.Header{Name: "caf\xe9.js", Typeflag: tar.TypeReg, Mode: 0o644, Size: 1, Format: tar.FormatGNU}))
_, err := writer.Write([]byte("x"))
require.NoError(t, err)
require.NoError(t, writer.Close())
dir := t.TempDir()
_, err = extractTree(t.Context(), &archive, dir, false)
require.NoError(t, err)
require.FileExists(t, filepath.Join(dir, sha, "caf\xe9.js"))
assert.True(t, treeIntact(dir, sha))
}

View File

@@ -10,6 +10,8 @@ import (
"encoding/base64"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"os/exec"
@@ -78,25 +80,25 @@ func (e *Error) Unwrap() error { return e.err }
func (e *Error) Commit() string { return e.commit }
// FindGitRevision get the current git revision
func FindGitRevision(ctx context.Context, file string) (shortSHA, sha string, err error) {
func FindGitRevision(ctx context.Context, file string) (string, error) {
logger := common.Logger(ctx)
sha, err = gitOutput(ctx, file, "rev-parse", "--verify", "HEAD^{commit}")
sha, err := gitOutput(ctx, file, "rev-parse", "--verify", "HEAD^{commit}")
if err != nil {
logger.WithError(err).Error("path", file, "not located inside a git repository")
return "", "", err
return "", err
}
if len(sha) < 7 {
return "", "", errors.New("HEAD could not be resolved")
return "", errors.New("HEAD could not be resolved")
}
logger.Debugf("Found revision: %s", sha)
return sha[:7], sha, nil
return sha, nil
}
// FindGitRef get the current git ref
func FindGitRef(ctx context.Context, file string) (string, error) {
logger := common.Logger(ctx)
logger.Debugf("Loading revision from git directory")
_, revision, err := FindGitRevision(ctx, file)
revision, err := FindGitRevision(ctx, file)
if err != nil {
return "", err
}
@@ -178,13 +180,10 @@ type NewGitCloneExecutorInput struct {
// 0 for full clone.
Depth int
// Quiet drops the informational clone line to debug level, for callers that log their own
// download summary (the setup section's action report).
Quiet bool
// For Gitea
InsecureSkipTLS bool
ClientCertFile, ClientKeyFile string // PEM, absolute as git runs in other directories
ClientCertFile, ClientKeyFile string // PEM, absolute as git runs in other directories
Transport http.RoundTripper // for FetchAction's archive downloads, nil for http.DefaultTransport
}
// CloneIfRequired reports whether an existing local clone was reused.
@@ -221,11 +220,7 @@ func CloneIfRequired(ctx context.Context, input NewGitCloneExecutorInput, logger
func NewGitCloneExecutor(input NewGitCloneExecutorInput) common.Executor {
return func(ctx context.Context) error {
logger := common.Logger(ctx)
if input.Quiet {
logger.Debugf("git clone '%s' # ref=%s", input.URL, input.Ref)
} else {
logger.Infof("git clone '%s' # ref=%s", input.URL, input.Ref)
}
logger.Infof("git clone '%s' # ref=%s", input.URL, input.Ref)
logger.Debugf(" cloning %s to %s", input.URL, input.Dir)
unlock, err := AcquireCloneLock(ctx, input.Dir)
if err != nil {
@@ -233,24 +228,10 @@ func NewGitCloneExecutor(input NewGitCloneExecutorInput) common.Executor {
}
defer unlock()
reused, err := CloneIfRequired(ctx, input, logger)
resolved, reused, err := resolveClone(ctx, input)
if err != nil {
return err
}
resolved, err := resolveRef(ctx, input.Dir, input.Ref)
if errors.Is(err, ErrShortRef) || (err != nil && input.OfflineMode) {
return err
}
if !input.OfflineMode && (err != nil || (reused && !isFullHash(input.Ref, resolved))) {
if err := fetch(ctx, input); err != nil {
return err
}
resolved, err = resolveRef(ctx, input.Dir, input.Ref)
}
if err != nil {
logger.Errorf("Unable to resolve %s: %v", input.Ref, err)
return err
}
if _, err := runGit(ctx, input.Dir, nil, "checkout", "--force", "--detach", resolved); err != nil {
return fmt.Errorf("checkout %s: %w", resolved, err)
}
@@ -264,6 +245,28 @@ func NewGitCloneExecutor(input NewGitCloneExecutorInput) common.Executor {
}
}
func resolveClone(ctx context.Context, input NewGitCloneExecutorInput) (string, bool, error) { // the caller holds input.Dir's lock
logger := common.Logger(ctx)
reused, err := CloneIfRequired(ctx, input, logger)
if err != nil {
return "", false, err
}
resolved, err := resolveRef(ctx, input.Dir, input.Ref)
if errors.Is(err, ErrShortRef) || (err != nil && input.OfflineMode) {
return "", false, err
}
if !input.OfflineMode && (err != nil || (reused && !isFullHash(input.Ref, resolved))) {
if err := fetch(ctx, input); err != nil {
return "", false, err
}
resolved, err = resolveRef(ctx, input.Dir, input.Ref)
}
if err != nil {
logger.Errorf("Unable to resolve %s: %v", input.Ref, err)
}
return resolved, reused, err
}
func clone(ctx context.Context, input NewGitCloneExecutorInput) error {
if input.Depth > 0 && isFullObjectID(input.Ref) {
if err := shallowPinnedClone(ctx, input); err == nil {
@@ -304,7 +307,7 @@ func shallowPinnedClone(ctx context.Context, input NewGitCloneExecutorInput) err
}
func fetch(ctx context.Context, input NewGitCloneExecutorInput) error {
args := []string{"fetch", "--force", "--prune"}
args := []string{"fetch", "--force", "--no-prune"}
if _, err := os.Stat(filepath.Join(input.Dir, ".git", "shallow")); err == nil {
args = append(args, "--depth", "1", "--no-tags")
}
@@ -318,9 +321,18 @@ func cachedRef(ref string) string {
func resolveRef(ctx context.Context, dir, ref string) (string, error) {
if !isFullObjectID(ref) {
for _, name := range []string{cachedRef(ref), "refs/tags/" + ref, "refs/remotes/origin/" + ref} {
if sha, err := gitOutput(ctx, dir, "rev-parse", "--verify", name+"^{commit}"); err == nil {
return sha, nil
revisions := []string{cachedRef(ref) + "^{commit}", "refs/tags/" + ref + "^{commit}", "refs/remotes/origin/" + ref + "^{commit}"}
if strings.ContainsAny(ref, "\x00\n") {
for _, revision := range revisions {
if sha, err := gitOutput(ctx, dir, "rev-parse", "--verify", revision); err == nil {
return sha, nil
}
}
} else if output, err := runGitWithStdin(ctx, dir, nil, strings.NewReader(strings.Join(revisions, "\n")+"\n"), "cat-file", "--batch-check=%(objectname)"); err == nil {
for sha := range strings.SplitSeq(string(output), "\n") {
if isFullObjectID(sha) {
return sha, nil
}
}
}
}
@@ -425,6 +437,10 @@ func ListFiles(ctx context.Context, dir string) ([]string, error) {
}
func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, args ...string) ([]byte, error) {
return runGitWithStdin(ctx, dir, input, nil, args...)
}
func runGitWithStdin(ctx context.Context, dir string, input *NewGitCloneExecutorInput, stdin io.Reader, args ...string) ([]byte, error) {
config := []string{"core.autocrlf=false", "core.eol=lf"} // keep action files LF
if input != nil {
config = append(config, RemoteConfig(input.Token, input.InsecureSkipTLS)...)
@@ -436,6 +452,7 @@ func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, ar
args = append([]string{"-C", dir}, args...)
}
cmd := exec.CommandContext(ctx, "git", args...)
cmd.Stdin = stdin
cmd.Env = slices.DeleteFunc(os.Environ(), func(entry string) bool {
name, _, _ := strings.Cut(entry, "=")
return slices.Contains(LocalEnvVars, name)

View File

@@ -21,7 +21,6 @@ import (
"testing/synctest"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/internal/pkg/lock"
@@ -30,6 +29,31 @@ import (
"github.com/stretchr/testify/require"
)
var gitFixtureDir string
func TestMain(m *testing.M) {
var err error
gitFixtureDir, err = os.MkdirTemp("", "runner-git-fixture-")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
code := 1
if err = gitCmd("init", "--quiet", "--template=", "--bare", "--initial-branch=main", filepath.Join(gitFixtureDir, "remote")); err == nil {
err = gitCmd("init", "--quiet", "--template=", "--initial-branch=main", filepath.Join(gitFixtureDir, "work"))
}
if err == nil {
code = m.Run()
} else {
fmt.Fprintln(os.Stderr, err)
}
if err := os.RemoveAll(gitFixtureDir); err != nil {
fmt.Fprintln(os.Stderr, err)
code = 1
}
os.Exit(code)
}
func TestFindGitSlug(t *testing.T) {
for _, testcase := range []struct {
remoteURL string
@@ -68,42 +92,20 @@ func TestErrorWrapsCommitAndCause(t *testing.T) {
require.Equal(t, "abc123", err.Commit())
}
func cleanGitHooks(dir string) error {
hooksDir := filepath.Join(dir, ".git", "hooks")
files, err := os.ReadDir(hooksDir)
if err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
for _, f := range files {
if f.IsDir() {
continue
}
relName := filepath.Join(hooksDir, f.Name())
if err := os.Remove(relName); err != nil {
return err
}
}
return nil
}
func TestFindGitMetadataOfSHA256RepositoryUsesOrigin(t *testing.T) {
t.Parallel()
basedir := t.TempDir()
const remoteURL = "https://github.com/owner/repo.git"
require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", basedir))
require.NoError(t, cleanGitHooks(basedir))
require.NoError(t, os.RemoveAll(filepath.Join(basedir, ".git", "hooks")))
require.NoError(t, gitCmd("-C", basedir, "commit", "--allow-empty", "-m", "init"))
require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", remoteURL))
require.NoError(t, gitCmd("-C", basedir, "config", "--add", "remote.origin.url", "https://github.com/other/repo.git"))
shortSHA, sha, err := FindGitRevision(t.Context(), basedir)
sha, err := FindGitRevision(t.Context(), basedir)
require.NoError(t, err)
require.Equal(t, gitRevParse(t, basedir, "HEAD"), sha)
require.Len(t, sha, 64)
require.Equal(t, sha[:7], shortSHA)
ref, err := FindGitRef(t.Context(), basedir)
require.NoError(t, err)
@@ -123,89 +125,65 @@ func TestGitFindRef(t *testing.T) {
basedir := t.TempDir()
for name, tt := range map[string]struct {
Prepare func(t *testing.T, dir string)
Assert func(t *testing.T, ref string, err error)
commands [][]string
ref string
}{
"new_repo": {
Prepare: func(t *testing.T, dir string) {},
Assert: func(t *testing.T, ref string, err error) {
require.Error(t, err)
},
},
"new_repo": {},
"new_repo_with_commit": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/heads/master", ref)
},
commands: [][]string{{"commit", "--allow-empty", "-m", "msg"}},
ref: "refs/heads/master",
},
"current_head_is_tag": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "commit msg"))
require.NoError(t, gitCmd("-C", dir, "tag", "v1.2.3"))
require.NoError(t, gitCmd("-C", dir, "checkout", "v1.2.3"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/tags/v1.2.3", ref)
commands: [][]string{
{"commit", "--allow-empty", "-m", "commit msg"},
{"tag", "v1.2.3"},
{"checkout", "v1.2.3"},
},
ref: "refs/tags/v1.2.3",
},
"current_head_is_same_as_tag": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "1.4.2 release"))
require.NoError(t, gitCmd("-C", dir, "tag", "v1.4.2"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/tags/v1.4.2", ref)
commands: [][]string{
{"commit", "--allow-empty", "-m", "1.4.2 release"},
{"tag", "v1.4.2"},
},
ref: "refs/tags/v1.4.2",
},
"current_head_is_not_tag": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg"))
require.NoError(t, gitCmd("-C", dir, "tag", "v1.4.2"))
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg2"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/heads/master", ref)
commands: [][]string{
{"commit", "--allow-empty", "-m", "msg"},
{"tag", "v1.4.2"},
{"commit", "--allow-empty", "-m", "msg2"},
},
ref: "refs/heads/master",
},
"current_head_is_another_branch": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "checkout", "-b", "mybranch"))
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/heads/mybranch", ref)
commands: [][]string{
{"checkout", "-b", "mybranch"},
{"commit", "--allow-empty", "-m", "msg"},
},
ref: "refs/heads/mybranch",
},
} {
t.Run(name, func(t *testing.T) {
dir := filepath.Join(basedir, name)
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, gitCmd("-C", dir, "init", "--initial-branch=master"))
require.NoError(t, cleanGitHooks(dir))
tt.Prepare(t, dir)
require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, os.WriteFile(filepath.Join(dir, ".git", "HEAD"), []byte("ref: refs/heads/master\n"), 0o644))
for _, command := range tt.commands {
require.NoError(t, gitCmd(append([]string{"-C", dir}, command...)...))
}
ref, err := FindGitRef(t.Context(), dir)
tt.Assert(t, ref, err)
if tt.ref == "" {
require.Error(t, err)
} else {
require.NoError(t, err)
require.Equal(t, tt.ref, ref)
}
})
}
}
func TestGitCloneExecutor(t *testing.T) {
// Build a local bare "remote" so this runs offline and fast. The cases below mirror
// the tag/branch/sha/short-sha ref paths the executor handles, formerly exercised by
// cloning actions/checkout and anchore/scan-action over the network.
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
remoteDir, workDir := newGitRemote(t)
require.NoError(t, os.WriteFile(filepath.Join(workDir, "action.sh"), []byte("#!/bin/sh\necho hi\n"), 0o755))
require.NoError(t, gitCmd("-C", workDir, "add", "action.sh"))
require.NoError(t, gitCmd("-C", workDir, "commit", "-m", "initial"))
@@ -213,14 +191,11 @@ func TestGitCloneExecutor(t *testing.T) {
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v2"))
// A branch with a dash in the name (mirrors the historical scan-action@act-fails case).
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "act-fails"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "branch-commit"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "act-fails"))
out, err := exec.Command("git", "-C", workDir, "rev-parse", "main").Output()
require.NoError(t, err)
fullSha := strings.TrimSpace(string(out))
fullSha := gitRevParse(t, workDir, "main")
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:refs/heads/"+fullSha[:4]))
@@ -229,6 +204,7 @@ func TestGitCloneExecutor(t *testing.T) {
t.Setenv("GIT_CONFIG_GLOBAL", hostConfig)
t.Setenv("GIT_DIR", t.TempDir())
var clones sync.WaitGroup
for name, tt := range map[string]struct {
Err error
Ref string
@@ -256,43 +232,40 @@ func TestGitCloneExecutor(t *testing.T) {
Ref: fullSha[:4],
},
} {
t.Run(name, func(t *testing.T) {
dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: "https://example.invalid/action",
Ref: tt.Ref,
Dir: dir,
})
clones.Go(func() {
t.Run(name, func(t *testing.T) {
dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: "https://example.invalid/action",
Ref: tt.Ref,
Dir: dir,
})
err := clone(t.Context())
if tt.Err != nil {
assert.Error(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, tt.Err, err)
assert.Equal(t, tt.Err, clone(t.Context()), "a retry on the cache keeps the error")
return
}
assert.Empty(t, err) //nolint:testifylint // pre-existing issue from nektos/act
marker := filepath.Join(dir, "marker")
require.NoError(t, os.WriteFile(marker, nil, 0o644))
require.NoError(t, clone(t.Context()))
assert.FileExists(t, marker, "an insteadOf rewrite must not evict the cache")
script, err := os.ReadFile(filepath.Join(dir, "action.sh"))
require.NoError(t, err)
assert.Equal(t, "#!/bin/sh\necho hi\n", string(script), "host core.autocrlf must not convert action files")
err := clone(t.Context())
if tt.Err != nil {
require.Error(t, err)
assert.Equal(t, tt.Err, err)
assert.Equal(t, tt.Err, clone(t.Context()), "a retry on the cache keeps the error")
return
}
require.NoError(t, err)
marker := filepath.Join(dir, "marker")
require.NoError(t, os.WriteFile(marker, nil, 0o644))
require.NoError(t, clone(t.Context()))
assert.FileExists(t, marker, "an insteadOf rewrite must not evict the cache")
script, err := os.ReadFile(filepath.Join(dir, "action.sh"))
require.NoError(t, err)
assert.Equal(t, "#!/bin/sh\necho hi\n", string(script), "host core.autocrlf must not convert action files")
})
})
}
clones.Wait()
}
func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
t.Parallel()
createRemote := func(message string) string {
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", message))
remoteDir, workDir := newGitRemote(t, message)
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
return remoteDir
@@ -342,18 +315,14 @@ func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
func TestGitCloneExecutorFollowsRemoteRefChanges(t *testing.T) {
t.Parallel()
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "initial"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "feature-1"))
remoteDir, workDir := newGitRemote(t, "initial", "feature-1")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{URL: remoteDir, Ref: "main", Dir: dir})
require.NoError(t, clone(t.Context()))
require.NoError(t, gitCmd("-C", dir, "config", "fetch.prune", "true"))
require.NoError(t, clone(t.Context()))
require.NoError(t, gitCmd("-C", workDir, "reset", "--hard", "HEAD~1"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "feature-rewritten"))
@@ -361,25 +330,27 @@ func TestGitCloneExecutorFollowsRemoteRefChanges(t *testing.T) {
require.NoError(t, clone(t.Context()), "a non-fast-forward ref must update")
assert.Equal(t, "feature-rewritten", gitHeadSubject(t, dir))
tagClone := NewGitCloneExecutor(NewGitCloneExecutorInput{URL: remoteDir, Ref: "v9", Dir: t.TempDir()})
tagDir := t.TempDir()
tagClone := NewGitCloneExecutor(NewGitCloneExecutorInput{URL: remoteDir, Ref: "v9", Dir: tagDir})
require.Error(t, tagClone(t.Context()))
require.NoError(t, gitCmd("-C", workDir, "tag", "v9"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v9"))
require.NoError(t, tagClone(t.Context()), "a tag published after a failed lookup must be fetched")
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "tag-moved"))
require.NoError(t, gitCmd("-C", workDir, "tag", "--force", "v9"))
require.NoError(t, gitCmd("-C", workDir, "push", "--force", "origin", "v9"))
for range 2 {
require.NoError(t, tagClone(t.Context()), "a moved tag must update and stay current")
assert.Equal(t, "tag-moved", gitHeadSubject(t, tagDir))
}
}
func TestGitCloneExecutorOfflineMode(t *testing.T) {
t.Parallel()
// Build a local "remote" with a single commit on main.
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "initial"))
remoteDir, workDir := newGitRemote(t, "initial")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
// Prime the cache with an online clone of main.
cacheDir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir,
@@ -388,8 +359,6 @@ func TestGitCloneExecutorOfflineMode(t *testing.T) {
})(t.Context()))
t.Run("cached branch resolves without fetching", func(t *testing.T) {
// Offline reuse of a cached branch must succeed even though ResolveRevision(input.Ref)
// finds no local refs/heads/<ref>.
err := NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir,
Ref: "main",
@@ -404,7 +373,6 @@ func TestGitCloneExecutorOfflineMode(t *testing.T) {
})
t.Run("unresolvable cached ref returns error", func(t *testing.T) {
// The ref was never cached; offline mode cannot resolve it and must return an error.
err := NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir,
Ref: "never-fetched",
@@ -415,56 +383,9 @@ func TestGitCloneExecutorOfflineMode(t *testing.T) {
})
}
func TestGitCloneExecutorQuietDemotesCloneLine(t *testing.T) {
t.Parallel()
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "initial"))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
// Quiet callers report the download themselves, so the clone line must not reach the job log.
for name, quiet := range map[string]bool{"quiet": true, "not quiet": false} {
t.Run(name, func(t *testing.T) {
hook := logtest.New()
logger := logging.New(logging.NewLevelHandler(hook, logging.LevelInfo))
ctx := common.WithLogger(t.Context(), logger.With("job", "j1"))
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir,
Ref: "main",
Dir: t.TempDir(),
Quiet: quiet,
})(ctx))
var cloneLines int
for _, entry := range hook.AllRecords() {
if strings.HasPrefix(entry.Message, "git clone ") {
cloneLines++
}
}
if quiet {
assert.Zero(t, cloneLines)
} else {
assert.Equal(t, 1, cloneLines)
}
})
}
}
func TestGitCloneExecutorShallow(t *testing.T) {
t.Parallel()
// Build a local "remote" with several commits on main plus a tag, so a full clone would pull noticeably more history than a shallow one.
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
for _, m := range []string{"c1", "c2", "c3"} {
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", m))
}
remoteDir, workDir := newGitRemote(t, "c1", "c2", "c3")
require.NoError(t, gitCmd("-C", workDir, "tag", "v1"))
unadvertisedSHA := gitRevParse(t, workDir, "HEAD~1")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
@@ -472,25 +393,17 @@ func TestGitCloneExecutorShallow(t *testing.T) {
shallowMarker := func(dir string) string { return filepath.Join(dir, ".git", "shallow") }
t.Run("branch is cloned shallowly", func(t *testing.T) {
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: "main", Dir: dir, Depth: 1,
})(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir), "only the tip commit should be present")
assert.Equal(t, "c3", gitHeadSubject(t, dir))
})
t.Run("tag is cloned shallowly", func(t *testing.T) {
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: "v1", Dir: dir, Depth: 1,
})(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir))
assert.Equal(t, "c3", gitHeadSubject(t, dir))
})
for name, ref := range map[string]string{"branch is cloned shallowly": "main", "tag is cloned shallowly": "v1"} {
t.Run(name, func(t *testing.T) {
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: ref, Dir: dir, Depth: 1,
})(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir), "only the tip commit should be present")
assert.Equal(t, "c3", gitHeadSubject(t, dir))
})
}
unadvertisedRemote := func(t *testing.T, allowUnadvertised bool) string {
remote := filepath.Join(t.TempDir(), "remote.git")
@@ -570,12 +483,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
t.Parallel()
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "pinned"))
remoteDir, workDir := newGitRemote(t, "pinned")
pinned := gitRevParse(t, workDir, "HEAD")
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "decoy"))
require.NoError(t, gitCmd("-C", workDir, "tag", pinned))
@@ -593,6 +501,21 @@ func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
}
}
func newGitRemote(t *testing.T, messages ...string) (string, string) {
t.Helper()
remoteDir, workDir := t.TempDir(), t.TempDir()
require.NoError(t, os.CopyFS(remoteDir, os.DirFS(filepath.Join(gitFixtureDir, "remote"))))
require.NoError(t, os.CopyFS(workDir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
config, err := os.ReadFile(filepath.Join(workDir, ".git", "config"))
require.NoError(t, err)
require.NoError(t, os.WriteFile(filepath.Join(workDir, ".git", "config"),
fmt.Appendf(config, "\n[remote \"origin\"]\n\turl = %s\n\tfetch = +refs/heads/*:refs/remotes/origin/*\n", strconv.Quote(remoteDir)), 0o644))
for _, message := range messages {
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", message))
}
return remoteDir, workDir
}
func gitRevParse(t *testing.T, dir, rev string) string {
t.Helper()
out, err := exec.Command("git", "-C", dir, "rev-parse", rev).Output()
@@ -620,9 +543,6 @@ func gitCmd(args ...string) error {
cmd := exec.Command("git", args...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
// Inject a deterministic identity and ignore the host's global/system config so commits
// succeed regardless of the host having no user.name/user.email (e.g. CI, GITHUB_ACTIONS
// unset) or a global commit.gpgsign, and without mutating the developer's ~/.gitconfig.
cmd.Env = append(os.Environ(),
"GIT_AUTHOR_NAME=Unit Test",
"GIT_AUTHOR_EMAIL=test@test.com",
@@ -724,7 +644,6 @@ func TestAcquireCloneLock(t *testing.T) {
})
}
// An unresponsive remote must not pin a job: the refresh has to be interruptible.
func TestNewGitCloneExecutorFetchSendsTokenAndHonoursContext(t *testing.T) {
t.Parallel()
block := make(chan struct{})

View File

@@ -14,13 +14,8 @@ const jobErrorContextKeyVal = jobErrorContextKey("job.error")
// JobError returns the job error for current context if any
func JobError(ctx context.Context) error {
val := ctx.Value(jobErrorContextKeyVal)
if val != nil {
if container, ok := val.(map[string]error); ok {
return container["error"]
}
}
return nil
container, _ := ctx.Value(jobErrorContextKeyVal).(map[string]error)
return container["error"]
}
func SetJobError(ctx context.Context, err error) {
@@ -31,6 +26,5 @@ func SetJobError(ctx context.Context, err error) {
// WithJobErrorContainer adds a value to the context as a container for an error
func WithJobErrorContainer(ctx context.Context) context.Context {
container := map[string]error{}
return context.WithValue(ctx, jobErrorContextKeyVal, container)
return context.WithValue(ctx, jobErrorContextKeyVal, map[string]error{})
}

View File

@@ -26,9 +26,7 @@ type lineWriter struct {
// NewLineWriter creates a new instance of a line writer
func NewLineWriter(handlers ...LineHandler) io.Writer {
w := new(lineWriter)
w.handlers = handlers
return w
return &lineWriter{handlers: handlers}
}
// FlushWriter flushes w if it implements Flusher. It is a no-op otherwise, so
@@ -39,24 +37,15 @@ func FlushWriter(w io.Writer) {
}
}
func (lw *lineWriter) Write(p []byte) (n int, err error) {
pBuf := bytes.NewBuffer(p)
written := 0
for {
line, err := pBuf.ReadString('\n')
w, _ := lw.buffer.WriteString(line)
written += w
if err != nil {
if err == io.EOF {
break
}
return written, err
func (lw *lineWriter) Write(p []byte) (int, error) {
for line := range bytes.SplitAfterSeq(p, []byte{'\n'}) {
_, _ = lw.buffer.Write(line)
if len(line) > 0 && line[len(line)-1] == '\n' {
lw.handleLine(lw.buffer.String())
lw.buffer.Reset()
}
lw.handleLine(lw.buffer.String())
lw.buffer.Reset()
}
return written, nil
return len(p), nil
}
// Flush emits any buffered, not-yet-newline-terminated content as a final line.
@@ -72,8 +61,7 @@ func (lw *lineWriter) Flush() {
func (lw *lineWriter) handleLine(line string) {
for _, h := range lw.handlers {
ok := h(line)
if !ok {
if !h(line) {
break
}
}

View File

@@ -9,24 +9,24 @@ import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLineWriter(t *testing.T) {
lines := make([]string, 0)
lineHandler := func(s string) bool {
func TestLineWriterBuffersAndFlushesLines(t *testing.T) {
var lines []string
lineWriter := NewLineWriter(func(s string) bool {
lines = append(lines, s)
return true
}
lineWriter := NewLineWriter(lineHandler)
})
assert := assert.New(t)
write := func(s string) {
n, err := lineWriter.Write([]byte(s))
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
assert.Equal(len(s), n, s)
}
write("")
write("hello")
write(" ")
write("world!!\nextra")
@@ -34,39 +34,17 @@ func TestLineWriter(t *testing.T) {
write(" line\n")
write("no newline here...")
assert.Len(lines, 4)
assert.Equal("hello world!!\n", lines[0])
assert.Equal("extra line\n", lines[1])
assert.Equal(" and another\n", lines[2])
assert.Equal("last line\n", lines[3])
}
expected := []string{"hello world!!\n", "extra line\n", " and another\n", "last line\n"}
assert.Equal(expected, lines)
func TestLineWriterFlush(t *testing.T) {
lines := make([]string, 0)
lineHandler := func(s string) bool {
lines = append(lines, s)
return true
}
lineWriter := NewLineWriter(lineHandler)
assert := assert.New(t)
_, err := lineWriter.Write([]byte("complete line\npartial line without newline"))
assert.NoError(err) //nolint:testifylint // pre-existing pattern from nektos/act
// Only the newline-terminated line is emitted before flushing.
assert.Equal([]string{"complete line\n"}, lines)
// Flushing emits the buffered, not-yet-terminated trailing line.
FlushWriter(lineWriter)
assert.Equal([]string{"complete line\n", "partial line without newline"}, lines)
expected = append(expected, "no newline here...")
assert.Equal(expected, lines)
// Flushing again is a no-op: nothing is buffered.
FlushWriter(lineWriter)
assert.Len(lines, 2)
assert.Equal(expected, lines)
}
func TestFlushWriterIgnoresNonFlusher(t *testing.T) {
// FlushWriter must be a safe no-op for writers that do not buffer lines.
assert.NotPanics(t, func() { FlushWriter(io.Discard) })
}

View File

@@ -34,10 +34,8 @@ const loggerHandlerKeyVal = loggerHandlerKey("logging.Handler")
// LoggerHandler returns the extra handler the job logger sends its records to, or nil.
// It affects the job logger, not the global logger.
func LoggerHandler(ctx context.Context) logging.Handler {
if h, ok := ctx.Value(loggerHandlerKeyVal).(logging.Handler); ok {
return h
}
return nil
h, _ := ctx.Value(loggerHandlerKeyVal).(logging.Handler)
return h
}
// WithLoggerHandler adds the extra handler for the job logger to the context

View File

@@ -9,6 +9,7 @@ import (
"fmt"
"io"
"log/slog"
"maps"
"reflect"
"runtime"
"slices"
@@ -94,11 +95,7 @@ func (h *consoleHandler) Handle(_ context.Context, r slog.Record) error {
}
prefixFieldClashes(data, caller != nil)
keys := make([]string, 0, len(data))
for k := range data {
keys = append(keys, k)
}
slices.Sort(keys)
keys := slices.Sorted(maps.Keys(data))
var b bytes.Buffer
if terminal && !opts.DisableColors {

View File

@@ -9,8 +9,6 @@ import (
"os"
"runtime"
"sync/atomic"
"github.com/sirupsen/logrus"
)
// TextOptions describes how the process-wide Logger writes a record as text.
@@ -41,11 +39,8 @@ func Default() *Logger { return std.Load() }
// initial console Logger.
func SetDefault(l *Logger) { std.Store(l) }
// SetLevel sets the minimum level of the initial process-wide Logger and of what libraries log through logrus.
func SetLevel(level Level) {
stdState.level.Set(slog.Level(level))
logrus.SetLevel(toLogrusLevel(level))
}
// SetLevel sets the minimum level of the initial process-wide Logger.
func SetLevel(level Level) { stdState.level.Set(slog.Level(level)) }
// GetLevel returns the minimum level of the initial process-wide Logger.
func GetLevel() Level { return Level(stdState.level.Level()) }
@@ -72,19 +67,16 @@ func SetReportCaller(on bool) {
stdState.mu.Lock()
defer stdState.mu.Unlock()
stdState.reportCaller = on
logrus.SetReportCaller(on)
}
func Enabled(level Level) bool { return Default().Enabled(level) }
func With(args ...any) *Logger { return Default().With(args...) }
func WithError(err error) *Logger { return Default().WithError(err) }
func Tracef(format string, args ...any) { Default().logf(LevelTrace, format, args) }
func Debugf(format string, args ...any) { Default().logf(LevelDebug, format, args) }
func Infof(format string, args ...any) { Default().logf(LevelInfo, format, args) }
func Warnf(format string, args ...any) { Default().logf(LevelWarn, format, args) }
func Errorf(format string, args ...any) { Default().logf(LevelError, format, args) }
func Trace(args ...any) { Default().log(LevelTrace, args) }
func Debug(args ...any) { Default().log(LevelDebug, args) }
func Info(args ...any) { Default().log(LevelInfo, args) }
func Warn(args ...any) { Default().log(LevelWarn, args) }

View File

@@ -43,7 +43,6 @@ func (l *Logger) Enabled(level Level) bool {
return l.h.Enabled(l.ctx, level)
}
func (l *Logger) Tracef(format string, args ...any) { l.logf(LevelTrace, format, args) }
func (l *Logger) Debugf(format string, args ...any) { l.logf(LevelDebug, format, args) }
func (l *Logger) Infof(format string, args ...any) { l.logf(LevelInfo, format, args) }
func (l *Logger) Warnf(format string, args ...any) { l.logf(LevelWarn, format, args) }

View File

@@ -13,7 +13,6 @@ import (
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -39,6 +38,7 @@ func (c countingStringer) String() string { *c.n++; return "x" }
func TestLogger(t *testing.T) {
h := logtest.NewWithLevel(logging.LevelTrace)
assert.Nil(t, h.LastRecord())
l := logging.New(h)
l.With("job", "j1").WithError(errors.New("boom")).Warnf("step %d", 2)
@@ -54,12 +54,9 @@ func TestLogger(t *testing.T) {
assert.Equal(t, "j1", job)
errv, _ := recs[0].Attr("error")
require.EqualError(t, errv.(error), "boom")
assert.Equal(t, "a1 2b", recs[1].Message) // fmt.Sprint spacing
assert.Equal(t, "a1 2b", recs[1].Message)
bad, _ := recs[2].Attr("!BADKEY")
assert.Equal(t, 7, bad)
h.Reset()
assert.Nil(t, h.LastRecord())
}
func TestLoggerSkipsFormattingWhenDisabled(t *testing.T) {
@@ -118,10 +115,8 @@ func TestDefaultLevel(t *testing.T) {
func TestSlogHandler(t *testing.T) {
var buf strings.Builder
slogger := slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
l := logging.New(logging.NewSlogHandler(slogger.Handler())).With("job", "j1")
l := logging.New(logging.NewSlogHandler(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug}))).With("job", "j1")
l.Tracef("dropped")
assert.False(t, l.Enabled(logging.LevelTrace))
l.WithError(errors.New("boom")).Errorf("fail %s", "x")
@@ -132,43 +127,6 @@ func TestSlogHandler(t *testing.T) {
assert.Contains(t, out, "error=boom")
}
func TestLogrusBridge(t *testing.T) {
prevLevel, prevDefault := logging.GetLevel(), logging.Default()
t.Cleanup(func() {
logging.SetLevel(prevLevel)
logging.SetReportCaller(false)
logging.SetDefault(prevDefault)
})
logging.SetLevel(logging.LevelInfo)
assert.False(t, logrus.IsLevelEnabled(logrus.DebugLevel))
logging.SetLevel(logging.LevelTrace)
logging.SetReportCaller(true)
h := logtest.NewWithLevel(logging.LevelTrace)
logging.SetDefault(logging.New(h))
logrus.WithField("lib", "docker").Warnf("from a library %d", 1)
_, _, line, _ := runtime.Caller(0)
logrus.Debug("below")
logrus.Trace("lowest")
recs := h.AllRecords()
require.Len(t, recs, 3)
assert.Equal(t, "from a library 1", recs[0].Message)
assert.Equal(t, logging.LevelWarn, recs[0].Level)
v, _ := recs[0].Attr("lib")
assert.Equal(t, "docker", v)
assert.Equal(t, logging.LevelDebug, recs[1].Level)
f := frameOf(t, recs[1].PC)
assert.True(t, strings.HasSuffix(f.Function, "TestLogrusBridge"), f.Function)
assert.Equal(t, line+1, f.Line)
assert.Equal(t, logging.LevelTrace, recs[2].Level)
h = logtest.New() // logrus passes Debug, the Logger does not
logging.SetDefault(logging.New(h))
logrus.Debug("dropped")
assert.Empty(t, h.AllRecords())
}
func TestSlogBridge(t *testing.T) {
h := logtest.SetDefault(t)
slog.With("lib", "actionslib").Warn("from a library", "n", 1)
@@ -185,19 +143,3 @@ func TestSlogBridge(t *testing.T) {
assert.EqualValues(t, 1, n)
assert.Equal(t, line-1, frameOf(t, recs[0].PC).Line)
}
func TestLogrusBridgeIntact(t *testing.T) {
assert.True(t, logging.LogrusBridgeIntact())
prev := logrus.StandardLogger().Out
t.Cleanup(func() { logrus.SetOutput(prev) })
logrus.SetOutput(&strings.Builder{}) // what a dependency doing this would do
assert.False(t, logging.LogrusBridgeIntact())
logrus.SetOutput(prev)
assert.True(t, logging.LogrusBridgeIntact())
prevLevel := logrus.GetLevel()
t.Cleanup(func() { logrus.SetLevel(prevLevel) })
logrus.SetLevel(logrus.PanicLevel)
assert.False(t, logging.LogrusBridgeIntact())
}

View File

@@ -1,102 +0,0 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package logging
import (
"context"
"io"
"github.com/sirupsen/logrus"
)
// Libraries the runner depends on, such as docker/cli and moby/go-archive, log through
// logrus's global logger. Hand what they log to the process-wide Logger, so it follows the same
// level, format and secret masking as the rest of the output.
func init() {
logrus.SetOutput(io.Discard)
logrus.SetFormatter(discardFormatter{})
logrus.SetLevel(toLogrusLevel(GetLevel()))
logrus.AddHook(bridgeHook{})
}
type discardFormatter struct{}
func (discardFormatter) Format(*logrus.Entry) ([]byte, error) { return nil, nil }
type bridgeHook struct{}
func (bridgeHook) Levels() []logrus.Level { return logrus.AllLevels }
func (bridgeHook) Fire(e *logrus.Entry) error {
l := Default()
level := fromLogrusLevel(e.Level)
ctx := e.Context
if ctx == nil {
ctx = context.Background()
}
if !l.h.Enabled(ctx, level) {
return nil
}
attrs := make([]Attr, 0, len(e.Data))
for k, v := range e.Data {
attrs = append(attrs, Attr{Key: k, Value: v})
}
r := Record{Time: e.Time, Level: level, Message: e.Message, Attrs: attrs}
if e.Caller != nil {
r.PC = e.Caller.PC + 1 // Frame.PC is one below the return address Record.PC holds
}
return l.h.Handle(ctx, r)
}
func toLogrusLevel(l Level) logrus.Level {
switch {
case l <= LevelTrace:
return logrus.TraceLevel
case l <= LevelDebug:
return logrus.DebugLevel
case l <= LevelInfo:
return logrus.InfoLevel
case l <= LevelWarn:
return logrus.WarnLevel
default:
return logrus.ErrorLevel
}
}
func fromLogrusLevel(l logrus.Level) Level {
switch l {
case logrus.TraceLevel:
return LevelTrace
case logrus.DebugLevel:
return LevelDebug
case logrus.InfoLevel:
return LevelInfo
case logrus.WarnLevel:
return LevelWarn
default:
return LevelError
}
}
// LogrusBridgeIntact reports whether logrus's global logger is still set up as the bridge needs,
// discarding its own output and following the process-wide level. A dependency that calls
// logrus.SetOutput or SetLevel breaks that, and its lines then show up twice, or at the wrong level.
func LogrusBridgeIntact() bool {
std := logrus.StandardLogger()
if std.Out != io.Discard || std.GetLevel() != toLogrusLevel(GetLevel()) {
return false
}
for _, level := range logrus.AllLevels {
found := false
for _, h := range std.Hooks[level] {
if _, ok := h.(bridgeHook); ok {
found = true
}
}
if !found {
return false
}
}
return true
}

View File

@@ -72,12 +72,6 @@ func (h *Handler) LastRecord() *logging.Record {
return &r
}
func (h *Handler) Reset() {
h.store.mu.Lock()
defer h.store.mu.Unlock()
h.store.records = nil
}
// SetDefault makes the process-wide Logger record into a new Handler until the test ends.
func SetDefault(t testing.TB) *Handler {
t.Helper()

View File

@@ -12,7 +12,7 @@ import (
type interfaceIP struct {
net.IP
net.Interface
Name string
}
// GetOutboundIP returns an outbound IP address of this machine.
@@ -43,7 +43,7 @@ func GetOutboundIP() net.IP {
ip = v.IP
}
if ip.IsGlobalUnicast() {
ips = append(ips, interfaceIP{ip, i})
ips = append(ips, interfaceIP{ip, i.Name})
}
}
}
@@ -59,29 +59,25 @@ func preferredIP(ips []interfaceIP) net.IP {
return nil
}
sort.Slice(ips, func(i, j int) bool {
ifi := ips[i].Interface
ifj := ips[j].Interface
first, second := ips[i], ips[j]
// ethernet is preferred
if vi, vj := strings.HasPrefix(ifi.Name, "e"), strings.HasPrefix(ifj.Name, "e"); vi != vj {
if vi, vj := strings.HasPrefix(first.Name, "e"), strings.HasPrefix(second.Name, "e"); vi != vj {
return vi
}
ipi := ips[i].IP
ipj := ips[j].IP
// IPv4 is preferred
if vi, vj := ipi.To4() != nil, ipj.To4() != nil; vi != vj {
if vi, vj := first.To4() != nil, second.To4() != nil; vi != vj {
return vi
}
// en0 is preferred to en1
if ifi.Name != ifj.Name {
return ifi.Name < ifj.Name
if first.Name != second.Name {
return first.Name < second.Name
}
// fallback
return ipi.String() < ipj.String()
return first.String() < second.String()
})
return ips[0].IP
}

View File

@@ -7,20 +7,33 @@
package container
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"maps"
"net"
"net/url"
"os"
"os/exec"
"os/user"
"path/filepath"
"runtime"
"strings"
"gitea.com/gitea/runner/act/common"
"github.com/distribution/reference"
"github.com/docker/cli/cli/config"
"github.com/moby/moby/api/types/registry"
)
func LoadDockerAuthConfig(ctx context.Context, image string) (registry.AuthConfig, error) {
logger := common.Logger(ctx)
// config.LoadDefaultConfigFile panics on nil io.Writer when the config
// file is malformed; use config.Load to route errors through the logger.
cfg, err := config.Load(config.Dir())
cfg, err := loadDockerConfig(ctx)
if err != nil {
logger.Warnf("Could not load docker config: %v", err)
return registry.AuthConfig{}, err
@@ -34,32 +47,27 @@ func LoadDockerAuthConfig(ctx context.Context, image string) (registry.AuthConfi
}
}
authConfig, err := cfg.GetAuthConfig(registryKey)
authConfig, err := cfg.authConfig(ctx, registryKey)
if err != nil {
logger.Warnf("Could not get auth config from docker config: %v", err)
return registry.AuthConfig{}, err
}
return registry.AuthConfig(authConfig), nil
return authConfig, nil
}
func LoadDockerAuthConfigs(ctx context.Context) map[string]registry.AuthConfig {
logger := common.Logger(ctx)
cfg, err := config.Load(config.Dir())
cfg, err := loadDockerConfig(ctx)
if err != nil {
logger.Warnf("Could not load docker config: %v", err)
return nil
}
creds, err := cfg.GetAllCredentials()
authConfigs, err := cfg.allAuthConfigs(ctx)
if err != nil {
logger.Warnf("Could not get docker auth configs: %v", err)
return nil
}
authConfigs := make(map[string]registry.AuthConfig, len(creds))
for k, v := range creds {
authConfigs[k] = registry.AuthConfig(v)
}
return authConfigs
}
@@ -69,3 +77,226 @@ func registryAuthConfigKey(domainName string) string {
}
return domainName
}
type dockerConfig struct {
Auths map[string]registry.AuthConfig `json:"auths"`
CredsStore string `json:"credsStore"`
CredHelpers map[string]string `json:"credHelpers"`
env map[string]registry.AuthConfig // from DOCKER_AUTH_CONFIG, nil when unset or unusable
}
func loadDockerConfig(ctx context.Context) (*dockerConfig, error) {
dir := os.Getenv("DOCKER_CONFIG")
if dir == "" {
home, _ := os.UserHomeDir()
if home == "" && runtime.GOOS != "windows" {
if current, err := user.Current(); err == nil {
home = current.HomeDir
}
}
dir = filepath.Join(home, ".docker")
}
cfg := &dockerConfig{}
filename := filepath.Join(dir, "config.json")
file, err := os.Open(filename)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return nil, fmt.Errorf("loading config file: %w", err)
}
if err == nil {
defer file.Close()
if err := json.NewDecoder(file).Decode(cfg); err != nil && !errors.Is(err, io.EOF) {
return nil, fmt.Errorf("parsing config file (%s): %w", filename, err)
}
}
for address, auth := range cfg.Auths {
if auth.Auth != "" {
if auth.Username, auth.Password, err = decodeAuth(auth.Auth); err != nil {
return nil, fmt.Errorf("parsing config file (%s): %w", filename, err)
}
}
auth.Auth, auth.ServerAddress = "", address
cfg.Auths[address] = auth
}
if value := os.Getenv("DOCKER_AUTH_CONFIG"); value != "" {
if cfg.env, err = parseDockerAuthConfigEnv(value); err != nil {
common.Logger(ctx).Warnf("Failed to create credential store from DOCKER_AUTH_CONFIG: %v", err)
}
}
return cfg, nil
}
func decodeAuth(auth string) (string, string, error) {
decoded, err := base64.StdEncoding.DecodeString(auth)
if err != nil {
return "", "", err
}
username, password, ok := strings.Cut(string(decoded), ":")
if !ok || username == "" {
return "", "", errors.New("invalid auth configuration file")
}
return username, strings.Trim(password, "\x00"), nil
}
func parseDockerAuthConfigEnv(value string) (map[string]registry.AuthConfig, error) {
var env struct {
Auths map[string]struct {
Auth string `json:"auth"`
} `json:"auths"`
}
decoder := json.NewDecoder(strings.NewReader(value))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&env); err != nil && !errors.Is(err, io.EOF) {
return nil, err
}
if decoder.More() {
return nil, errors.New("DOCKER_AUTH_CONFIG does not support more than one JSON object")
}
auths := make(map[string]registry.AuthConfig, len(env.Auths))
for address, entry := range env.Auths {
if entry.Auth == "" {
return nil, fmt.Errorf("DOCKER_AUTH_CONFIG environment variable is missing key `auth` for %s", address)
}
username, password, err := decodeAuth(entry.Auth)
if err != nil {
return nil, err
}
auths[address] = registry.AuthConfig{Username: username, Password: password, ServerAddress: address}
}
return auths, nil
}
func (c *dockerConfig) authConfig(ctx context.Context, key string) (registry.AuthConfig, error) {
if auth, ok := c.env[key]; ok {
return auth, nil
}
helper := c.CredsStore
if named, ok := c.CredHelpers[key]; ok {
helper = named
}
if helper == "" {
return c.fileAuth(key), nil
}
auth := c.fileAuth(key)
stored, err := helperCredentials(ctx, helper, key)
if err != nil {
return auth, err
}
auth.Username, auth.Password, auth.IdentityToken, auth.ServerAddress = stored.Username, stored.Password, stored.IdentityToken, stored.ServerAddress
return auth, nil
}
func (c *dockerConfig) fileAuth(key string) registry.AuthConfig {
if auth, ok := c.Auths[key]; ok {
return auth
}
for address, auth := range c.Auths {
if registryHostname(address) == key {
return auth
}
}
return registry.AuthConfig{}
}
func (c *dockerConfig) allAuthConfigs(ctx context.Context) (map[string]registry.AuthConfig, error) {
var auths map[string]registry.AuthConfig
if c.CredsStore == "" {
auths = maps.Clone(c.Auths)
} else {
var err error
if auths, err = c.helperAuthConfigs(ctx); err != nil {
if c.env == nil {
return nil, err
}
common.Logger(ctx).Warnf("Could not list the credentials of docker-credential-%s: %v", c.CredsStore, err)
}
}
if auths == nil {
auths = map[string]registry.AuthConfig{}
}
maps.Copy(auths, c.env)
if c.CredsStore == "" {
c.Auths = maps.Clone(auths) // docker's file store holds the DOCKER_AUTH_CONFIG entries after listing, and credHelpers set to "" read it
}
for address := range c.CredHelpers {
auth, err := c.authConfig(ctx, registryAuthConfigKey(address))
if err != nil {
common.Logger(ctx).Warnf("Failed to get credentials for registry: %s: %v", address, err)
continue
}
auths[address] = auth
}
return auths, nil
}
func (c *dockerConfig) helperAuthConfigs(ctx context.Context) (map[string]registry.AuthConfig, error) {
out, err := runCredentialHelper(ctx, c.CredsStore, "list", "unused")
if err != nil {
return nil, fmt.Errorf("error listing credentials - err: %w, out: `%s`", err, strings.TrimSpace(string(out)))
}
var addresses map[string]string
if err := json.NewDecoder(bytes.NewReader(out)).Decode(&addresses); err != nil {
return nil, err
}
auths := make(map[string]registry.AuthConfig, len(addresses))
for address := range addresses {
stored, err := helperCredentials(ctx, c.CredsStore, address)
if err != nil {
return nil, err
}
auth := c.Auths[address]
auth.Username, auth.Password, auth.IdentityToken = stored.Username, stored.Password, stored.IdentityToken
if auth.ServerAddress == "" {
auth.ServerAddress = stored.ServerAddress
}
auths[address] = auth
}
return auths, nil
}
func helperCredentials(ctx context.Context, helper, address string) (registry.AuthConfig, error) {
out, err := runCredentialHelper(ctx, helper, "get", address)
if err != nil {
if strings.TrimSpace(string(out)) == "credentials not found in native keychain" {
return registry.AuthConfig{}, nil
}
return registry.AuthConfig{}, fmt.Errorf("error getting credentials - err: %w, out: `%s`", err, strings.TrimSpace(string(out)))
}
var stored struct{ Username, Secret string }
if err := json.NewDecoder(bytes.NewReader(out)).Decode(&stored); err != nil {
return registry.AuthConfig{}, err
}
if stored.Username == "<token>" {
return registry.AuthConfig{IdentityToken: stored.Secret, ServerAddress: address}, nil
}
return registry.AuthConfig{Username: stored.Username, Password: stored.Secret, ServerAddress: address}, nil
}
func runCredentialHelper(ctx context.Context, helper, action, input string) ([]byte, error) {
cmd := exec.CommandContext(ctx, "docker-credential-"+helper, action)
cmd.Stdin = strings.NewReader(input)
cmd.Stderr = os.Stderr
return cmd.Output()
}
func registryHostname(address string) string {
if _, rest, ok := strings.Cut(address, "://"); ok {
if u, err := url.Parse(address); err == nil && u.Hostname() != "" {
if u.Port() == "" {
return u.Hostname()
}
return net.JoinHostPort(u.Hostname(), u.Port())
}
if host, _, _ := strings.Cut(rest, "/"); host != "" {
if colon := strings.LastIndex(host, ":"); strings.Count(host, ":") > 1 && !strings.HasPrefix(host, "[") && isDigits(host[colon+1:]) {
return net.JoinHostPort(host[:colon], host[colon+1:])
}
return host
}
}
host, _, _ := strings.Cut(address, "/")
return host
}
func isDigits(s string) bool {
return s != "" && strings.Trim(s, "0123456789") == ""
}

View File

@@ -0,0 +1,46 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !windows && !(WITHOUT_DOCKER || !(linux || darwin || netbsd))
package container
import (
"encoding/base64"
"os"
"path/filepath"
"testing"
"github.com/moby/moby/api/types/registry"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLoadDockerAuthConfigFromFileHelpersAndEnv(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "docker-credential-fake"), []byte("#!/bin/sh\nread -r server\n[ \"$1\" = get ] && [ \"$server\" = helper.example.com ] && exec echo '{\"Username\":\"helped\",\"Secret\":\"pw\"}'\necho 'credentials not found in native keychain'\nexit 1\n"), 0o755))
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
auth := func(credentials string) string { return base64.StdEncoding.EncodeToString([]byte(credentials)) }
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte(`{"auths": {"https://legacy.example.com/v1/": {"auth": "`+auth("legacy:pw")+`"}}, "credHelpers": {"helper.example.com": "fake", "gone.example.com": "fake", "env-legacy.example.com": ""}}`), 0o600))
t.Setenv("DOCKER_CONFIG", dir)
t.Setenv("DOCKER_AUTH_CONFIG", `{"auths": {"env.example.com": {"auth": "`+auth("env:pw")+`"}, "https://env-legacy.example.com/v1/": {"auth": "`+auth("env:pw")+`"}}}`)
for image, want := range map[string]registry.AuthConfig{
"legacy.example.com/app": {Username: "legacy", Password: "pw", ServerAddress: "https://legacy.example.com/v1/"},
"helper.example.com/app": {Username: "helped", Password: "pw", ServerAddress: "helper.example.com"},
"env.example.com/app": {Username: "env", Password: "pw", ServerAddress: "env.example.com"},
"gone.example.com/app": {},
} {
got, err := LoadDockerAuthConfig(t.Context(), image)
require.NoError(t, err)
assert.Equal(t, want, got, image)
}
assert.Equal(t, map[string]registry.AuthConfig{
"https://legacy.example.com/v1/": {Username: "legacy", Password: "pw", ServerAddress: "https://legacy.example.com/v1/"},
"helper.example.com": {Username: "helped", Password: "pw", ServerAddress: "helper.example.com"},
"gone.example.com": {},
"env.example.com": {Username: "env", Password: "pw", ServerAddress: "env.example.com"},
"https://env-legacy.example.com/v1/": {Username: "env", Password: "pw", ServerAddress: "https://env-legacy.example.com/v1/"},
"env-legacy.example.com": {Username: "env", Password: "pw", ServerAddress: "https://env-legacy.example.com/v1/"},
}, LoadDockerAuthConfigs(t.Context()))
}

View File

@@ -7,15 +7,18 @@
package container
import (
"archive/tar"
"context"
"io"
"io/fs"
"os"
"path/filepath"
"runtime"
"strings"
"time"
"gitea.com/gitea/runner/act/common"
"github.com/moby/go-archive"
"github.com/moby/go-archive/compression"
"github.com/moby/moby/client"
"github.com/moby/patternmatcher"
"github.com/moby/patternmatcher/ignorefile"
@@ -43,9 +46,8 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
logger.Debugf("Building image from '%v'", input.ContextDir)
tags := []string{input.ImageTag}
options := client.ImageBuildOptions{
Tags: tags,
Tags: []string{input.ImageTag},
Remove: true,
AuthConfigs: LoadDockerAuthConfigs(ctx),
Dockerfile: input.Dockerfile,
@@ -72,12 +74,10 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
logger.Debugf("Creating image from context dir '%s' with tag '%s' and platform '%s'", input.ContextDir, input.ImageTag, input.Platform)
resp, err := cli.ImageBuild(ctx, buildContext, options)
err = logDockerResponse(logger, resp.Body, err != nil)
if err != nil {
return err
}
return nil
return logDockerResponse(logger, resp.Body, false)
}
}
@@ -115,14 +115,140 @@ func createBuildContext(ctx context.Context, contextDir, relDockerfile string) (
includes = append(includes, ".dockerignore", relDockerfile)
}
buildCtx, err := archive.TarWithOptions(contextDir, &archive.TarOptions{
Compression: compression.None,
ExcludePatterns: excludes,
IncludeFiles: includes,
})
pm, err := patternmatcher.New(excludes)
if err != nil {
return nil, err
}
return buildCtx, nil
reader, writer := io.Pipe()
go func() { writer.CloseWithError(writeBuildContext(ctx, writer, contextDir, includes, pm)) }()
return reader, nil
}
func writeBuildContext(ctx context.Context, w io.Writer, contextDir string, includes []string, pm *patternmatcher.PatternMatcher) error {
tw := tar.NewWriter(w)
seen := map[string]bool{}
hardlinks := map[uint64]string{}
buffer := make([]byte, 32*1024)
for _, include := range includes {
include = filepath.FromSlash(include)
var parents []parentMatch
root := strings.TrimSuffix(contextDir, string(filepath.Separator)) + string(filepath.Separator) + include // unlike filepath.Join, follows a symlinked context dir as docker build does
err := filepath.WalkDir(root, func(file string, entry fs.DirEntry, err error) error {
if err != nil {
return leaveOut(ctx, file, err)
}
rel, err := filepath.Rel(contextDir, file)
if err != nil || rel == "." {
return nil
}
if rel != include {
for len(parents) > 0 && !strings.HasPrefix(rel, parents[len(parents)-1].dir+string(filepath.Separator)) {
parents = parents[:len(parents)-1]
}
var parentInfo patternmatcher.MatchInfo
if len(parents) > 0 {
parentInfo = parents[len(parents)-1].info
}
skip, info, err := pm.MatchesUsingParentResults(rel, parentInfo)
if err != nil {
return err
}
if entry.IsDir() {
parents = append(parents, parentMatch{rel, info})
}
if skip {
if entry.IsDir() && !hasExceptionWithin(pm, rel) {
return filepath.SkipDir
}
return nil
}
}
if seen[rel] {
return nil
}
seen[rel] = true
return addBuildContextEntry(ctx, tw, file, rel, entry, hardlinks, buffer)
})
if err != nil {
return err
}
}
return tw.Close()
}
type parentMatch struct {
dir string
info patternmatcher.MatchInfo
}
func hasExceptionWithin(pm *patternmatcher.PatternMatcher, dir string) bool {
dirPrefix := dir + string(filepath.Separator)
for _, pattern := range pm.Patterns() {
if pattern.Exclusion() && strings.HasPrefix(pattern.String()+string(filepath.Separator), dirPrefix) {
return true
}
}
return false
}
func leaveOut(ctx context.Context, file string, err error) error {
common.Logger(ctx).Warnf("Leaving %s out of the build context: %v", file, err)
return nil
}
func addBuildContextEntry(ctx context.Context, tw *tar.Writer, file, rel string, entry fs.DirEntry, hardlinks map[uint64]string, buffer []byte) error {
info, err := entry.Info()
if err != nil {
return leaveOut(ctx, file, err)
}
if info.Mode()&fs.ModeSocket != 0 {
return nil // tar has no socket type
}
var link string
if info.Mode()&fs.ModeSymlink != 0 {
if link, err = os.Readlink(file); err != nil {
return leaveOut(ctx, file, err)
}
}
header, err := tar.FileInfoHeader(noNameLookups{info}, link)
if err != nil {
return err
}
header.Name = filepath.ToSlash(rel)
inode, linked := hardlinkInode(info)
if target, ok := hardlinks[inode]; linked && ok {
header.Typeflag, header.Linkname, header.Size = tar.TypeLink, target, 0
}
var content *os.File
if header.Typeflag == tar.TypeReg && header.Size > 0 {
if content, err = os.Open(file); err != nil {
return leaveOut(ctx, file, err)
}
defer content.Close()
}
if linked && header.Typeflag != tar.TypeLink {
hardlinks[inode] = header.Name
}
if info.IsDir() {
header.Name += "/"
}
header.Format = tar.FormatPAX
header.ModTime = header.ModTime.Truncate(time.Second)
header.AccessTime, header.ChangeTime = time.Time{}, time.Time{}
if runtime.GOOS == "windows" {
header.Mode = header.Mode&0o755 | 0o111 // Windows has no exec bit, so mark everything executable like docker build
}
if err := tw.WriteHeader(header); err != nil {
return err
}
if content != nil {
_, err = io.CopyBuffer(tw, struct{ io.Reader }{content}, buffer) // hides os.File's WriteTo, which would allocate a buffer per file
}
return err
}
type noNameLookups struct{ fs.FileInfo } // leaves Uname and Gname empty like docker build, so packing never waits on NSS
func (noNameLookups) Uname() (string, error) { return "", nil }
func (noNameLookups) Gname() (string, error) { return "", nil }

View File

@@ -0,0 +1,71 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
package container
import (
"archive/tar"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/require"
)
func TestDockerBuildReportsDaemonError(t *testing.T) {
daemon := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
writer.Header().Set("API-Version", "1.47")
if strings.HasSuffix(request.URL.Path, "/build") {
writer.Header().Set("Content-Type", "application/json")
writer.WriteHeader(http.StatusInternalServerError)
_, _ = io.WriteString(writer, `{"message":"build rejected"}`)
}
}))
t.Cleanup(daemon.Close)
t.Setenv("DOCKER_HOST", daemon.URL)
t.Setenv("DOCKER_CONFIG", t.TempDir())
t.Setenv("DOCKER_AUTH_CONFIG", "")
require.ErrorContains(t, NewDockerBuildExecutor(NewDockerBuildExecutorInput{
ImageTag: "action:test", Dockerfile: "Dockerfile", BuildContext: strings.NewReader(""),
})(t.Context()), "build rejected")
}
func TestCreateBuildContextFollowsDockerignoreAndSymlinkedRootButKeepsDockerfile(t *testing.T) {
dir := t.TempDir()
for name, content := range map[string]string{
".dockerignore": "*.log\nbuild\n!build/keep.txt\nDockerfile", "Dockerfile": "FROM scratch", "a.txt": "a",
"a.log": "log", "build/drop.txt": "drop", "build/keep.txt": "keep",
} {
require.NoError(t, os.MkdirAll(filepath.Join(dir, filepath.Dir(name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644))
}
require.NoError(t, os.Mkdir(filepath.Join(dir, "empty"), 0o755))
contexts := []string{dir}
if link := filepath.Join(t.TempDir(), "link"); os.Symlink(dir, link) == nil {
contexts = append(contexts, link+string(filepath.Separator))
}
for _, contextDir := range contexts {
buildContext, err := createBuildContext(t.Context(), contextDir, "Dockerfile")
require.NoError(t, err)
var names []string
for reader := tar.NewReader(buildContext); ; {
header, err := reader.Next()
if err == io.EOF {
break
}
require.NoError(t, err)
names = append(names, header.Name)
}
require.NoError(t, buildContext.Close())
require.Equal(t, []string{".dockerignore", "a.txt", "build/keep.txt", "empty/", "Dockerfile"}, names, contextDir)
}
}

View File

@@ -0,0 +1,18 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd)
package container
import (
"io/fs"
"syscall"
)
func hardlinkInode(info fs.FileInfo) (uint64, bool) {
if stat, ok := info.Sys().(*syscall.Stat_t); ok && !info.IsDir() && stat.Nlink > 1 {
return stat.Ino, true
}
return 0, false
}

View File

@@ -0,0 +1,35 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd)
package container
import (
"archive/tar"
"os"
"path/filepath"
"syscall"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestCreateBuildContextKeepsHardlinkedSpecialFilesLinked(t *testing.T) {
dir := t.TempDir()
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "first"), 0o644))
require.NoError(t, os.Link(filepath.Join(dir, "first"), filepath.Join(dir, "second")))
buildContext, err := createBuildContext(t.Context(), dir, "Dockerfile")
require.NoError(t, err)
defer buildContext.Close()
reader := tar.NewReader(buildContext)
first, err := reader.Next()
require.NoError(t, err)
assert.Equal(t, byte(tar.TypeFifo), first.Typeflag)
second, err := reader.Next()
require.NoError(t, err)
assert.Equal(t, byte(tar.TypeLink), second.Typeflag)
assert.Equal(t, first.Name, second.Linkname)
}

View File

@@ -0,0 +1,10 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER
package container
import "io/fs"
func hardlinkInode(_info fs.FileInfo) (uint64, bool) { return 0, false }

View File

@@ -4,11 +4,12 @@
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
// This file is exact copy of https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts.go with:
// This file is based on https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts.go with:
// * appended with license information
// * regexp and loader.ParseVolume in place of the import-restricted internal/lazyregexp and internal/volumespec
// * regexp and ParseVolume in place of the import-restricted internal/lazyregexp and internal/volumespec
// * cdiDeviceName in place of the container-device-interface parser's IsQualifiedName
// * jsontext in place of json.Compact for the seccomp profile
// * invalidParameter from the package's errors.go, and convertPortSet/convertPortMap for the callers in docker_run.go
// * invalidParameter from the package's errors.go, and convertPortSet/convertPortMap for nat port types
//
// docker/cli is licensed under the Apache License, Version 2.0.
// See DOCKER_LICENSE for the full license text.
@@ -33,14 +34,12 @@ import (
"time"
cerrdefs "github.com/containerd/errdefs"
"github.com/docker/cli/cli/compose/loader"
"github.com/docker/cli/opts"
"github.com/docker/go-connections/nat"
"github.com/moby/moby/api/types/container"
"github.com/moby/moby/api/types/mount"
"github.com/moby/moby/api/types/network"
"github.com/spf13/pflag"
cdi "tags.cncf.io/container-device-interface/pkg/parser"
)
const (
@@ -155,9 +154,6 @@ type containerOptions struct {
init bool
umask opts.UmaskOpt
annotations *opts.MapOpts
Image string
Args []string
}
// addFlags adds all command line flags that will be used by parse to the FlagSet
@@ -388,7 +384,7 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
volumes := copts.volumes.GetMap()
// add any bind targets to the list of container volumes
for bind := range volumes {
parsed, err := loader.ParseVolume(bind)
parsed, err := ParseVolume(bind)
if err != nil {
return nil, err
}
@@ -396,7 +392,7 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
if parsed.Source != "" {
toBind := bind
if parsed.Type == string(mount.TypeBind) {
if parsed.Type == mount.TypeBind {
if hostPart, targetPath, ok := strings.Cut(bind, ":"); ok {
if !filepath.IsAbs(hostPart) && strings.HasPrefix(hostPart, ".") {
if absHostPart, err := filepath.Abs(hostPart); err == nil {
@@ -423,12 +419,7 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
tmpfs[k] = v
}
var runCmd, entrypoint []string
if len(copts.Args) > 0 {
runCmd = copts.Args
}
var entrypoint []string
if copts.entrypoint != "" {
entrypoint = []string{copts.entrypoint}
} else if flags.Changed("entrypoint") {
@@ -449,36 +440,15 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
if err != nil {
return nil, err
}
portBindings := network.PortMap{}
for port, bindings := range natPortBindings {
p, err := network.ParsePort(string(port))
if err != nil {
return nil, err
}
portBindings[p] = []network.PortBinding{}
for _, b := range bindings {
var hostIP netip.Addr
if b.HostIP != "" {
hostIP, err = netip.ParseAddr(b.HostIP)
if err != nil {
return nil, err
}
}
portBindings[p] = append(portBindings[p], network.PortBinding{
HostIP: hostIP,
HostPort: b.HostPort,
})
}
portBindings, err := convertPortMap(natPortBindings)
if err != nil {
return nil, err
}
// Add published ports as exposed ports.
exposedPorts := network.PortSet{}
for port := range ports {
p, err := network.ParsePort(string(port))
if err != nil {
return nil, err
}
exposedPorts[p] = struct{}{}
exposedPorts, err := convertPortSet(ports)
if err != nil {
return nil, err
}
// Merge in exposed ports to the map of published ports
@@ -504,7 +474,7 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
deviceMappings := make([]container.DeviceMapping, 0, len(devices))
cdiDeviceNames := make([]string, 0, len(devices))
for _, device := range devices {
if cdi.IsQualifiedName(device) {
if cdiDeviceName.MatchString(device) {
cdiDeviceNames = append(cdiDeviceNames, device)
continue
}
@@ -619,11 +589,10 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
deviceRequests := copts.gpus.Value()
if len(cdiDeviceNames) > 0 {
cdiDeviceRequest := container.DeviceRequest{
deviceRequests = append(deviceRequests, container.DeviceRequest{
Driver: "cdi",
DeviceIDs: cdiDeviceNames,
}
deviceRequests = append(deviceRequests, cdiDeviceRequest)
})
}
resources := container.Resources{
@@ -669,8 +638,6 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
AttachStdout: attachStdout,
AttachStderr: attachStderr,
Env: envVariables,
Cmd: runCmd,
Image: copts.Image,
Volumes: volumes,
Entrypoint: entrypoint,
WorkingDir: copts.workingDir,
@@ -811,10 +778,8 @@ func parseNetworkOpts(copts *containerOptions) (map[string]*network.EndpointSett
// For backward compatibility: if no custom options are provided for the network,
// and only a single network is specified, omit the endpoint-configuration
// on the client (the daemon will still create it when creating the container)
if i == 0 && len(copts.netMode.Value()) == 1 {
if ep == nil || reflect.ValueOf(*ep).IsZero() {
continue
}
if i == 0 && len(copts.netMode.Value()) == 1 && reflect.ValueOf(*ep).IsZero() {
continue
}
endpoints[n.Target] = ep
}
@@ -845,14 +810,12 @@ func applyContainerOptions(n *opts.NetworkAttachmentOpts, copts *containerOption
return invalidParameter(errors.New("conflicting options: cannot specify both --link-local-ip and per-network link-local IP addresses"))
}
if copts.aliases.Len() > 0 {
n.Aliases = make([]string, copts.aliases.Len())
copy(n.Aliases, copts.aliases.GetSlice())
n.Aliases = slices.Clone(copts.aliases.GetSlice())
}
// For a user-defined network, "--link" is an endpoint option, it creates an alias. But,
// for the default bridge it defines a legacy-link.
if container.NetworkMode(n.Target).IsUserDefined() && copts.links.Len() > 0 {
n.Links = make([]string, copts.links.Len())
copy(n.Links, copts.links.GetSlice())
n.Links = slices.Clone(copts.links.GetSlice())
}
if copts.ipv4Address != nil {
if ipv4, ok := netip.AddrFromSlice(copts.ipv4Address.To4()); ok {
@@ -891,7 +854,6 @@ func parseNetworkAttachmentOpt(ep opts.NetworkAttachmentOpts) (*network.Endpoint
}
epConfig.Aliases = append(epConfig.Aliases, ep.Aliases...)
if len(ep.DriverOpts) > 0 {
epConfig.DriverOpts = make(map[string]string)
epConfig.DriverOpts = ep.DriverOpts
}
if len(ep.Links) > 0 {
@@ -935,11 +897,10 @@ func convertToStandardNotation(ports []string) ([]string, error) {
}
func parseLoggingOpts(loggingDriver string, loggingOpts []string) (map[string]string, error) {
loggingOptsMap := opts.ConvertKVStringsToMap(loggingOpts)
if loggingDriver == "none" && len(loggingOpts) > 0 {
return map[string]string{}, fmt.Errorf("invalid logging opts for driver %s", loggingDriver)
}
return loggingOptsMap, nil
return opts.ConvertKVStringsToMap(loggingOpts), nil
}
// takes a local seccomp daemon, reads the file contents for sending to the daemon
@@ -1197,6 +1158,8 @@ func convertPortSet(ports nat.PortSet) (network.PortSet, error) {
return converted, nil
}
var cdiDeviceName = regexp.MustCompile(`^[a-zA-Z]([a-zA-Z0-9_.-]*[a-zA-Z0-9])?/[a-zA-Z]([a-zA-Z0-9_.-]*[a-zA-Z0-9])?=[a-zA-Z0-9]([a-zA-Z0-9_.:-]*[a-zA-Z0-9])?$`)
func convertPortMap(bindings nat.PortMap) (network.PortMap, error) {
converted := make(network.PortMap, len(bindings))
for port, portBindings := range bindings {

View File

@@ -2,7 +2,7 @@
// Copyright 2022 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
// This file is exact copy of https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts_test.go with:
// This file is based on https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts_test.go with:
// * appended with license information
// * added tests for the locally changed parseDevice, validateDevice and invalidParameter
//
@@ -25,13 +25,11 @@ import (
"testing"
"time"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/moby/moby/api/types/container"
networktypes "github.com/moby/moby/api/types/network"
"github.com/spf13/pflag"
"gotest.tools/v3/assert"
is "gotest.tools/v3/assert/cmp"
"gotest.tools/v3/skip"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func mustParseMAC(s string) networktypes.HardwareAddr {
@@ -71,7 +69,6 @@ func parseRun(args []string) (*container.Config, *container.HostConfig, *network
if err := flags.Parse(args); err != nil {
return nil, nil, nil, err
}
// TODO(dnephin): fix tests to accept ContainerConfig; see https://github.com/moby/moby/pull/31621
containerCfg, err := parse(flags, copts, runtime.GOOS)
if err != nil {
return nil, nil, nil, err
@@ -83,14 +80,13 @@ func setupRunFlags() (*pflag.FlagSet, *containerOptions) {
flags := pflag.NewFlagSet("run", pflag.ContinueOnError)
flags.SetOutput(io.Discard)
flags.Usage = nil
copts := addFlags(flags)
return flags, copts
return flags, addFlags(flags)
}
func mustParse(t *testing.T, args string) (*container.Config, *container.HostConfig, *networktypes.NetworkingConfig) {
t.Helper()
config, hostConfig, nwConfig, err := parseRun(append(strings.Split(args, " "), "ubuntu", "bash"))
assert.NilError(t, err)
config, hostConfig, nwConfig, err := parseRun(strings.Split(args, " "))
require.NoError(t, err)
return config, hostConfig, nwConfig
}
@@ -156,12 +152,12 @@ func TestParseRunLinks(t *testing.T) {
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
_, hostConfig, netConfig := mustParse(t, tc.input)
assert.Check(t, is.DeepEqual(hostConfig.Links, tc.expHostConfigLinks))
assert.Check(t, is.Len(netConfig.EndpointsConfig, len(tc.expNetConfigLinks)))
assert.Equal(t, tc.expHostConfigLinks, hostConfig.Links)
assert.Len(t, netConfig.EndpointsConfig, len(tc.expNetConfigLinks))
for netName, expLinks := range tc.expNetConfigLinks {
nc, ok := netConfig.EndpointsConfig[netName]
assert.Assert(t, ok)
assert.Check(t, is.DeepEqual(nc.Links, expLinks))
require.True(t, ok)
assert.Equal(t, expLinks, nc.Links)
}
})
}
@@ -169,65 +165,33 @@ func TestParseRunLinks(t *testing.T) {
func TestParseRunWithoutUmask(t *testing.T) {
_, hostConfig, _, err := parseRun([]string{"ubuntu", "bash"})
assert.NilError(t, err)
assert.Assert(t, hostConfig.Umask == nil)
require.NoError(t, err)
require.Nil(t, hostConfig.Umask)
}
func TestParseRunUmask(t *testing.T) {
_, hostConfig, _, err := parseRun([]string{"--umask", "0022", "ubuntu", "bash"})
assert.NilError(t, err)
assert.Assert(t, hostConfig.Umask != nil)
assert.Equal(t, uint32(0o22), *hostConfig.Umask)
require.NoError(t, err)
require.NotNil(t, hostConfig.Umask)
require.Equal(t, uint32(0o22), *hostConfig.Umask)
}
func TestParseRunAttach(t *testing.T) {
tests := []struct {
input string
expected container.Config
for _, tc := range []struct {
input string
stdin, stdout, stderr bool
}{
{
input: "",
expected: container.Config{
AttachStdout: true,
AttachStderr: true,
},
},
{
input: "-i",
expected: container.Config{
AttachStdin: true,
AttachStdout: true,
AttachStderr: true,
},
},
{
input: "-a stdin",
expected: container.Config{
AttachStdin: true,
},
},
{
input: "-a stdin -a stdout",
expected: container.Config{
AttachStdin: true,
AttachStdout: true,
},
},
{
input: "-a stdin -a stdout -a stderr",
expected: container.Config{
AttachStdin: true,
AttachStdout: true,
AttachStderr: true,
},
},
}
for _, tc := range tests {
{"", false, true, true},
{"-i", true, true, true},
{"-a stdin", true, false, false},
{"-a stdin -a stdout", true, true, false},
{"-a stdin -a stdout -a stderr", true, true, true},
} {
t.Run(tc.input, func(t *testing.T) {
config, _, _ := mustParse(t, tc.input)
assert.Equal(t, config.AttachStdin, tc.expected.AttachStdin)
assert.Equal(t, config.AttachStdout, tc.expected.AttachStdout)
assert.Equal(t, config.AttachStderr, tc.expected.AttachStderr)
require.Equal(t, tc.stdin, config.AttachStdin)
require.Equal(t, tc.stdout, config.AttachStdout)
require.Equal(t, tc.stderr, config.AttachStderr)
})
}
}
@@ -273,109 +237,52 @@ func TestParseRunWithInvalidArgs(t *testing.T) {
flags, _ := setupRunFlags()
for _, tc := range tests {
t.Run(strings.Join(tc.args, " "), func(t *testing.T) {
assert.Error(t, flags.Parse(tc.args), tc.error)
require.EqualError(t, flags.Parse(tc.args), tc.error)
})
}
}
func TestParseWithVolumes(t *testing.T) { //nolint:gocyclo // verbatim copy from docker/cli tests
// A single volume
arr, tryit := setupPlatformVolume([]string{`/tmp`}, []string{`c:\tmp`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds != nil {
t.Fatalf("Error parsing volume flags, %q should not mount-bind anything. Received %v", tryit, hostConfig.Binds)
} else if _, exists := config.Volumes[arr[0]]; !exists {
t.Fatalf("Error parsing volume flags, %q is missing from volumes. Received %v", tryit, config.Volumes)
}
// Two volumes
arr, tryit = setupPlatformVolume([]string{`/tmp`, `/var`}, []string{`c:\tmp`, `c:\var`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds != nil {
t.Fatalf("Error parsing volume flags, %q should not mount-bind anything. Received %v", tryit, hostConfig.Binds)
} else if _, exists := config.Volumes[arr[0]]; !exists {
t.Fatalf("Error parsing volume flags, %s is missing from volumes. Received %v", arr[0], config.Volumes)
} else if _, exists := config.Volumes[arr[1]]; !exists {
t.Fatalf("Error parsing volume flags, %s is missing from volumes. Received %v", arr[1], config.Volumes)
}
// A single bind mount
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp`}, []string{os.Getenv("TEMP") + `:c:\containerTmp`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || hostConfig.Binds[0] != arr[0] {
t.Fatalf("Error parsing volume flags, %q should mount-bind the path before the colon into the path after the colon. Received %v %v", arr[0], hostConfig.Binds, config.Volumes)
}
// Two bind mounts.
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp`, `/hostVar:/containerVar`}, []string{os.Getenv("ProgramData") + `:c:\ContainerPD`, os.Getenv("TEMP") + `:c:\containerTmp`})
if _, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || compareRandomizedStrings(hostConfig.Binds[0], hostConfig.Binds[1], arr[0], arr[1]) != nil {
t.Fatalf("Error parsing volume flags, `%s and %s` did not mount-bind correctly. Received %v", arr[0], arr[1], hostConfig.Binds)
}
// Two bind mounts, first read-only, second read-write.
// TODO Windows: The Windows version uses read-write as that's the only mode it supports. Can change this post TP4
arr, tryit = setupPlatformVolume(
[]string{`/hostTmp:/containerTmp:ro`, `/hostVar:/containerVar:rw`},
[]string{os.Getenv("TEMP") + `:c:\containerTmp:rw`, os.Getenv("ProgramData") + `:c:\ContainerPD:rw`})
if _, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || compareRandomizedStrings(hostConfig.Binds[0], hostConfig.Binds[1], arr[0], arr[1]) != nil {
t.Fatalf("Error parsing volume flags, `%s and %s` did not mount-bind correctly. Received %v", arr[0], arr[1], hostConfig.Binds)
}
// Similar to previous test but with alternate modes which are only supported by Linux
if runtime.GOOS != "windows" {
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp:ro,Z`, `/hostVar:/containerVar:rw,Z`}, []string{})
if _, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || compareRandomizedStrings(hostConfig.Binds[0], hostConfig.Binds[1], arr[0], arr[1]) != nil {
t.Fatalf("Error parsing volume flags, `%s and %s` did not mount-bind correctly. Received %v", arr[0], arr[1], hostConfig.Binds)
func TestParseWithVolumes(t *testing.T) {
for _, tc := range []struct {
unix, windows []string
binds int
}{
{[]string{`/tmp`}, []string{`c:\tmp`}, 0},
{[]string{`/tmp`, `/var`}, []string{`c:\tmp`, `c:\var`}, 0},
{[]string{`/hostTmp:/containerTmp`}, []string{os.Getenv("TEMP") + `:c:\containerTmp`}, 1},
{[]string{`/hostTmp:/containerTmp`, `/hostVar:/containerVar`}, []string{os.Getenv("ProgramData") + `:c:\ContainerPD`, os.Getenv("TEMP") + `:c:\containerTmp`}, 2},
{[]string{`/hostTmp:/containerTmp:ro`, `/hostVar:/containerVar:rw`}, []string{os.Getenv("TEMP") + `:c:\containerTmp:rw`, os.Getenv("ProgramData") + `:c:\ContainerPD:rw`}, 2},
{[]string{`/hostTmp:/containerTmp:ro,Z`, `/hostVar:/containerVar:rw,Z`}, nil, 2},
{[]string{`/hostTmp:/containerTmp:Z`, `/hostVar:/containerVar:z`}, nil, 2},
{[]string{`/hostTmp:/containerTmp`, `/containerVar`}, []string{os.Getenv("TEMP") + `:c:\containerTmp`, `c:\containerTmp`}, 1},
{nil, []string{os.Getenv("SystemDrive") + `\:d:`}, 1},
} {
volumes := tc.unix
if runtime.GOOS == "windows" {
volumes = tc.windows
}
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp:Z`, `/hostVar:/containerVar:z`}, []string{})
if _, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || compareRandomizedStrings(hostConfig.Binds[0], hostConfig.Binds[1], arr[0], arr[1]) != nil {
t.Fatalf("Error parsing volume flags, `%s and %s` did not mount-bind correctly. Received %v", arr[0], arr[1], hostConfig.Binds)
if len(volumes) == 0 {
continue
}
t.Run(strings.Join(volumes, ","), func(t *testing.T) {
var args []string
for _, volume := range volumes {
args = append(args, "-v", volume)
}
config, hostConfig, _, err := parseRun(args)
require.NoError(t, err)
if tc.binds == 0 {
assert.Nil(t, hostConfig.Binds)
} else {
assert.ElementsMatch(t, volumes[:tc.binds], hostConfig.Binds)
}
expectedVolumes := map[string]struct{}{}
for _, volume := range volumes[tc.binds:] {
expectedVolumes[volume] = struct{}{}
}
assert.Equal(t, expectedVolumes, config.Volumes)
})
}
// One bind mount and one volume
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp`, `/containerVar`}, []string{os.Getenv("TEMP") + `:c:\containerTmp`, `c:\containerTmp`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || len(hostConfig.Binds) > 1 || hostConfig.Binds[0] != arr[0] {
t.Fatalf("Error parsing volume flags, %s and %s should only one and only one bind mount %s. Received %s", arr[0], arr[1], arr[0], hostConfig.Binds)
} else if _, exists := config.Volumes[arr[1]]; !exists {
t.Fatalf("Error parsing volume flags %s and %s. %s is missing from volumes. Received %v", arr[0], arr[1], arr[1], config.Volumes)
}
// Root to non-c: drive letter (Windows specific)
if runtime.GOOS == "windows" {
arr, tryit = setupPlatformVolume([]string{}, []string{os.Getenv("SystemDrive") + `\:d:`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || len(hostConfig.Binds) > 1 || hostConfig.Binds[0] != arr[0] || len(config.Volumes) != 0 {
t.Fatalf("Error parsing %s. Should have a single bind mount and no volumes", arr[0])
}
}
}
// setupPlatformVolume takes two arrays of volume specs - a Unix style
// spec and a Windows style spec. Depending on the platform being unit tested,
// it returns one of them, along with a volume string that would be passed
// on the docker CLI (e.g. -v /bar -v /foo).
func setupPlatformVolume(u, w []string) ([]string, string) {
var a []string
if runtime.GOOS == "windows" {
a = w
} else {
a = u
}
s := ""
for _, v := range a {
s = s + "-v " + v + " "
}
return a, s
}
// check if (a == c && b == d) || (a == d && b == c)
// because maps are randomized
func compareRandomizedStrings(a, b, c, d string) error {
if a == c && b == d {
return nil
}
if a == d && b == c {
return nil
}
return errors.New("strings don't match")
}
// Simple parse with MacAddress validation
@@ -396,66 +303,49 @@ func TestRunFlagsParseWithMemory(t *testing.T) {
flags, _ := setupRunFlags()
args := []string{"--memory=invalid", "img", "cmd"}
err := flags.Parse(args)
assert.ErrorContains(t, err, `invalid argument "invalid" for "-m, --memory" flag`)
require.ErrorContains(t, err, `invalid argument "invalid" for "-m, --memory" flag`)
_, hostconfig, _ := mustParse(t, "--memory=1G")
assert.Check(t, is.Equal(int64(1073741824), hostconfig.Memory))
assert.Equal(t, int64(1073741824), hostconfig.Memory)
}
func TestParseWithMemorySwap(t *testing.T) {
flags, _ := setupRunFlags()
args := []string{"--memory-swap=invalid", "img", "cmd"}
err := flags.Parse(args)
assert.ErrorContains(t, err, `invalid argument "invalid" for "--memory-swap" flag`)
require.ErrorContains(t, err, `invalid argument "invalid" for "--memory-swap" flag`)
_, hostconfig, _ := mustParse(t, "--memory-swap=1G")
assert.Check(t, is.Equal(int64(1073741824), hostconfig.MemorySwap))
assert.Equal(t, int64(1073741824), hostconfig.MemorySwap)
_, hostconfig, _ = mustParse(t, "--memory-swap=-1")
assert.Check(t, is.Equal(int64(-1), hostconfig.MemorySwap))
assert.Equal(t, int64(-1), hostconfig.MemorySwap)
}
func TestParseHostname(t *testing.T) {
validHostnames := map[string]string{
"hostname": "hostname",
"host-name": "host-name",
"hostname123": "hostname123",
"123hostname": "123hostname",
"hostname-of-63-bytes-long-should-be-valid-and-without-any-error": "hostname-of-63-bytes-long-should-be-valid-and-without-any-error",
}
hostnameWithDomain := "--hostname=hostname.domainname"
hostnameWithDomainTld := "--hostname=hostname.domainname.tld"
for hostname, expectedHostname := range validHostnames {
if config, _, _ := mustParse(t, "--hostname="+hostname); config.Hostname != expectedHostname {
t.Fatalf("Expected the config to have 'hostname' as %q, got %q", expectedHostname, config.Hostname)
}
}
if config, _, _ := mustParse(t, hostnameWithDomain); config.Hostname != "hostname.domainname" || config.Domainname != "" {
t.Fatalf("Expected the config to have 'hostname' as hostname.domainname, got %q", config.Hostname)
}
if config, _, _ := mustParse(t, hostnameWithDomainTld); config.Hostname != "hostname.domainname.tld" || config.Domainname != "" {
t.Fatalf("Expected the config to have 'hostname' as hostname.domainname.tld, got %q", config.Hostname)
}
}
func TestParseHostnameDomainname(t *testing.T) {
validDomainnames := map[string]string{
"domainname": "domainname",
"domain-name": "domain-name",
"domainname123": "domainname123",
"123domainname": "123domainname",
"domainname-63-bytes-long-should-be-valid-and-without-any-errors": "domainname-63-bytes-long-should-be-valid-and-without-any-errors",
}
for domainname, expectedDomainname := range validDomainnames {
if config, _, _ := mustParse(t, "--domainname="+domainname); config.Domainname != expectedDomainname {
t.Fatalf("Expected the config to have 'domainname' as %q, got %q", expectedDomainname, config.Domainname)
}
}
if config, _, _ := mustParse(t, "--hostname=some.prefix --domainname=domainname"); config.Hostname != "some.prefix" || config.Domainname != "domainname" {
t.Fatalf("Expected the config to have 'hostname' as 'some.prefix' and 'domainname' as 'domainname', got %q and %q", config.Hostname, config.Domainname)
}
if config, _, _ := mustParse(t, "--hostname=another-prefix --domainname=domainname.tld"); config.Hostname != "another-prefix" || config.Domainname != "domainname.tld" {
t.Fatalf("Expected the config to have 'hostname' as 'another-prefix' and 'domainname' as 'domainname.tld', got %q and %q", config.Hostname, config.Domainname)
for _, tc := range []struct {
input, hostname, domainname string
}{
{"--hostname=hostname", "hostname", ""},
{"--hostname=host-name", "host-name", ""},
{"--hostname=hostname123", "hostname123", ""},
{"--hostname=123hostname", "123hostname", ""},
{"--hostname=hostname-of-63-bytes-long-should-be-valid-and-without-any-error", "hostname-of-63-bytes-long-should-be-valid-and-without-any-error", ""},
{"--hostname=hostname.domainname", "hostname.domainname", ""},
{"--hostname=hostname.domainname.tld", "hostname.domainname.tld", ""},
{"--domainname=domainname", "", "domainname"},
{"--domainname=domain-name", "", "domain-name"},
{"--domainname=domainname123", "", "domainname123"},
{"--domainname=123domainname", "", "123domainname"},
{"--domainname=domainname-63-bytes-long-should-be-valid-and-without-any-errors", "", "domainname-63-bytes-long-should-be-valid-and-without-any-errors"},
{"--hostname=some.prefix --domainname=domainname", "some.prefix", "domainname"},
{"--hostname=another-prefix --domainname=domainname.tld", "another-prefix", "domainname.tld"},
} {
t.Run(tc.input, func(t *testing.T) {
config, _, _ := mustParse(t, tc.input)
require.Equal(t, tc.hostname, config.Hostname)
require.Equal(t, tc.domainname, config.Domainname)
})
}
}
@@ -474,7 +364,7 @@ func TestParseWithExpose(t *testing.T) {
for expose, expectedError := range tests {
t.Run(expose, func(t *testing.T) {
_, _, _, err := parseRun([]string{fmt.Sprintf("--expose=%v", expose), "img", "cmd"})
assert.Error(t, err, expectedError)
require.EqualError(t, err, expectedError)
})
}
})
@@ -489,30 +379,38 @@ func TestParseWithExpose(t *testing.T) {
for expose, exposedPorts := range tests {
t.Run(expose, func(t *testing.T) {
config, _, _, err := parseRun([]string{fmt.Sprintf("--expose=%v", expose), "img", "cmd"})
assert.NilError(t, err)
require.NoError(t, err)
for _, port := range exposedPorts {
_, ok := config.ExposedPorts[port]
assert.Check(t, ok, "missing port %q in exposed ports: %#+v", port, config.ExposedPorts[port])
assert.True(t, ok, "missing port %q in exposed ports: %#+v", port, config.ExposedPorts[port])
}
})
}
})
t.Run("merge with published", func(t *testing.T) {
// Merge with actual published port
config, _, _, err := parseRun([]string{"--publish=80", "--expose=80-81/tcp", "img", "cmd"})
assert.NilError(t, err)
assert.Check(t, is.Len(config.ExposedPorts, 2))
config, hostConfig, _, err := parseRun([]string{"--publish=80", "--publish=127.0.0.1:8080:80", "--publish=[::1]:8081:80", "--expose=80-81/tcp"})
require.NoError(t, err)
assert.Equal(t, networktypes.PortMap{
networktypes.MustParsePort("80/tcp"): {
{},
{HostIP: netip.MustParseAddr("127.0.0.1"), HostPort: "8080"},
{HostIP: netip.MustParseAddr("::1"), HostPort: "8081"},
},
}, hostConfig.PortBindings)
assert.Len(t, config.ExposedPorts, 2)
ports := []networktypes.Port{networktypes.MustParsePort("80/tcp"), networktypes.MustParsePort("81/tcp")}
for _, port := range ports {
_, ok := config.ExposedPorts[port]
assert.Check(t, ok, "missing port %q in exposed ports: %#+v", port, config.ExposedPorts[port])
assert.True(t, ok, "missing port %q in exposed ports: %#+v", port, config.ExposedPorts[port])
}
})
}
func TestParseDevice(t *testing.T) {
skip.If(t, runtime.GOOS != "linux") // Windows and macOS validate server-side
if runtime.GOOS != "linux" {
t.Skip("Windows and macOS validate server-side")
}
testCases := []struct {
devices []string
deviceMapping *container.DeviceMapping
@@ -586,17 +484,17 @@ func TestParseDevice(t *testing.T) {
_, hostconfig, _, err := parseRun(args)
assert.NilError(t, err)
require.NoError(t, err)
if tc.deviceMapping != nil {
if assert.Check(t, is.Len(hostconfig.Devices, 1)) {
assert.Check(t, is.DeepEqual(*tc.deviceMapping, hostconfig.Devices[0]))
if assert.Len(t, hostconfig.Devices, 1) {
assert.Equal(t, *tc.deviceMapping, hostconfig.Devices[0])
}
} else {
assert.Check(t, is.Len(hostconfig.Devices, 0))
assert.Empty(t, hostconfig.Devices)
}
assert.Check(t, is.DeepEqual(tc.deviceRequests, hostconfig.DeviceRequests))
assert.Equal(t, tc.deviceRequests, hostconfig.DeviceRequests)
})
}
}
@@ -665,11 +563,11 @@ func TestParseDeviceByServerOS(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
got, err := parseDevice(tc.device, tc.serverOS)
if tc.wantErr != "" {
assert.Error(t, err, tc.wantErr)
require.EqualError(t, err, tc.wantErr)
return
}
assert.NilError(t, err)
assert.Equal(t, got, tc.want)
require.NoError(t, err)
require.Equal(t, tc.want, got)
})
}
}
@@ -848,13 +746,13 @@ func TestParseNetworkConfig(t *testing.T) {
_, hConfig, nwConfig, err := parseRun(tc.flags)
if tc.expectedErr != "" {
assert.Error(t, err, tc.expectedErr)
require.EqualError(t, err, tc.expectedErr)
return
}
assert.NilError(t, err)
assert.DeepEqual(t, hConfig.NetworkMode, tc.expectedHostCfg.NetworkMode)
assert.DeepEqual(t, nwConfig.EndpointsConfig, tc.expected, cmpopts.EquateComparable(netip.Addr{}))
require.NoError(t, err)
require.Equal(t, tc.expectedHostCfg.NetworkMode, hConfig.NetworkMode)
require.Equal(t, tc.expected, nwConfig.EndpointsConfig)
})
}
}
@@ -863,24 +761,24 @@ func TestParseModes(t *testing.T) {
// pid ko
flags, copts := setupRunFlags()
args := []string{"--pid=container:", "img", "cmd"}
assert.NilError(t, flags.Parse(args))
require.NoError(t, flags.Parse(args))
_, err := parse(flags, copts, runtime.GOOS)
assert.ErrorContains(t, err, "--pid: invalid PID mode")
require.ErrorContains(t, err, "--pid: invalid PID mode")
// pid ok
_, hostconfig, _, err := parseRun([]string{"--pid=host", "img", "cmd"})
assert.NilError(t, err)
require.NoError(t, err)
if !hostconfig.PidMode.Valid() {
t.Fatalf("Expected a valid PidMode, got %v", hostconfig.PidMode)
}
// uts ko
_, _, _, err = parseRun([]string{"--uts=container:", "img", "cmd"}) //nolint:dogsled // verbatim copy from docker/cli tests
assert.ErrorContains(t, err, "--uts: invalid UTS mode")
require.ErrorContains(t, err, "--uts: invalid UTS mode")
// uts ok
_, hostconfig, _, err = parseRun([]string{"--uts=host", "img", "cmd"})
assert.NilError(t, err)
require.NoError(t, err)
if !hostconfig.UTSMode.Valid() {
t.Fatalf("Expected a valid UTSMode, got %v", hostconfig.UTSMode)
}
@@ -892,11 +790,11 @@ func TestRunFlagsParseShmSize(t *testing.T) {
args := []string{"--shm-size=a128m", "img", "cmd"}
expectedErr := `invalid argument "a128m" for "--shm-size" flag:`
err := flags.Parse(args)
assert.ErrorContains(t, err, expectedErr)
require.ErrorContains(t, err, expectedErr)
// shm-size ok
_, hostconfig, _, err := parseRun([]string{"--shm-size=128m", "img", "cmd"})
assert.NilError(t, err)
require.NoError(t, err)
if hostconfig.ShmSize != 134217728 {
t.Fatalf("Expected a valid ShmSize, got %d", hostconfig.ShmSize)
}
@@ -967,11 +865,11 @@ func TestParseRestartPolicy(t *testing.T) {
t.Run(tc.input, func(t *testing.T) {
_, hostConfig, _, err := parseRun([]string{"--restart=" + tc.input, "img", "cmd"})
if tc.expectedErr != "" {
assert.Check(t, is.Error(err, tc.expectedErr))
assert.Check(t, is.Nil(hostConfig))
assert.Nil(t, hostConfig)
assert.EqualError(t, err, tc.expectedErr)
} else {
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(hostConfig.RestartPolicy, tc.expected))
require.NoError(t, err)
assert.Equal(t, tc.expected, hostConfig.RestartPolicy)
}
})
}
@@ -980,45 +878,33 @@ func TestParseRestartPolicy(t *testing.T) {
func TestParseRestartPolicyAutoRemove(t *testing.T) {
_, _, _, err := parseRun([]string{"--rm", "--restart=always", "img", "cmd"}) //nolint:dogsled // verbatim copy from docker/cli tests
const expected = "conflicting options: cannot specify both --restart and --rm"
assert.Check(t, is.Error(err, expected))
assert.EqualError(t, err, expected)
}
func TestParseHealth(t *testing.T) {
checkOk := func(args ...string) *container.HealthConfig {
config, _, _, err := parseRun(args)
if err != nil {
t.Fatalf("%#v: %v", args, err)
}
return config.Healthcheck
}
checkError := func(expected string, args ...string) {
config, _, _, err := parseRun(args)
if err == nil {
t.Fatalf("Expected error, but got %#v", config)
}
if err.Error() != expected {
t.Fatalf("Expected %#v, got %#v", expected, err)
}
}
health := checkOk("--no-healthcheck", "img", "cmd")
if health == nil || len(health.Test) != 1 || health.Test[0] != "NONE" {
t.Fatalf("--no-healthcheck failed: %#v", health)
}
health = checkOk("--health-cmd=/check.sh -q", "img", "cmd")
if len(health.Test) != 2 || health.Test[0] != "CMD-SHELL" || health.Test[1] != "/check.sh -q" {
t.Fatalf("--health-cmd: got %#v", health.Test)
}
if health.Timeout != 0 {
t.Fatalf("--health-cmd: timeout = %s", health.Timeout)
}
checkError("--no-healthcheck conflicts with --health-* options",
"--no-healthcheck", "--health-cmd=/check.sh -q", "img", "cmd")
health = checkOk("--health-timeout=2s", "--health-retries=3", "--health-interval=4.5s", "--health-start-period=5s", "--health-start-interval=1s", "img", "cmd")
if health.Timeout != 2*time.Second || health.Retries != 3 || health.Interval != 4500*time.Millisecond || health.StartPeriod != 5*time.Second || health.StartInterval != 1*time.Second {
t.Fatalf("--health-*: got %#v", health)
for _, tc := range []struct {
args []string
expected *container.HealthConfig
wantErr string
}{
{[]string{"--no-healthcheck", "img", "cmd"}, &container.HealthConfig{Test: []string{"NONE"}}, ""},
{[]string{"--health-cmd=/check.sh -q", "img", "cmd"}, &container.HealthConfig{Test: []string{"CMD-SHELL", "/check.sh -q"}}, ""},
{[]string{"--no-healthcheck", "--health-cmd=/check.sh -q", "img", "cmd"}, nil, "--no-healthcheck conflicts with --health-* options"},
{
[]string{"--health-timeout=2s", "--health-retries=3", "--health-interval=4.5s", "--health-start-period=5s", "--health-start-interval=1s", "img", "cmd"},
&container.HealthConfig{Timeout: 2 * time.Second, Retries: 3, Interval: 4500 * time.Millisecond, StartPeriod: 5 * time.Second, StartInterval: time.Second},
"",
},
} {
t.Run(strings.Join(tc.args, " "), func(t *testing.T) {
config, _, _, err := parseRun(tc.args)
if tc.wantErr != "" {
require.EqualError(t, err, tc.wantErr)
} else {
require.NoError(t, err)
require.Equal(t, tc.expected, config.Healthcheck)
}
})
}
}
@@ -1037,93 +923,66 @@ func TestParseLoggingOpts(t *testing.T) {
}
}
func TestParseEnvfileVariables(t *testing.T) { //nolint:dupl // verbatim copy from docker/cli tests
expErr := "--env-file: open nonexistent: no such file or directory"
if runtime.GOOS == "windows" {
expErr = "--env-file: open nonexistent: The system cannot find the file specified."
}
// env ko
if _, _, _, err := parseRun([]string{"--env-file=nonexistent", "img", "cmd"}); err == nil || err.Error() != expErr {
t.Fatalf("Expected an error with message '%s', got %v", expErr, err)
}
// env ok
config, _, _, err := parseRun([]string{"--env-file=testdata/valid.env", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
if len(config.Env) != 1 || config.Env[0] != "ENV1=value1" {
t.Fatalf("Expected a config with [ENV1=value1], got %v", config.Env)
}
config, _, _, err = parseRun([]string{"--env-file=testdata/valid.env", "--env=ENV2=value2", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
if len(config.Env) != 2 || config.Env[0] != "ENV1=value1" || config.Env[1] != "ENV2=value2" {
t.Fatalf("Expected a config with [ENV1=value1 ENV2=value2], got %v", config.Env)
func TestParseFileVariables(t *testing.T) {
for _, tc := range []struct {
flag, file, key string
}{
{"env", "valid.env", "ENV"},
{"label", "valid.label", "LABEL"},
} {
t.Run(tc.flag, func(t *testing.T) {
expectedError := "no such file or directory"
if runtime.GOOS == "windows" {
expectedError = "The system cannot find the file specified."
}
_, _, _, err := parseRun([]string{"--" + tc.flag + "-file=nonexistent"})
require.EqualError(t, err, "--"+tc.flag+"-file: open nonexistent: "+expectedError)
for _, extra := range []bool{false, true} {
args := []string{"--" + tc.flag + "-file=testdata/" + tc.file}
expected := []string{tc.key + "1=value1"}
if extra {
args = append(args, "--"+tc.flag+"="+tc.key+"2=value2")
expected = append(expected, tc.key+"2=value2")
}
config, _, _, err := parseRun(args)
require.NoError(t, err)
if tc.flag == "env" {
assert.Equal(t, expected, config.Env)
} else {
assert.Len(t, config.Labels, len(expected))
for _, label := range expected {
key, value, _ := strings.Cut(label, "=")
assert.Equal(t, value, config.Labels[key])
}
}
}
})
}
}
func TestParseEnvfileVariablesWithBOMUnicode(t *testing.T) {
// UTF8 with BOM
config, _, _, err := parseRun([]string{"--env-file=testdata/utf8.env", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
env := []string{"FOO=BAR", "HELLO=" + string([]byte{0xe6, 0x82, 0xa8, 0xe5, 0xa5, 0xbd}), "BAR=FOO"}
if len(config.Env) != len(env) {
t.Fatalf("Expected a config with %d env variables, got %v: %v", len(env), len(config.Env), config.Env)
}
for i, v := range env {
if config.Env[i] != v {
t.Fatalf("Expected a config with [%s], got %v", v, []byte(config.Env[i]))
}
}
config, _, _, err := parseRun([]string{"--env-file=testdata/utf8.env"})
require.NoError(t, err)
assert.Equal(t, []string{"FOO=BAR", "HELLO=您好", "BAR=FOO"}, config.Env)
// UTF16 with BOM
e := "invalid utf8 bytes at line"
if _, _, _, err := parseRun([]string{"--env-file=testdata/utf16.env", "img", "cmd"}); err == nil || !strings.Contains(err.Error(), e) {
t.Fatalf("Expected an error with message '%s', got %v", e, err)
}
// UTF16BE with BOM
if _, _, _, err := parseRun([]string{"--env-file=testdata/utf16be.env", "img", "cmd"}); err == nil || !strings.Contains(err.Error(), e) {
t.Fatalf("Expected an error with message '%s', got %v", e, err)
}
}
func TestParseLabelfileVariables(t *testing.T) { //nolint:dupl // verbatim copy from docker/cli tests
expErr := "--label-file: open nonexistent: no such file or directory"
if runtime.GOOS == "windows" {
expErr = "--label-file: open nonexistent: The system cannot find the file specified."
}
// label ko
if _, _, _, err := parseRun([]string{"--label-file=nonexistent", "img", "cmd"}); err == nil || err.Error() != expErr {
t.Fatalf("Expected an error with message '%s', got %v", expErr, err)
}
// label ok
config, _, _, err := parseRun([]string{"--label-file=testdata/valid.label", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
if len(config.Labels) != 1 || config.Labels["LABEL1"] != "value1" {
t.Fatalf("Expected a config with [LABEL1:value1], got %v", config.Labels)
}
config, _, _, err = parseRun([]string{"--label-file=testdata/valid.label", "--label=LABEL2=value2", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
if len(config.Labels) != 2 || config.Labels["LABEL1"] != "value1" || config.Labels["LABEL2"] != "value2" {
t.Fatalf("Expected a config with [LABEL1:value1 LABEL2:value2], got %v", config.Labels)
for _, file := range []string{"utf16.env", "utf16be.env"} {
t.Run(file, func(t *testing.T) {
_, _, _, err := parseRun([]string{"--env-file=testdata/" + file})
require.ErrorContains(t, err, "invalid utf8 bytes at line")
})
}
}
func TestParseEntryPoint(t *testing.T) {
config, _, _, err := parseRun([]string{"--entrypoint=anything", "cmd", "img"})
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(config.Entrypoint, []string{"anything"}))
require.NoError(t, err)
assert.Equal(t, []string{"anything"}, config.Entrypoint)
}
func TestValidateDevice(t *testing.T) {
skip.If(t, runtime.GOOS != "linux") // Windows and macOS validate server-side
if runtime.GOOS != "linux" {
t.Skip("Windows and macOS validate server-side")
}
valid := []string{
"/home",
"/home:/home",
@@ -1222,22 +1081,22 @@ func TestValidateDeviceByServerOS(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
got, err := validateDevice(tc.value, tc.serverOS)
if tc.wantError != "" {
assert.Error(t, err, tc.wantError)
require.EqualError(t, err, tc.wantError)
return
}
assert.NilError(t, err)
assert.Equal(t, got, tc.want)
require.NoError(t, err)
require.Equal(t, tc.want, got)
})
}
}
func TestDeviceCgroupRulesAndInvalidParameter(t *testing.T) {
got, err := validateDeviceCgroupRule("c 1:3 rwm")
assert.NilError(t, err)
assert.Equal(t, got, "c 1:3 rwm")
require.NoError(t, err)
require.Equal(t, "c 1:3 rwm", got)
_, err = validateDeviceCgroupRule("invalid")
assert.Error(t, err, "invalid device cgroup format 'invalid'")
require.EqualError(t, err, "invalid device cgroup format 'invalid'")
if invalidParameter(nil) != nil {
t.Fatal("invalidParameter(nil) should be nil")
@@ -1245,9 +1104,9 @@ func TestDeviceCgroupRulesAndInvalidParameter(t *testing.T) {
cause := errors.New("bad input")
err = invalidParameter(cause)
var invalid interface{ InvalidParameter() }
assert.Assert(t, errors.As(err, &invalid))
assert.Assert(t, errors.Is(err, cause))
assert.Equal(t, invalidParameter(err), err) // already invalid, so not wrapped twice
require.ErrorAs(t, err, &invalid)
require.ErrorIs(t, err, cause)
require.Condition(t, func() bool { return invalidParameter(err) == err })
}
func TestParseSystemPaths(t *testing.T) {
@@ -1298,9 +1157,9 @@ func TestParseSystemPaths(t *testing.T) {
for _, tc := range tests {
securityOpts, maskedPaths, readonlyPaths := parseSystemPaths(tc.in)
assert.DeepEqual(t, securityOpts, tc.out)
assert.DeepEqual(t, maskedPaths, tc.masked)
assert.DeepEqual(t, readonlyPaths, tc.readonly)
require.Equal(t, tc.out, securityOpts)
require.Equal(t, tc.masked, maskedPaths)
require.Equal(t, tc.readonly, readonlyPaths)
}
}
@@ -1322,9 +1181,9 @@ func TestConvertToStandardNotation(t *testing.T) {
for key, ports := range valid {
convertedPorts, err := convertToStandardNotation(ports)
if err != nil {
assert.NilError(t, err)
require.NoError(t, err)
}
assert.DeepEqual(t, strings.Split(key, " "), convertedPorts)
require.Equal(t, strings.Split(key, " "), convertedPorts)
}
for _, ports := range invalid {

View File

@@ -58,18 +58,14 @@ func RemoveImage(ctx context.Context, imageName string, force, pruneChildren boo
}
defer cli.Close()
if _, err := cli.ImageInspect(ctx, imageName); cerrdefs.IsNotFound(err) {
if _, err = cli.ImageRemove(ctx, imageName, client.ImageRemoveOptions{
Force: force,
PruneChildren: pruneChildren,
}); cerrdefs.IsNotFound(err) {
return false, nil
} else if err != nil {
return false, err
}
if _, err = cli.ImageRemove(ctx, imageName, client.ImageRemoveOptions{
Force: force,
PruneChildren: pruneChildren,
}); err != nil {
return false, err
}
return true, nil
}

View File

@@ -5,14 +5,17 @@
package container
import (
"bytes"
"context"
"fmt"
"os"
"os/exec"
"strings"
"testing"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
mobyclient "github.com/moby/moby/client"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -21,49 +24,55 @@ func init() {
logging.SetLevel(logging.LevelDebug)
}
// buildScratchImage builds a tiny empty image for the given platform locally (FROM scratch, no
// network or emulation since there is nothing to run) and returns its tag, removing it after
// the test.
func buildScratchImage(t *testing.T, platform string) string {
func loadScratchImage(t *testing.T, platform string) string {
t.Helper()
tag := fmt.Sprintf("act-test-exists-%s:latest", strings.TrimPrefix(platform, "linux/"))
cmd := exec.Command("docker", "build", "--platform", platform, "-t", tag, "-")
cmd.Stdin = strings.NewReader("FROM scratch\nLABEL act-test=1\n")
// Force BuildKit: it records the requested architecture in the image config for a
// FROM-scratch build, whereas the classic builder ignores --platform and tags it with the
// host arch, which would break the per-platform existence assertions below.
cmd.Env = append(os.Environ(), "DOCKER_BUILDKIT=1")
out, err := cmd.CombinedOutput()
require.NoError(t, err, string(out))
t.Cleanup(func() { _ = exec.Command("docker", "rmi", "-f", tag).Run() })
tag := fmt.Sprintf("act-test-exists-%s-%d:latest", strings.TrimPrefix(platform, "linux/"), time.Now().UnixNano())
var archive bytes.Buffer
require.NoError(t, writeFilesTar(t.Context(), &archive, 0, 0,
&FileEntry{Name: "config.json", Mode: 0o600, Body: fmt.Sprintf(`{"architecture":%q,"os":"linux","rootfs":{"type":"layers","diff_ids":[]}}`, strings.TrimPrefix(platform, "linux/"))},
&FileEntry{Name: "manifest.json", Mode: 0o600, Body: fmt.Sprintf(`[{"Config":"config.json","RepoTags":[%q],"Layers":[]}]`, tag)},
))
cli, err := GetDockerClient(t.Context())
require.NoError(t, err)
t.Cleanup(func() { _ = cli.Close() })
response, err := cli.ImageLoad(t.Context(), &archive)
require.NoError(t, err)
defer response.Close()
require.NoError(t, logDockerResponse(common.Logger(t.Context()), response, false))
t.Cleanup(func() {
_, _ = cli.ImageRemove(context.WithoutCancel(t.Context()), tag, mobyclient.ImageRemoveOptions{Force: true})
})
return tag
}
func TestImageExistsLocally(t *testing.T) {
t.Parallel()
requireDocker(t)
ctx := t.Context()
// a non-existent image is reported absent
missing, err := ImageExistsLocally(ctx, "library/alpine:this-random-tag-will-never-exist", "linux/amd64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.False(t, missing)
// Build tiny images for two architectures locally so per-platform existence can be checked
// offline (formerly pulled node:24-bookworm-slim for amd64 and arm64 over the network).
amd64Ref := buildScratchImage(t, "linux/amd64")
arm64Ref := buildScratchImage(t, "linux/arm64")
amd64Ref := loadScratchImage(t, "linux/amd64")
arm64Ref := loadScratchImage(t, "linux/arm64")
amd64Exists, err := ImageExistsLocally(ctx, amd64Ref, "linux/amd64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.True(t, amd64Exists)
// a non-host architecture image is detected for its own architecture
arm64Exists, err := ImageExistsLocally(ctx, arm64Ref, "linux/arm64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.True(t, arm64Exists)
// a present image is reported absent for a different platform
wrongPlatform, err := ImageExistsLocally(ctx, amd64Ref, "linux/arm64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.False(t, wrongPlatform)
removed, err := RemoveImage(ctx, amd64Ref, false, false)
require.NoError(t, err)
assert.True(t, removed)
removed, err = RemoveImage(ctx, amd64Ref, false, false)
require.NoError(t, err)
assert.False(t, removed)
}

View File

@@ -33,18 +33,10 @@ func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, isE
defer dockerResponse.Close()
scanner := bufio.NewScanner(dockerResponse)
msg := dockerMessage{}
scanner.Buffer(nil, 64*1024*1024) // a build echoes long instructions in one record
for scanner.Scan() {
line := scanner.Bytes()
msg.ID = ""
msg.Stream = ""
msg.Error = ""
msg.ErrorDetail.Message = ""
msg.Status = ""
msg.Progress = ""
var msg dockerMessage
if err := json.Unmarshal(line, &msg); err != nil {
writeLog(logger, false, "Unable to unmarshal line [%s] ==> %v", string(line), err)
continue
@@ -74,7 +66,7 @@ func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, isE
}
}
return nil
return scanner.Err()
}
func writeLog(logger *logging.Logger, isError bool, format string, args ...any) {

View File

@@ -53,7 +53,9 @@ func removeOrphanNetworks(ctx context.Context, cli client.APIClient, runnerUUID,
networks:
for _, n := range networks.Items {
result, err := cli.NetworkInspect(ctx, n.ID, client.NetworkInspectOptions{})
if err != nil {
if cerrdefs.IsNotFound(err) {
continue
} else if err != nil {
errs = append(errs, fmt.Errorf("failed to inspect network %s: %w", n.Name, err))
continue
}
@@ -71,7 +73,9 @@ networks:
continue networks
}
}
if _, err := cli.NetworkRemove(ctx, n.ID, client.NetworkRemoveOptions{}); err != nil {
if _, err := cli.NetworkRemove(ctx, n.ID, client.NetworkRemoveOptions{}); cerrdefs.IsNotFound(err) {
continue
} else if err != nil {
errs = append(errs, fmt.Errorf("failed to remove network %s: %w", n.Name, err))
continue
}
@@ -81,14 +85,14 @@ networks:
}
func NewDockerNetworkCreateExecutor(name string, opts NewDockerNetworkCreateExecutorInput) common.Executor {
return func(ctx context.Context) error {
return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx)
if err != nil {
return err
}
defer cli.Close()
return createDockerNetwork(ctx, cli, name, opts)
}
}).IfNot(common.Dryrun)
}
func createDockerNetwork(ctx context.Context, cli client.APIClient, name string, opts NewDockerNetworkCreateExecutorInput) error {
@@ -153,7 +157,7 @@ func isAddressPoolExhausted(err error) bool {
}
func NewDockerNetworkConnectExecutor(networkName, container string) common.Executor {
return func(ctx context.Context) error {
return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx)
if err != nil {
return err
@@ -163,7 +167,7 @@ func NewDockerNetworkConnectExecutor(networkName, container string) common.Execu
common.Logger(ctx).Warnf("cannot connect %s to network %s, so jobs on it cannot reach the cache: %v", container, networkName, err)
}
return nil
}
}).IfNot(common.Dryrun)
}
func connectNetwork(ctx context.Context, cli client.APIClient, networkName, container string) error {
@@ -178,14 +182,14 @@ func connectNetwork(ctx context.Context, cli client.APIClient, networkName, cont
}
func NewDockerNetworkDisconnectExecutor(networkName, container string) common.Executor {
return func(ctx context.Context) error {
return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx)
if err != nil {
return err
}
defer cli.Close()
return disconnectNetwork(ctx, cli, networkName, container)
}
}).IfNot(common.Dryrun)
}
func disconnectNetwork(ctx context.Context, cli client.APIClient, networkName, container string) error {
@@ -197,42 +201,41 @@ func disconnectNetwork(ctx context.Context, cli client.APIClient, networkName, c
}
func NewDockerNetworkRemoveExecutor(name string) common.Executor {
return func(ctx context.Context) error {
return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx)
if err != nil {
return err
}
defer cli.Close()
return removeNetwork(ctx, cli, name)
}).IfNot(common.Dryrun)
}
// Make sure that all network of the specified name are removed
// cli.NetworkRemove refuses to remove a network if there are duplicates
networks, err := cli.NetworkList(ctx, client.NetworkListOptions{})
if err != nil {
func removeNetwork(ctx context.Context, cli client.APIClient, name string) error {
networks, err := cli.NetworkList(ctx, client.NetworkListOptions{}) // NetworkRemove by name refuses duplicates, so remove each by ID
if err != nil {
return err
}
var errs []error
for _, n := range networks.Items {
if n.Name != name {
continue
}
result, err := cli.NetworkInspect(ctx, n.ID, client.NetworkInspectOptions{})
if cerrdefs.IsNotFound(err) {
continue
} else if err != nil {
return err
}
// For Gitea, reduce log noise
// common.Logger(ctx).Debugf("%v", networks)
var errs []error
for _, n := range networks.Items {
if n.Name == name {
result, err := cli.NetworkInspect(ctx, n.ID, client.NetworkInspectOptions{})
if err != nil {
return err
}
// it holds a subnet out of the daemon's pool until something reclaims it
if len(result.Network.Containers) != 0 {
common.Logger(ctx).Warnf("Refusing to remove network %s because it still has active endpoints, the idle cleanup reclaims it once they are gone", name)
continue
}
if _, err = cli.NetworkRemove(ctx, n.ID, client.NetworkRemoveOptions{}); err != nil {
errs = append(errs, fmt.Errorf("failed to remove network %s: %w", name, err))
}
}
if len(result.Network.Containers) != 0 {
common.Logger(ctx).Warnf("Refusing to remove network %s because it still has active endpoints, the orphan cleanup reclaims it once they are gone", name)
continue
}
if _, err = cli.NetworkRemove(ctx, n.ID, client.NetworkRemoveOptions{}); err != nil && !cerrdefs.IsNotFound(err) {
errs = append(errs, fmt.Errorf("failed to remove network %s: %w", name, err))
}
return errors.Join(errs...)
}
return errors.Join(errs...)
}
// IsolatedCacheContainer returns the container holding addr, when jobs on jobNetwork cannot reach it there.

View File

@@ -116,7 +116,7 @@ func TestConnectAndDisconnectNetwork(t *testing.T) {
// Of this runner's networks, only the ones nothing but its own cache container is attached to
// and old enough to predate any job now starting are the runner's to reclaim. An unexpected
// NetworkRemove fails the test on its own, since testify has no expectation to match it against.
func TestRemoveOrphanNetworks(t *testing.T) {
func TestRemoveNetworks(t *testing.T) {
ctx := t.Context()
cutoff := time.Date(2026, time.April, 29, 20, 0, 0, 0, time.UTC)
client := &mockDockerClient{}
@@ -128,11 +128,13 @@ func TestRemoveOrphanNetworks(t *testing.T) {
{ID: "starting"},
{ID: "cached"},
{ID: "shared"},
{ID: "gone"},
{ID: "raced"},
}}, nil)
client.On("NetworkInspect", ctx, "orphan", mobyclient.NetworkInspectOptions{}).
Return(mobyclient.NetworkInspectResult{}, nil)
Return(mobyclient.NetworkInspectResult{}, nil).Twice()
client.On("NetworkInspect", ctx, "busy", mobyclient.NetworkInspectOptions{}).
Return(mobyclient.NetworkInspectResult{Network: network.Inspect{Containers: map[string]network.EndpointResource{"c": {}}}}, nil)
Return(mobyclient.NetworkInspectResult{Network: network.Inspect{Containers: map[string]network.EndpointResource{"c": {}}}}, nil).Twice()
client.On("NetworkInspect", ctx, "starting", mobyclient.NetworkInspectOptions{}).
Return(mobyclient.NetworkInspectResult{Network: network.Inspect{
Network: network.Network{Created: cutoff.Add(time.Second)},
@@ -142,13 +144,28 @@ func TestRemoveOrphanNetworks(t *testing.T) {
Return(mobyclient.NetworkInspectResult{Network: network.Inspect{Containers: map[string]network.EndpointResource{"0123456789abcdef": {}}}}, nil)
client.On("NetworkInspect", ctx, "shared", mobyclient.NetworkInspectOptions{}).
Return(mobyclient.NetworkInspectResult{Network: network.Inspect{Containers: map[string]network.EndpointResource{"0123456789abcdef": {}, "job": {}}}}, nil)
client.On("NetworkInspect", ctx, "gone", mobyclient.NetworkInspectOptions{}).
Return(mobyclient.NetworkInspectResult{}, cerrdefs.ErrNotFound).Twice()
client.On("NetworkInspect", ctx, "raced", mobyclient.NetworkInspectOptions{}).
Return(mobyclient.NetworkInspectResult{}, nil).Twice()
client.On("NetworkRemove", ctx, "raced", mobyclient.NetworkRemoveOptions{}).
Return(mobyclient.NetworkRemoveResult{}, cerrdefs.ErrNotFound).Twice()
client.On("NetworkRemove", ctx, "orphan", mobyclient.NetworkRemoveOptions{}).
Return(mobyclient.NetworkRemoveResult{}, nil)
Return(mobyclient.NetworkRemoveResult{}, nil).Twice()
client.On("NetworkDisconnect", ctx, "cached", mobyclient.NetworkDisconnectOptions{Container: "0123456789abcdef", Force: true}).
Return(nil).Once()
client.On("NetworkRemove", ctx, "cached", mobyclient.NetworkRemoveOptions{}).
Return(mobyclient.NetworkRemoveResult{}, nil).Once()
require.NoError(t, removeOrphanNetworks(ctx, client, "runner-1", "0123456789ab", cutoff))
client.On("NetworkList", ctx, mobyclient.NetworkListOptions{}).Return(mobyclient.NetworkListResult{Items: []network.Summary{
{ID: "orphan", Name: "job"},
{ID: "busy", Name: "job"},
{ID: "gone", Name: "job"},
{ID: "raced", Name: "job"},
{ID: "shared", Name: "other"},
}}, nil)
require.NoError(t, removeNetwork(ctx, client, "job"))
client.AssertExpectations(t)
}

View File

@@ -6,6 +6,7 @@ package container
import (
"testing"
specs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -17,32 +18,17 @@ func TestParsePlatform(t *testing.T) {
assert.Nil(t, got)
})
t.Run("os/arch", func(t *testing.T) {
got, err := parsePlatform("linux/amd64")
require.NoError(t, err)
require.NotNil(t, got)
assert.Equal(t, "linux", got.OS)
assert.Equal(t, "amd64", got.Architecture)
assert.Empty(t, got.Variant)
})
t.Run("os/arch/variant", func(t *testing.T) {
got, err := parsePlatform("linux/arm/v7")
require.NoError(t, err)
require.NotNil(t, got)
assert.Equal(t, "linux", got.OS)
assert.Equal(t, "arm", got.Architecture)
assert.Equal(t, "v7", got.Variant)
})
t.Run("input is lowercased", func(t *testing.T) {
got, err := parsePlatform("Linux/AMD64/V8")
require.NoError(t, err)
require.NotNil(t, got)
assert.Equal(t, "linux", got.OS)
assert.Equal(t, "amd64", got.Architecture)
assert.Equal(t, "v8", got.Variant)
})
for input, want := range map[string]specs.Platform{
"linux/amd64": {OS: "linux", Architecture: "amd64"},
"linux/arm/v7": {OS: "linux", Architecture: "arm", Variant: "v7"},
"Linux/AMD64/V8": {OS: "linux", Architecture: "amd64", Variant: "v8"},
} {
t.Run(input, func(t *testing.T) {
got, err := parsePlatform(input)
require.NoError(t, err)
assert.Equal(t, &want, got)
})
}
for _, bad := range []string{
"amd64",

View File

@@ -48,6 +48,20 @@ var (
rawStreamPath = regexp.MustCompile(`^(/v[0-9.]+)?/(containers/[^/]+/attach|exec/[^/]+/start)$`)
)
var dockerProxyDirectories, dockerProxyProbes sync.Map
type dockerProxyDirectory struct {
dir, daemonDir string
socket os.FileInfo
}
type dockerProxyProbe struct {
directory dockerProxyDirectory
done chan struct{}
seen bool
err error
}
func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
if host := os.Getenv("DOCKER_HOST"); runtime.GOOS != "linux" || host != "" && !strings.HasPrefix(host, "unix://") {
return nil
@@ -63,16 +77,17 @@ func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
if !ok {
return nil
}
if info, err := os.Stat(daemonSocket); err != nil || info.Mode()&os.ModeSocket == 0 {
socket, err := os.Stat(daemonSocket)
if err != nil || socket.Mode()&os.ModeSocket == 0 {
return nil
}
dir, daemonDir := runnerContainerWorkdir(probeCtx, cli)
seen := daemonDir != ""
workdir, _ := os.Getwd()
key := [3]string{cli.DaemonHost(), workdir, os.TempDir()}
cached, _ := dockerProxyDirectories.Load(key)
directory, seen := cached.(dockerProxyDirectory)
seen = seen && os.SameFile(directory.socket, socket) // daemon replacement invalidates cached paths
if !seen {
if dir, err = filepath.Abs(os.TempDir()); err == nil {
daemonDir = dir
seen, err = daemonSeesDir(probeCtx, cli, dir, daemonDir)
}
directory, seen, err = probeDockerProxyDirectory(probeCtx, cli, key, socket)
}
if err != nil {
common.Logger(ctx).Infof("docker proxy probe failed, jobs get the daemon socket directly: %v", err)
@@ -85,15 +100,51 @@ func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
if ctx.Err() != nil {
return nil
}
proxy, err := StartDockerProxy(daemonSocket, dir, job)
proxy, err := StartDockerProxy(daemonSocket, directory.dir, job)
if err != nil {
dockerProxyDirectories.Delete(key)
common.Logger(ctx).Warnf("docker proxy not started, the job gets the daemon socket directly: %v", err)
return nil
}
proxy.Socket = daemonDir + strings.TrimPrefix(proxy.Socket, dir)
dockerProxyDirectories.Store(key, directory)
proxy.Socket = directory.daemonDir + strings.TrimPrefix(proxy.Socket, directory.dir)
return proxy
}
func probeDockerProxyDirectory(ctx context.Context, cli client.APIClient, key [3]string, socket os.FileInfo) (dockerProxyDirectory, bool, error) {
for {
probe := &dockerProxyProbe{directory: dockerProxyDirectory{socket: socket}, done: make(chan struct{})}
actual, loaded := dockerProxyProbes.LoadOrStore(key, probe)
if loaded {
probe, _ = actual.(*dockerProxyProbe)
if !os.SameFile(probe.directory.socket, socket) {
dockerProxyProbes.CompareAndDelete(key, probe)
continue
}
select {
case <-ctx.Done():
return dockerProxyDirectory{}, false, ctx.Err()
case <-probe.done:
}
if ctx.Err() == nil && (errors.Is(probe.err, context.Canceled) || errors.Is(probe.err, context.DeadlineExceeded)) {
continue
}
} else {
probe.directory.dir, probe.directory.daemonDir = runnerContainerWorkdir(ctx, cli)
probe.seen = probe.directory.daemonDir != ""
if !probe.seen {
if probe.directory.dir, probe.err = filepath.Abs(key[2]); probe.err == nil {
probe.directory.daemonDir = probe.directory.dir
probe.seen, probe.err = daemonSeesDir(ctx, cli, probe.directory.dir, probe.directory.daemonDir)
}
}
dockerProxyProbes.CompareAndDelete(key, probe)
close(probe.done)
}
return probe.directory, probe.seen, probe.err
}
}
// runnerContainerWorkdir looks the runner's container up by hostname to find the daemon's path to its working directory.
func runnerContainerWorkdir(ctx context.Context, cli client.APIClient) (workdir, daemonDir string) {
workdir, err := os.Getwd()
@@ -144,12 +195,18 @@ func daemonSeesDir(ctx context.Context, cli client.APIClient, dir, daemonDir str
if len(images.Items) == 0 {
return false, errors.New("no image available for the docker proxy probe")
}
created, err := cli.ContainerCreate(ctx, client.ContainerCreateOptions{
Config: &container.Config{Image: images.Items[0].ID, Cmd: []string{"true"}},
HostConfig: &container.HostConfig{Mounts: []mount.Mount{
{Type: mount.TypeBind, Source: daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}, // not the marker itself, podman creates a missing bind source where docker rejects it
}},
})
var created client.ContainerCreateResult
for _, image := range images.Items {
created, err = cli.ContainerCreate(ctx, client.ContainerCreateOptions{
Config: &container.Config{Image: image.ID, Cmd: []string{"true"}},
HostConfig: &container.HostConfig{Mounts: []mount.Mount{
{Type: mount.TypeBind, Source: daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}, // not the marker itself, podman creates a missing bind source where docker rejects it
}},
})
if !cerrdefs.IsNotFound(err) {
break
}
}
if cerrdefs.IsInvalidArgument(err) {
return false, nil
}

View File

@@ -20,7 +20,9 @@ import (
"runtime"
"strconv"
"strings"
"sync/atomic"
"testing"
"testing/synctest"
"time"
cerrdefs "github.com/containerd/errdefs"
@@ -68,6 +70,7 @@ func TestDockerProxy(t *testing.T) {
bodies := make(chan []byte, 8)
createStarted, releaseCreate := make(chan struct{}), make(chan struct{})
streamsDone := make(chan struct{}, 2)
var probes atomic.Int32
daemonSocket := filepath.Join(shortTempDir(t), "d.sock")
listener, err := net.Listen("unix", daemonSocket)
require.NoError(t, err)
@@ -75,12 +78,28 @@ func TestDockerProxy(t *testing.T) {
switch {
case strings.HasSuffix(r.URL.Path, "/create"):
body, _ := io.ReadAll(r.Body)
if bytes.Contains(body, []byte(`"Image":"probe-image"`)) {
_, _ = w.Write([]byte(`{"Id":"probe"}`))
return
}
if r.URL.RawQuery == "wait" {
close(createStarted)
<-releaseCreate
}
bodies <- body
w.WriteHeader(http.StatusCreated)
case strings.HasSuffix(r.URL.Path, "/images/json"):
if probes.Add(1) == 1 {
http.Error(w, "probe unavailable", http.StatusServiceUnavailable)
return
}
_, _ = w.Write([]byte(`[{"Id":"probe-image"}]`))
case strings.HasSuffix(r.URL.Path, "/json"):
http.NotFound(w, r)
case strings.HasSuffix(r.URL.Path, "/archive"):
w.Header().Set("X-Docker-Container-Path-Stat", "e30=")
case r.Method == http.MethodDelete:
w.WriteHeader(http.StatusNoContent)
case r.URL.Path == "/exec/detached/start" || r.URL.Path == "/exec/error/start":
if r.URL.Path == "/exec/error/start" {
w.Header().Set("Content-Type", "application/vnd.docker.raw-stream")
@@ -211,6 +230,31 @@ func TestDockerProxy(t *testing.T) {
assert.Equal(t, "OK GET /v1.47/_ping", string(body))
})
t.Run("discovery caches only successes", func(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("automatic proxy discovery requires Linux")
}
t.Setenv("DOCKER_HOST", "unix://"+daemonSocket)
t.Setenv("TMPDIR", shortTempDir(t))
require.Nil(t, NewDockerProxy(t.Context(), "failed"))
for _, job := range []string{"first", "second"} {
proxy := NewDockerProxy(t.Context(), job)
require.NotNil(t, proxy)
require.NoError(t, proxy.Close(context.Background()))
}
assert.EqualValues(t, 2, probes.Load())
dockerProxyDirectories.Range(func(key, value any) bool {
directory, _ := value.(dockerProxyDirectory)
directory.socket, _ = os.Stat(t.TempDir())
dockerProxyDirectories.Store(key, directory)
return true
})
proxy := NewDockerProxy(t.Context(), "replaced daemon")
require.NotNil(t, proxy)
require.NoError(t, proxy.Close(context.Background()))
assert.EqualValues(t, 3, probes.Load())
})
t.Run("tunnels raw streams through stdin EOF", func(t *testing.T) {
for _, stream := range []struct{ path, upgrade string }{
{path: "/v1.47/exec/abc/start"},
@@ -328,6 +372,9 @@ func TestDockerProxyWithDaemon(t *testing.T) {
job := "proxy-test-" + strconv.FormatInt(time.Now().UnixNano(), 36)
proxy, err := StartDockerProxy(daemonSocketPath(t, direct), dir, job)
if os.Geteuid() != 0 && errors.Is(err, os.ErrPermission) {
t.Skipf("non-root runners that cannot take the daemon socket's ownership get it directly: %v", err)
}
require.NoError(t, err)
t.Cleanup(func() { _ = proxy.Close(context.Background()) })
viaProxy, err := mobyclient.New(mobyclient.WithHost("unix://" + proxy.Socket))
@@ -440,7 +487,7 @@ func (c *probeClient) ContainerInspect(context.Context, string, mobyclient.Conta
}
func (c *probeClient) ImageList(context.Context, mobyclient.ImageListOptions) (mobyclient.ImageListResult, error) {
return mobyclient.ImageListResult{Items: []image.Summary{{ID: "probe-image"}}}, nil
return mobyclient.ImageListResult{Items: []image.Summary{{ID: "removed-image"}, {ID: "probe-image"}}}, nil
}
func (c *probeClient) ContainerCreate(_ context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
@@ -455,6 +502,139 @@ func (c *probeClient) ContainerStatPath(_ context.Context, _ string, opts mobycl
return mobyclient.ContainerStatPathResult{}, c.stat(opts.Path)
}
func TestDockerProxyDirectoryProbes(t *testing.T) {
type result struct {
seen bool
err error
}
for name, createErr := range map[string]error{"unavailable directory": nil, "daemon error": errors.New("probe failed")} {
t.Run(name, func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
socket, err := os.Stat(t.TempDir())
require.NoError(t, err)
key := [3]string{"daemon", dir, dir}
release := make(chan struct{})
var calls, removals atomic.Int32
visible := false
cli := &probeClient{
create: func(_opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
calls.Add(1)
<-release
return mobyclient.ContainerCreateResult{ID: "probe"}, createErr
},
stat: func(_path string) error {
if visible {
return nil
} else {
return cerrdefs.ErrNotFound
}
},
remove: func(_ctx context.Context, _id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
removals.Add(1)
return mobyclient.ContainerRemoveResult{}, nil
},
}
results := make(chan result, 8)
for range cap(results) {
go func() {
_, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
results <- result{seen, err}
}()
}
synctest.Wait()
assert.EqualValues(t, 1, calls.Load())
ctx, cancel := context.WithCancel(t.Context())
cancel()
_, seen, err := probeDockerProxyDirectory(ctx, cli, key, socket)
require.ErrorIs(t, err, context.Canceled)
assert.False(t, seen)
replacement, err := os.Stat(t.TempDir())
require.NoError(t, err)
replaced := make(chan result, 1)
go func() {
_, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, replacement)
replaced <- result{seen, err}
}()
synctest.Wait()
assert.EqualValues(t, 2, calls.Load())
close(release)
for range cap(results) {
result := <-results
require.ErrorIs(t, result.err, createErr)
assert.False(t, result.seen)
}
replacementResult := <-replaced
require.ErrorIs(t, replacementResult.err, createErr)
assert.False(t, replacementResult.seen)
assert.EqualValues(t, 2, calls.Load())
if createErr == nil {
assert.EqualValues(t, 2, removals.Load())
} else {
assert.Zero(t, removals.Load())
}
createErr, visible = nil, true
directory, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
require.NoError(t, err)
assert.True(t, seen)
assert.Equal(t, dir, directory.dir)
assert.Equal(t, dir, directory.daemonDir)
assert.EqualValues(t, 3, calls.Load())
entries, err := os.ReadDir(dir)
require.NoError(t, err)
assert.Empty(t, entries)
})
})
}
t.Run("cancelled probe does not cancel another caller", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
socket, err := os.Stat(t.TempDir())
require.NoError(t, err)
key := [3]string{"daemon", dir, dir}
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
release := make(chan struct{})
calls := 0
cli := &probeClient{
create: func(_opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
calls++
if calls == 1 {
<-release
return mobyclient.ContainerCreateResult{}, ctx.Err()
} else {
return mobyclient.ContainerCreateResult{ID: "probe"}, nil
}
},
stat: func(_path string) error { return nil },
remove: func(_ctx context.Context, _id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
return mobyclient.ContainerRemoveResult{}, nil
},
}
leader, follower := make(chan result, 1), make(chan result, 1)
go func() {
_, seen, err := probeDockerProxyDirectory(ctx, cli, key, socket)
leader <- result{seen, err}
}()
synctest.Wait()
go func() {
_, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
follower <- result{seen, err}
}()
synctest.Wait()
assert.Equal(t, 1, calls)
cancel()
close(release)
leaderResult, followerResult := <-leader, <-follower
require.ErrorIs(t, leaderResult.err, context.Canceled)
assert.False(t, leaderResult.seen)
require.NoError(t, followerResult.err)
assert.True(t, followerResult.seen)
assert.Equal(t, 2, calls)
})
})
}
func TestDaemonSeesDir(t *testing.T) {
dir := t.TempDir()
markers := make(map[string]bool)
@@ -474,9 +654,14 @@ func TestDaemonSeesDir(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
removed := false
var attemptedImages []string
cli := &probeClient{
create: func(opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
assert.Equal(t, []mount.Mount{{Type: mount.TypeBind, Source: testCase.daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}}, opts.HostConfig.Mounts)
attemptedImages = append(attemptedImages, opts.Config.Image)
if opts.Config.Image == "removed-image" {
return mobyclient.ContainerCreateResult{}, cerrdefs.ErrNotFound
}
cancel()
if _, err := os.Stat(testCase.daemonDir); err != nil {
return mobyclient.ContainerCreateResult{}, cerrdefs.ErrInvalidArgument
@@ -504,6 +689,7 @@ func TestDaemonSeesDir(t *testing.T) {
}
seen, err := daemonSeesDir(ctx, cli, dir, testCase.daemonDir)
require.ErrorIs(t, err, testCase.removeErr)
assert.Equal(t, []string{"removed-image", "probe-image"}, attemptedImages)
assert.Equal(t, testCase.seen, seen)
assert.Equal(t, testCase.removed, removed)
entries, err := os.ReadDir(dir)

View File

@@ -114,7 +114,7 @@ func getImagePullOptions(ctx context.Context, input NewDockerPullExecutorInput)
if err != nil {
return imagePullOptions, err
}
if authConfig.Username == "" && authConfig.Password == "" {
if authConfig.Username == "" && authConfig.Password == "" && authConfig.IdentityToken == "" && authConfig.RegistryToken == "" {
return imagePullOptions, nil
}
logger.Info("using DockerAuthConfig authentication for docker pull")

View File

@@ -6,13 +6,17 @@ package container
import (
"io"
"os"
"path/filepath"
"strings"
"testing"
"testing/iotest"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"github.com/docker/cli/cli/config"
"github.com/moby/moby/api/pkg/authconfig"
"github.com/moby/moby/api/types/registry"
assert "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -44,10 +48,7 @@ func TestCleanImage(t *testing.T) {
func TestGetImagePullOptions(t *testing.T) {
ctx := t.Context()
orig := config.Dir()
t.Cleanup(func() { config.SetDir(orig) })
config.SetDir("/non-existent/docker")
t.Setenv("DOCKER_CONFIG", "/non-existent/docker")
options, err := getImagePullOptions(ctx, NewDockerPullExecutorInput{})
assert.NoError(t, err, "Failed to create ImagePullOptions") //nolint:testifylint // pre-existing issue from nektos/act
@@ -61,13 +62,28 @@ func TestGetImagePullOptions(t *testing.T) {
assert.NoError(t, err, "Failed to create ImagePullOptions") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, "eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwicGFzc3dvcmQiOiJwYXNzd29yZCJ9", options.RegistryAuth, "Username and Password should be provided")
config.SetDir("testdata/docker-pull-options")
t.Setenv("DOCKER_CONFIG", "testdata/docker-pull-options")
options, err = getImagePullOptions(ctx, NewDockerPullExecutorInput{
Image: "nektos/act",
})
assert.NoError(t, err, "Failed to create ImagePullOptions") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, "eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwicGFzc3dvcmQiOiJwYXNzd29yZFxuIiwic2VydmVyYWRkcmVzcyI6Imh0dHBzOi8vaW5kZXguZG9ja2VyLmlvL3YxLyJ9", options.RegistryAuth, "RegistryAuth should be taken from local docker config")
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte(`{"auths":{"identity.example.com":{"identitytoken":"identity-token"},"registry.example.com":{"registrytoken":"registry-token"}}}`), 0o600))
t.Setenv("DOCKER_CONFIG", dir)
for host, want := range map[string]registry.AuthConfig{
"identity.example.com": {IdentityToken: "identity-token", ServerAddress: "identity.example.com"},
"registry.example.com": {RegistryToken: "registry-token", ServerAddress: "registry.example.com"},
} {
options, err := getImagePullOptions(ctx, NewDockerPullExecutorInput{Image: host + "/app"})
require.NoError(t, err)
require.NotEmpty(t, options.RegistryAuth, host)
got, err := authconfig.Decode(options.RegistryAuth)
require.NoError(t, err)
assert.Equal(t, want, *got)
}
}
// A digest-pinned image is immutable, so its local copy is always current.
@@ -86,4 +102,8 @@ func TestLogDockerResponseError(t *testing.T) {
require.EqualError(t, streamErr(`{"error":"toomanyrequests: rate limit exceeded"}`), "toomanyrequests: rate limit exceeded")
require.EqualError(t, streamErr(`{"errorDetail":{"message":"unexpected EOF"}}`), "unexpected EOF")
require.NoError(t, streamErr(`{"status":"Downloading"}`))
require.NoError(t, streamErr(`{"stream":"`+strings.Repeat("x", 70000)+`"}`))
require.ErrorIs(t, logDockerResponse(logger, io.NopCloser(io.MultiReader(
strings.NewReader(`{"status":"Downloading"}`+"\n"), iotest.ErrReader(io.ErrUnexpectedEOF),
)), false), io.ErrUnexpectedEOF)
}

View File

@@ -13,6 +13,7 @@ import (
"errors"
"fmt"
"io"
"net"
"os"
"path"
"path/filepath"
@@ -27,11 +28,8 @@ import (
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"dario.cat/mergo"
"github.com/bmatcuk/doublestar/v4"
cerrdefs "github.com/containerd/errdefs"
"github.com/docker/cli/cli/compose/loader"
"github.com/docker/cli/cli/connhelper"
"github.com/docker/cli/opts"
"github.com/moby/moby/api/pkg/stdcopy"
"github.com/moby/moby/api/types/container"
@@ -52,15 +50,12 @@ const drainGracePeriod = 2 * time.Second
// NewContainer creates a reference to a container
func NewContainer(input *NewContainerInput) ExecutionsEnvironment {
cr := new(containerReference)
cr.input = input
// Resolved up front because the image pull runs before the container is created.
cf := createFlagsFromOptions(input.allOptions())
if cf.platform != "" {
cr.input.Platform = cf.platform
input.Platform = cf.platform
}
cr.pullPolicy = cf.pull
return cr
return &containerReference{input: input, pullPolicy: cf.pull}
}
// supportsContainerImagePlatform reports whether the Docker server API version
@@ -210,18 +205,30 @@ func (cr *containerReference) DumpLogs(ctx context.Context) error {
if cr.id == "" {
return cr.missingContainerError("read its logs")
}
tty, err := cr.containerTTY(ctx)
if err != nil {
return err
}
logs, err := cr.cli.ContainerLogs(ctx, cr.id, client.ContainerLogsOptions{ShowStdout: true, ShowStderr: true})
if err != nil {
return err
}
defer logs.Close()
return cr.copyOutput(logs)
return cr.copyOutput(logs, tty)
}
func (cr *containerReference) containerTTY(ctx context.Context) (bool, error) {
inspect, err := cr.cli.ContainerInspect(ctx, cr.id, client.ContainerInspectOptions{})
if err != nil {
return false, fmt.Errorf("failed to inspect container: %w", err)
}
return inspect.Container.Config != nil && inspect.Container.Config.Tty, nil
}
// copyOutput writes a container stream to the writers the container was created with,
// demultiplexing it unless the container has a TTY, which sends a single raw stream.
func (cr *containerReference) copyOutput(stream io.Reader) error {
func (cr *containerReference) copyOutput(stream io.Reader, tty bool) error {
outWriter := cr.input.Stdout
if outWriter == nil {
outWriter = os.Stdout
@@ -232,7 +239,7 @@ func (cr *containerReference) copyOutput(stream io.Reader) error {
}
var err error
if !cr.input.AllocatePTY || os.Getenv("NORAW") != "" {
if !tty || os.Getenv("NORAW") != "" {
_, err = stdcopy.StdCopy(outWriter, errWriter, stream)
} else {
_, err = io.Copy(outWriter, stream)
@@ -301,7 +308,7 @@ func containerInfoFromInspect(inspect container.InspectResponse) *Info {
}
func (cr *containerReference) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(cr, srcPath, env).IfNot(common.Dryrun)
return parseEnvFile(cr, srcPath, env, false).IfNot(common.Dryrun)
}
func (cr *containerReference) UpdateFromImageEnv(env *map[string]string) common.Executor {
@@ -372,15 +379,13 @@ func GetDockerClient(ctx context.Context) (cli client.APIClient, err error) {
dockerHost := os.Getenv("DOCKER_HOST")
if strings.HasPrefix(dockerHost, "ssh://") {
var helper *connhelper.ConnectionHelper
helper, err = connhelper.GetConnectionHelper(dockerHost)
if err != nil {
var args []string
if args, err = sshArgs(dockerHost); err != nil {
return nil, err
}
cli, err = client.New(
client.WithHost(helper.Host),
client.WithDialContext(helper.Dialer),
client.WithHost("http://docker.example.com"), // the dialer below ignores the address
client.WithDialContext(func(context.Context, string, string) (net.Conn, error) { return dialCommand("ssh", args...) }),
)
} else {
cli, err = client.New(client.FromEnv)
@@ -398,17 +403,16 @@ func GetDockerClient(ctx context.Context) (cli client.APIClient, err error) {
return cli, nil
}
func GetHostInfo(ctx context.Context) (info system.Info, err error) {
var cli client.APIClient
cli, err = GetDockerClient(ctx)
func GetHostInfo(ctx context.Context) (system.Info, error) {
cli, err := GetDockerClient(ctx)
if err != nil {
return info, err
return system.Info{}, err
}
defer cli.Close()
result, err := cli.Info(ctx, client.InfoOptions{})
if err != nil {
return info, err
return system.Info{}, err
}
return result.Info, nil
@@ -480,22 +484,23 @@ func (cr *containerReference) find() common.Executor {
}
cr.id = ""
}
containers, err := cr.cli.ContainerList(ctx, client.ContainerListOptions{
All: true,
})
result, err := cr.cli.ContainerInspect(ctx, cr.input.Name, client.ContainerInspectOptions{})
if err == nil && result.Container.Name == "/"+cr.input.Name {
cr.id = result.Container.ID
return nil
} else if cerrdefs.IsNotFound(err) {
return nil
}
containers, err := cr.cli.ContainerList(ctx, client.ContainerListOptions{All: true}) // inspect also matches ids, and podman fails it with a 500 for a container removed mid-request
if err != nil {
return fmt.Errorf("failed to list containers: %w", err)
}
for _, c := range containers.Items {
for _, name := range c.Names {
if name[1:] == cr.input.Name {
cr.id = c.ID
return nil
}
for _, candidate := range containers.Items {
if slices.Contains(candidate.Names, "/"+cr.input.Name) {
cr.id = candidate.ID
break
}
}
return nil
}
}
@@ -506,7 +511,7 @@ func (cr *containerReference) remove() common.Executor {
if idOrName == "" && cr.input != nil {
idOrName = cr.input.Name
}
if idOrName == "" {
if idOrName == "" || cr.cli == nil {
return nil
}
@@ -562,32 +567,32 @@ func (input *NewContainerInput) allOptions() string {
return strings.TrimSpace(input.RunnerOptions + " " + input.WorkflowOptions)
}
func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config *container.Config, hostConfig *container.HostConfig) (*container.Config, *container.HostConfig, error) {
func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config *container.Config, hostConfig *container.HostConfig) error {
logger := common.Logger(ctx)
options := cr.input.allOptions()
if options == "" {
return config, hostConfig, nil
return nil
}
// For Gitea, checked here because the parse below is what would read those files
if err := rejectHostReadingOptions(cr.input.WorkflowOptions); err != nil {
return nil, nil, err
return err
}
// parse configuration from CLI container.options
flags, copts, cf, err := parseContainerOptions(options)
if err != nil {
return nil, nil, err
return err
}
if err := cf.validate(); err != nil {
return nil, nil, fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
return fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
}
if len(copts.netMode.Value()) == 0 {
if err = copts.netMode.Set(cr.input.NetworkMode); err != nil {
return nil, nil, fmt.Errorf("cannot parse networkmode=%s. This is an internal error and should not happen: '%w'", cr.input.NetworkMode, err)
return fmt.Errorf("cannot parse networkmode=%s. This is an internal error and should not happen: '%w'", cr.input.NetworkMode, err)
}
}
@@ -600,7 +605,7 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
containerConfig, err := parse(flags, copts, runtime.GOOS)
if err != nil {
return nil, nil, fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
return fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
}
// workflow aliases join the runner's own, deduped so a re-create cannot grow the input
for _, endpoint := range containerConfig.NetworkingConfig.EndpointsConfig {
@@ -615,30 +620,24 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
if !hostConfig.Privileged {
trusted, err := parseOptionsHostConfig(cr.input.RunnerOptions)
if err != nil {
return nil, nil, err
return err
}
sanitizeOptionsHostConfig(logger, containerConfig.HostConfig, trusted)
}
logger.Debugf("Custom container.Config from options ==> %+v", containerConfig.Config)
err = mergo.Merge(config, containerConfig.Config, mergo.WithOverride, mergo.WithAppendSlice)
if err != nil {
return nil, nil, fmt.Errorf("cannot merge container.Config options: '%s': '%w'", options, err)
}
overlay(reflect.ValueOf(config).Elem(), reflect.ValueOf(containerConfig.Config).Elem(), true)
logger.Debugf("Merged container.Config ==> %+v", config)
logger.Debugf("Custom container.HostConfig from options ==> %+v", containerConfig.HostConfig)
cr.sanitizeConfig(ctx, nil, containerConfig.HostConfig) // a dropped volume must not displace the runner's own
cr.sanitizeConfig(ctx, containerConfig.HostConfig) // a dropped volume must not displace the runner's own
overlayVolumes(hostConfig, containerConfig.HostConfig)
binds := hostConfig.Binds
mounts := hostConfig.Mounts
networkMode := hostConfig.NetworkMode
err = mergo.Merge(hostConfig, containerConfig.HostConfig, mergo.WithOverride)
if err != nil {
return nil, nil, fmt.Errorf("cannot merge container.HostConfig options: '%s': '%w'", options, err)
}
overlay(reflect.ValueOf(hostConfig).Elem(), reflect.ValueOf(containerConfig.HostConfig).Elem(), false)
hostConfig.Binds = binds
hostConfig.Mounts = mounts
if cf.name != "" {
@@ -654,7 +653,7 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
hostConfig.NetworkMode = networkMode
logger.Debugf("Merged container.HostConfig ==> %+v", hostConfig)
return config, hostConfig, nil
return nil
}
func (cr *containerReference) create(capAdd, capDrop []string) common.Executor {
@@ -720,8 +719,7 @@ func (cr *containerReference) create(capAdd, capDrop []string) common.Executor {
AutoRemove: input.AutoRemove,
}
config, hostConfig, err = cr.mergeContainerConfigs(ctx, config, hostConfig)
if err != nil {
if err := cr.mergeContainerConfigs(ctx, config, hostConfig); err != nil {
return err
}
@@ -730,18 +728,15 @@ func (cr *containerReference) create(capAdd, capDrop []string) common.Executor {
}
// For Gitea
config, hostConfig = cr.sanitizeConfig(ctx, config, hostConfig)
cr.sanitizeConfig(ctx, hostConfig)
var networkingConfig *network.NetworkingConfig
n := hostConfig.NetworkMode
// IsUserDefined and IsHost are broken on windows
if n.IsUserDefined() && n != "host" && len(input.NetworkAliases) > 0 {
endpointConfig := &network.EndpointSettings{
Aliases: input.NetworkAliases,
}
networkingConfig = &network.NetworkingConfig{
EndpointsConfig: map[string]*network.EndpointSettings{
input.NetworkMode: endpointConfig,
input.NetworkMode: {Aliases: input.NetworkAliases},
},
}
}
@@ -782,15 +777,11 @@ func (cr *containerReference) exec(cmd []string, env map[string]string, user, wo
logger.Debugf("Exec command '%s'", cmd)
isTerminal := cr.input.AllocatePTY
var wd string
if workdir != "" {
if strings.HasPrefix(workdir, "/") {
wd = workdir
} else {
wd = fmt.Sprintf("%s/%s", cr.input.WorkingDir, workdir)
}
} else {
wd = cr.input.WorkingDir
wd := cr.input.WorkingDir
if strings.HasPrefix(workdir, "/") {
wd = workdir
} else if workdir != "" {
wd = fmt.Sprintf("%s/%s", cr.input.WorkingDir, workdir)
}
logger.Debugf("Working directory '%s'", wd)
@@ -815,8 +806,7 @@ func (cr *containerReference) exec(cmd []string, env map[string]string, user, wo
}
defer resp.Close()
err = cr.waitForCommand(ctx, resp.HijackedResponse)
if err != nil {
if err := cr.waitForCommand(ctx, resp.HijackedResponse); err != nil {
return err
}
@@ -873,7 +863,7 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
cmdResponse := make(chan error, 1)
go func() {
cmdResponse <- cr.copyOutput(resp.Reader)
cmdResponse <- cr.copyOutput(resp.Reader, cr.input.AllocatePTY)
}()
select {
@@ -889,7 +879,10 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
// truncate the tail of the log. The goroutine exits once the hijacked
// stream is closed by resp.Close() in the caller's defer.
select {
case <-cmdResponse:
case err := <-cmdResponse:
if err != nil {
logger.Debugf("Failed to drain command output after cancellation: %v", err)
}
case <-time.After(drainGracePeriod):
logger.Warn("Timed out draining command output after cancellation")
}
@@ -898,11 +891,7 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
// from executing
return ctx.Err()
case err := <-cmdResponse:
if err != nil {
logger.Error(err)
}
return nil
return err
}
}
@@ -918,17 +907,14 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
return err
}
logger.Debugf("Writing tarball %s from %s", tarFile.Name(), srcPath)
defer func(tarFile *os.File) {
name := tarFile.Name()
err := tarFile.Close()
if !errors.Is(err, os.ErrClosed) {
defer func() {
if err := tarFile.Close(); err != nil && !errors.Is(err, os.ErrClosed) {
logger.Error(err)
}
err = os.Remove(name)
if err != nil {
if err := os.Remove(tarFile.Name()); err != nil {
logger.Error(err)
}
}(tarFile)
}()
extractionRoot := "/"
if actPath := cr.GetActPath(); strings.HasPrefix(dstPath+"/", actPath+"/") {
extractionRoot = actPath // keeps tar entries from crossing the image's /var/run symlink
@@ -938,8 +924,7 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
}
logger.Debugf("Extracting content from '%s' to '%s'", tarFile.Name(), dstPath)
_, err = tarFile.Seek(0, 0)
if err != nil {
if _, err := tarFile.Seek(0, 0); err != nil {
return fmt.Errorf("failed to seek tar archive: %w", err)
}
return cr.upload(ctx, extractionRoot, tarFile)
@@ -970,6 +955,10 @@ func (cr *containerReference) upload(ctx context.Context, dstPath string, conten
func (cr *containerReference) attach() common.Executor {
return func(ctx context.Context) error {
tty, err := cr.containerTTY(ctx)
if err != nil {
return err
}
out, err := cr.cli.ContainerAttach(ctx, cr.id, client.ContainerAttachOptions{
Stream: true,
Stdout: true,
@@ -980,9 +969,12 @@ func (cr *containerReference) attach() common.Executor {
}
done := make(chan struct{})
cr.attachDone = done
stopClose := context.AfterFunc(ctx, out.Close)
go func() {
defer close(done)
if copyErr := cr.copyOutput(out.Reader); copyErr != nil {
defer out.Close()
defer stopClose()
if copyErr := cr.copyOutput(out.Reader, tty); copyErr != nil && ctx.Err() == nil {
common.Logger(ctx).Error(copyErr)
}
}()
@@ -1017,6 +1009,9 @@ func (cr *containerReference) wait() common.Executor {
return fmt.Errorf("failed to wait for container: %w", err)
}
case status := <-waitResult.Result:
if status.Error != nil {
return fmt.Errorf("failed to wait for container: %s", status.Error.Message)
}
statusCode = status.StatusCode
}
@@ -1107,9 +1102,7 @@ func parseOptionsHostConfig(options string) (*container.HostConfig, error) {
return containerConfig.HostConfig, nil
}
// For Gitea
// sanitizeConfig remove the invalid configurations from `config` and `hostConfig`
func (cr *containerReference) sanitizeConfig(ctx context.Context, config *container.Config, hostConfig *container.HostConfig) (*container.Config, *container.HostConfig) {
func (cr *containerReference) sanitizeConfig(ctx context.Context, hostConfig *container.HostConfig) {
logger := common.Logger(ctx)
matcher := newValidVolumeMatcher(ctx, cr.input.ValidVolumes)
@@ -1117,7 +1110,7 @@ func (cr *containerReference) sanitizeConfig(ctx context.Context, config *contai
// sanitize binds
sanitizedBinds := make([]string, 0, len(hostConfig.Binds))
for _, bind := range hostConfig.Binds {
parsed, err := loader.ParseVolume(bind)
parsed, err := ParseVolume(bind)
if err != nil {
logger.Warnf("parse volume [%s] error: %v", bind, err)
continue
@@ -1127,8 +1120,8 @@ func (cr *containerReference) sanitizeConfig(ctx context.Context, config *contai
sanitizedBinds = append(sanitizedBinds, bind)
continue
}
if matcher.isValid(parsed.Source, mount.Type(parsed.Type), parsed.ReadOnly) {
if matcher.needsNoCopy(parsed.Source, mount.Type(parsed.Type)) && (parsed.Volume == nil || !parsed.Volume.NoCopy) {
if matcher.isValid(parsed.Source, parsed.Type, parsed.ReadOnly) {
if matcher.needsNoCopy(parsed.Source, parsed.Type) && !parsed.NoCopy {
bind += ",nocopy"
}
sanitizedBinds = append(sanitizedBinds, bind)
@@ -1158,19 +1151,65 @@ func (cr *containerReference) sanitizeConfig(ctx context.Context, config *contai
}
}
hostConfig.Mounts = sanitizedMounts
return config, hostConfig
}
func SanitizeBinds(ctx context.Context, validVolumes, binds []string) []string {
cr := &containerReference{input: &NewContainerInput{ValidVolumes: validVolumes}}
_, hostConfig := cr.sanitizeConfig(ctx, nil, &container.HostConfig{Binds: binds})
hostConfig := &container.HostConfig{Binds: binds}
cr.sanitizeConfig(ctx, hostConfig)
return hostConfig.Binds
}
func overlay(dst, src reflect.Value, appendSlices bool) {
switch dst.Kind() {
case reflect.Struct:
opaque := true
for i := range dst.NumField() {
if dst.Type().Field(i).IsExported() {
opaque = false
overlay(dst.Field(i), src.Field(i), appendSlices)
}
}
if opaque {
dst.Set(src)
}
case reflect.Map:
if src.IsNil() {
return
}
if dst.IsNil() {
dst.Set(reflect.MakeMap(dst.Type()))
}
for key, value := range src.Seq2() {
dst.SetMapIndex(key, value)
}
case reflect.Slice:
if appendSlices {
dst.Set(reflect.AppendSlice(dst, src))
} else if src.Len() > 0 {
dst.Set(src)
}
case reflect.Pointer:
if src.IsNil() {
return
}
if dst.IsNil() {
dst.Set(src)
return
}
overlay(dst.Elem(), src.Elem(), appendSlices)
case reflect.Array:
dst.Set(src)
default:
if !src.IsZero() {
dst.Set(src)
}
}
}
// bindTarget returns the container path a bind mounts onto, empty if it cannot be parsed.
func bindTarget(bind string) string {
parsed, err := loader.ParseVolume(bind)
parsed, err := ParseVolume(bind)
if err != nil {
return ""
}

View File

@@ -13,9 +13,13 @@ import (
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"gitea.com/gitea/runner/act/common"
@@ -88,6 +92,16 @@ func (m *mockDockerClient) ContainerAttach(ctx context.Context, containerID stri
return args.Get(0).(mobyclient.ContainerAttachResult), args.Error(1)
}
func (m *mockDockerClient) ContainerLogs(ctx context.Context, containerID string, opts mobyclient.ContainerLogsOptions) (mobyclient.ContainerLogsResult, error) {
args := m.Called(ctx, containerID, opts)
return args.Get(0).(mobyclient.ContainerLogsResult), args.Error(1)
}
func (m *mockDockerClient) ContainerStart(ctx context.Context, containerID string, opts mobyclient.ContainerStartOptions) (mobyclient.ContainerStartResult, error) {
args := m.Called(ctx, containerID, opts)
return args.Get(0).(mobyclient.ContainerStartResult), args.Error(1)
}
func (m *mockDockerClient) ContainerWait(ctx context.Context, containerID string, opts mobyclient.ContainerWaitOptions) mobyclient.ContainerWaitResult {
args := m.Called(ctx, containerID, opts)
return args.Get(0).(mobyclient.ContainerWaitResult)
@@ -154,6 +168,7 @@ func (r *interruptReader) Read(_ []byte) (int, error) {
type mockConn struct {
net.Conn
mock.Mock
closed atomic.Bool
}
func (m *mockConn) Write(b []byte) (n int, err error) {
@@ -162,6 +177,7 @@ func (m *mockConn) Write(b []byte) (n int, err error) {
}
func (m *mockConn) Close() (err error) {
m.closed.Store(true)
return nil
}
@@ -182,11 +198,9 @@ func TestDockerExecAbort(t *testing.T) {
}, nil)
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
id: "123",
cli: client,
input: &NewContainerInput{},
}
channel := make(chan error)
@@ -209,12 +223,10 @@ func TestDockerExecAbort(t *testing.T) {
func TestDockerExecFailure(t *testing.T) {
ctx := t.Context()
conn := &mockConn{}
client := &mockDockerClient{}
client.On("ExecCreate", ctx, "123", mock.AnythingOfType("client.ExecCreateOptions")).Return(mobyclient.ExecCreateResult{ID: "id"}, nil)
client.On("ExecAttach", ctx, "id", mock.AnythingOfType("client.ExecAttachOptions")).Return(mobyclient.ExecAttachResult{
Conn: conn,
Conn: &mockConn{},
Reader: bufio.NewReader(strings.NewReader("output")),
}, nil)
client.On("ExecInspect", ctx, "id", mobyclient.ExecInspectOptions{}).Return(mobyclient.ExecInspectResult{
@@ -222,11 +234,9 @@ func TestDockerExecFailure(t *testing.T) {
}, nil)
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
id: "123",
cli: client,
input: &NewContainerInput{},
}
err := cr.exec([]string{""}, map[string]string{}, "user", "workdir")(ctx)
@@ -235,7 +245,21 @@ func TestDockerExecFailure(t *testing.T) {
assert.Equal(t, ExitCodeError(1), exitErr)
assert.Equal(t, "Process completed with exit code 1.", err.Error())
conn.AssertExpectations(t)
client.AssertExpectations(t)
}
func TestDockerExecStreamFailure(t *testing.T) {
ctx := t.Context()
client := &mockDockerClient{}
client.On("ExecCreate", ctx, "123", mock.AnythingOfType("client.ExecCreateOptions")).
Return(mobyclient.ExecCreateResult{ID: "id"}, nil)
client.On("ExecAttach", ctx, "id", mock.AnythingOfType("client.ExecAttachOptions")).
Return(mobyclient.ExecAttachResult{
Conn: &mockConn{},
Reader: bufio.NewReader(bytes.NewReader(stdcopyFrame(stdcopy.Systemerr, "unable to start exec"))),
}, nil)
require.ErrorContains(t, (&containerReference{id: "123", cli: client, input: &NewContainerInput{}}).
exec([]string{"sh"}, nil, "", "")(ctx), "unable to start exec")
client.AssertExpectations(t)
}
@@ -250,86 +274,89 @@ func stdcopyFrame(stream stdcopy.StdType, payload string) []byte {
return b
}
// TestDockerAttachFlushesTrailingLine verifies that wait() blocks until the
// attach() streaming goroutine has drained and flushed the container's output,
// so a final line without a trailing newline is not lost.
func TestDockerAttachFlushesTrailingLine(t *testing.T) {
ctx := t.Context()
for _, testcase := range []struct {
name string
tty bool
}{
{name: "multiplexed"},
{name: "tty from container options", tty: true},
} {
t.Run(testcase.name, func(t *testing.T) {
ctx := t.Context()
output := []byte("line one\nlast line without newline")
if !testcase.tty {
output = stdcopyFrame(stdcopy.Stdout, string(output))
}
var lines []string
logWriter := common.NewLineWriter(func(line string) bool {
lines = append(lines, line)
return true
})
client := &mockDockerClient{}
client.On("ContainerInspect", ctx, "123", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{Container: container.InspectResponse{Config: &container.Config{Tty: testcase.tty}}}, nil)
conn := &mockConn{}
client.On("ContainerAttach", ctx, "123", mock.AnythingOfType("client.ContainerAttachOptions")).
Return(mobyclient.ContainerAttachResult{
Conn: conn,
Reader: bufio.NewReader(bytes.NewReader(output)),
}, nil)
client.On("ContainerLogs", ctx, "123", mobyclient.ContainerLogsOptions{ShowStdout: true, ShowStderr: true}).
Return(io.NopCloser(bytes.NewReader(output)), nil)
statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 0}
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{Result: statusCh})
cr := &containerReference{id: "123", cli: client, input: &NewContainerInput{Stdout: logWriter, Stderr: logWriter}}
framed := bytes.NewBuffer(stdcopyFrame(stdcopy.Stdout, "line one\nlast line without newline"))
var lines []string
logWriter := common.NewLineWriter(func(s string) bool {
lines = append(lines, s)
return true
})
client := &mockDockerClient{}
client.On("ContainerAttach", ctx, "123", mock.AnythingOfType("client.ContainerAttachOptions")).
Return(mobyclient.ContainerAttachResult{
Conn: &mockConn{},
Reader: bufio.NewReader(framed),
}, nil)
statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 0}
errCh := make(chan error, 1)
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{
Result: (<-chan container.WaitResponse)(statusCh),
Error: (<-chan error)(errCh),
require.NoError(t, cr.attach()(ctx))
require.NoError(t, cr.wait()(ctx))
assert.Equal(t, []string{"line one\n", "last line without newline"}, lines)
assert.True(t, conn.closed.Load())
require.NoError(t, cr.DumpLogs(ctx))
assert.Equal(t, []string{"line one\n", "last line without newline", "line one\n", "last line without newline"}, lines)
client.AssertExpectations(t)
})
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
Stdout: logWriter,
Stderr: logWriter,
},
}
}
require.NoError(t, cr.attach()(ctx))
require.NoError(t, cr.wait()(ctx))
// wait() must have blocked until the goroutine drained AND flushed; the
// trailing, non-newline-terminated line must therefore be present. Reading
// lines here is race-free because wait() synchronizes on attachDone, which
// the goroutine closes after the final append.
assert.Equal(t, []string{"line one\n", "last line without newline"}, lines)
func TestDockerAttachStreamClosesWithItsContext(t *testing.T) {
conn, peer := net.Pipe()
t.Cleanup(func() { _ = conn.Close() })
t.Cleanup(func() { _ = peer.Close() })
client := &mockDockerClient{}
client.On("ContainerInspect", mock.Anything, "123", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{Container: container.InspectResponse{Config: &container.Config{}}}, nil)
client.On("ContainerAttach", mock.Anything, "123", mock.AnythingOfType("client.ContainerAttachOptions")).
Return(mobyclient.ContainerAttachResult{HijackedResponse: mobyclient.NewHijackedResponse(conn, "")}, nil)
client.On("ContainerStart", mock.Anything, "123", mobyclient.ContainerStartOptions{}).
Return(mobyclient.ContainerStartResult{}, errors.New("container start failed"))
cr := &containerReference{id: "123", cli: client, input: &NewContainerInput{}}
ctx, cancel := context.WithCancel(t.Context())
require.ErrorContains(t, cr.Start(true)(ctx), "container start failed")
cancel()
<-cr.attachDone
client.AssertExpectations(t)
}
func TestDockerWaitFailure(t *testing.T) {
ctx := t.Context()
statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 2}
errCh := make(chan error, 1)
client := &mockDockerClient{}
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{
Result: (<-chan container.WaitResponse)(statusCh),
Error: (<-chan error)(errCh),
})
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
}
Return(mobyclient.ContainerWaitResult{Result: statusCh})
cr := &containerReference{id: "123", cli: client}
err := cr.wait()(ctx)
var exitErr ExitCodeError
require.ErrorAs(t, err, &exitErr)
assert.Equal(t, ExitCodeError(2), exitErr)
assert.Equal(t, "Process completed with exit code 2.", err.Error())
statusCh <- container.WaitResponse{Error: &container.WaitExitError{Message: "container wait failed"}}
require.EqualError(t, cr.wait()(ctx), "failed to wait for container: container wait failed")
client.AssertExpectations(t)
}
@@ -398,62 +425,79 @@ func TestRemoveWithoutIDUsesName(t *testing.T) {
client.AssertExpectations(t)
}
// find() must drop a stale cached id so later Copy/Exec don't hit the
// daemon with a torn-down container.
func TestRemovePreservesClientCreationError(t *testing.T) {
t.Setenv("DOCKER_HOST", "://")
require.ErrorContains(t, (&containerReference{input: &NewContainerInput{Name: "job-1"}}).
Remove()(t.Context()), "failed to connect to docker daemon")
}
func TestFindRevalidatesStaleID(t *testing.T) {
ctx := t.Context()
inspectOpts := mobyclient.ContainerInspectOptions{}
notFound := cerrdefs.ErrNotFound.WithMessage("No such container")
boom := errors.New("daemon unreachable")
newCR := func(id string) (*containerReference, *mockDockerClient) {
client := &mockDockerClient{}
return &containerReference{id: id, cli: client, input: &NewContainerInput{Name: "job-1"}}, client
for _, tc := range []struct {
name, cachedID string
cachedErr error
lookup container.InspectResponse
lookupErr error
listed []container.Summary
wantID string
retryList bool
}{
{name: "stale id cleared, name lookup empty", cachedID: "stale", cachedErr: notFound, lookupErr: notFound},
{name: "stale id cleared, name lookup repopulates", cachedID: "stale", cachedErr: notFound, lookup: container.InspectResponse{ID: "fresh", Name: "/job-1"}, wantID: "fresh"},
{name: "id match falls back to listing", lookup: container.InspectResponse{ID: "job-1abc", Name: "/somebody-else"}, listed: []container.Summary{{ID: "fresh", Names: []string{"/job-1"}}}, wantID: "fresh"},
{name: "live id kept", cachedID: "live", wantID: "live"},
{name: "transient inspect error trusts cache", cachedID: "live", cachedErr: boom, wantID: "live"},
{name: "name lookup error falls back to listing", lookupErr: boom, listed: []container.Summary{{ID: "other", Names: []string{"/somebody-else"}}, {ID: "fresh", Names: []string{"/job-1"}}}, wantID: "fresh", retryList: true},
} {
t.Run(tc.name, func(t *testing.T) {
client := &mockDockerClient{}
cr := &containerReference{id: tc.cachedID, cli: client, input: &NewContainerInput{Name: "job-1"}}
if tc.cachedID != "" {
client.On("ContainerInspect", ctx, tc.cachedID, inspectOpts).Return(mobyclient.ContainerInspectResult{}, tc.cachedErr)
}
if tc.cachedID == "" || cerrdefs.IsNotFound(tc.cachedErr) {
client.On("ContainerInspect", ctx, "job-1", inspectOpts).Return(mobyclient.ContainerInspectResult{Container: tc.lookup}, tc.lookupErr)
}
if tc.listed != nil {
client.On("ContainerList", ctx, mobyclient.ContainerListOptions{All: true}).Return(mobyclient.ContainerListResult{Items: tc.listed}, nil).Once()
}
require.NoError(t, cr.find()(ctx))
assert.Equal(t, tc.wantID, cr.id)
if tc.retryList {
client.On("ContainerList", ctx, mobyclient.ContainerListOptions{All: true}).Return(mobyclient.ContainerListResult{}, boom)
cr.id = ""
require.ErrorIs(t, cr.find()(ctx), boom)
}
client.AssertExpectations(t)
})
}
listOpts := mobyclient.ContainerListOptions{All: true}
inspectOpts := mobyclient.ContainerInspectOptions{}
}
t.Run("stale id cleared, name lookup empty", func(t *testing.T) {
cr, client := newCR("stale")
client.On("ContainerInspect", ctx, "stale", inspectOpts).Return(mobyclient.ContainerInspectResult{}, notFound)
client.On("ContainerList", ctx, listOpts).Return(mobyclient.ContainerListResult{}, nil)
require.NoError(t, cr.find()(ctx))
assert.Empty(t, cr.id)
client.AssertExpectations(t)
})
t.Run("stale id cleared, name lookup repopulates", func(t *testing.T) {
cr, client := newCR("stale")
client.On("ContainerInspect", ctx, "stale", inspectOpts).Return(mobyclient.ContainerInspectResult{}, notFound)
client.On("ContainerList", ctx, listOpts).Return(mobyclient.ContainerListResult{Items: []container.Summary{
{ID: "other", Names: []string{"/somebody-else"}},
{ID: "fresh", Names: []string{"/job-1"}},
}}, nil)
require.NoError(t, cr.find()(ctx))
assert.Equal(t, "fresh", cr.id)
client.AssertExpectations(t)
})
t.Run("live id kept", func(t *testing.T) {
cr, client := newCR("live")
client.On("ContainerInspect", ctx, "live", inspectOpts).Return(mobyclient.ContainerInspectResult{}, nil)
require.NoError(t, cr.find()(ctx))
assert.Equal(t, "live", cr.id)
client.AssertExpectations(t)
})
t.Run("transient inspect error trusts cache", func(t *testing.T) {
cr, client := newCR("live")
client.On("ContainerInspect", ctx, "live", inspectOpts).Return(mobyclient.ContainerInspectResult{}, boom)
require.NoError(t, cr.find()(ctx))
assert.Equal(t, "live", cr.id)
client.AssertExpectations(t)
})
t.Run("list error propagates", func(t *testing.T) {
cr, client := newCR("")
client.On("ContainerList", ctx, listOpts).Return(mobyclient.ContainerListResult{}, boom)
require.ErrorIs(t, cr.find()(ctx), boom)
client.AssertExpectations(t)
})
func TestRunnerArchCachedPerRun(t *testing.T) {
var infos atomic.Int32
daemon := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
writer.Header().Set("Content-Type", "application/json")
writer.Header().Set("API-Version", "1.47")
if strings.HasSuffix(request.URL.Path, "/info") {
infos.Add(1)
_, _ = io.WriteString(writer, `{"Architecture":"aarch64"}`)
}
}))
t.Cleanup(daemon.Close)
t.Setenv("DOCKER_HOST", daemon.URL)
extensions := &LinuxContainerEnvironmentExtensions{}
run := WithRunnerArch(t.Context())
var lookups sync.WaitGroup
for range 4 {
lookups.Go(func() { assert.Equal(t, "ARM64", extensions.GetRunnerContext(run)["arch"]) })
}
lookups.Wait()
assert.Equal(t, "ARM64", extensions.GetRunnerContext(t.Context())["arch"])
assert.EqualValues(t, 2, infos.Load())
}
// Every daemon entry point fails fast with a clear, container-named
@@ -461,7 +505,8 @@ func TestFindRevalidatesStaleID(t *testing.T) {
func TestRejectsMissingContainer(t *testing.T) {
ctx := t.Context()
client := &mockDockerClient{}
client.On("ContainerList", ctx, mobyclient.ContainerListOptions{All: true}).Return(mobyclient.ContainerListResult{}, nil)
client.On("ContainerInspect", ctx, "job-1", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{}, cerrdefs.ErrNotFound)
cr := &containerReference{cli: client, input: &NewContainerInput{Name: "job-1"}}
check := func(op string, err error) {
t.Helper()
@@ -494,14 +539,13 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
{name: "sibling prefix", directory: true, destination: "/var/run/actor/action", extractionRoot: "/", entry: "var/run/actor/action/x"},
} {
t.Run(testcase.name, func(t *testing.T) {
ctx := t.Context()
hook := logtest.NewWithLevel(logging.LevelError)
ctx := common.WithLogger(t.Context(), logging.New(hook))
client := &mockDockerClient{}
client.On("ContainerInspect", ctx, "stale", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{}, cerrdefs.ErrNotFound.WithMessage("gone"))
client.On("ContainerList", ctx, mobyclient.ContainerListOptions{All: true}).
Return(mobyclient.ContainerListResult{Items: []container.Summary{
{ID: "fresh", Names: []string{"/job-1"}},
}}, nil)
client.On("ContainerInspect", ctx, "job-1", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{Container: container.InspectResponse{ID: "fresh", Name: "/job-1"}}, nil)
client.On("CopyToContainer", ctx, "fresh", mock.Anything).
Run(func(args mock.Arguments) {
opts := args.Get(2).(mobyclient.CopyToContainerOptions)
@@ -520,14 +564,12 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
require.NoError(t, cr.Copy(testcase.destination, &FileEntry{Name: "x", Mode: 0o644})(ctx))
}
assert.Equal(t, "fresh", cr.id)
assert.Empty(t, hook.AllRecords())
client.AssertExpectations(t)
})
}
}
// Type assert containerReference implements ExecutionsEnvironment
var _ ExecutionsEnvironment = &containerReference{}
func TestCheckVolumes(t *testing.T) {
testCases := []struct {
desc string
@@ -622,10 +664,11 @@ func TestCheckVolumes(t *testing.T) {
ValidVolumes: tc.validVolumes,
},
}
_, hostConf := cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{Binds: tc.binds, Mounts: tc.mounts})
hostConf := &container.HostConfig{Binds: tc.binds, Mounts: tc.mounts}
cr.sanitizeConfig(ctx, hostConf)
assert.Equal(t, tc.expectedBinds, hostConf.Binds)
assert.ElementsMatch(t, tc.expectedMounts, hostConf.Mounts)
assert.Len(t, hook.AllRecords(), len(tc.binds)+len(tc.mounts)-len(tc.expectedBinds)-len(tc.expectedMounts)) // every drop is warned about
assert.Len(t, hook.AllRecords(), len(tc.binds)+len(tc.mounts)-len(tc.expectedBinds)-len(tc.expectedMounts))
})
}
}
@@ -645,8 +688,6 @@ func TestMergeContainerConfigsDropsVolumeDriversFromWorkflows(t *testing.T) {
assert.Equal(t, "job-escape", hostConfig.Mounts[0].Source)
}
// Both of these are read here, on the runner, so a workflow could read the runner's files
// and environment with them.
func TestMergeContainerConfigsKeepsTheRunnersFilesAndEnvToItself(t *testing.T) {
hostFile := filepath.Join(t.TempDir(), "host.env")
require.NoError(t, os.WriteFile(hostFile, []byte("STOLEN=from-the-host\n"), 0o600))
@@ -656,16 +697,12 @@ func TestMergeContainerConfigsKeepsTheRunnersFilesAndEnvToItself(t *testing.T) {
logger := logging.New(logtest.New())
cr := &containerReference{input: &NewContainerInput{NetworkMode: "bridge", WorkflowOptions: option}}
_, _, err := cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{})
require.ErrorContains(t, err, "not allowed in a workflow")
require.ErrorContains(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{}), "not allowed in a workflow")
// the runner reading its own files is what those options are for
cr = &containerReference{input: &NewContainerInput{NetworkMode: "bridge", RunnerOptions: option}}
_, _, err = cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{})
require.NoError(t, err)
require.NoError(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{}))
}
// a bare name is no longer resolved from the runner's environment, for either source
logger := logging.New(logtest.New())
cr := &containerReference{input: &NewContainerInput{
NetworkMode: "bridge",
@@ -673,8 +710,8 @@ func TestMergeContainerConfigsKeepsTheRunnersFilesAndEnvToItself(t *testing.T) {
WorkflowOptions: "--env RUNNER_SECRET --env GIVEN=value",
}}
config, _, err := cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{})
require.NoError(t, err)
config := &container.Config{}
require.NoError(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), config, &container.HostConfig{}))
assert.Equal(t, []string{"RUNNER_SECRET", "RUNNER_SECRET", "GIVEN=value"}, config.Env)
}
@@ -711,7 +748,6 @@ func TestSanitizeOptionsHostConfig(t *testing.T) {
assert.Equal(t, &container.HostConfig{}, hostConfig)
}
// mergeOptions merges both option sources into a bare container, returning the result and its log.
func mergeOptions(t *testing.T, runnerOptions, workflowOptions string, privileged bool) (*container.HostConfig, *logtest.Handler) {
t.Helper()
logger, hook := logtest.NewLogger()
@@ -723,12 +759,12 @@ func mergeOptions(t *testing.T, runnerOptions, workflowOptions string, privilege
ValidVolumes: []string{"**"},
}}
_, hostConfig, err := cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{
hostConfig := &container.HostConfig{
Privileged: privileged,
UsernsMode: container.UsernsMode("private"),
NetworkMode: container.NetworkMode("bridge"),
})
require.NoError(t, err)
}
require.NoError(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, hostConfig))
return hostConfig, hook
}
@@ -782,23 +818,26 @@ func TestCheckVolumesRejectsEscapingHostPaths(t *testing.T) {
}
escapingPath := allowed + string(filepath.Separator) + ".." + string(filepath.Separator) + "denied"
_, hostConf := cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{
hostConf := &container.HostConfig{
Binds: []string{escapingPath + ":/mnt"},
})
}
cr.sanitizeConfig(ctx, hostConf)
assert.Empty(t, hostConf.Binds)
linkPath := filepath.Join(allowed, "link")
if err := os.Symlink(denied, linkPath); err != nil {
t.Skipf("cannot create symlink: %v", err)
}
_, hostConf = cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{
hostConf = &container.HostConfig{
Binds: []string{linkPath + ":/mnt"},
})
}
cr.sanitizeConfig(ctx, hostConf)
assert.Empty(t, hostConf.Binds)
_, hostConf = cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{
hostConf = &container.HostConfig{
Binds: []string{filepath.Join(linkPath, "missing") + ":/mnt"},
})
}
cr.sanitizeConfig(ctx, hostConf)
assert.Empty(t, hostConf.Binds)
}
@@ -808,11 +847,12 @@ func TestSanitizeConfigKeepsTmpfsMounts(t *testing.T) {
for _, validVolumes := range [][]string{nil, {"data"}, {"**:ro"}} {
logger, hook := logtest.NewLogger()
_, sanitized := (&containerReference{input: &NewContainerInput{ValidVolumes: validVolumes}}).sanitizeConfig(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{Mounts: []mount.Mount{
sanitized := &container.HostConfig{Mounts: []mount.Mount{
hostConfig.Mounts[0],
{Type: mount.TypeVolume, Target: "/anonymous"},
{Type: mount.TypeTmpfs, Source: "/etc", Target: "/etc-copy"},
}})
}}
(&containerReference{input: &NewContainerInput{ValidVolumes: validVolumes}}).sanitizeConfig(common.WithLogger(t.Context(), logger), sanitized)
assert.Equal(t, hostConfig.Mounts, sanitized.Mounts)
assert.Len(t, hook.AllRecords(), 2)
}
@@ -908,11 +948,11 @@ func TestMergeContainerConfigsVolumesReplaceRunnerMounts(t *testing.T) {
},
}
_, hostConf, err := cr.mergeContainerConfigs(ctx, &container.Config{}, &container.HostConfig{
hostConf := &container.HostConfig{
Binds: []string{"/var/run/docker.sock:/var/run/docker.sock"},
Mounts: []mount.Mount{{Type: mount.TypeVolume, Source: "act-toolcache", Target: "/opt/hostedtoolcache"}},
})
require.NoError(t, err)
}
require.NoError(t, cr.mergeContainerConfigs(ctx, &container.Config{}, hostConf))
assert.Equal(t, []string{"/var/run/docker.sock:/var/run/docker.sock", "/host/tools:/opt/hostedtoolcache"}, hostConf.Binds)
assert.Empty(t, hostConf.Mounts)
}
@@ -964,8 +1004,7 @@ func TestMergeContainerConfigsKeepsNetworkAliasesFromOptions(t *testing.T) {
NetworkMode: "job-network", NetworkAliases: []string{"redis"}, WorkflowOptions: "--network-alias redis-primary",
}}
_, _, err := cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{})
require.NoError(t, err)
require.NoError(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{}))
assert.Equal(t, []string{"redis", "redis-primary"}, cr.input.NetworkAliases)
}

111
act/container/docker_ssh.go Normal file
View File

@@ -0,0 +1,111 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
package container
import (
"bytes"
"errors"
"fmt"
"io"
"net"
"os/exec"
"strings"
"sync"
"syscall"
"time"
"github.com/docker/cli/cli/connhelper/ssh"
)
func sshArgs(daemonURL string) ([]string, error) {
spec, err := ssh.ParseURL(daemonURL)
if err != nil {
return nil, err
}
remote := []string{"docker", "system", "dial-stdio"}
if strings.Trim(spec.Path, "/") != "" {
remote = []string{"docker", "--host=unix://" + spec.Path, "system", "dial-stdio"}
}
return spec.Command([]string{"-o", "ConnectTimeout=30", "-T"}, remote...)
}
func dialCommand(name string, args ...string) (net.Conn, error) {
conn := &commandConn{cmd: exec.Command(name, args...)}
conn.cmd.SysProcAttr = sshSysProcAttr() // a session of its own like docker's, so a ProxyCommand never reads the runner's terminal: https://github.com/docker/cli/issues/1707
conn.cmd.Stderr = &conn.stderr
conn.cmd.WaitDelay = time.Second // a ProxyCommand child keeping stderr open would otherwise block Wait after ssh exits
conn.wait = sync.OnceValue(func() error {
if err := conn.cmd.Wait(); err != nil && !errors.Is(err, exec.ErrWaitDelay) {
return fmt.Errorf("%s: %w: %s", strings.Join(conn.cmd.Args, " "), err, strings.TrimSpace(conn.stderr.String()))
}
return nil
})
var err error
if conn.stdin, err = conn.cmd.StdinPipe(); err != nil {
return nil, err
}
if conn.stdout, err = conn.cmd.StdoutPipe(); err != nil {
return nil, err
}
return conn, conn.cmd.Start()
}
type commandConn struct {
cmd *exec.Cmd
stdin io.WriteCloser
stdout io.ReadCloser
stderr cappedBuffer
wait func() error
}
func (c *commandConn) waitWithin(timeout time.Duration) error {
timer := time.AfterFunc(timeout, func() { _ = c.cmd.Process.Kill() })
defer timer.Stop()
return c.wait()
}
func (c *commandConn) Read(p []byte) (int, error) {
n, err := c.stdout.Read(p)
if errors.Is(err, io.EOF) {
if waitErr := c.waitWithin(10 * time.Second); waitErr != nil {
return n, waitErr
}
}
return n, err
}
func (c *commandConn) Write(p []byte) (int, error) { return c.stdin.Write(p) }
// CloseWrite lets the daemon see the end of a hijacked stream, like a TCP half-close.
func (c *commandConn) CloseWrite() error { return c.stdin.Close() }
func (c *commandConn) Close() error {
_ = c.stdin.Close()
if c.cmd.Process.Signal(syscall.SIGTERM) != nil { // SIGTERM lets ssh take its ProxyCommand down, Windows only kills
_ = c.cmd.Process.Kill()
}
_ = c.waitWithin(3 * time.Second)
return nil
}
func (c *commandConn) LocalAddr() net.Addr { return &net.UnixAddr{Name: "stdio", Net: "unix"} }
func (c *commandConn) RemoteAddr() net.Addr { return &net.UnixAddr{Name: "stdio", Net: "unix"} }
func (c *commandConn) SetDeadline(_deadline time.Time) error { return nil }
func (c *commandConn) SetReadDeadline(_deadline time.Time) error { return nil }
func (c *commandConn) SetWriteDeadline(_deadline time.Time) error { return nil }
type cappedBuffer struct{ bytes.Buffer }
func (b *cappedBuffer) Write(p []byte) (int, error) {
if b.Len() > 4096 {
b.Reset()
}
return b.Buffer.Write(p)
}

View File

@@ -0,0 +1,12 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER
package container
import "syscall"
func sshSysProcAttr() *syscall.SysProcAttr {
return &syscall.SysProcAttr{Setsid: true, Pdeathsig: syscall.SIGKILL}
}

View File

@@ -0,0 +1,52 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
package container
import (
"io"
"os/exec"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestSSHArgs(t *testing.T) {
args, err := sshArgs("ssh://me@host:2222/run/user/1000/my docker.sock")
require.NoError(t, err)
assert.Equal(t, []string{"-l", "me", "-p", "2222", "-o", "ConnectTimeout=30", "-T", "--", "host", "docker '--host=unix:///run/user/1000/my docker.sock' system dial-stdio"}, args)
args, err = sshArgs("ssh://[::1]")
require.NoError(t, err)
assert.Equal(t, []string{"-o", "ConnectTimeout=30", "-T", "--", "::1", "docker system dial-stdio"}, args)
for _, invalid := range []string{"ssh://me:secret@host", "ssh://host?x=1", "ssh://host#x", "ssh://"} {
_, err := sshArgs(invalid)
assert.Error(t, err, invalid)
}
}
func TestDialCommandStreamsStdioAndReportsExit(t *testing.T) {
t.Parallel()
if _, err := exec.LookPath("sh"); err != nil {
t.Skip("needs a POSIX shell")
}
conn, err := dialCommand("sh", "-c", "cat")
require.NoError(t, err)
_, err = conn.Write([]byte("ping"))
require.NoError(t, err)
require.NoError(t, conn.(interface{ CloseWrite() error }).CloseWrite())
echoed, err := io.ReadAll(conn)
require.NoError(t, err)
assert.Equal(t, "ping", string(echoed))
require.NoError(t, conn.Close())
conn, err = dialCommand("sh", "-c", "exec 3<&0; cat <&3 >/dev/null & echo 'no route to host' >&2; exit 255")
require.NoError(t, err)
_, err = io.ReadAll(conn)
require.ErrorContains(t, err, "no route to host")
require.NoError(t, conn.Close())
}

View File

@@ -0,0 +1,12 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (darwin || netbsd)
package container
import "syscall"
func sshSysProcAttr() *syscall.SysProcAttr {
return &syscall.SysProcAttr{Setsid: true}
}

View File

@@ -0,0 +1,10 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER
package container
import "syscall"
func sshSysProcAttr() *syscall.SysProcAttr { return nil }

View File

@@ -7,6 +7,7 @@ package container
import (
"archive/tar"
"bytes"
"cmp"
"context"
"errors"
"fmt"
@@ -64,15 +65,11 @@ func (e *HostEnvironment) processGroup(ctx context.Context) *process.Group {
}
func (e *HostEnvironment) Create(_, _ []string) common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
func (e *HostEnvironment) Close() common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
func (e *HostEnvironment) Copy(destPath string, files ...*FileEntry) common.Executor {
@@ -168,15 +165,11 @@ func (e *HostEnvironment) GetContainerArchive(ctx context.Context, srcPath strin
}
func (e *HostEnvironment) Pull(_ bool) common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
func (e *HostEnvironment) Start(_ bool) common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
type ptyWriter struct {
@@ -193,7 +186,6 @@ func (w *ptyWriter) Write(buf []byte) (int, error) {
}
if w.dirtyLine || len(buf) > 1 && buf[len(buf)-2] != '\n' {
_, _ = w.Out.Write([]byte("\n"))
return n, io.EOF
}
return n, io.EOF
}
@@ -233,49 +225,38 @@ func setupPty(cmd *exec.Cmd, cmdline string) (*os.File, *os.File, error) {
return ppty, tty, nil
}
func writeKeepAlive(ppty io.Writer) {
c := 1
var err error
for c == 1 && err == nil {
c, err = ppty.Write([]byte{4})
<-time.After(time.Second)
}
}
func copyPtyOutput(writer io.Writer, ppty io.Reader, finishLog context.CancelFunc) {
defer func() {
finishLog()
}()
if _, err := io.Copy(writer, ppty); err != nil {
return
func writeKeepAlive(ctx context.Context, ppty io.Writer) {
for ctx.Err() == nil {
if count, err := ppty.Write([]byte{4}); count != 1 || err != nil {
return
}
select {
case <-ctx.Done():
return
case <-time.After(time.Second):
}
}
}
func (e *HostEnvironment) UpdateFromImageEnv(_ *map[string]string) common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
func getEnvListFromMap(env map[string]string) []string {
envList := make([]string, 0)
for k, v := range env {
envList = append(envList, fmt.Sprintf("%s=%s", k, v))
envList := make([]string, 0, len(env))
for key, value := range env {
envList = append(envList, key+"="+value)
}
return envList
}
func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline string, env map[string]string, _, workdir string) error {
envList := getEnvListFromMap(env)
var wd string
func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline string, env map[string]string, workdir string) error {
wd := e.Path
if workdir != "" {
if filepath.IsAbs(workdir) {
wd = workdir
} else {
wd = workdir
if !filepath.IsAbs(workdir) {
wd = filepath.Join(e.Path, workdir)
}
} else {
wd = e.Path
}
// Flush any buffered, not-yet-newline-terminated trailing line, as the docker backend
// does in waitForCommand, so the final line of a command's output is not lost.
@@ -288,9 +269,8 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
cmd := exec.CommandContext(ctx, f)
cmd.Path = f
cmd.Args = command
cmd.Stdin = nil
cmd.Stdout = e.StdOut
cmd.Env = envList
cmd.Env = getEnvListFromMap(env)
cmd.Stderr = e.StdOut
cmd.Dir = wd
cmd.SysProcAttr = process.SysProcAttr(cmdline, false)
@@ -322,8 +302,11 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
writer = &ptyWriter{Out: e.StdOut}
var finishLog context.CancelFunc
logctx, finishLog = context.WithCancel(context.Background())
go copyPtyOutput(writer, ppty, finishLog)
go writeKeepAlive(ppty)
go func() {
defer finishLog()
_, _ = io.Copy(writer, ppty)
}()
go writeKeepAlive(logctx, ppty)
}
if err := cmd.Start(); err != nil {
return err
@@ -338,32 +321,32 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
} else {
defer k.Close()
}
err = cmd.Wait()
if err != nil {
if exitErr, ok := errors.AsType[*exec.ExitError](err); ok {
return ExitCodeError(exitErr.ExitCode())
}
return err
}
err = treeKill.Wait(ctx, cmd)
if tty != nil {
writer.AutoStop.Store(true)
if _, err := tty.WriteString("\x04"); err != nil {
common.Logger(ctx).Debug("Failed to write EOT")
}
<-logctx.Done()
select {
case <-logctx.Done():
case <-ctx.Done(): // a cancelled step does not wait on output nobody consumes
}
ppty.Close()
ppty = nil
}
if exitErr, ok := errors.AsType[*exec.ExitError](err); ok {
return ExitCodeError(exitErr.ExitCode())
}
return err
}
func (e *HostEnvironment) Exec(command []string /*cmdline string, */, env map[string]string, user, workdir string) common.Executor {
return e.ExecWithCmdLine(command, "", env, user, workdir)
func (e *HostEnvironment) Exec(command []string, env map[string]string, _user, workdir string) common.Executor {
return e.ExecWithCmdLine(command, "", env, workdir)
}
func (e *HostEnvironment) ExecWithCmdLine(command []string, cmdline string, env map[string]string, user, workdir string) common.Executor {
func (e *HostEnvironment) ExecWithCmdLine(command []string, cmdline string, env map[string]string, workdir string) common.Executor {
return func(ctx context.Context) error {
if err := e.exec(ctx, command, cmdline, env, user, workdir); err != nil {
if err := e.exec(ctx, command, cmdline, env, workdir); err != nil {
select {
case <-ctx.Done():
return fmt.Errorf("this step has been cancelled: %w", err)
@@ -376,7 +359,7 @@ func (e *HostEnvironment) ExecWithCmdLine(command []string, cmdline string, env
}
func (e *HostEnvironment) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(e, srcPath, env)
return parseEnvFile(e, srcPath, env, runtime.GOOS == "windows")
}
// removeAll is a var so tests can substitute a blocking stub.
@@ -605,8 +588,7 @@ func (*HostEnvironment) GetPathVariableName() string {
}
func (e *HostEnvironment) DefaultPathVariable() string {
v, _ := os.LookupEnv(e.GetPathVariableName())
return v
return os.Getenv(e.GetPathVariableName())
}
func (*HostEnvironment) JoinPathVariable(paths ...string) string {
@@ -616,30 +598,22 @@ func (*HostEnvironment) JoinPathVariable(paths ...string) string {
// Reference for Arch values for runner.arch
// https://docs.github.com/en/actions/learn-github-actions/contexts#runner-context
func goArchToActionArch(arch string) string {
archMapper := map[string]string{
return cmp.Or(map[string]string{
"x86_64": "X64",
"amd64": "X64",
"386": "X86",
"arm": "ARM",
"aarch64": "ARM64",
"arm64": "ARM64",
}
if arch, ok := archMapper[arch]; ok {
return arch
}
return arch
}[arch], arch)
}
func goOsToActionOs(os string) string {
osMapper := map[string]string{
return cmp.Or(map[string]string{
"linux": "Linux",
"darwin": "macOS",
"windows": "Windows",
}
if os, ok := osMapper[os]; ok {
return os
}
return os
}[os], os)
}
func (e *HostEnvironment) GetRunnerContext(_ context.Context) map[string]any {

View File

@@ -10,11 +10,11 @@ import (
"context"
"io"
"os"
"path"
"path/filepath"
"runtime"
"strings"
"testing"
"testing/synctest"
"time"
"gitea.com/gitea/runner/act/common"
@@ -24,7 +24,6 @@ import (
"github.com/stretchr/testify/require"
)
// Type assert HostEnvironment implements ExecutionsEnvironment
var _ ExecutionsEnvironment = &HostEnvironment{}
func TestActionPlatformNames(t *testing.T) {
@@ -61,63 +60,68 @@ func TestCopyDir(t *testing.T) {
}
func TestGetContainerArchive(t *testing.T) {
dir := t.TempDir()
ctx := t.Context()
e := &HostEnvironment{
Path: filepath.Join(dir, "path"),
TmpDir: filepath.Join(dir, "tmp"),
ToolCache: filepath.Join(dir, "tool_cache"),
ActPath: filepath.Join(dir, "act_path"),
StdOut: os.Stdout,
Workdir: path.Join("testdata", "scratch"),
}
_ = os.MkdirAll(e.Path, 0o700)
_ = os.MkdirAll(e.TmpDir, 0o700)
_ = os.MkdirAll(e.ToolCache, 0o700)
_ = os.MkdirAll(e.ActPath, 0o700)
e := &HostEnvironment{Path: t.TempDir()}
expectedContent := []byte("sdde/7sh")
err := os.WriteFile(filepath.Join(e.Path, "action.yml"), expectedContent, 0o600)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
archive, err := e.GetContainerArchive(ctx, e.Path)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
defer archive.Close()
reader := tar.NewReader(archive)
h, err := reader.Next()
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
assert.Equal(t, "action.yml", h.Name)
content, err := io.ReadAll(reader)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
assert.Equal(t, expectedContent, content)
_, err = reader.Next()
assert.ErrorIs(t, err, io.EOF)
}
func TestHostEnvironmentExecExitCode(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
dir := t.TempDir()
ctx := t.Context()
e := &HostEnvironment{
Path: filepath.Join(dir, "path"),
TmpDir: filepath.Join(dir, "tmp"),
ToolCache: filepath.Join(dir, "tool_cache"),
ActPath: filepath.Join(dir, "act_path"),
StdOut: io.Discard,
Workdir: filepath.Join(dir, "path"),
}
for _, p := range []string{e.Path, e.TmpDir, e.ToolCache, e.ActPath} {
assert.NoError(t, os.MkdirAll(p, 0o700)) //nolint:testifylint // test setup
}
for name, allocatePTY := range map[string]bool{"without PTY": false, "with PTY": true} {
t.Run(name, func(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
reader, writer := io.Pipe()
defer reader.Close()
e := &HostEnvironment{Path: t.TempDir(), StdOut: writer, AllocatePTY: allocatePTY}
t.Cleanup(func() { assert.NoError(t, e.Remove()(t.Context())) })
err := e.Exec([]string{"sh", "-c", "exit 3"}, map[string]string{"PATH": os.Getenv("PATH")}, "", "")(ctx)
var exitErr ExitCodeError
require.ErrorAs(t, err, &exitErr)
assert.Equal(t, ExitCodeError(3), exitErr)
assert.Equal(t, "Process completed with exit code 3.", err.Error())
result := make(chan error, 1)
go func() {
defer writer.Close()
result <- e.Exec([]string{"sh", "-c", "printf 'failed\\n'; exit 3"}, map[string]string{"PATH": os.Getenv("PATH")}, "", "")(t.Context())
}()
first := make([]byte, 1)
_, err := io.ReadFull(reader, first)
require.NoError(t, err)
synctest.Wait()
select {
case err := <-result:
t.Fatalf("Exec returned before its output was consumed: %v", err)
default:
}
remaining, err := io.ReadAll(reader)
require.NoError(t, err)
err = <-result
var exitErr ExitCodeError
require.ErrorAs(t, err, &exitErr)
assert.Equal(t, ExitCodeError(3), exitErr)
assert.Equal(t, "Process completed with exit code 3.", err.Error())
assert.Equal(t, "failed\n", string(first)+string(remaining))
})
})
}
}
func TestHostEnvironmentAllocatePTY(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
@@ -130,20 +134,13 @@ func TestHostEnvironmentAllocatePTY(t *testing.T) {
{name: "on", allocPTY: true, expect: "TTY"},
} {
t.Run(tc.name, func(t *testing.T) {
dir := t.TempDir()
t.Parallel()
buf := &bytes.Buffer{}
e := &HostEnvironment{
Path: filepath.Join(dir, "path"),
TmpDir: filepath.Join(dir, "tmp"),
ToolCache: filepath.Join(dir, "tool_cache"),
ActPath: filepath.Join(dir, "act_path"),
Path: t.TempDir(),
StdOut: buf,
Workdir: filepath.Join(dir, "path"),
AllocatePTY: tc.allocPTY,
}
for _, p := range []string{e.Path, e.TmpDir, e.ToolCache, e.ActPath} {
require.NoError(t, os.MkdirAll(p, 0o700))
}
err := e.Exec(
[]string{"sh", "-c", "[ -t 1 ] && printf TTY || printf NOTTY"},
@@ -156,158 +153,91 @@ func TestHostEnvironmentAllocatePTY(t *testing.T) {
}
}
func TestHostEnvironmentRemovePreservesWorkdirByDefault(t *testing.T) {
ctx := common.WithLogger(t.Context(), logging.Default())
base := t.TempDir()
miscRoot := filepath.Join(base, "misc")
path := filepath.Join(miscRoot, "hostexecutor")
require.NoError(t, os.MkdirAll(path, 0o700))
workdir := filepath.Join(base, "workspace", "owner", "repo")
require.NoError(t, os.MkdirAll(workdir, 0o700))
e := &HostEnvironment{
Path: path,
Workdir: workdir,
CleanUp: func() {
_ = os.RemoveAll(miscRoot)
},
StdOut: os.Stdout,
func TestHostEnvironmentRemoveWorkdir(t *testing.T) {
for name, owned := range map[string]bool{"preserves caller workdir": false, "cleans owned workdir": true} {
t.Run(name, func(t *testing.T) {
miscRoot, workdir := t.TempDir(), t.TempDir()
e := &HostEnvironment{
Path: filepath.Join(miscRoot, "hostexecutor"),
Workdir: workdir,
CleanWorkdir: owned,
CleanUp: func() { _ = os.RemoveAll(miscRoot) },
}
require.NoError(t, os.MkdirAll(e.Path, 0o700))
require.NoError(t, e.Remove()(t.Context()))
_, err := os.Stat(workdir)
if owned {
assert.ErrorIs(t, err, os.ErrNotExist)
} else {
require.NoError(t, err)
}
})
}
require.NoError(t, e.Remove()(ctx))
_, err := os.Stat(workdir)
require.NoError(t, err)
}
func TestHostEnvironmentRemoveCleansWorkdirWhenOwned(t *testing.T) {
ctx := common.WithLogger(t.Context(), logging.Default())
base := t.TempDir()
miscRoot := filepath.Join(base, "misc")
path := filepath.Join(miscRoot, "hostexecutor")
require.NoError(t, os.MkdirAll(path, 0o700))
workdir := filepath.Join(base, "workspace", "123", "owner", "repo")
require.NoError(t, os.MkdirAll(workdir, 0o700))
e := &HostEnvironment{
Path: path,
Workdir: workdir,
CleanWorkdir: true,
CleanUp: func() {
_ = os.RemoveAll(miscRoot)
},
StdOut: os.Stdout,
}
require.NoError(t, e.Remove()(ctx))
_, err := os.Stat(workdir)
assert.ErrorIs(t, err, os.ErrNotExist)
}
func TestRemoveAllWithContextDoesNotHangOnStuckDelete(t *testing.T) {
release := make(chan struct{})
stubDone := make(chan struct{})
synctest.Test(t, func(t *testing.T) {
release := make(chan struct{})
stubDone := make(chan struct{})
orig := removeAll
removeAll = func(string) error {
defer close(stubDone)
<-release
return nil
}
// removeAllWithContext intentionally leaks the delete goroutine on timeout,
// and that goroutine still references removeAll. Unblock it and wait for it
// to return before restoring the var, so the restore can't race the read.
t.Cleanup(func() {
close(release)
<-stubDone
removeAll = orig
orig := removeAll
removeAll = func(string) error {
defer close(stubDone)
<-release
return nil
}
t.Cleanup(func() {
close(release)
<-stubDone
removeAll = orig
})
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
err := removeAllWithContext(ctx, t.TempDir())
require.ErrorIs(t, err, context.DeadlineExceeded)
})
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
err := removeAllWithContext(ctx, t.TempDir())
require.ErrorIs(t, err, context.DeadlineExceeded)
}
// TestHostEnvironmentRemoveDoesNotHangOnStuckCleanUp guards against a stalled
// CleanUp callback (e.g. an os.RemoveAll blocked by an AV/EDR filter driver or
// an unresponsive mount) wedging the runner slot forever at "Cleaning up
// container". Remove must time out the callback and complete job teardown.
func TestHostEnvironmentRemoveDoesNotHangOnStuckCleanUp(t *testing.T) {
// Keep the suite fast: shrink the per-phase teardown timeout for this test.
orig := hostCleanupTimeout
hostCleanupTimeout = 100 * time.Millisecond
t.Cleanup(func() { hostCleanupTimeout = orig })
ctx := common.WithLogger(t.Context(), logging.Default())
base := t.TempDir()
path := filepath.Join(base, "misc", "hostexecutor")
require.NoError(t, os.MkdirAll(path, 0o700))
release := make(chan struct{})
t.Cleanup(func() { close(release) }) // unblock the leaked goroutine at test end
e := &HostEnvironment{
Path: path,
CleanUp: func() {
<-release // simulate a delete syscall stuck indefinitely
},
StdOut: os.Stdout,
}
done := make(chan error, 1)
go func() { done <- e.Remove()(ctx) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatal("Remove() hung on a stuck CleanUp callback")
}
}
// TestHostEnvironmentRemoveDoesNotHangOnStuckPathRemoval guards against a
// stalled os.RemoveAll on the misc/workspace paths (same AV/EDR wedge as
// #1023) wedging job completion after the CleanUp callback has already timed
// out or finished.
func TestHostEnvironmentRemoveDoesNotHangOnStuckPathRemoval(t *testing.T) {
origTimeout := hostCleanupTimeout
hostCleanupTimeout = 100 * time.Millisecond
t.Cleanup(func() { hostCleanupTimeout = origTimeout })
release := make(chan struct{})
stubDone := make(chan struct{})
origRemoveAll := removeAll
removeAll = func(string) error {
defer close(stubDone)
<-release
return nil
}
// The stuck delete goroutine outlives the timed-out Remove and still reads
// removeAll; unblock it and wait before restoring to avoid a restore/read race.
t.Cleanup(func() {
close(release)
<-stubDone
removeAll = origRemoveAll
})
ctx := common.WithLogger(t.Context(), logging.Default())
base := t.TempDir()
path := filepath.Join(base, "misc", "hostexecutor")
require.NoError(t, os.MkdirAll(path, 0o700))
e := &HostEnvironment{
Path: path,
StdOut: os.Stdout,
}
done := make(chan error, 1)
go func() { done <- e.Remove()(ctx) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatal("Remove() hung on a stuck path removal")
func TestHostEnvironmentRemoveDoesNotHang(t *testing.T) {
for _, phase := range []string{"CleanUp callback", "path removal"} {
t.Run(phase, func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
origTimeout := hostCleanupTimeout
hostCleanupTimeout = 100 * time.Millisecond
t.Cleanup(func() { hostCleanupTimeout = origTimeout })
release, stubDone := make(chan struct{}), make(chan struct{})
block := func() {
defer close(stubDone)
<-release
}
e := &HostEnvironment{Path: filepath.Join(t.TempDir(), "misc", "hostexecutor"), StdOut: os.Stdout}
if phase == "CleanUp callback" {
e.CleanUp = block
} else {
origRemoveAll := removeAll
removeAll = func(string) error {
block()
return nil
}
t.Cleanup(func() { removeAll = origRemoveAll })
}
t.Cleanup(func() {
close(release)
<-stubDone
})
require.NoError(t, os.MkdirAll(e.Path, 0o700))
done := make(chan error, 1)
go func() { done <- e.Remove()(common.WithLogger(t.Context(), logging.Default())) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatalf("Remove() hung on a stuck %s", phase)
}
})
})
}
}
@@ -315,17 +245,11 @@ func TestBuildWindowsWorkspaceKillScript(t *testing.T) {
t.Run("single dir", func(t *testing.T) {
s := buildWindowsWorkspaceKillScript([]string{`C:\workspace\job1`})
assert.Contains(t, s, `$paths = @('C:\workspace\job1')`)
// Self-PID guard is essential — without it the script could taskkill
// the PowerShell process running it.
assert.Contains(t, s, "$selfPid = $PID")
assert.Contains(t, s, "$_.ProcessId -eq $selfPid")
// Must match both ExecutablePath (binaries from the workspace) and
// CommandLine (system binaries invoked with workspace paths in args),
// both bounded by dir+separator so a name-prefix sibling is spared.
assert.Contains(t, s, `$prefix = $p + '\'`)
assert.Contains(t, s, "$_.ExecutablePath.StartsWith($prefix")
assert.Contains(t, s, "$_.CommandLine.IndexOf($prefix")
// Each matched PID must be tree-killed, not just stopped.
assert.Contains(t, s, "taskkill.exe /PID $_.ProcessId /T /F")
})
@@ -338,25 +262,16 @@ func TestBuildWindowsWorkspaceKillScript(t *testing.T) {
assert.Contains(t, s, `'C:\work\path'`)
assert.Contains(t, s, `'C:\work\workdir'`)
assert.Contains(t, s, `'C:\Users\runner\AppData\Local\Temp\job-42'`)
// Commas between entries — no trailing comma, no leading comma.
assert.Contains(t, s, `'C:\work\path','C:\work\workdir',`)
})
t.Run("path with single quote is escaped", func(t *testing.T) {
// In PowerShell single-quoted strings the only special char is the
// quote itself, escaped by doubling. A workspace path that ever
// contained `'` would inject a command into the script otherwise.
s := buildWindowsWorkspaceKillScript([]string{`C:\work\it's\path`})
assert.Contains(t, s, `'C:\work\it''s\path'`)
// And it must NOT appear unescaped — otherwise the quote would
// terminate the literal early.
assert.NotContains(t, s, `'C:\work\it's\path'`)
})
t.Run("path with wildcard metacharacters is matched literally", func(t *testing.T) {
// A path containing [ ] ? * must be embedded verbatim and matched with
// ordinal String methods, not -like, otherwise the metacharacters would
// be interpreted as wildcards and the leftover process could escape.
s := buildWindowsWorkspaceKillScript([]string{`C:\work\[job]?1`})
assert.Contains(t, s, `'C:\work\[job]?1'`)
assert.NotContains(t, s, "-like")
@@ -366,7 +281,6 @@ func TestBuildWindowsWorkspaceKillScript(t *testing.T) {
t.Run("empty dir list still produces a valid script", func(t *testing.T) {
s := buildWindowsWorkspaceKillScript(nil)
// Empty array literal — script runs, matches nothing, is a no-op.
assert.Contains(t, s, "$paths = @()")
assert.Contains(t, s, "Get-CimInstance Win32_Process")
})

View File

@@ -93,17 +93,29 @@ func (p *KubernetesPod) JobContainer(input *NewContainerInput) ExecutionsEnviron
return p.job
}
var invalidLabelChars = regexp.MustCompile(`[^a-z0-9-]+`)
var (
invalidLabelChars = regexp.MustCompile(`[^a-z0-9-]+`)
dnsSubdomain = regexp.MustCompile(`^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$`)
)
// KubernetesServiceName returns the container name and hostname of service id, made a DNS label as both must be.
func KubernetesServiceName(id string) string {
// kubernetesServiceName returns the container name of service id, made a DNS label as it must be.
func kubernetesServiceName(id string) string {
name := strings.Trim(invalidLabelChars.ReplaceAllString(strings.ToLower(id), "-"), "-")
return strings.TrimRight(name[:min(63, len(name))], "-")
}
// ServiceContainer returns a container of the pod the job reaches at hostname KubernetesServiceName(id).
// KubernetesServiceHostnames returns the hostnames of service id, its container name and, where hostAliases accept it as a DNS subdomain, id lowercased.
func KubernetesServiceHostnames(id string) []string {
name, host := kubernetesServiceName(id), strings.ToLower(id)
if host != name && len(host) <= 253 && dnsSubdomain.MatchString(host) { // https://github.com/kubernetes/apimachinery/blob/v0.35.1/pkg/util/validation/validation.go#L198-L207
return []string{name, host}
}
return []string{name}
}
// ServiceContainer returns a container of the pod the job reaches at KubernetesServiceHostnames(id).
func (p *KubernetesPod) ServiceContainer(id string, input *NewContainerInput) ExecutionsEnvironment {
service := &kubernetesContainer{pod: p, id: id, name: KubernetesServiceName(id), input: input}
service := &kubernetesContainer{pod: p, id: id, name: kubernetesServiceName(id), input: input}
p.services = append(p.services, service)
return service
}
@@ -131,7 +143,7 @@ func (p *KubernetesPod) create(ctx context.Context) error {
return err
}
for _, service := range p.services {
if !strings.EqualFold(service.id, service.name) {
if !slices.ContainsFunc(KubernetesServiceHostnames(service.id), func(hostname string) bool { return strings.EqualFold(hostname, service.id) }) { // glibc and Go match /etc/hosts case-insensitively
common.Logger(ctx).Warnf("Service %s is reachable as %s or localhost, as a Kubernetes hostname cannot be %q", service.id, service.name, service.id)
}
}
@@ -165,7 +177,8 @@ func (p *KubernetesPod) manifest(secret string, secretData map[string]string, me
serviceTemplate = templateContainers[index]
templateSpec["containers"] = slices.Delete(slices.Clone(templateContainers), index, index+1)
}
var containers, hostnames, volumes []any
var containers, volumes []any
var hostnames []string
names := map[string]bool{}
for _, container := range templateContainers {
names[manifestField(container, "name")] = true
@@ -184,7 +197,7 @@ func (p *KubernetesPod) manifest(secret string, secretData map[string]string, me
continue
}
containers = append(containers, mergeManifest(serviceTemplate, spec, ""))
hostnames = append(hostnames, c.name)
hostnames = append(hostnames, KubernetesServiceHostnames(c.id)...)
}
for _, name := range []string{"workspace", "act"} {
if !slices.ContainsFunc(templateVolumes, func(volume any) bool { return manifestField(volume, "name") == name }) {
@@ -374,16 +387,41 @@ func (c *kubernetesContainer) Start(_ bool) common.Executor {
if err := c.pod.waitStarted(ctx); err != nil {
return err
}
var stdout bytes.Buffer
if err := c.pod.client.exec(ctx, c.pod.name, c.name, []string{"sh", "-c", `printf '%s\n%s' "$(uname -m)" "$PATH"`}, nil, &stdout, io.Discard); err != nil {
output, err := c.pod.probe(ctx, c.name)
if err != nil {
return err
}
machine, path, _ := strings.Cut(stdout.String(), "\n")
machine, path, _ := strings.Cut(output, "\n")
c.pod.arch, c.pod.path = goArchToActionArch(machine), path
return nil
}
}
// probe retries exec while it fails short of the container, as a new node runs pods before its kubelet has a serving certificate.
func (p *KubernetesPod) probe(ctx context.Context, container string) (string, error) {
var failure string
deadline := time.Now().Add(time.Minute)
for delay := 100 * time.Millisecond; ; delay = min(2*delay, 2*time.Second) {
var stdout bytes.Buffer
err := p.client.exec(ctx, p.name, container, []string{"sh", "-c", `printf '%s\n%s' "$(uname -m)" "$PATH"`}, nil, &stdout, io.Discard)
var exitErr ExitCodeError
var apiErr *apiError
if err == nil || ctx.Err() != nil || errors.As(err, &exitErr) || time.Now().After(deadline) ||
errors.As(err, &apiErr) && apiErr.code < http.StatusInternalServerError {
return stdout.String(), err
}
if err.Error() != failure {
failure = err.Error()
common.Logger(ctx).Infof("Waiting for exec into pod %s: %v", p.name, err)
}
select {
case <-ctx.Done():
return "", ctx.Err()
case <-time.After(delay):
}
}
}
func (*kubernetesContainer) Pull(bool) common.Executor {
return func(context.Context) error { return nil }
}
@@ -462,7 +500,7 @@ func (c *kubernetesContainer) GetContainerArchive(ctx context.Context, srcPath s
}
func (c *kubernetesContainer) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(c, srcPath, env)
return parseEnvFile(c, srcPath, env, false)
}
func (c *kubernetesContainer) UpdateFromImageEnv(env *map[string]string) common.Executor {

View File

@@ -43,8 +43,9 @@ type podManifest struct {
}
type fakeCluster struct {
mu sync.Mutex
requests map[string][]string
mu sync.Mutex
requests map[string][]string
execFailures []http.HandlerFunc
}
func (f *fakeCluster) record(request, content string) {
@@ -53,7 +54,42 @@ func (f *fakeCluster) record(request, content string) {
f.requests[request] = append(f.requests[request], content)
}
func fakeKubernetes(t *testing.T, podStatus string) (string, *fakeCluster) {
func (f *fakeCluster) failExec(failures ...http.HandlerFunc) {
f.mu.Lock()
defer f.mu.Unlock()
f.execFailures = failures
}
func (f *fakeCluster) nextExecFailure() http.HandlerFunc {
f.mu.Lock()
defer f.mu.Unlock()
if len(f.execFailures) == 0 {
return nil
}
failure := f.execFailures[0]
f.execFailures = f.execFailures[1:]
return failure
}
// execFailure answers like the API server, code 0 for a failure reported after the websocket upgrade.
func execFailure(t *testing.T, code int, status string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if code != 0 {
w.WriteHeader(code)
_, _ = io.WriteString(w, status)
return
}
conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{Subprotocols: []string{execProtocol}})
if err != nil {
t.Error(err)
return
}
defer func() { _ = conn.CloseNow() }()
_ = conn.Write(r.Context(), websocket.MessageBinary, append([]byte{3}, status...))
}
}
func fakeKubernetes(t *testing.T, podStatus string, env ...string) (string, *fakeCluster) {
cluster := &fakeCluster{requests: map[string][]string{}}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer test-token" {
@@ -70,7 +106,11 @@ func fakeKubernetes(t *testing.T, podStatus string) (string, *fakeCluster) {
_, _ = io.WriteString(w, `{"items":[{"metadata":{"name":"old","creationTimestamp":"2020-01-01T00:00:00Z"}},{"metadata":{"name":"new","creationTimestamp":"2999-01-01T00:00:00Z"}}]}`)
case strings.HasSuffix(r.URL.Path, "/exec"):
cluster.record("exec", r.URL.RawQuery)
serveLocalExec(t, w, r)
if failure := cluster.nextExecFailure(); failure != nil {
failure(w, r)
} else {
serveLocalExec(t, w, r, env)
}
case strings.HasSuffix(r.URL.Path, "/log"):
_, _ = io.WriteString(w, "log of "+r.URL.Query().Get("container"))
case r.Method == http.MethodGet && strings.HasPrefix(r.URL.Path, pods+"/"):
@@ -92,7 +132,7 @@ users: [{name: test, user: {tokenFile: token}}]
return kubeconfig, cluster
}
func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request) {
func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request, env []string) {
conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{Subprotocols: []string{execProtocol}})
if err != nil {
t.Error(err)
@@ -114,7 +154,7 @@ func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request) {
stdin.Write(frame[1:])
}
cmd := exec.Command(command[0], command[1:]...)
cmd.Env = append(os.Environ(), "COPYFILE_DISABLE=1")
cmd.Env = append(append(os.Environ(), "COPYFILE_DISABLE=1"), env...)
var stdout, stderr bytes.Buffer
cmd.Stdin, cmd.Stdout, cmd.Stderr = &stdin, &stdout, &stderr
status := `{"status":"Success"}`
@@ -129,16 +169,31 @@ func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request) {
}
}
func TestKubernetesServiceHostnames(t *testing.T) {
for id, hostnames := range map[string][]string{
"db": {"db"},
"My_DB": {"my-db"},
"devstoreaccount1.azurite.local": {"devstoreaccount1-azurite-local", "devstoreaccount1.azurite.local"},
"DB.Local": {"db-local", "db.local"},
strings.Repeat("a", 64): {strings.Repeat("a", 63), strings.Repeat("a", 64)},
strings.Repeat("a", 254): {strings.Repeat("a", 63)},
} {
assert.Equal(t, hostnames, KubernetesServiceHostnames(id), id)
}
}
func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
ctx := t.Context()
kubeconfig, cluster := fakeKubernetes(t, `{"spec":{"containers":[{},{}]},"status":{"phase":"Running","containerStatuses":[
{"name":"job","ready":true,"state":{"running":{}}},
{"name":"my-db","ready":true,"state":{"running":{}}}]}}`)
workdir, err := filepath.EvalSymlinks(t.TempDir())
require.NoError(t, err)
path := workdir + ":" + os.Getenv("PATH")
kubeconfig, cluster := fakeKubernetes(t, `{"spec":{"containers":[{},{}]},"status":{"phase":"Running","containerStatuses":[
{"name":"job","ready":true,"state":{"running":{}}},
{"name":"my-db","ready":true,"state":{"running":{}}}]}}`, "PATH="+path)
options := KubernetesOptions{Kubeconfig: kubeconfig, RunnerUUID: "runner-uuid", PodTemplates: []map[string]any{
{"spec": map[string]any{
"automountServiceAccountToken": true,
@@ -199,8 +254,12 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
assert.Contains(t, cluster.requests["POST /api/v1/namespaces/ci/pods"][0], `"args":["postgres","-D","$$(PGDATA)"]`)
require.NoError(t, os.WriteFile(filepath.Join(workdir, "env"), []byte("#!/bin/sh\necho \"$@\" > \"$0-args\"\nexec /usr/bin/env \"$@\"\n"), 0o755))
t.Setenv("PATH", workdir+":"+os.Getenv("PATH"))
cluster.failExec(
execFailure(t, http.StatusInternalServerError, `{"message":"error dialing backend: remote error: tls: internal error"}`),
execFailure(t, 0, `{"status":"Failure","reason":"InternalError","message":"Internal error occurred: error sending request: remote error: tls: internal error"}`),
)
require.NoError(t, job.Start(false)(ctx))
assert.Len(t, cluster.requests["exec"], 3)
stepEnv := map[string]string{"INPUT_WHO-TO-GREET": "it's\nme", "SECRET": "hunter2", "UID": "5"}
require.NoError(t, job.Exec([]string{"printenv", "SECRET"}, stepEnv, "", "")(ctx))
@@ -242,7 +301,7 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
pathEnv := map[string]string{}
require.NoError(t, job.UpdateFromImageEnv(&pathEnv)(ctx))
assert.Equal(t, os.Getenv("PATH"), pathEnv["PATH"])
assert.Equal(t, path, pathEnv["PATH"])
info, err := service.Inspect(ctx)
require.NoError(t, err)
@@ -282,6 +341,21 @@ func TestKubernetesPodFailsBeforeSteps(t *testing.T) {
}
require.ErrorContains(t, job.Create(nil, nil)(t.Context()), test.err)
}
running, cluster := fakeKubernetes(t, `{"spec":{"containers":[{}]},"status":{"containerStatuses":[{"name":"job","state":{"running":{}}}]}}`)
for _, test := range []struct {
failure http.HandlerFunc
err string
}{
{execFailure(t, http.StatusForbidden, `{"message":"pods \"job\" is forbidden"}`), "is forbidden"},
{execFailure(t, 0, `{"status":"Failure","details":{"causes":[{"reason":"ExitCode","message":"126"}]}}`), ExitCodeError(126).Error()},
} {
cluster.failExec(test.failure)
pod, err := NewKubernetesPod(KubernetesOptions{Kubeconfig: running})
require.NoError(t, err)
job := pod.JobContainer(&NewContainerInput{Image: "node"})
require.NoError(t, job.Create(nil, nil)(t.Context()))
require.ErrorContains(t, job.Start(false)(t.Context()), test.err)
}
pod, err := NewKubernetesPod(KubernetesOptions{Kubeconfig: kubeconfig, Namespace: "ci"})
require.NoError(t, err)
job := pod.JobContainer(&NewContainerInput{Image: "missing"})

View File

@@ -16,6 +16,8 @@ import (
type LinuxContainerEnvironmentExtensions struct{}
var windowsPathRegex = regexp.MustCompile(`^([a-zA-Z]):\\(.+)$`)
// Resolves the equivalent host path inside the container
// This is required for windows and WSL 2 to translate things like C:\Users\Myproject to /mnt/users/Myproject
// For use in docker volumes and binds
@@ -31,8 +33,6 @@ func (*LinuxContainerEnvironmentExtensions) ToContainerPath(path string) string
return ""
}
// Test if the path is a windows path
windowsPathRegex := regexp.MustCompile(`^([a-zA-Z]):\\(.+)$`)
windowsPathComponents := windowsPathRegex.FindStringSubmatch(abspath)
// Return as-is if no match
@@ -43,11 +43,7 @@ func (*LinuxContainerEnvironmentExtensions) ToContainerPath(path string) string
// Convert to WSL2-compatible path if it is a windows path
// NOTE: Cannot use filepath because it will use the wrong path separators assuming we want the path to be windows
// based if running on Windows, and because we are feeding this to Docker, GoLang auto-path-translate doesn't work.
driveLetter := strings.ToLower(windowsPathComponents[1])
translatedPath := strings.ReplaceAll(windowsPathComponents[2], `\`, `/`)
// Should make something like /mnt/c/Users/person/My Folder/MyActProject
result := strings.Join([]string{"/mnt", driveLetter, translatedPath}, `/`)
return result
return "/mnt/" + strings.ToLower(windowsPathComponents[1]) + "/" + strings.ReplaceAll(windowsPathComponents[2], `\`, `/`)
}
func (*LinuxContainerEnvironmentExtensions) GetActPath() string {
@@ -69,10 +65,41 @@ func (*LinuxContainerEnvironmentExtensions) JoinPathVariable(paths ...string) st
// DefaultToolCache is where the runner mounts the tool cache inside job containers.
const DefaultToolCache = "/opt/hostedtoolcache"
type runnerArchKey struct{}
func WithRunnerArch(ctx context.Context) context.Context { // shares one architecture lookup per run
if ctx.Value(runnerArchKey{}) != nil {
return ctx
}
return context.WithValue(ctx, runnerArchKey{}, &runnerArchCache{lock: make(chan struct{}, 1)})
}
type runnerArchCache struct {
lock chan struct{} // lets cancelled jobs stop waiting for the cache
arch string
}
func cachedRunnerArch(ctx context.Context) string {
cache, ok := ctx.Value(runnerArchKey{}).(*runnerArchCache)
if !ok {
return RunnerArch(ctx)
}
select {
case cache.lock <- struct{}{}:
defer func() { <-cache.lock }()
case <-ctx.Done():
return ""
}
if cache.arch == "" {
cache.arch = RunnerArch(ctx)
}
return cache.arch
}
func (*LinuxContainerEnvironmentExtensions) GetRunnerContext(ctx context.Context) map[string]any {
return map[string]any{
"os": "Linux",
"arch": RunnerArch(ctx),
"arch": cachedRunnerArch(ctx),
"temp": "/tmp",
"tool_cache": DefaultToolCache,
}

View File

@@ -5,15 +5,13 @@
package container
import (
"fmt"
"os"
"runtime"
"strings"
"testing"
"gitea.com/gitea/runner/act/common/logging"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestContainerPath(t *testing.T) {
@@ -23,42 +21,27 @@ func TestContainerPath(t *testing.T) {
workDir string
}
linuxcontainerext := &LinuxContainerEnvironmentExtensions{}
cwd, err := os.Getwd()
require.NoError(t, err)
tests := []containerPathJob{
{"/home/act/go/src/github.com/nektos/act", "/home/act/go/src/github.com/nektos/act", ""},
{"/home/act", `/home/act/`, ""},
{cwd, ".", ""},
}
if runtime.GOOS == "windows" {
cwd, err := os.Getwd()
if err != nil {
logging.Error(err)
}
rootDrive := os.Getenv("SystemDrive")
rootDriveLetter := strings.ReplaceAll(strings.ToLower(rootDrive), `:`, "")
for _, v := range []containerPathJob{
tests = []containerPathJob{
{"/mnt/c/Users/act/go/src/github.com/nektos/act", "C:\\Users\\act\\go\\src\\github.com\\nektos\\act\\", ""},
{"/mnt/f/work/dir", `F:\work\dir`, ""},
{"/mnt/c/windows/to/unix", "windows\\to\\unix", rootDrive + "\\"},
{fmt.Sprintf("/mnt/%v/act", rootDriveLetter), "act", rootDrive + "\\"},
} {
if v.workDir != "" {
t.Chdir(v.workDir)
}
assert.Equal(t, v.destinationPath, linuxcontainerext.ToContainerPath(v.sourcePath))
{"/mnt/" + strings.ReplaceAll(strings.ToLower(rootDrive), `:`, "") + "/act", "act", rootDrive + "\\"},
}
t.Chdir(cwd)
} else {
cwd, err := os.Getwd()
if err != nil {
logging.Error(err)
}
for _, v := range []containerPathJob{
{"/home/act/go/src/github.com/nektos/act", "/home/act/go/src/github.com/nektos/act", ""},
{"/home/act", `/home/act/`, ""},
{cwd, ".", ""},
} {
assert.Equal(t, v.destinationPath, linuxcontainerext.ToContainerPath(v.sourcePath))
}
for _, test := range tests {
if test.workDir != "" {
t.Chdir(test.workDir)
}
assert.Equal(t, test.destinationPath, (&LinuxContainerEnvironmentExtensions{}).ToContainerPath(test.sourcePath))
}
}
@@ -67,5 +50,4 @@ type typeAssertMockContainer struct {
LinuxContainerEnvironmentExtensions
}
// Type assert Container + LinuxContainerEnvironmentExtensions implements ExecutionsEnvironment
var _ ExecutionsEnvironment = &typeAssertMockContainer{}

View File

@@ -7,6 +7,7 @@ package container
import (
"archive/tar"
"bufio"
"bytes"
"context"
"fmt"
"io"
@@ -18,7 +19,8 @@ import (
"golang.org/x/text/transform"
)
func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Executor {
// parseEnvFile reads a GITHUB_ENV style file, splitting lines like GitHub's runner on Windows or elsewhere.
func parseEnvFile(e Container, srcPath string, env *map[string]string, windows bool) common.Executor {
localEnv := *env
return func(ctx context.Context) error {
envTar, err := e.GetContainerArchive(ctx, srcPath)
@@ -38,8 +40,22 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
s := bufio.NewScanner(decoded)
// Default 64 KiB max token size is too small for realistic env-file lines; allow up to 16 MiB.
s.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
scanned := 0
s.Split(func(data []byte, atEOF bool) (int, []byte, error) {
if index := bytes.IndexByte(data[scanned:], '\n'); index >= 0 {
advance := scanned + index + 1
scanned = 0
return advance, data[:advance], nil
}
scanned = len(data)
if atEOF && len(data) > 0 {
scanned = 0
return len(data), data, nil
}
return 0, nil, nil
})
for s.Scan() {
line := s.Text()
line := trimEnvLineEnding(s.Text(), windows)
// GitHub's runner ignores blank lines
if strings.TrimSpace(line) == "" {
continue
@@ -52,10 +68,13 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
case multiLineEnv != -1:
var multiLineEnvContent []string
multiLineEnvDelimiter := line[multiLineEnv+2:]
if multiLineEnv == 0 || multiLineEnvDelimiter == "" {
return fmt.Errorf("invalid format '%v', name and delimiter must not be empty", line)
}
delimiterFound := false
for s.Scan() {
content := s.Text()
if content == multiLineEnvDelimiter {
if trimEnvLineEnding(content, windows) == multiLineEnvDelimiter {
delimiterFound = true
break
}
@@ -67,7 +86,7 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
if !delimiterFound {
return fmt.Errorf("invalid format delimiter '%v' not found before end of file", multiLineEnvDelimiter)
}
localEnv[line[:multiLineEnv]] = strings.Join(multiLineEnvContent, "\n")
localEnv[line[:multiLineEnv]] = trimEnvLineEnding(strings.Join(multiLineEnvContent, ""), windows)
default:
return fmt.Errorf("invalid format '%v', expected a line with '=' or '<<'", line)
}
@@ -75,7 +94,13 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
if err := s.Err(); err != nil {
return fmt.Errorf("reading env file: %w", err)
}
env = &localEnv
return nil
}
}
func trimEnvLineEnding(line string, windows bool) string {
if windows && strings.HasSuffix(line, "\r\n") {
return line[:len(line)-2]
}
return strings.TrimSuffix(line, "\n")
}

View File

@@ -12,7 +12,6 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/text/encoding"
"golang.org/x/text/encoding/unicode"
)
@@ -22,117 +21,78 @@ func newTestHostEnv(t *testing.T) (*HostEnvironment, string) {
return e, filepath.Join(e.Path, "envfile")
}
func TestParseEnvFileSingleLine(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO=bar\nBAZ=qux\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "bar", env["FOO"])
assert.Equal(t, "qux", env["BAZ"])
}
func TestParseEnvFileMultiLine(t *testing.T) {
e, envPath := newTestHostEnv(t)
content := "FOO<<EOF\nline1\nline2\nEOF\n"
require.NoError(t, os.WriteFile(envPath, []byte(content), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "line1\nline2", env["FOO"])
}
func TestParseEnvFileLargeValueWithinLimit(t *testing.T) {
e, envPath := newTestHostEnv(t)
big := strings.Repeat("x", 2*1024*1024)
content := "FOO<<EOF\n" + big + "\nEOF\n"
require.NoError(t, os.WriteFile(envPath, []byte(content), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, big, env["FOO"])
func TestParseEnvFile(t *testing.T) {
t.Parallel()
largeValue := strings.Repeat("x", 2*1024*1024)
encodeUTF16 := func(endianness unicode.Endianness) string {
content, err := unicode.UTF16(endianness, unicode.UseBOM).NewEncoder().Bytes([]byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nEOF\r\n"))
require.NoError(t, err)
return string(content)
}
for _, test := range []struct {
name, content string
want map[string]string
}{
{"single line", "FOO=bar\nBAZ=qux\n", map[string]string{"FOO": "bar", "BAZ": "qux"}},
{"single line CRLF and final bare CR", "FOO=bar\r\nBAZ=qux\r", map[string]string{"FOO": "bar\r", "BAZ": "qux\r"}},
{"single line without final newline", "FOO=bar", map[string]string{"FOO": "bar"}},
{"blank lines", "\nFOO=bar\n\n \nBAZ=qux\n\n", map[string]string{"FOO": "bar", "BAZ": "qux"}},
{"multiline", "FOO<<EOF\nline1\nline2\nEOF\n", map[string]string{"FOO": "line1\nline2"}},
{"multiline without final newline", "FOO<<EOF\nline1\r\nline2\r\nEOF", map[string]string{"FOO": "line1\r\nline2\r"}},
{"empty multiline without final newline", "FOO<<EOF\nEOF", map[string]string{"FOO": ""}},
{"multiline blank lines", "FOO<<EOF\nline1\n\nline2\nEOF\n", map[string]string{"FOO": "line1\n\nline2"}},
{"multiline leading blank line", "FOO<<EOF\n\nline2\nEOF\n", map[string]string{"FOO": "\nline2"}},
{"multiline CRLF", "FOO<<EOF\r\nline1\r\nline2\r\nEOF\r\n", map[string]string{"FOO": "line1\r\nline2\r"}},
{"multiline mixed line endings", "FOO<<EOF\nline1\r\nline2\n\r\nEOF\n", map[string]string{"FOO": "line1\r\nline2\n\r"}},
{"large value within limit", "FOO<<EOF\n" + largeValue + "\nEOF\n", map[string]string{"FOO": largeValue}},
{"UTF8 BOM", "\xef\xbb\xbfFOO=bar\n", map[string]string{"FOO": "bar"}},
{"UTF16 little endian", encodeUTF16(unicode.LittleEndian), map[string]string{"FOO": "bar\r", "MULTI": "line1\r"}},
{"UTF16 big endian", encodeUTF16(unicode.BigEndian), map[string]string{"FOO": "bar\r", "MULTI": "line1\r"}},
} {
t.Run(test.name, func(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte(test.content), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env, false)(t.Context()))
assert.Equal(t, test.want, env)
})
}
t.Run("Windows line endings", func(t *testing.T) {
hostEnv, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nline2\nEOF\r\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(hostEnv, envPath, &env, true)(t.Context()))
assert.Equal(t, map[string]string{"FOO": "bar", "MULTI": "line1\r\nline2"}, env)
})
}
func TestParseEnvFileLineExceedsBufferReportsScannerError(t *testing.T) {
t.Parallel()
e, envPath := newTestHostEnv(t)
tooBig := strings.Repeat("x", 17*1024*1024) // over the 16 MiB cap
content := "FOO<<EOF\n" + tooBig + "\nEOF\n"
require.NoError(t, os.WriteFile(envPath, []byte(content), 0o600))
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\n"+strings.Repeat("x", 17*1024*1024)+"\nEOF\n"), 0o600))
env := map[string]string{}
err := parseEnvFile(e, envPath, &env)(t.Context())
err := parseEnvFile(e, envPath, &env, false)(t.Context())
require.ErrorIs(t, err, bufio.ErrTooLong)
assert.Contains(t, err.Error(), "reading env file")
}
// Regression test: a blank line used to fail the job at "Complete Job", after
// every step had already been recorded as successful.
func TestParseEnvFileBlankLines(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("\nFOO=bar\n\n \nBAZ=qux\n\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "bar", env["FOO"])
assert.Equal(t, "qux", env["BAZ"])
}
// blank lines inside a heredoc value are content, not separators
func TestParseEnvFileMultiLineKeepsBlankLines(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\nline1\n\nline2\nEOF\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "line1\n\nline2", env["FOO"])
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\n\nline2\nEOF\n"), 0o600))
env = map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "\nline2", env["FOO"])
}
func TestParseEnvFileUTF8BOM(t *testing.T) {
e, envPath := newTestHostEnv(t)
content := append([]byte{0xEF, 0xBB, 0xBF}, []byte("FOO=bar\n")...)
require.NoError(t, os.WriteFile(envPath, content, 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "bar", env["FOO"])
}
// Windows host mode: PowerShell 5.1 redirection writes UTF-16, which used to be
// unrecognisable as KEY=VALUE, so the writes were silently ignored.
func TestParseEnvFileUTF16(t *testing.T) {
tests := []struct {
name string
encoder *encoding.Encoder
}{
{"little endian", unicode.UTF16(unicode.LittleEndian, unicode.UseBOM).NewEncoder()},
{"big endian", unicode.UTF16(unicode.BigEndian, unicode.UseBOM).NewEncoder()},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
func TestParseEnvFileInvalidHeredoc(t *testing.T) {
for name, content := range map[string]string{
"missing delimiter": "FOO<<EOF\nline1\nline2\n",
"empty name": "<<EOF\nline1\nEOF\n",
"empty delimiter": "FOO<<\nline1\n\n",
"delimiter with carriage return": "FOO<<EOF\nline1\nEOF\r\n",
} {
t.Run(name, func(t *testing.T) {
e, envPath := newTestHostEnv(t)
content, err := tt.encoder.Bytes([]byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nEOF\r\n"))
require.NoError(t, err)
require.NoError(t, os.WriteFile(envPath, content, 0o600))
require.NoError(t, os.WriteFile(envPath, []byte(content), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "bar", env["FOO"])
assert.Equal(t, "line1", env["MULTI"])
err := parseEnvFile(e, envPath, &env, false)(t.Context())
require.Error(t, err)
assert.Contains(t, err.Error(), "delimiter")
assert.Empty(t, env)
})
}
}
func TestParseEnvFileMissingDelimiter(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\nline1\nline2\n"), 0o600))
env := map[string]string{}
err := parseEnvFile(e, envPath, &env)(t.Context())
require.Error(t, err)
assert.Contains(t, err.Error(), "delimiter")
}

View File

@@ -1 +0,0 @@
testfile

View File

@@ -0,0 +1,85 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"errors"
"fmt"
"slices"
"strings"
"unicode"
"unicode/utf8"
"github.com/moby/moby/api/types/mount"
)
type VolumeSpec struct {
Type mount.Type
Source string
Target string
ReadOnly bool
NoCopy bool
}
// ParseVolume parses spec like `docker run -v` does, without knowing the target platform.
func ParseVolume(spec string) (VolumeSpec, error) {
switch len(spec) {
case 0:
return VolumeSpec{}, errors.New("invalid empty volume spec")
case 1, 2:
return VolumeSpec{Type: mount.TypeVolume, Target: spec}, nil
}
spec = string([]rune(spec)) // each invalid UTF-8 byte becomes U+FFFD, as in the JSON the daemon receives
var parts []string
for rest := spec; ; {
start := 0
if hasDriveLetter(rest) {
start = strings.IndexByte(rest, ':') + 1
}
end := strings.IndexByte(rest[start:], ':')
if end < 0 {
parts = append(parts, rest)
break
}
parts = append(parts, rest[:start+end])
rest = rest[start+end+1:]
}
if len(parts) > 3 {
return VolumeSpec{}, fmt.Errorf("invalid spec: %s: too many colons", spec)
}
if slices.Contains(parts, "") {
return VolumeSpec{}, fmt.Errorf("invalid spec: %s: empty section between colons", spec)
}
volume := VolumeSpec{Type: mount.TypeVolume, Target: parts[0]}
if len(parts) > 1 {
volume.Source, volume.Target = parts[0], parts[1]
if isHostPath(volume.Source) {
volume.Type = mount.TypeBind
}
}
if len(parts) == 3 {
for option := range strings.SplitSeq(parts[2], ",") {
switch option {
case "ro":
volume.ReadOnly = true
case "rw":
volume.ReadOnly = false
case "nocopy":
volume.NoCopy = true
}
}
}
return volume, nil
}
func hasDriveLetter(s string) bool {
letter, size := utf8.DecodeRuneInString(s)
return size < len(s) && s[size] == ':' && unicode.IsLetter(letter)
}
func isHostPath(source string) bool {
return strings.ContainsAny(source[:1], "./~") || strings.HasPrefix(source, `\\`) || hasDriveLetter(source)
}

View File

@@ -0,0 +1,18 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"testing"
"github.com/moby/moby/api/types/mount"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestParseVolumeReplacesInvalidUTF8LikeTheDaemonSees(t *testing.T) {
parsed, err := ParseVolume("/data/\xff\xfe:/mnt:ro")
require.NoError(t, err)
assert.Equal(t, VolumeSpec{Type: mount.TypeBind, Source: "/data/<2F><>", Target: "/mnt", ReadOnly: true}, parsed)
}

View File

@@ -43,7 +43,6 @@ func (tc TarCollector) WriteFile(fpath string, fi fs.FileInfo, linkName string,
// update the name to correctly reflect the desired destination when untaring
header.Name = path.Join(tc.DstDir, fpath)
header.Mode = int64(fi.Mode())
header.ModTime = fi.ModTime()
header.Uid = tc.UID
header.Gid = tc.GID
@@ -58,10 +57,8 @@ func (tc TarCollector) WriteFile(fpath string, fi fs.FileInfo, linkName string,
}
// copy file data into tar writer
if _, err := io.Copy(tc.TarWriter, f); err != nil {
return err
}
return nil
_, err = io.Copy(tc.TarWriter, f)
return err
}
type CopyCollector struct {
@@ -87,10 +84,8 @@ func (cc *CopyCollector) WriteFile(fpath string, fi fs.FileInfo, linkName string
return err
}
defer df.Close()
if _, err := io.Copy(df, f); err != nil {
return err
}
return nil
_, err = io.Copy(df, f)
return err
}
type FileCollector struct {

View File

@@ -28,12 +28,12 @@ var (
type pinnedKey struct{}
// WithPinnedCheckout keeps a run's sha and ref fixed when HEAD moves.
func WithPinnedCheckout(ctx context.Context) context.Context {
// WithPinnedCheckout keeps a run's sha, ref and repository fixed when HEAD moves, in pins unless ctx has some.
func WithPinnedCheckout(ctx context.Context, pins *sync.Map) context.Context {
if ctx.Value(pinnedKey{}) != nil {
return ctx
}
return context.WithValue(ctx, pinnedKey{}, &sync.Map{})
return context.WithValue(ctx, pinnedKey{}, pins)
}
func pinned(ctx context.Context, key string, lookup func() (string, error)) (string, error) {
@@ -57,12 +57,29 @@ func SetRef(ctx context.Context, ghc *model.GithubContext, repoPath string) {
case "pull_request_target":
ghc.Ref = "refs/heads/" + ghc.BaseRef
case "pull_request", "pull_request_review", "pull_request_review_comment":
ghc.Ref = fmt.Sprintf("refs/pull/%.0f/merge", ghc.Event["number"])
number := ghc.Event["number"]
if number == nil {
number = model.NestedMapLookup(ghc.Event, "pull_request", "number")
}
ghc.Ref = fmt.Sprintf("refs/pull/%.0f/merge", number)
if ghc.EventName == "pull_request" && ghc.Event["action"] == "closed" && model.NestedMapLookup(ghc.Event, "pull_request", "merged") == true {
ghc.Ref = "refs/heads/" + ghc.BaseRef
}
case "deployment", "deployment_status":
ghc.Ref = model.AsString(model.NestedMapLookup(ghc.Event, "deployment", "ref"))
case "release":
ghc.Ref = "refs/tags/" + model.AsString(model.NestedMapLookup(ghc.Event, "release", "tag_name"))
case "push", "create", "workflow_dispatch":
case "create":
ghc.Ref = model.AsString(ghc.Event["ref"])
if ghc.Ref != "" && !strings.HasPrefix(ghc.Ref, "refs/") {
switch model.AsString(ghc.Event["ref_type"]) {
case "branch":
ghc.Ref = "refs/heads/" + ghc.Ref
case "tag":
ghc.Ref = "refs/tags/" + ghc.Ref
}
}
case "push", "workflow_dispatch":
ghc.Ref = model.AsString(ghc.Event["ref"])
default:
defaultBranch := model.AsString(model.NestedMapLookup(ghc.Event, "repository", "default_branch"))
@@ -117,8 +134,7 @@ func SetSha(ctx context.Context, ghc *model.GithubContext, repoPath string) {
if ghc.Sha == "" {
sha, err := pinned(ctx, "sha:"+repoPath, func() (string, error) {
_, revision, err := findGitRevision(ctx, repoPath)
return revision, err
return findGitRevision(ctx, repoPath)
})
if err != nil {
logger.Warnf("unable to get git revision: %v", err)
@@ -132,12 +148,12 @@ func SetSha(ctx context.Context, ghc *model.GithubContext, repoPath string) {
// remote in repoPath when it is not set yet, and derives its owner.
func SetRepositoryAndOwner(ctx context.Context, ghc *model.GithubContext, repoPath string) {
if ghc.Repository == "" {
repo, err := findGithubRepo(ctx, repoPath)
repo, err := pinned(ctx, "repo:"+repoPath, func() (string, error) { return findGithubRepo(ctx, repoPath) })
if err != nil {
common.Logger(ctx).Warnf("unable to get git repo (repoPath: %v): %v", repoPath, err)
return
}
ghc.Repository = repo
}
ghc.RepositoryOwner = strings.Split(ghc.Repository, "/")[0]
ghc.RepositoryOwner, _, _ = strings.Cut(ghc.Repository, "/")
}

View File

@@ -7,30 +7,21 @@ package ghcontext
import (
"context"
"errors"
"sync"
"testing"
"gitea.com/gitea/runner/act/common/logging"
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
)
func TestSetRef(t *testing.T) {
logging.SetLevel(logging.LevelDebug)
oldFindGitRef := findGitRef
oldFindGitRevision := findGitRevision
defer func() { findGitRef = oldFindGitRef }()
defer func() { findGitRevision = oldFindGitRevision }()
findGitRef = func(ctx context.Context, file string) (string, error) {
findGitRef = func(_ctx context.Context, _file string) (string, error) {
return "refs/heads/master", nil
}
findGitRevision = func(ctx context.Context, file string) (string, string, error) {
return "", "1234fakesha", nil
}
tables := []struct {
eventName string
event map[string]any
@@ -51,6 +42,30 @@ func TestSetRef(t *testing.T) {
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "pull_request_review",
event: map[string]any{"pull_request": map[string]any{"number": 1234.}},
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "pull_request",
event: map[string]any{"number": 1234., "action": "closed", "pull_request": map[string]any{"merged": true}},
ref: "refs/heads/master",
refName: "master",
},
{
eventName: "pull_request",
event: map[string]any{"number": 1234., "action": "closed", "pull_request": map[string]any{"merged": false}},
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "pull_request_review_comment",
event: map[string]any{"pull_request": map[string]any{"number": 1234.}},
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "deployment",
event: map[string]any{
@@ -79,6 +94,24 @@ func TestSetRef(t *testing.T) {
ref: "refs/heads/somebranch",
refName: "somebranch",
},
{
eventName: "create",
event: map[string]any{"ref": "feature/new", "ref_type": "branch"},
ref: "refs/heads/feature/new",
refName: "feature/new",
},
{
eventName: "create",
event: map[string]any{"ref": "v1", "ref_type": "tag"},
ref: "refs/tags/v1",
refName: "v1",
},
{
eventName: "create",
event: map[string]any{"ref": "refs/tags/v1", "ref_type": "tag"},
ref: "refs/tags/v1",
refName: "v1",
},
{
eventName: "unknown",
event: map[string]any{
@@ -114,7 +147,7 @@ func TestSetRef(t *testing.T) {
}
t.Run("no-default-branch", func(t *testing.T) {
findGitRef = func(ctx context.Context, file string) (string, error) {
findGitRef = func(_ctx context.Context, _file string) (string, error) {
return "", errors.New("no default branch")
}
@@ -130,19 +163,15 @@ func TestSetRef(t *testing.T) {
}
func TestSetSha(t *testing.T) {
logging.SetLevel(logging.LevelDebug)
oldFindGitRef := findGitRef
oldFindGitRevision := findGitRevision
oldFindGithubRepo := findGithubRepo
defer func() { findGitRef = oldFindGitRef }()
defer func() { findGitRevision = oldFindGitRevision }()
defer func() { findGithubRepo = oldFindGithubRepo }()
findGitRef = func(ctx context.Context, file string) (string, error) {
return "refs/heads/master", nil
}
findGitRevision = func(ctx context.Context, file string) (string, string, error) {
return "", "1234fakesha", nil
findGitRevision = func(_ctx context.Context, _file string) (string, error) {
return "1234fakesha", nil
}
tables := []struct {
@@ -177,11 +206,6 @@ func TestSetSha(t *testing.T) {
},
sha: "deployment-sha",
},
{
eventName: "release",
event: map[string]any{},
sha: "1234fakesha",
},
{
eventName: "push",
event: map[string]any{
@@ -195,18 +219,12 @@ func TestSetSha(t *testing.T) {
event: map[string]any{},
sha: "1234fakesha",
},
{
eventName: "no-event",
event: map[string]any{},
sha: "1234fakesha",
},
}
for _, table := range tables {
t.Run(table.eventName, func(t *testing.T) {
ghc := &model.GithubContext{
EventName: table.eventName,
BaseRef: "master",
Event: table.event,
}
@@ -216,16 +234,19 @@ func TestSetSha(t *testing.T) {
})
}
t.Run("pinned checkout keeps the first sha and ref", func(t *testing.T) {
ctx := WithPinnedCheckout(t.Context())
t.Run("pinned checkout keeps the first sha, ref and repository", func(t *testing.T) {
ctx := WithPinnedCheckout(t.Context(), &sync.Map{})
for _, head := range []string{"first", "moved"} {
findGitRevision = func(context.Context, string) (string, string, error) { return "", head, nil }
findGitRevision = func(context.Context, string) (string, error) { return head, nil }
findGitRef = func(context.Context, string) (string, error) { return "refs/heads/" + head, nil }
findGithubRepo = func(context.Context, string) (string, error) { return "owner/" + head, nil }
ghc := &model.GithubContext{Event: map[string]any{}}
SetSha(ctx, ghc, "/some/dir")
SetRef(ctx, ghc, "/some/dir")
SetRepositoryAndOwner(ctx, ghc, "/some/dir")
assert.Equal(t, "first", ghc.Sha)
assert.Equal(t, "refs/heads/first", ghc.Ref)
assert.Equal(t, "owner/first", ghc.Repository)
}
})
}

View File

@@ -13,35 +13,23 @@ import (
"testing"
)
func TestLookPath2SearchesPathAndEmptyElement(t *testing.T) {
func TestLookPath2SearchesPathAndEmptyElementOrUsesDirectPath(t *testing.T) {
dir := t.TempDir()
exe := filepath.Join(dir, "tool")
if err := os.WriteFile(exe, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
got, err := LookPath2("tool", map[string]string{"PATH": string(filepath.ListSeparator) + dir})
if err != nil {
t.Fatal(err)
}
if got != exe {
t.Fatalf("LookPath2() = %q, want %q", got, exe)
}
}
func TestLookPath2DirectPathDoesNotSearchPath(t *testing.T) {
dir := t.TempDir()
exe := filepath.Join(dir, "tool")
if err := os.WriteFile(exe, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
got, err := LookPath2(exe, map[string]string{"PATH": ""})
if err != nil {
t.Fatal(err)
}
if got != exe {
t.Fatalf("LookPath2() = %q, want %q", got, exe)
for file, searchPath := range map[string]string{"tool": string(filepath.ListSeparator) + dir, exe: ""} {
t.Run(file, func(t *testing.T) {
got, err := LookPath2(file, map[string]string{"PATH": searchPath})
if err != nil {
t.Fatal(err)
}
if got != exe {
t.Fatalf("LookPath2() = %q, want %q", got, exe)
}
})
}
}

View File

@@ -28,21 +28,15 @@ func chkStat(file string) error {
}
func hasExt(file string) bool {
i := strings.LastIndex(file, ".")
if i < 0 {
return false
}
return strings.LastIndexAny(file, `:\/`) < i
return strings.LastIndex(file, ".") > strings.LastIndexAny(file, `:\/`)
}
func findExecutable(file string, exts []string) (string, error) {
if len(exts) == 0 {
return file, chkStat(file)
}
if hasExt(file) {
if chkStat(file) == nil {
return file, nil
}
if hasExt(file) && chkStat(file) == nil {
return file, nil
}
for _, e := range exts {
if f := file + e; chkStat(f) == nil {

View File

@@ -8,7 +8,6 @@ import (
"context"
"crypto/rand"
"crypto/sha256"
"embed"
"encoding/hex"
"errors"
"fmt"
@@ -20,12 +19,15 @@ import (
"path/filepath"
"regexp"
"runtime"
"slices"
"strings"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/container"
_ "embed"
"gitea.dev/actionslib/pkg/model"
"github.com/kballard/go-shellquote"
)
@@ -47,11 +49,13 @@ type fileWriter func(filename string, data []byte, perm fs.FileMode) error
type runAction func(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor
//go:embed res/trampoline.js
var trampoline embed.FS
var trampoline string
var (
ContainerImageExistsLocally = container.ImageExistsLocally
ContainerNewDockerBuildExecutor = container.NewDockerBuildExecutor
dockerActionNamePattern = regexp.MustCompile("[^a-zA-Z0-9]")
inputEnvPattern = regexp.MustCompile("[^A-Z0-9-]")
)
func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath string, readFile actionYamlReader, writeFile fileWriter) (*model.Action, error) {
@@ -71,52 +75,52 @@ func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath
reader, closer, err = readFile("action.yaml")
addError("action.yaml", err)
if os.IsNotExist(err) {
_, closer, err := readFile("Dockerfile")
addError("Dockerfile", err)
dockerfile := "Dockerfile"
_, closer, err := readFile(dockerfile)
addError(dockerfile, err)
if os.IsNotExist(err) {
dockerfile = "dockerfile"
_, closer, err = readFile(dockerfile)
addError(dockerfile, err)
}
if err == nil {
closer.Close()
action := &model.Action{
Name: "(Synthetic)",
Runs: model.ActionRuns{
Using: "docker",
Image: "Dockerfile",
Image: dockerfile,
},
}
logger.Debugf("Using synthetic action %v for Dockerfile", action)
logger.Debugf("Using synthetic action %v for %s", action, dockerfile)
return action, nil
}
if step.With != nil {
if val, ok := step.With["args"]; ok {
var b []byte
if b, err = trampoline.ReadFile("res/trampoline.js"); err != nil {
return nil, err
}
err2 := writeFile(filepath.Join(actionDir, actionPath, "trampoline.js"), b, 0o400)
if err2 != nil {
return nil, err2
}
action := &model.Action{
Name: "(Synthetic)",
Inputs: map[string]model.Input{
"cwd": {
Description: "(Actual working directory)",
Required: false,
Default: filepath.Join(actionDir, actionPath),
},
"command": {
Description: "(Actual program)",
Required: false,
Default: val,
},
},
Runs: model.ActionRuns{
Using: "node12",
Main: "trampoline.js",
},
}
logger.Debugf("Using synthetic action %v", action)
return action, nil
if val, ok := step.With["args"]; ok {
if err := writeFile(filepath.Join(actionDir, actionPath, "trampoline.js"), []byte(trampoline), 0o644); err != nil {
return nil, err
}
action := &model.Action{
Name: "(Synthetic)",
Inputs: map[string]model.Input{
"args": {},
"cwd": {
Description: "(Actual working directory)",
Required: false,
Default: filepath.Join(actionDir, actionPath),
},
"command": {
Description: "(Actual program)",
Required: false,
Default: val,
},
},
Runs: model.ActionRuns{
Using: "node12",
Main: "trampoline.js",
},
}
logger.Debugf("Using synthetic action %v", action)
return action, nil
}
}
}
@@ -125,8 +129,7 @@ func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath
}
defer closer.Close()
action, err := model.ReadAction(reader)
return action, err
return model.ReadAction(reader)
}
func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actionPath, containerActionDir string) error {
@@ -151,12 +154,17 @@ func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actio
}
defer unlock()
if !rc.Config.NoActionPatch {
// A concurrent job's prepare resets this directory, so patch under the copy's lock.
defer patchActions(ctx, actionScriptPaths(filepath.Join(actionDir, actionPath), step.getActionModel()))()
if err := rc.JobContainer.CopyDir(containerActionDirCopy, actionDir+"/", false, true)(ctx); err != nil { // an exported commit has no .git to apply gitignore with
return err
}
return rc.JobContainer.CopyDir(containerActionDirCopy, actionDir+"/", rc.Config.UseGitIgnore, true)(ctx)
if rc.Config.NoActionPatch {
return nil
}
patched := patchedScripts(ctx, actionDir, actionScriptPaths(actionDir, filepath.Join(actionDir, actionPath), step.getActionModel()))
if len(patched) == 0 {
return nil
}
return rc.JobContainer.Copy(containerActionDirCopy, patched...)(ctx)
}
func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor {
@@ -166,7 +174,7 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
return func(ctx context.Context) error {
logger := common.Logger(ctx)
actionPath := ""
if remoteAction != nil && remoteAction.Path != "" {
if remoteAction != nil {
actionPath = remoteAction.Path
}
@@ -177,9 +185,10 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
if err := populateEnvsFromInput(ctx, step.getEnv(), action, rc); err != nil {
return err
}
warnUnexpectedInputs(ctx, action, stepModel.With)
printRunActionHeader(ctx, stepModel, action, *step.getEnv(), rc)
actionLocation := path.Join(actionDir, actionPath)
actionName, containerActionDir := getContainerActionPaths(stepModel, actionLocation, rc)
actionName, containerActionDir := getContainerActionPaths(step)
logger.Debugf("type=%v actionDir=%s actionPath=%s workdir=%s actionCacheDir=%s actionName=%s containerActionDir=%s", stepModel.Type(), actionDir, actionPath, rc.Config.Workdir, rc.ActionCacheDir(), actionName, containerActionDir)
@@ -196,7 +205,7 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
return rc.JobContainer.Exec(containerArgs, *step.getEnv(), "", "")(ctx)
case x.IsDocker():
location := actionLocation
location := path.Join(actionDir, actionPath)
if remoteAction == nil {
location = containerActionDir
}
@@ -215,12 +224,9 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
rc.ApplyExtraPath(ctx, step.getEnv())
execFileName := action.Runs.Main + ".out"
buildArgs := []string{"go", "build", "-o", execFileName, action.Runs.Main}
execArgs := []string{filepath.Join(containerActionDir, execFileName)}
return common.NewPipelineExecutor(
rc.JobContainer.Exec(buildArgs, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec(execArgs, *step.getEnv(), "", ""),
rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Main}, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec([]string{filepath.Join(containerActionDir, execFileName)}, *step.getEnv(), "", ""),
)(ctx)
default:
return fmt.Errorf("the runs.using key must be one of: %v, got %s", []string{
@@ -259,16 +265,14 @@ func dockerActionImageTag(repository, actionName string, localAction bool) strin
name = path.Join(repository, actionName)
}
// The human-readable name is sanitized by collapsing every non-alphanumeric character to "-".
sanitized := regexp.MustCompile("[^a-zA-Z0-9]").ReplaceAllString(name, "-")
sanitized := dockerActionNamePattern.ReplaceAllString(name, "-")
if localAction {
// For local actions a short hash of the raw repository and action path is appended so the tag stays unique per repository.
sum := sha256.Sum256([]byte(repository + "\x00" + actionName))
sanitized += "-" + hex.EncodeToString(sum[:])[:12]
}
// "-dockeraction" ensures that "./", "./test " won't get converted to "act-:latest", "act-test-:latest" which are invalid docker image names
image := fmt.Sprintf("%s-dockeraction:%s", sanitized, "latest")
image = "act-" + strings.TrimLeft(image, "-")
return strings.ToLower(image)
return strings.ToLower("act-" + strings.TrimLeft(sanitized+"-dockeraction:latest", "-"))
}
// One image per step and job: its stages share a build, and concurrent jobs never run each other's.
@@ -312,9 +316,12 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
return err
}
correctArchExists, err := ContainerImageExistsLocally(ctx, image, rc.Config.ContainerArchitecture)
if err != nil {
return err
correctArchExists := anyArchExists
if anyArchExists && rc.Config.ContainerArchitecture != "" && rc.Config.ContainerArchitecture != "any" {
correctArchExists, err = ContainerImageExistsLocally(ctx, image, rc.Config.ContainerArchitecture)
if err != nil {
return err
}
}
if anyArchExists && !correctArchExists {
@@ -361,14 +368,11 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
logger.Debugf("image '%s' for architecture '%s' already exists", image, rc.Config.ContainerArchitecture)
}
}
cmd, err := shellquote.Split(step.getStepModel().With["args"])
if err != nil {
return err
}
ee, err := evalDockerEnv(ctx, step, action)
if err != nil {
return err
}
var cmd []string
if action.Runs.Args != nil {
// a fresh slice, the manifest is evaluated again for every stage
cmd = make([]string, len(action.Runs.Args))
@@ -377,6 +381,11 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
return fmt.Errorf("unable to interpolate runs.args: %w", err)
}
}
} else {
cmd, err = shellquote.Split(stepInput(step.getStepModel(), "args"))
if err != nil {
return err
}
}
entrypoint, err := dockerEntrypoint(step, stage)
if err != nil {
@@ -406,7 +415,7 @@ func dockerEntrypoint(step actionStep, stage stepStage) ([]string, error) {
default:
entrypoint = runs.Entrypoint
if entrypoint == "" {
if fields := strings.Fields(step.getStepModel().With["entrypoint"]); len(fields) > 0 {
if fields := strings.Fields(stepInput(step.getStepModel(), "entrypoint")); len(fields) > 0 {
return fields, nil
}
}
@@ -424,9 +433,15 @@ func evalDockerEnv(ctx context.Context, step step, action *model.Action) (*expre
var err error
inputs := make(map[string]string)
providedInputs := make(map[string]bool, len(step.getStepModel().With))
for key := range step.getStepModel().With {
providedInputs[strings.ToLower(key)] = true
}
eval := rc.NewExpressionEvaluator(ctx)
// Set Defaults
for k, input := range action.Inputs {
if providedInputs[strings.ToLower(k)] {
continue
}
if inputs[k], err = eval.Interpolate(ctx, input.Default); err != nil {
return nil, fmt.Errorf("unable to interpolate the default of input %s: %w", k, err)
}
@@ -485,20 +500,19 @@ func newStepContainer(ctx context.Context, step step, image string, cmd, entrypo
}
func populateEnvsFromSavedState(env *map[string]string, step actionStep, rc *RunContext) {
state, ok := rc.IntraActionState[step.getStepModel().ID]
if ok {
for name, value := range state {
envName := "STATE_" + name
(*env)[envName] = value
}
for name, value := range rc.IntraActionState[step.getStepModel().ID] {
(*env)["STATE_"+name] = value
}
}
func inputEnvKey(inputID string) string {
return "INPUT_" + inputEnvPattern.ReplaceAllString(strings.ToUpper(inputID), "_")
}
func populateEnvsFromInput(ctx context.Context, env *map[string]string, action *model.Action, rc *RunContext) error {
eval := rc.NewExpressionEvaluator(ctx)
for inputID, input := range action.Inputs {
envKey := regexp.MustCompile("[^A-Z0-9-]").ReplaceAllString(strings.ToUpper(inputID), "_")
envKey = "INPUT_" + envKey
envKey := inputEnvKey(inputID)
if _, ok := (*env)[envKey]; !ok {
var err error
if (*env)[envKey], err = eval.Interpolate(ctx, input.Default); err != nil {
@@ -509,25 +523,55 @@ func populateEnvsFromInput(ctx context.Context, env *map[string]string, action *
return nil
}
func getContainerActionPaths(step *model.Step, actionDir string, rc *RunContext) (string, string) {
actionName := ""
containerActionDir := "."
if step.Type() != model.StepTypeUsesActionRemote {
actionName = getOsSafeRelativePath(actionDir, rc.Config.Workdir)
containerActionDir = rc.JobContainer.ToContainerPath(rc.Config.Workdir) + "/" + actionName
actionName = "./" + actionName
} else if step.Type() == model.StepTypeUsesActionRemote {
actionName = getOsSafeRelativePath(actionDir, rc.ActionCacheDir())
containerActionDir = rc.JobContainer.GetActPath() + "/actions/" + actionName
// lookupFold finds name in values, ignoring case where the environment does, as GitHub's inputs always do.
func lookupFold[V any](values map[string]V, name string, caseInsensitive bool) (V, bool) {
if value, ok := values[name]; ok || !caseInsensitive {
return value, ok
}
if actionName == "" {
actionName = filepath.Base(actionDir)
if runtime.GOOS == "windows" {
actionName = strings.ReplaceAll(actionName, "\\", "/")
for key, value := range values {
if strings.EqualFold(key, name) {
return value, true
}
}
return actionName, containerActionDir
var zero V
return zero, false
}
func stepInput(step *model.Step, name string) string {
value, _ := lookupFold(step.With, name, true)
return value
}
func warnUnexpectedInputs(ctx context.Context, action *model.Action, inputs map[string]string) {
docker := action.Runs.Using.IsDocker()
var unexpectedInputs []string
for input := range inputs {
_, declared := lookupFold(action.Inputs, input, true)
containerInput := docker && (strings.EqualFold(input, "entryPoint") || strings.EqualFold(input, "args"))
if !declared && !containerInput {
unexpectedInputs = append(unexpectedInputs, input)
}
}
if len(unexpectedInputs) == 0 {
return
}
validInputs := slices.Sorted(maps.Keys(action.Inputs))
if docker {
validInputs = append([]string{"entryPoint", "args"}, validInputs...)
}
slices.Sort(unexpectedInputs)
common.Logger(ctx).With(rawOutputField, true).Warnf("##[warning]%s", EscapeCommandData(fmt.Sprintf(
"Unexpected input(s) '%s', valid inputs are ['%s']", strings.Join(unexpectedInputs, "', '"), strings.Join(validInputs, "', '"))))
}
func getContainerActionPaths(step actionStep) (string, string) {
rc := step.getRunContext()
if sar, ok := step.(*stepActionRemote); ok { // a copy per action, as its repository's directory is shared
actionName := path.Join(sar.containerKey(), sar.remoteAction.Path)
return actionName, rc.JobContainer.GetActPath() + "/actions/" + actionName
}
actionName := getOsSafeRelativePath(filepath.Join(rc.Config.Workdir, step.getStepModel().Uses), rc.Config.Workdir)
return "./" + actionName, rc.JobContainer.ToContainerPath(rc.Config.Workdir) + "/" + actionName
}
func getOsSafeRelativePath(s, prefix string) string {
@@ -573,13 +617,13 @@ func actionStagePaths(step actionStep) (actionDir, actionPath, actionName, conta
stepModel := step.getStepModel()
if sar, ok := step.(*stepActionRemote); ok {
actionDir = sar.actionDir()
actionDir = sar.actionDir
actionPath = sar.remoteAction.Path
} else {
actionDir = filepath.Join(rc.Config.Workdir, stepModel.Uses)
}
actionName, containerActionDir = getContainerActionPaths(stepModel, path.Join(actionDir, actionPath), rc)
actionName, containerActionDir = getContainerActionPaths(step)
return actionDir, actionPath, actionName, containerActionDir
}
@@ -612,6 +656,7 @@ func runPreStep(step actionStep) common.Executor {
return err
}
}
printRunActionHeader(ctx, step.getStepModel(), action, *step.getEnv(), rc)
switch {
case x.IsNode():
if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil {
@@ -648,12 +693,9 @@ func runPreStep(step actionStep) common.Executor {
rc.ApplyExtraPath(ctx, step.getEnv())
execFileName := action.Runs.Pre + ".out"
buildArgs := []string{"go", "build", "-o", execFileName, action.Runs.Pre}
execArgs := []string{filepath.Join(containerActionDir, execFileName)}
return common.NewPipelineExecutor(
rc.JobContainer.Exec(buildArgs, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec(execArgs, *step.getEnv(), "", ""),
rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Pre}, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec([]string{filepath.Join(containerActionDir, execFileName)}, *step.getEnv(), "", ""),
)(ctx)
default:
return nil
@@ -663,8 +705,11 @@ func runPreStep(step actionStep) common.Executor {
func shouldRunPostStep(step actionStep) common.Conditional {
return func(ctx context.Context) bool {
if remote, ok := step.(*stepActionRemote); ok && remote.preStepStarted && !remote.action.Runs.Using.IsComposite() {
return true
}
log := common.Logger(ctx)
stepResults := step.getRunContext().getStepsContext()
stepResults := step.getRunContext().StepResults
stepResult := stepResults[step.getStepModel().ID]
if stepResult == nil {
@@ -710,6 +755,9 @@ func runPostStep(step actionStep) common.Executor {
actionDir, actionPath, _, containerActionDir := actionStagePaths(step)
x := action.Runs.Using
if !x.IsComposite() { // a composite's children print their own
logger.With(rawOutputField, true).Info("Post job cleanup.")
}
switch {
case x.IsNode():
@@ -742,12 +790,9 @@ func runPostStep(step actionStep) common.Executor {
rc.ApplyExtraPath(ctx, step.getEnv())
execFileName := action.Runs.Post + ".out"
buildArgs := []string{"go", "build", "-o", execFileName, action.Runs.Post}
execArgs := []string{filepath.Join(containerActionDir, execFileName)}
return common.NewPipelineExecutor(
rc.JobContainer.Exec(buildArgs, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec(execArgs, *step.getEnv(), "", ""),
rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Post}, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec([]string{filepath.Join(containerActionDir, execFileName)}, *step.getEnv(), "", ""),
)(ctx)
default:

View File

@@ -8,7 +8,6 @@ import (
"context"
"errors"
"fmt"
"regexp"
"slices"
"strconv"
"strings"
@@ -32,8 +31,7 @@ func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step
ee := parent.NewStepExpressionEvaluator(ctx, step)
for inputID, input := range step.getActionModel().Inputs {
envKey := regexp.MustCompile("[^A-Z0-9-]").ReplaceAllString(strings.ToUpper(inputID), "_")
envKey = "INPUT_" + strings.ToUpper(envKey)
envKey := inputEnvKey(inputID)
// lookup if key is defined in the step but the already
// evaluated value from the environment
@@ -54,6 +52,11 @@ func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step
}
}
}
for envKey, value := range (&model.Step{With: step.getStepModel().With}).GetEnv() {
if _, ok := env[envKey]; !ok {
env[envKey] = value
}
}
gh := step.getGithubContext(ctx)
env["GITHUB_ACTION_REPOSITORY"] = gh.ActionRepository
env["GITHUB_ACTION_REF"] = gh.ActionRef
@@ -211,19 +214,16 @@ func (rc *RunContext) compositeExecutor(action *model.Action) *compositeSteps {
}
stepID := step.getStepModel().ID
stepPre := rc.newCompositeCommandExecutor(step.pre())
preSteps = append(preSteps, newCompositeStepLogExecutor(stepPre, stepID))
preSteps = append(preSteps, newCompositeStepLogExecutor(rc.newCompositeCommandExecutor(step.pre()), stepID))
steps = append(steps, newCompositeStepLogExecutor(rc.newCompositeCommandExecutor(step.main()), stepID))
// run the post executor in reverse order
stepPost := rc.newCompositeCommandExecutor(step.post())
if postExecutor != nil {
stepPost := rc.newCompositeCommandExecutor(step.post())
postExecutor = newCompositeStepLogExecutor(stepPost.Finally(postExecutor), stepID)
} else {
stepPost := rc.newCompositeCommandExecutor(step.post())
postExecutor = newCompositeStepLogExecutor(stepPost, stepID)
stepPost = stepPost.Finally(postExecutor)
}
postExecutor = newCompositeStepLogExecutor(stepPost, stepID)
}
steps = append(steps, common.JobError)
@@ -241,14 +241,7 @@ func (rc *RunContext) compositeExecutor(action *model.Action) *compositeSteps {
func (rc *RunContext) newCompositeCommandExecutor(executor common.Executor) common.Executor {
return func(ctx context.Context) error {
ctx = WithCompositeLogger(ctx, &rc.Masks)
logWriter := rc.commandLogWriter(ctx)
oldout, olderr := rc.JobContainer.ReplaceLogWriter(logWriter, logWriter)
defer rc.JobContainer.ReplaceLogWriter(oldout, olderr)
return executor(ctx)
return rc.withCommandLogWriter(WithCompositeLogger(ctx, &rc.Masks), executor)
}
}
@@ -257,12 +250,12 @@ func newCompositeStepLogExecutor(runStep common.Executor, stepID string) common.
ctx = WithCompositeStepLogger(ctx, stepID)
logger := common.Logger(ctx)
err := runStep(ctx)
if err == nil {
err = ctx.Err()
}
if err != nil {
logger.Errorf("##[error]%s", EscapeCommandData(err.Error()))
common.SetJobError(ctx, err)
} else if ctx.Err() != nil {
logger.Errorf("##[error]%s", EscapeCommandData(ctx.Err().Error()))
common.SetJobError(ctx, ctx.Err())
}
return nil
}

View File

@@ -4,10 +4,13 @@
package runner
import (
"context"
"testing"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
@@ -17,10 +20,11 @@ import (
func TestCompositeActionParity(t *testing.T) {
t.Run("inherits contexts without leaking inputs", func(t *testing.T) {
ctx := t.Context()
hook := logtest.NewWithLevel(logging.LevelWarn)
ctx := common.WithLogger(t.Context(), logging.New(hook))
strategy := &model.Strategy{MaxParallelString: "3", RawMatrix: yaml.Node{Kind: yaml.MappingNode}}
parent := &RunContext{
Config: &Config{},
Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Matrix: map[string]any{"os": "linux"},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Name: "workflow", Jobs: map[string]*model.Job{"job": {Strategy: strategy}}}},
JobContainer: &jobContainerMock{},
@@ -28,21 +32,26 @@ func TestCompositeActionParity(t *testing.T) {
jobIndex: 1,
jobTotal: 4,
}
composite, err := newCompositeRunContext(ctx, parent, &stepActionRemote{
Step: &model.Step{With: map[string]string{"SHARED": "outer"}},
step := &stepActionLocal{
Step: &model.Step{Uses: "./action", With: map[string]string{"SHARED": "outer", "extra": "literal${{ github.job }}"}},
RunContext: parent,
action: &model.Action{Inputs: map[string]model.Input{"shared": {Default: "outer-default"}}},
action: &model.Action{Inputs: map[string]model.Input{"shared": {Default: "outer-default"}}, Runs: model.ActionRuns{Using: "composite"}},
env: map[string]string{"INPUT_SHARED": "outer"},
}, "/action")
}
require.NoError(t, runActionImpl(step, "/action", nil)(ctx))
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, "##[warning]Unexpected input(s) 'extra', valid inputs are ['shared']", hook.LastRecord().Message)
composite, err := step.getCompositeRunContext(ctx)
require.NoError(t, err)
assert.Same(t, strategy, composite.Run.Job().Strategy)
assert.True(t, composite.IsHostEnv())
interpolated, err := composite.NewExpressionEvaluator(ctx).Interpolate(ctx,
"${{ matrix.os }}|${{ strategy.max-parallel }}|${{ strategy.job-index }}|${{ inputs.shared }}")
"${{ matrix.os }}|${{ strategy.max-parallel }}|${{ strategy.job-index }}|${{ inputs.shared }}|${{ inputs.extra }}")
require.NoError(t, err)
assert.Equal(t, "linux|3|1|outer", interpolated)
assert.Equal(t, "linux|3|1|outer|literal${{ github.job }}", interpolated)
assert.NotContains(t, composite.Env, "INPUT_SHARED")
assert.NotContains(t, composite.Env, "INPUT_EXTRA")
nestedEnv := composite.GetEnv()
require.NoError(t, populateEnvsFromInput(ctx, &nestedEnv, &model.Action{Inputs: map[string]model.Input{"shared": {Default: "inner-default"}}}, composite))
@@ -50,9 +59,11 @@ func TestCompositeActionParity(t *testing.T) {
})
t.Run("propagates pre failures", func(t *testing.T) {
setCloneExecutor(t, func(git.NewGitCloneExecutorInput) common.Executor { return common.NewErrorExecutor(assert.AnError) })
setFetch(t, func(_ctx context.Context, _input git.NewGitCloneExecutorInput) (string, string, error) {
return "", "", assert.AnError
})
rc := &RunContext{
Config: &Config{GitHubInstance: "github.com", ActionCacheDir: t.TempDir()},
Config: &Config{GitHubInstance: "github.com", ActionCacheDir: t.TempDir(), PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
JobContainer: &jobContainerMock{},
}
@@ -69,9 +80,9 @@ func TestAppendUniqueMasks(t *testing.T) {
want []string
}{
{
name: "appends new masks",
name: "appends each new mask once",
dst: []string{"a"},
src: []string{"b", "c"},
src: []string{"b", "c", "b", "a"},
want: []string{"a", "b", "c"},
},
{
@@ -80,12 +91,6 @@ func TestAppendUniqueMasks(t *testing.T) {
src: []string{"a", "b"},
want: []string{"a", "b"},
},
{
name: "deduplicates within src",
dst: []string{"a"},
src: []string{"b", "b", "a"},
want: []string{"a", "b"},
},
{
name: "empty src leaves dst unchanged",
dst: []string{"a"},
@@ -101,26 +106,6 @@ func TestAppendUniqueMasks(t *testing.T) {
}
}
// TestAppendUniqueMasksNoExponentialGrowth reproduces the exponential growth of
// the parent's Masks slice observed with nested/repeated composite actions. A
// composite RunContext is seeded with its parent's masks and the whole seeded
// slice was previously appended back into the parent, doubling its length on
// every composite action.
func TestAppendUniqueMasksNoExponentialGrowth(t *testing.T) {
parentMasks := []string{"secret"}
for range 20 {
// compositeRC.Masks starts as a copy of the parent's masks (it is
// seeded with parent.Masks in newCompositeRunContext).
compositeMasks := make([]string, len(parentMasks))
copy(compositeMasks, parentMasks)
parentMasks = appendUniqueMasks(parentMasks, compositeMasks)
}
assert.Equal(t, []string{"secret"}, parentMasks)
}
func TestCompositeStepIfReadsJobStatusExceptInItsOwnMainSteps(t *testing.T) {
failStep := func(rc *RunContext) {
rc.StepResults["failed"] = &model.StepResult{Conclusion: model.StepStatusFailure}

View File

@@ -8,6 +8,7 @@ import (
"context"
"io"
"io/fs"
"net/http"
"os"
"os/exec"
"path/filepath"
@@ -19,6 +20,8 @@ import (
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/model"
@@ -85,6 +88,19 @@ runs:
},
},
},
{
name: "readLowercaseDockerfile",
step: &model.Step{},
filename: "dockerfile",
fileContent: "FROM ubuntu:20.04",
expected: &model.Action{
Name: "(Synthetic)",
Runs: model.ActionRuns{
Using: "docker",
Image: "dockerfile",
},
},
},
{
name: "readWithArgs",
step: &model.Step{
@@ -95,6 +111,7 @@ runs:
expected: &model.Action{
Name: "(Synthetic)",
Inputs: map[string]model.Input{
"args": {},
"cwd": {
Description: "(Actual working directory)",
Required: false,
@@ -126,15 +143,15 @@ runs:
return strings.NewReader(tt.fileContent), closerFunc(func() { closed = true }), nil
}
writeFile := func(filename string, data []byte, perm fs.FileMode) error {
writeFile := func(filename string, _data []byte, perm fs.FileMode) error {
assert.Equal(t, "actionDir/actionPath/trampoline.js", filename)
assert.Equal(t, fs.FileMode(0o400), perm)
assert.Equal(t, fs.FileMode(0o644), perm)
return nil
}
action, err := readActionImpl(t.Context(), tt.step, "actionDir", "actionPath", readFile, writeFile)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
assert.Equal(t, tt.expected, action)
assert.Equal(t, tt.filename != "", closed)
@@ -144,91 +161,48 @@ runs:
func TestActionRunner(t *testing.T) {
t.Chdir(t.TempDir())
table := []struct {
for _, tt := range []struct {
name string
step actionStep
inputs map[string]model.Input
with map[string]string
state map[string]string
expectedEnv map[string]string
warning string
}{
{
name: "with-input",
step: &stepActionRemote{
Step: &model.Step{
Uses: "org/repo/path@ref",
},
RunContext: &RunContext{
Config: &Config{},
Run: &model.Run{
JobID: "job",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"job": {
Name: "job",
},
},
},
},
},
action: &model.Action{
Inputs: map[string]model.Input{
"key": {
Default: "default value",
},
},
Runs: model.ActionRuns{
Using: "node16",
},
},
env: map[string]string{},
},
name: "with-input",
inputs: map[string]model.Input{"key": {Default: "default value"}},
expectedEnv: map[string]string{"INPUT_KEY": "default value"},
},
{
name: "restore-saved-state",
step: &stepActionRemote{
Step: &model.Step{
ID: "step",
Uses: "org/repo/path@ref",
},
RunContext: &RunContext{
ActionPath: "path",
Config: &Config{},
Run: &model.Run{
JobID: "job",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"job": {
Name: "job",
},
},
},
},
CurrentStep: "post-step",
StepResults: map[string]*model.StepResult{
"step": {},
},
IntraActionState: map[string]map[string]string{
"step": {
"name": "state value",
},
},
},
action: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
},
},
env: map[string]string{},
},
name: "declared input ignores case",
inputs: map[string]model.Input{"key": {Default: "default value"}},
with: map[string]string{"KEY": "provided value"},
expectedEnv: map[string]string{"INPUT_KEY": "provided value"},
},
{
name: "unexpected inputs warn once across stages",
inputs: map[string]model.Input{"key": {}},
with: map[string]string{"KEY": "provided value", "UNKNOWN": "value", "other%0A\r\n": "value"},
warning: "##[warning]Unexpected input(s) 'UNKNOWN', 'other%250A%0D%0A', valid inputs are ['key']",
},
{
name: "no declared inputs",
with: map[string]string{"unknown": "value"},
warning: "##[warning]Unexpected input(s) 'unknown', valid inputs are ['']",
},
{
name: "restore-saved-state",
state: map[string]string{"name": "state value"},
expectedEnv: map[string]string{"STATE_name": "state value"},
},
}
for _, tt := range table {
} {
t.Run(tt.name, func(t *testing.T) {
ctx := t.Context()
hook := logtest.NewWithLevel(logging.LevelWarn)
ctx := common.WithLogger(t.Context(), logging.New(hook))
cm := &containerMock{}
cm.On("CopyDir", "/var/run/act/actions/dir/", "dir/", false, true).Return(func(ctx context.Context) error { return nil })
containerActionDir := "/var/run/act/actions/" + model.UsesHash("/org/repo/path@ref") + "/"
cm.On("CopyDir", containerActionDir, "dir/", false, true).Return(func(_ctx context.Context) error { return nil })
envMatcher := mock.MatchedBy(func(env map[string]string) bool {
for k, v := range tt.expectedEnv {
@@ -238,14 +212,37 @@ func TestActionRunner(t *testing.T) {
}
return true
})
cm.On("Exec", []string{"node", "--preserve-symlinks-main", containerActionDir + "path"}, envMatcher, "", "").Return(func(_ctx context.Context) error { return nil })
cm.On("Exec", nodeActionCommand(containerActionDir+"path/pre.js"), mock.Anything, "", "").Return(func(_ctx context.Context) error { return nil })
cm.On("Exec", nodeActionCommand(containerActionDir+"path/post.js"), envMatcher, "", "").Return(func(_ctx context.Context) error { return nil })
cm.On("Exec", []string{"node", "--preserve-symlinks-main", "/var/run/act/actions/dir/path"}, envMatcher, "", "").Return(func(ctx context.Context) error { return nil })
tt.step.getRunContext().JobContainer = cm
err := runActionImpl(tt.step, "dir", mustNewRemoteAction(t, "org/repo/path@ref"))(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
remoteAction := mustNewRemoteAction(t, "org/repo/path@ref")
step := &stepActionRemote{
Step: &model.Step{ID: "step", Uses: "org/repo/path@ref", With: tt.with},
RunContext: &RunContext{
Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
IntraActionState: map[string]map[string]string{"step": tt.state},
JobContainer: cm,
},
action: &model.Action{Inputs: tt.inputs, Runs: model.ActionRuns{Using: "node16", Pre: "pre.js", Post: "post.js"}},
actionDir: "dir",
env: (&model.Step{With: tt.with}).GetEnv(),
remoteAction: remoteAction,
}
require.NoError(t, runPreStep(step)(ctx))
assert.Empty(t, hook.AllRecords())
require.NoError(t, runActionImpl(step, "dir", remoteAction)(ctx))
require.NoError(t, runPostStep(step)(ctx))
if tt.warning == "" {
assert.Empty(t, hook.AllRecords())
} else {
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, tt.warning, hook.LastRecord().Message)
rawOutput, ok := hook.LastRecord().Attr(rawOutputField)
require.True(t, ok)
assert.Equal(t, true, rawOutput)
}
cm.AssertExpectations(t)
})
}
@@ -319,7 +316,6 @@ func TestNewStepContainerDoesNotUseDockerSecrets(t *testing.T) {
newStepContainer(ctx, step, "registry.example.com/action:tag", nil, nil, "")
// DOCKER_USERNAME/DOCKER_PASSWORD should not be injected as pull credentials for docker action containers.
assert.Empty(t, captured.Username)
assert.Empty(t, captured.Password)
assert.True(t, captured.AutoRemove)
@@ -433,7 +429,7 @@ func TestExecAsDockerHoldsCloneLockForRemoteUncached(t *testing.T) {
step := &stepActionRemote{
Step: &model.Step{ID: "1", Uses: "remote/action@v1", With: map[string]string{}},
RunContext: &RunContext{
Config: &Config{},
Config: &Config{Env: testCheckoutEnv()},
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{
@@ -471,19 +467,76 @@ func TestExecAsDockerHoldsCloneLockForRemoteUncached(t *testing.T) {
cancel()
releaseOnce()
<-done
assert.ErrorIs(t, <-done, context.Canceled)
}
func TestExecAsDockerImageChecks(t *testing.T) {
originalExists, originalBuild, originalContainer := ContainerImageExistsLocally, ContainerNewDockerBuildExecutor, ContainerNewContainer
t.Cleanup(func() {
ContainerImageExistsLocally, ContainerNewDockerBuildExecutor, ContainerNewContainer = originalExists, originalBuild, originalContainer
})
ContainerNewContainer = func(*container.NewContainerInput) container.ExecutionsEnvironment { return fakeContainer{} }
for _, testcase := range []struct {
name, platform string
exists, matches, rebuild bool
wantBuild, wantRemove bool
wantChecks []string
}{
{name: "missing image", platform: "linux/arm64", wantBuild: true, wantChecks: []string{"any"}},
{name: "unspecified platform", exists: true, matches: true, wantChecks: []string{"any"}},
{name: "any platform", platform: "any", exists: true, matches: true, wantChecks: []string{"any"}},
{name: "matching platform", platform: "linux/arm64", exists: true, matches: true, wantChecks: []string{"any", "linux/arm64"}},
{name: "different platform", platform: "linux/arm64", exists: true, wantBuild: true, wantRemove: true, wantChecks: []string{"any", "linux/arm64"}},
{name: "forced rebuild", exists: true, matches: true, rebuild: true, wantBuild: true, wantChecks: []string{"any"}},
} {
t.Run(testcase.name, func(t *testing.T) {
var checks []string
ContainerImageExistsLocally = func(_ctx context.Context, _image, platform string) (bool, error) {
checks = append(checks, platform)
return testcase.exists && (platform == "any" || testcase.matches), nil
}
var built, removed bool
ContainerNewDockerBuildExecutor = func(container.NewDockerBuildExecutorInput) common.Executor {
return func(context.Context) error { built = true; return nil }
}
fakeDockerDaemon(t, func(writer http.ResponseWriter, request *http.Request) {
if request.Method == http.MethodDelete {
removed = true
_, _ = io.WriteString(writer, `[]`)
} else {
_, _ = io.WriteString(writer, `{"Architecture":"aarch64"}`)
}
})
step := &stepActionRemote{
Step: &model.Step{ID: "action"},
RunContext: &RunContext{
Config: &Config{
PresetGitHubContext: &model.GithubContext{},
ContainerArchitecture: testcase.platform,
ForceRebuild: testcase.rebuild,
},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
JobContainer: &jobContainerMock{},
},
action: &model.Action{Runs: model.ActionRuns{Using: "docker", Image: "Dockerfile"}},
env: map[string]string{},
}
dir := t.TempDir()
require.NoError(t, execAsDocker(t.Context(), step, "action", dir, dir, false, stepStageMain))
assert.Equal(t, testcase.wantChecks, checks)
assert.Equal(t, testcase.wantBuild, built)
assert.Equal(t, testcase.wantRemove, removed)
})
}
}
func TestDockerActionImageTag(t *testing.T) {
// Remote actions already carry a unique, ref-scoped actionName (the uses
// hash), so the tag must be left untouched for backwards compatibility.
assert.Equal(t,
"act-abc123-dockeraction:latest",
dockerActionImageTag("owner/repo", "abc123", false),
)
// Local actions keep a human-readable, repository-namespaced prefix and gain a short hash suffix that makes the tag unique per (repository, actionName).
// See https://gitea.com/gitea/runner/issues/1039.
assert.Equal(t,
"act-owner-repo-baca2daaa2fe-dockeraction:latest",
dockerActionImageTag("owner/repo", "./", true),
@@ -493,16 +546,11 @@ func TestDockerActionImageTag(t *testing.T) {
dockerActionImageTag("owner/repo", "./sub", true),
)
// Sanitizing every non-alphanumeric character to "-" is lossy, so distinct inputs can collapse to the same readable prefix.
// The hash suffix must keep such cases apart, otherwise an image built for one repository is reused for another.
collisions := [][2]struct {
repoName string
actionName string
}{
// Two different repositories, both `uses: ./`: "a/b-c" and "a-b/c" both sanitize to "a-b-c".
{{"a/b-c", "./"}, {"a-b/c", "./"}},
// A repository's root action vs another repository's sub-path action:
// "owner/repo-a" + "./" and "owner/repo" + "./a" both sanitize to "owner-repo-a".
{{"owner/repo-a", "./"}, {"owner/repo", "./a"}},
}
for _, c := range collisions {
@@ -514,7 +562,6 @@ func TestDockerActionImageTag(t *testing.T) {
)
}
// Distinct local actions within the same repository keep distinct tags.
assert.NotEqual(t,
dockerActionImageTag("owner/repo", "./", true),
dockerActionImageTag("owner/repo", "./sub", true),
@@ -535,15 +582,20 @@ func TestDockerActionImageTag(t *testing.T) {
func TestExecAsDockerStageEntrypoint(t *testing.T) {
orig := ContainerNewContainer
defer func() { ContainerNewContainer = orig }()
fakeDockerDaemon(t, func(writer http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(writer, `{"Architecture":"x86_64"}`)
})
for _, tc := range []struct {
name string
stage stepStage
with map[string]string
inputs map[string]model.Input
runs model.ActionRuns
env map[string]string
wantCmd []string
wantEntrypoint []string
warning string
}{
{
name: "main stage prefers manifest values",
@@ -558,17 +610,36 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
wantEntrypoint: []string{"main.sh"},
},
{
name: "main stage uses caller fallbacks",
name: "main stage uses caller fallbacks in any case and warns about undeclared inputs",
stage: stepStageMain,
with: map[string]string{"args": "caller --flag", "entrypoint": "input.sh --verbose"},
with: map[string]string{"ARGS": "caller --flag", "entryPoint": "input.sh --verbose", "unknown": "value"},
inputs: map[string]model.Input{"message": {}},
runs: model.ActionRuns{Env: map[string]string{"ACTION_ONLY": "manifest"}},
wantCmd: []string{"caller", "--flag"},
wantEntrypoint: []string{"input.sh", "--verbose"},
warning: "##[warning]Unexpected input(s) 'unknown', valid inputs are ['entryPoint', 'args', 'message']",
},
{
name: "manifest args keep caller input quotes literal",
stage: stepStageMain,
with: map[string]string{"args": "it's literal"},
runs: model.ActionRuns{Args: []string{"${{ inputs.args }}"}},
env: map[string]string{"INPUT_ARGS": "it's literal"},
wantCmd: []string{"it's literal"},
},
{
name: "caller input skips its default expression",
stage: stepStageMain,
with: map[string]string{"MESSAGE": "provided"},
inputs: map[string]model.Input{"message": {Default: "${{ fromJSON('') }}"}},
runs: model.ActionRuns{Args: []string{"${{ inputs.message }}"}},
env: map[string]string{"INPUT_MESSAGE": "provided"},
wantCmd: []string{"provided"},
},
{
name: "explicit empty manifest args suppress caller args",
stage: stepStageMain,
with: map[string]string{"args": "caller"},
with: map[string]string{"args": "it's ignored"},
runs: model.ActionRuns{Args: []string{}},
wantCmd: []string{},
},
@@ -594,6 +665,8 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
},
} {
t.Run(tc.name, func(t *testing.T) {
hook := logtest.NewWithLevel(logging.LevelWarn)
ctx := common.WithLogger(t.Context(), logging.New(hook))
tc.runs.Using, tc.runs.Image = "docker", "docker://node:14"
cm := &containerMock{}
var input *container.NewContainerInput
@@ -605,12 +678,13 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
step := &stepActionRemote{
Step: &model.Step{ID: "1", Uses: "org/action@v1", With: tc.with},
RunContext: &RunContext{
Config: &Config{},
Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
JobContainer: cm,
},
action: &model.Action{Runs: tc.runs},
env: mergeMaps(tc.env),
action: &model.Action{Inputs: tc.inputs, Runs: tc.runs},
env: mergeMaps(tc.env),
remoteAction: mustNewRemoteAction(t, "org/action@v1"),
}
cm.On("Pull", false).Return(func(context.Context) error { return nil })
@@ -619,7 +693,17 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
cm.On("Start", true).Return(func(context.Context) error { return nil })
cm.On("Close").Return(func(context.Context) error { return nil })
require.NoError(t, execAsDocker(t.Context(), step, "action", t.TempDir(), t.TempDir(), false, tc.stage))
if tc.stage == stepStageMain {
require.NoError(t, runActionImpl(step, t.TempDir(), step.remoteAction)(ctx))
} else {
require.NoError(t, execAsDocker(ctx, step, "action", t.TempDir(), t.TempDir(), false, tc.stage))
}
if tc.warning == "" {
assert.Empty(t, hook.AllRecords())
} else {
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, tc.warning, hook.LastRecord().Message)
}
require.NotNil(t, input)
assert.Equal(t, tc.wantCmd, input.Cmd)
assert.Equal(t, tc.wantEntrypoint, input.Entrypoint)

View File

@@ -66,145 +66,97 @@ func TestCancelledJobStatusEnablesAlwaysAndCancelledSteps(t *testing.T) {
assert.Equal(t, "cancelled", rc.Run.Job().Result)
}
// TestMainStepsExecutorRunsAlwaysStepsAfterCancel verifies that newMainStepsExecutor does
// not abandon the remaining steps when the run is cancelled mid-pipeline. The later step
// still runs (so a main-stage always() step is reached), it runs under a fresh,
// non-cancelled context, and the job is marked cancelled. The interrupt error is still
// propagated so callers up the chain see the cancellation.
func TestMainStepsExecutorRunsAlwaysStepsAfterCancel(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
func TestMainStepsExecutorRunsAlwaysStepsAfterInterruption(t *testing.T) {
for name, wantError := range map[string]error{"cancel": context.Canceled, "timeout": context.DeadlineExceeded} {
t.Run(name, func(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
var ctx context.Context
var interrupt func()
if wantError == context.Canceled {
ctx, interrupt = context.WithCancel(t.Context())
defer interrupt()
} else {
deadline := newControllableDeadlineContext(t.Context())
ctx, interrupt = deadline, deadline.expire
}
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
var ran []string
var laterStepCtxErr error
err := newMainStepsExecutor(rc, []common.Executor{
func(_ctx context.Context) error {
ran = append(ran, "step1")
interrupt()
return nil
},
func(ctx context.Context) error {
ran = append(ran, "always-step")
laterStepCtxErr = ctx.Err()
return nil
},
})(ctx)
var ran []string
var laterStepCtxErr error
steps := []common.Executor{
func(_ context.Context) error {
ran = append(ran, "step1")
cancel() // server cancellation lands while step1 runs
return nil
},
func(c context.Context) error {
ran = append(ran, "always-step")
laterStepCtxErr = c.Err()
return nil
},
require.ErrorIs(t, err, wantError)
assert.Equal(t, []string{"step1", "always-step"}, ran)
require.NoError(t, laterStepCtxErr)
assert.Equal(t, wantError == context.Canceled, rc.jobCancelled)
setJobResult(t.Context(), rc, rc, true)
if wantError == context.DeadlineExceeded {
assert.True(t, rc.jobFailed)
assert.Equal(t, "failure", rc.getJobContext().Status)
assert.Equal(t, "failure", rc.Run.Job().Result)
} else {
assert.Equal(t, "cancelled", rc.Run.Job().Result)
}
})
}
err := newMainStepsExecutor(rc, steps)(ctx)
require.ErrorIs(t, err, context.Canceled, "interrupt error is propagated")
assert.Equal(t, []string{"step1", "always-step"}, ran, "the always() step still runs after cancel")
require.NoError(t, laterStepCtxErr, "remaining steps run under a fresh, non-cancelled context")
assert.True(t, rc.jobCancelled, "the job is marked cancelled")
}
// TestMainStepsExecutorMarksFailedOnTimeoutBetweenSteps guards the timeout path's symmetry with the cancel path.
// When the job deadline (timeout-minutes) lands in the gap between two steps, the job must be marked as failed (not cancelled),
// so always()/failure() cleanup steps run while default success() steps skip, and so the timed-out job is not reported as success.
func TestMainStepsExecutorMarksFailedOnTimeoutBetweenSteps(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
func TestStepsExecutorRunsMainStepsAfterPreInterruption(t *testing.T) {
for name, tt := range map[string]struct {
cancel bool
preErr error
wantError error
}{
"cancel": {cancel: true, wantError: context.Canceled},
"failure": {preErr: assert.AnError, wantError: assert.AnError},
} {
t.Run(name, func(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
ctx := t.Context()
var cancel context.CancelFunc
if tt.cancel {
ctx, cancel = context.WithCancel(ctx)
defer cancel()
}
ctx := newControllableDeadlineContext(t.Context())
var ran []string
var mainStepCtxErr error
err := newStepsExecutor(rc, []common.Executor{
func(_ctx context.Context) error {
ran = append(ran, "pre1")
if tt.cancel {
cancel()
}
return tt.preErr
},
}, []common.Executor{
func(ctx context.Context) error {
ran = append(ran, "always-step")
mainStepCtxErr = ctx.Err()
return nil
},
})(ctx)
var ran []string
var laterStepCtxErr error
steps := []common.Executor{
func(context.Context) error {
ran = append(ran, "step1")
ctx.expire()
return nil
},
func(c context.Context) error {
ran = append(ran, "always-step")
laterStepCtxErr = c.Err()
return nil
},
require.ErrorIs(t, err, tt.wantError)
assert.Equal(t, []string{"pre1", "always-step"}, ran)
require.NoError(t, mainStepCtxErr)
assert.Equal(t, tt.cancel, rc.jobCancelled)
})
}
err := newMainStepsExecutor(rc, steps)(ctx)
require.ErrorIs(t, err, context.DeadlineExceeded, "the timeout error is propagated")
assert.Equal(t, []string{"step1", "always-step"}, ran, "the always() step still runs after a timeout")
require.NoError(t, laterStepCtxErr, "remaining steps run under a fresh, non-expired context")
assert.True(t, rc.jobFailed, "a job timeout marks the job failed")
assert.False(t, rc.jobCancelled, "a timeout is not a cancellation")
// The status the real main-step `if` evaluation sees: "failure", so default success() steps skip while always()/failure() steps run.
assert.Equal(t, "failure", rc.getJobContext().Status)
}
// TestStepsExecutorRunsMainStepsAfterPreCancel verifies that a cancellation landing during the
// pre phase does not abandon the main steps: newStepsExecutor still runs the main-steps executor,
// so a main-stage always()/cancelled() step is reached (under a fresh, non-cancelled context),
// the job is marked cancelled, and the cancellation is propagated. Before the fix the `.Then(...)`
// short-circuit skipped the main steps entirely when a pre step was cancelled.
func TestStepsExecutorRunsMainStepsAfterPreCancel(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
var ran []string
var mainStepCtxErr error
preSteps := []common.Executor{
func(_ context.Context) error {
ran = append(ran, "pre1")
cancel() // server cancellation lands during the pre phase
return nil
},
}
steps := []common.Executor{
func(c context.Context) error {
ran = append(ran, "always-step")
mainStepCtxErr = c.Err()
return nil
},
}
err := newStepsExecutor(rc, preSteps, steps)(ctx)
require.ErrorIs(t, err, context.Canceled, "the cancellation is propagated")
assert.Equal(t, []string{"pre1", "always-step"}, ran, "the main always() step runs after a pre-phase cancel")
require.NoError(t, mainStepCtxErr, "the main step runs under a fresh, non-cancelled context")
assert.True(t, rc.jobCancelled, "the job is marked cancelled")
}
// TestStepsExecutorRunsMainStepsAfterPreFailure verifies that a failing pre step does not abandon
// the main steps: they still run (so a main-stage always()/failure() step is reached), and the
// pre-step error is propagated so the job is reported as failed. The main steps' own `if`
// evaluation is what skips success()-default steps, so running them here is safe.
func TestStepsExecutorRunsMainStepsAfterPreFailure(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
var ran []string
preSteps := []common.Executor{
func(_ context.Context) error {
ran = append(ran, "pre1")
return assert.AnError
},
}
steps := []common.Executor{
func(_ context.Context) error {
ran = append(ran, "always-step")
return nil
},
}
err := newStepsExecutor(rc, preSteps, steps)(t.Context())
require.ErrorIs(t, err, assert.AnError, "the pre-step error is propagated")
assert.Equal(t, []string{"pre1", "always-step"}, ran, "the main always() step runs after a pre-step failure")
assert.False(t, rc.jobCancelled, "a pre-step failure is not a cancellation")
}
// TestPreStepFailureAffectsMainStepIfStatus verifies the status path used by real

View File

@@ -5,61 +5,74 @@
package runner
import (
"cmp"
"context"
"fmt"
"regexp"
"maps"
"os"
"strings"
"unicode"
"gitea.com/gitea/runner/act/common"
)
var commandPatternGA *regexp.Regexp
var commandPatternADO *regexp.Regexp
func init() {
commandPatternGA = regexp.MustCompile("^::([^ ]+?)( (.+?))?::([^\r\n]*)[\r\n]*$")
// excluding ']' ends the command info at the first bracket, as GitHub does
commandPatternADO = regexp.MustCompile("^##\\[([^ \\]]+)( ([^\\]]*))?]([^\r\n]*)[\r\n]*$")
}
func tryParseRawActionCommand(line string) (command string, kvPairs map[string]string, arg string, ok bool) {
command, kvPairs, arg, _, ok = tryParseActionCommand(line)
return command, kvPairs, arg, ok
}
func tryParseActionCommand(line string) (command string, kvPairs map[string]string, arg string, legacy, ok bool) {
if m := commandPatternGA.FindStringSubmatch(line); m != nil {
command = m[1]
kvPairs = parseKeyValuePairs(m[3], ",")
arg = m[4]
ok = true
} else if m := commandPatternADO.FindStringSubmatch(line); m != nil {
command = m[1]
kvPairs = parseKeyValuePairs(m[3], ";")
arg = m[4]
legacy = true
ok = true
// parseActionCommand parses like GitHub's runner, but rejects a bare \r GitHub would split at and keeps a value's leading '=' so masks match.
func parseActionCommand(line, resumeCommand string) (command string, properties map[string]string, data string, ok bool) {
line = strings.TrimRight(line, "\r\n")
if strings.ContainsAny(line, "\r\n") {
return "", nil, "", false
}
return command, kvPairs, arg, legacy, ok
trimmed := strings.TrimLeftFunc(line, unicode.IsSpace)
for _, legacy := range []bool{false, true} {
header, found := strings.CutPrefix(trimmed, "::")
separator, propertySeparator, unescapeProperty, unescapeData := "::", ",", commandPropertyUnescaper.Replace, UnescapeCommandData
if legacy {
_, header, found = strings.Cut(line, "##[")
separator, propertySeparator, unescapeProperty, unescapeData = "]", ";", UnescapeLegacyCommand, UnescapeLegacyCommand
}
if !found {
continue
}
header, data, found := strings.Cut(header, separator)
if !found {
continue
}
command, propertyList, _ := strings.Cut(header, " ")
command = strings.ToLower(command)
switch command {
case "stop-commands", "set-env", "set-output", "save-state", "add-path", "add-mask", "add-matcher",
"remove-matcher", "warning", "error", "notice", "debug", "group", "endgroup", "echo":
default:
if resumeCommand == "" || !strings.EqualFold(command, resumeCommand) {
continue
}
}
if !legacy {
propertyList = strings.TrimSpace(propertyList)
}
properties := map[string]string{}
for property := range strings.SplitSeq(propertyList, propertySeparator) {
if key, value, _ := strings.Cut(strings.TrimLeft(property, "="), "="); value != "" {
properties[strings.ToLower(key)] = unescapeProperty(value)
}
}
return command, properties, unescapeData(data), true
}
return "", nil, "", false
}
// commandHandler reports whether a line is logged, keeping per-handler command state as GitHub does.
func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
logger := common.Logger(ctx)
rawLogger := common.Logger(ctx).With(rawOutputField, true)
debug := rc.topLevelRunContext().Config.RunnerDebug()
echo := debug
resumeCommand := ""
return func(line string) bool {
command, kvPairs, arg, legacy, ok := tryParseActionCommand(line)
command, kvPairs, arg, ok := parseActionCommand(line, resumeCommand)
if !ok {
return true
}
command = strings.ToLower(command)
if resumeCommand != "" {
// There should not be any emojis in the log output for Gitea.
// Return true (not false) so the line still reaches the raw_output
// log handler; otherwise everything between ::stop-commands:: and
// its end token is silently dropped from the step log.
logger.Infof("%s", line)
// Resumed here rather than from the switch, because the end token is arbitrary
// and a token naming a real command would otherwise never resume.
if strings.EqualFold(command, resumeCommand) {
@@ -67,15 +80,9 @@ func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
}
return true
}
if legacy {
arg = UnescapeLegacyCommand(arg)
kvPairs = unescapeKvPairs(kvPairs, UnescapeLegacyCommand)
} else {
arg = UnescapeCommandData(arg)
kvPairs = unescapeKvPairs(kvPairs, unescapeCommandProperty)
}
echoed := echo
if (command == "set-env" || command == "add-path") && rc.refuseUnsecureCommand(ctx, command) {
return true
return echoed
}
switch command {
case "set-env":
@@ -83,26 +90,67 @@ func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
case "set-output":
rc.setOutput(ctx, kvPairs, arg)
case "add-path":
rc.addPath(ctx, arg)
rc.addPath(arg)
case "save-state":
rc.saveState(ctx, kvPairs, arg)
case "echo":
switch strings.ToLower(strings.TrimSpace(arg)) {
case "on":
echo = true
case "off":
echo = false
}
case "add-mask":
rc.AddMask(arg)
logger.Infof("%s", "***")
// The raw line is still forwarded, carrying the secret: that is how the reporter
// learns the mask, and it drops the row rather than writing it out.
rc.AddMask(ctx, arg)
if echoed {
rawLogger.Infof("::add-mask::***")
}
return false
case "stop-commands":
resumeCommand = arg
logger.Infof("%s", line)
case "save-state":
logger.Infof("%s", line)
rc.saveState(ctx, kvPairs, arg)
default:
// ::debug::, ::error::, ::warning::, ::add-matcher:: and anything unrecognised are
// passed through for the reporter and Gitea's web UI to render.
logger.Infof("%s", line)
if len(arg) > 6 { // as on GitHub, so the log does not give away the resume token
rc.AddMask(ctx, arg)
}
return true
case "debug":
return debug
case "notice", "warning", "error":
rawLogger.Infof("%s", formatAnnotation(command, kvPairs, arg))
return false
case "group", "endgroup":
return true
}
return echoed
}
}
// return true to let gitea's logger handle these special outputs also
return true
// formatAnnotation folds the location and title into the message, as Gitea's web UI drops command properties.
func formatAnnotation(level string, properties map[string]string, message string) string {
prefix := properties["file"]
if prefix != "" {
if line := properties["line"]; line != "" {
prefix += ":" + line
if col := cmp.Or(properties["col"], properties["column"]); col != "" {
prefix += ":" + col
}
}
prefix += ": "
}
if title := properties["title"]; title != "" {
prefix += title + ": "
}
return "::" + level + "::" + EscapeCommandData(prefix+message)
}
// serviceCommandHandler applies only ::add-mask::, so a service cannot set the job's env, outputs or state.
func (rc *RunContext) serviceCommandHandler(ctx context.Context) common.LineHandler {
return func(line string) bool {
command, _, arg, ok := parseActionCommand(line, "")
if !ok || command != "add-mask" {
return true
}
rc.AddMask(ctx, arg)
return false
}
}
@@ -112,9 +160,18 @@ const allowUnsecureCommandsVar = "ACTIONS_ALLOW_UNSECURE_COMMANDS"
// not run, recording the error that fails the step. GitHub disabled both because a step that
// echoes untrusted content can use them to set NODE_OPTIONS or PATH for every later step.
func (rc *RunContext) refuseUnsecureCommand(ctx context.Context, command string) bool {
if rc.allowUnsecureCommandsOptIn() {
if isTruthyEnv(os.Getenv(allowUnsecureCommandsVar)) { // the runner's own opt-in wins, as on GitHub
return false
}
caseInsensitive := rc.JobContainer != nil && rc.JobContainer.IsEnvironmentCaseInsensitive()
for _, env := range []map[string]string{rc.currentStepEnv(), rc.Env, rc.GlobalEnv} {
if value, ok := lookupFold(env, allowUnsecureCommandsVar, caseInsensitive); ok {
if isTruthyEnv(value) {
return false
}
break
}
}
// The step executor logs the failure itself, so keep this line's wording distinct.
common.Logger(ctx).With(rawOutputField, true).Errorf("##[error]%s", EscapeCommandData(fmt.Sprintf(
@@ -130,14 +187,6 @@ func (rc *RunContext) refuseUnsecureCommand(ctx context.Context, command string)
return true
}
// allowUnsecureCommandsOptIn reports whether the workflow itself asked for the deprecated
// commands, from any env scope, as it can on GitHub.
func (rc *RunContext) allowUnsecureCommandsOptIn() bool {
return isTruthyEnv(rc.currentStepEnv()[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.Env[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.GlobalEnv[allowUnsecureCommandsVar])
}
// isTruthyEnv mirrors GitHub's bool.TryParse: only "true", in any casing.
func isTruthyEnv(v string) bool {
return strings.EqualFold(strings.TrimSpace(v), "true")
@@ -152,6 +201,18 @@ func (rc *RunContext) takeUnsecureCommandError() error {
return err
}
// runWithCommands runs executor judging its workflow commands by env, failing it on a refused one.
func (rc *RunContext) runWithCommands(ctx context.Context, env map[string]string, executor common.Executor) error {
rc.setCurrentStepEnv(maps.Clone(env)) // the executor keeps writing to env while the command handler reads it
defer rc.setCurrentStepEnv(nil)
_ = rc.takeUnsecureCommandError() // a refusal from before belongs to no step
err := executor(ctx)
if refusal := rc.takeUnsecureCommandError(); err == nil {
return refusal
}
return err
}
func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg string, fromCommand bool) {
name := kvPairs["name"]
if strings.EqualFold(name, "NODE_OPTIONS") {
@@ -162,7 +223,6 @@ func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg
common.Logger(ctx).With(rawOutputField, true).Errorf("##[error]%s", EscapeCommandData(message))
return
}
common.Logger(ctx).Infof("::set-env:: %s=%s", name, arg)
if rc.Env == nil {
rc.Env = make(map[string]string)
}
@@ -185,22 +245,18 @@ func (rc *RunContext) setEnvFile(ctx context.Context, kvPairs map[string]string,
}
func (rc *RunContext) setOutput(ctx context.Context, kvPairs map[string]string, arg string) {
logger := common.Logger(ctx)
stepID := rc.CurrentStep
outputName := kvPairs["name"]
result, ok := rc.StepResults[stepID]
if !ok {
logger.Infof("No outputs registered for step '%s'", stepID)
common.Logger(ctx).Infof("No outputs registered for step '%s'", stepID)
return
}
logger.Infof("::set-output:: %s=%s", outputName, arg)
result.Outputs[outputName] = arg
result.Outputs[kvPairs["name"]] = arg
}
func (rc *RunContext) addPath(ctx context.Context, arg string) {
common.Logger(ctx).Infof("::add-path:: %s", arg)
func (rc *RunContext) addPath(arg string) {
extraPath := []string{arg}
for _, v := range rc.ExtraPath {
if v != arg {
@@ -210,18 +266,6 @@ func (rc *RunContext) addPath(ctx context.Context, arg string) {
rc.ExtraPath = extraPath
}
func parseKeyValuePairs(kvPairs, separator string) map[string]string {
rtn := make(map[string]string)
kvPairList := strings.SplitSeq(kvPairs, separator)
for kvPair := range kvPairList {
kv := strings.SplitN(kvPair, "=", 2)
if len(kv) == 2 {
rtn[kv[0]] = kv[1]
}
}
return rtn
}
// A Replacer never rescans what it wrote, so "%250A" stays a literal "%0A".
var (
commandDataEscaper = strings.NewReplacer("%", "%25", "\r", "%0D", "\n", "%0A")
@@ -240,22 +284,11 @@ func UnescapeCommandData(arg string) string {
return commandDataUnescaper.Replace(arg)
}
func unescapeCommandProperty(arg string) string {
return commandPropertyUnescaper.Replace(arg)
}
// UnescapeLegacyCommand decodes a "##[cmd]" line, which also spells ";" and "]" escaped.
func UnescapeLegacyCommand(arg string) string {
return legacyCommandUnescaper.Replace(arg)
}
func unescapeKvPairs(kvPairs map[string]string, unescape func(string) string) map[string]string {
for k, v := range kvPairs {
kvPairs[k] = unescape(v)
}
return kvPairs
}
func (rc *RunContext) saveState(_ context.Context, kvPairs map[string]string, arg string) {
stepID := rc.CurrentStep
if stepID != "" {

View File

@@ -5,12 +5,13 @@
package runner
import (
"bytes"
"io"
"os"
"strconv"
"strings"
"testing"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.dev/actionslib/pkg/model"
@@ -18,9 +19,55 @@ import (
"github.com/stretchr/testify/require"
)
func TestParseActionCommand(t *testing.T) {
for _, test := range []struct {
name, line, resumeCommand, command string
properties map[string]string
data string
ok bool
}{
{"plain", "plain output", "", "", nil, "", false},
{"unknown v2", "::unknown::v", "", "", nil, "", false},
{"missing v2 terminator", "::set-output name=x", "", "", nil, "", false},
{"empty command", "::::", "", "", nil, "", false},
{"text before v2", "debug ::set-output name=x::v", "", "", nil, "", false},
{"missing legacy terminator", "##[debug", "", "", nil, "", false},
{"tab is not a name separator", "::set-output\tname=x::v", "", "", nil, "", false},
{"v2 leading unicode whitespace and line endings", " \t\u2003::set-output name=tab::v\r\n\r\n", "", "set-output", map[string]string{"name": "tab"}, "v", true},
{"legacy prefix and casing", "prefix ##[SET-OUTPUT NAME=mid;]v\r\n", "", "set-output", map[string]string{"name": "mid"}, "v", true},
{"only first legacy prefix", "##[unknown]##[set-output name=x]v", "", "", nil, "", false},
{"v2 delimiter boundaries", "::set-output name=a:b,other=c:d:: a::b ", "", "set-output", map[string]string{"name": "a:b", "other": "c:d"}, " a::b ", true},
{"legacy first closing bracket", "##[set-output name=x]v]tail", "", "set-output", map[string]string{"name": "x"}, "v]tail", true},
{"v2 semicolons are literal", "::set-output name=x;y=z::v", "", "set-output", map[string]string{"name": "x;y=z"}, "v", true},
{"legacy commas are literal", "##[set-output name=x,y=z]v", "", "set-output", map[string]string{"name": "x,y=z"}, "v", true},
{"values keep their leading equals", "::set-output name==x=y,=key=v,empty=,=x,spaced= =x::v", "", "set-output", map[string]string{"name": "=x=y", "key": "v", "spaced": " =x"}, "v", true},
{"v2 trims whole list only", "::set-output \tfile=x, line=2,name = value \t::v", "", "set-output", map[string]string{"file": "x", " line": "2", "name ": " value"}, "v", true},
{"legacy preserves list whitespace", "##[set-output name=q ]v", "", "set-output", map[string]string{" name": "q "}, "v", true},
{"case insensitive duplicate keys", "::SET-OUTPUT NAME=first,Name=last::v", "", "set-output", map[string]string{"name": "last"}, "v", true},
{"keys stay escaped", "::set-output na%3Ame=x::v", "", "set-output", map[string]string{"na%3ame": "x"}, "v", true},
{"v2 decode table", "::set-output name=%0D%0A%3A%2C%25%253A%252C%250A%2525%3a%2c%0d%0a%3B%5D::%0D%0A%25%3A%2C%3B%5D%250A", "", "set-output", map[string]string{"name": "\r\n:,%%3A%2C%0A%25%3a%2c%0d%0a%3B%5D"}, "\r\n%%3A%2C%3B%5D%0A", true},
{"legacy decode table", "##[set-output name=%3B%0D%0A%5D%25%253B%255D%250A%2525%3b%5d%0d%0a%3A%2C]v%5Dw%3Bx", "", "set-output", map[string]string{"name": ";\r\n]%%3B%5D%0A%25%3b%5d%0d%0a%3A%2C"}, "v]w;x", true},
{"unknown v2 falls back to legacy", "::unknown::prefix ##[set-output name=x]v", "", "set-output", map[string]string{"name": "x"}, "v", true},
{"unterminated v2 falls back to legacy", "::debug ##[set-output name=x]v", "", "set-output", map[string]string{"name": "x"}, "v", true},
{"v2 wins over legacy", "::debug::##[set-output name=x]v", "", "debug", map[string]string{}, "##[set-output name=x]v", true},
{"internal line breaks are not commands", "::debug::hidden\rplain\r\n", "", "", nil, "", false},
{"v2 resume token", "::ReSuMe-ToKeN::", "RESUME-TOKEN", "resume-token", map[string]string{}, "", true},
{"legacy resume token", "prefix ##[RESUME-TOKEN]v", "resume-token", "resume-token", map[string]string{}, "v", true},
{"different resume token", "::other-token::", "resume-token", "", nil, "", false},
} {
t.Run(test.name, func(t *testing.T) {
command, properties, data, ok := parseActionCommand(test.line, test.resumeCommand)
assert.Equal(t, test.command, command)
assert.Equal(t, test.properties, properties)
assert.Equal(t, test.data, data)
assert.Equal(t, test.ok, ok)
})
}
}
// unsecureRC opts into ::set-env:: and ::add-path::, which are refused without it.
func unsecureRC() *RunContext {
return &RunContext{Env: map[string]string{allowUnsecureCommandsVar: "true"}}
return &RunContext{Config: &Config{}, Env: map[string]string{allowUnsecureCommandsVar: "true"}}
}
func TestSetEnv(t *testing.T) {
@@ -37,212 +84,37 @@ func TestSetEnv(t *testing.T) {
a.NotContains(rc.Env, "NODE_OPTIONS")
a.NotContains(rc.Env, "node_options")
entries := hook.AllRecords()
require.Len(t, entries, 3)
a.Equal("##[error]Can't update NODE_OPTIONS environment variable using ::set-env:: command.", entries[1].Message)
a.Equal("##[error]Can't store NODE_OPTIONS output parameter using '$GITHUB_ENV' command.", entries[2].Message)
}
func TestStopCommandsKeepsSuppressedLinesInLog(t *testing.T) {
a := assert.New(t)
ctx := t.Context()
rc := unsecureRC()
handler := rc.commandHandler(ctx)
// Stop command processing until the matching end token is seen.
a.True(handler("::stop-commands::my-end-token\n"))
// A command-shaped line while stopped must not be executed (env unchanged),
// but must still return true so it reaches the raw_output log handler and is
// not dropped from the step log.
a.True(handler("::set-env name=x::valz\n"))
a.NotContains(rc.Env, "x")
// The matching end token resumes command processing.
a.True(handler("::my-end-token::\n"))
// Commands are processed again after resuming.
a.True(handler("::set-env name=y::valy\n"))
a.Equal("valy", rc.Env["y"])
require.Len(t, entries, 2)
a.Equal("##[error]Can't update NODE_OPTIONS environment variable using ::set-env:: command.", entries[0].Message)
a.Equal("##[error]Can't store NODE_OPTIONS output parameter using '$GITHUB_ENV' command.", entries[1].Message)
}
func TestSetOutput(t *testing.T) {
a := assert.New(t)
ctx := t.Context()
rc := new(RunContext)
rc.StepResults = make(map[string]*model.StepResult)
handler := rc.commandHandler(ctx)
rc := &RunContext{Config: &Config{}, CurrentStep: "my-step", StepResults: map[string]*model.StepResult{"my-step": {Outputs: map[string]string{}}}}
handler := rc.commandHandler(t.Context())
rc.CurrentStep = "my-step"
rc.StepResults[rc.CurrentStep] = &model.StepResult{
Outputs: make(map[string]string),
}
handler("::set-output name=x::valz\n")
a.Equal("valz", rc.StepResults["my-step"].Outputs["x"])
handler("::set-output name=x::percent2%25\n")
a.Equal("percent2%", rc.StepResults["my-step"].Outputs["x"])
handler("::set-output name=x::percent2%25%0Atest\n")
a.Equal("percent2%\ntest", rc.StepResults["my-step"].Outputs["x"])
handler("::set-output name=x::percent2%25%0Atest another3%25test\n")
a.Equal("percent2%\ntest another3%test", rc.StepResults["my-step"].Outputs["x"])
handler("::set-output name=x%3A::percent2%25%0Atest\n")
a.Equal("percent2%\ntest", rc.StepResults["my-step"].Outputs["x:"])
handler("::set-output name=x%3A%2C%0A%25%0D%3A::percent2%25%0Atest\n")
a.Equal("percent2%\ntest", rc.StepResults["my-step"].Outputs["x:,\n%\r:"])
handler("::set-output name=symbol::std::vector")
a.Equal("std::vector", rc.StepResults["my-step"].Outputs["symbol"])
handler("::set-output name=a=b::value\n")
a.Equal("value", rc.StepResults["my-step"].Outputs["a=b"])
handler("##[set-output name=legacy%3B%5D]value%3B%5D")
a.Equal("value;]", rc.StepResults["my-step"].Outputs["legacy;]"])
handler("##[set-output name=bracket]value]tail")
a.Equal("value]tail", rc.StepResults["my-step"].Outputs["bracket"])
handler("::set-output name=modern%3B%5D::value%3B%5D\n")
a.Equal("value%3B%5D", rc.StepResults["my-step"].Outputs["modern%3B%5D"])
handler("::set-output NAME=modern%3A::v%0D%0A%25%3B%5D%250A\n")
handler("##[set-output name=legacy%3B%5D]v%0D%0A%25%3B%5D%250A\n")
assert.Equal(t, map[string]string{"modern:": "v\r\n%%3B%5D%0A", "legacy;]": "v\r\n%;]%0A"}, rc.StepResults["my-step"].Outputs)
}
func TestAddpath(t *testing.T) {
a := assert.New(t)
ctx := t.Context()
rc := unsecureRC()
handler := rc.commandHandler(ctx)
handler("::add-path::/zoo\n")
a.Equal("/zoo", rc.ExtraPath[0])
handler("::add-path::/boo\n")
a.Equal("/boo", rc.ExtraPath[0])
}
func TestStopCommands(t *testing.T) {
logger, hook := logtest.NewLogger()
a := assert.New(t)
ctx := common.WithLogger(t.Context(), logger)
rc := unsecureRC()
handler := rc.commandHandler(ctx)
handler("::set-env name=x::valz\n")
a.Equal("valz", rc.Env["x"])
handler("::stop-commands::MY-END-TOKEN\n")
handler("::set-env name=x::abcd\n")
a.Equal("valz", rc.Env["x"])
handler("::my-end-token::\n")
handler("::set-env name=x::abcd\n")
a.Equal("abcd", rc.Env["x"])
messages := make([]string, 0)
for _, entry := range hook.AllRecords() {
messages = append(messages, entry.Message)
for _, prefix := range []string{"::add-path::", "##[add-path]"} {
t.Run(prefix, func(t *testing.T) {
rc := unsecureRC()
handler := rc.commandHandler(t.Context())
handler(prefix + "/zoo\n")
handler(prefix + "/boo\n")
assert.Equal(t, []string{"/boo", "/zoo"}, rc.ExtraPath)
handler(prefix + "/zoo\n")
assert.Equal(t, []string{"/zoo", "/boo"}, rc.ExtraPath)
})
}
a.Contains(messages, "::set-env name=x::abcd\n")
}
// The end token is arbitrary, so one that happens to name a real command must still resume
// rather than being swallowed by that command's case.
func TestStopCommandsResumesOnCommandNamedToken(t *testing.T) {
a := assert.New(t)
rc := unsecureRC()
handler := rc.commandHandler(t.Context())
handler("::stop-commands::add-mask\n")
handler("::set-env name=x::suppressed\n")
a.NotContains(rc.Env, "x")
handler("::add-mask::\n")
handler("::set-env name=x::resumed\n")
a.Equal("resumed", rc.Env["x"])
}
func TestAddpathADO(t *testing.T) {
a := assert.New(t)
ctx := t.Context()
rc := unsecureRC()
handler := rc.commandHandler(ctx)
handler("##[add-path]/zoo\n")
a.Equal("/zoo", rc.ExtraPath[0])
handler("##[add-path]/boo\n")
a.Equal("/boo", rc.ExtraPath[0])
}
func TestAddmask(t *testing.T) {
logger, hook := logtest.NewLogger()
a := assert.New(t)
ctx := t.Context()
loggerCtx := common.WithLogger(ctx, logger)
rc := new(RunContext)
handler := rc.commandHandler(loggerCtx)
handler("::ADD-MASK::my::secret")
a.Equal("***", hook.LastRecord().Message)
a.Equal([]string{"my::secret"}, rc.Masks)
}
// based on https://stackoverflow.com/a/10476304
func captureOutput(t *testing.T, f func()) string {
old := os.Stdout
r, w, _ := os.Pipe()
os.Stdout = w
f()
outC := make(chan string)
go func() {
var buf bytes.Buffer
_, err := io.Copy(&buf, r)
if err != nil {
a := assert.New(t)
a.Fail("io.Copy failed")
}
outC <- buf.String()
}()
w.Close()
os.Stdout = old
out := <-outC
return out
}
func TestAddmaskUsemask(t *testing.T) {
rc := new(RunContext)
rc.StepResults = make(map[string]*model.StepResult)
rc.CurrentStep = "my-step"
rc.StepResults[rc.CurrentStep] = &model.StepResult{
Outputs: make(map[string]string),
}
a := assert.New(t)
config := &Config{
Secrets: map[string]string{},
InsecureSecrets: false,
}
re := captureOutput(t, func() {
ctx := t.Context()
ctx = WithJobLogger(ctx, "0", "testjob", config, &rc.Masks, map[string]any{})
handler := rc.commandHandler(ctx)
handler("::add-mask::secret\n")
handler("::set-output:: token=secret\n")
})
a.Equal("[testjob] ***\n[testjob] ::set-output:: = token=***\n", re)
}
func TestSaveState(t *testing.T) {
rc := &RunContext{
Config: &Config{},
CurrentStep: "step",
StepResults: map[string]*model.StepResult{},
}
@@ -264,31 +136,37 @@ func TestEscapeCommandData(t *testing.T) {
func TestUnsecureCommands(t *testing.T) {
tests := []struct {
name string
jobEnv map[string]string
stepEnv map[string]string
optedIn bool
name string
jobEnv map[string]string
globalEnv map[string]string
stepEnv map[string]string
processEnv string
caseInsensitive bool
optedIn bool
}{
{name: "refused with no opt-in"},
// GitHub reads the opt-in with bool.TryParse, so "1" is not one.
{name: "refused for a value bool.TryParse rejects", jobEnv: map[string]string{allowUnsecureCommandsVar: "1"}},
{name: "opted in through the step environment", stepEnv: map[string]string{allowUnsecureCommandsVar: "true"}, optedIn: true},
{name: "opted in through the job environment", jobEnv: map[string]string{allowUnsecureCommandsVar: "TRUE"}, optedIn: true},
{name: "step opt-out overrides the job and global environment", jobEnv: map[string]string{allowUnsecureCommandsVar: "true"}, globalEnv: map[string]string{allowUnsecureCommandsVar: "true"}, stepEnv: map[string]string{allowUnsecureCommandsVar: "false"}},
{name: "the runner's own opt-in overrides a step opt-out", stepEnv: map[string]string{allowUnsecureCommandsVar: "false"}, processEnv: "true", optedIn: true},
{name: "a differently cased opt-in counts where env names are case-insensitive", jobEnv: map[string]string{"Actions_Allow_Unsecure_Commands": "true"}, caseInsensitive: true, optedIn: true},
{name: "a differently cased opt-in is ignored where env names are case-sensitive", jobEnv: map[string]string{"Actions_Allow_Unsecure_Commands": "true"}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Setenv(allowUnsecureCommandsVar, tt.processEnv)
a := assert.New(t)
rc := &RunContext{Env: tt.jobEnv}
rc := &RunContext{Config: &Config{}, Env: tt.jobEnv, GlobalEnv: tt.globalEnv, JobContainer: fakeContainer{caseInsensitive: tt.caseInsensitive}}
rc.setCurrentStepEnv(tt.stepEnv)
handler := rc.commandHandler(t.Context())
handler("::set-env name=x::valz\n")
handler("::add-path::/opt/bin\n")
a.False(handler("::set-env name=x::valz\n"))
a.False(handler("::add-path::/opt/bin\n"))
if !tt.optedIn {
a.Empty(rc.Env["x"])
a.Empty(rc.ExtraPath)
// The refusal fails the step that produced it, once.
require.ErrorContains(t, rc.takeUnsecureCommandError(), "set-env")
a.NoError(rc.takeUnsecureCommandError())
return
@@ -299,3 +177,109 @@ func TestUnsecureCommands(t *testing.T) {
})
}
}
type maskRecorder struct {
logging.Handler
masks []string
}
func (m *maskRecorder) AddMask(mask string) { m.masks = append(m.masks, mask) }
func TestCommandLogWriter(t *testing.T) {
for _, tt := range []struct {
name string
debug bool
steps [][]string
log []string
masks []string
}{
{
name: "applied commands are hidden unless echoed",
steps: [][]string{{
"::set-env name=x::y", "::add-path::/bin", "::set-output name=a::b", "##[save-state name=a]b", "::add-matcher::m.json", "::remove-matcher owner=m::",
"::echo::On%0A",
"::set-env name=x::y", "::add-path::/bin", "::set-output name=a::b", "##[save-state name=a]b", "::add-matcher::m.json", "::remove-matcher owner=m::",
"::ADD-MASK::my::secret", "::echo::bogus", "::echo::off", "::add-mask ::empty-properties", "::unknown::v", "plain",
}},
log: []string{
"::set-env name=x::y", "::add-path::/bin", "::set-output name=a::b", "##[save-state name=a]b", "::add-matcher::m.json", "::remove-matcher owner=m::",
"::add-mask::***", "::echo::bogus", "::echo::off", "::unknown::v", "plain",
},
masks: []string{"my::secret", "empty-properties"},
},
{
name: "step debug shows debug lines and echoes commands until echo is off",
debug: true,
steps: [][]string{{"::debug::first%0Asecond", "::set-output name=a::b", "::echo::off", "::debug::again", "::set-output name=a::b"}},
log: []string{"::debug::first%0Asecond", "::set-output name=a::b", "::echo::off", "::debug::again"},
},
{
name: "debug lines are hidden without step debug and an invalid echo changes nothing",
steps: [][]string{{"::echo::bogus", "::set-output name=a::b", "::echo::on", "::debug::d"}},
},
{
name: "annotations fold their location once whether echoed or not",
steps: [][]string{{
"::echo::on",
"::error file=main.go,line=12,col=5,title=vet::undefined: x",
"::warning File=a%3Ab.go,Line=1,column=2,title=100%252C::50%25%0Anext",
"::warning file=x,line=1,col=2,column=3::m",
"::notice line=9,col=4,title=title::m",
"::error file=x,col=2::m",
"::error file=x,line=1::m",
"::notice::std::vector",
"##[error file=x%3By]m%5D",
}},
log: []string{
"::error::main.go:12:5: vet: undefined: x", "::warning::a:b.go:1:2: 100%252C: 50%25%0Anext", "::warning::x:1:2: m",
"::notice::title: m", "::error::x: m", "::error::x:1: m", "::notice::std::vector", "::error::x;y: m]",
},
},
{
name: "stop-commands keeps lines verbatim until resumed",
steps: [][]string{{"::stop-commands::resume-token", "::add-mask::kept", "::set-output name=a::b", "::RESUME-TOKEN::", "::add-mask::masked"}},
log: []string{"::stop-commands::resume-token", "::add-mask::kept", "::set-output name=a::b", "::RESUME-TOKEN::"},
masks: []string{"resume-token", "masked"},
},
{
name: "a short stop token naming a command stays unmasked and resumes in any case",
steps: [][]string{{"::stop-commands::NOTICE", "::notice::", "::add-mask::masked"}},
log: []string{"::stop-commands::NOTICE", "::notice::"},
masks: []string{"masked"},
},
{
name: "state does not carry over to the next step",
steps: [][]string{{"::echo::on", "::stop-commands::token77"}, {"::set-output name=a::b", "::add-mask::masked"}},
log: []string{"::stop-commands::token77"},
masks: []string{"token77", "masked"},
},
} {
t.Run(tt.name, func(t *testing.T) {
logger, hook := logtest.NewLogger()
recorder := &maskRecorder{}
ctx := common.WithLoggerHandler(common.WithLogger(t.Context(), logger), recorder)
rc := &RunContext{
Config: &Config{},
Parent: &RunContext{Config: &Config{Secrets: map[string]string{"ACTIONS_STEP_DEBUG": strconv.FormatBool(tt.debug)}}},
Env: map[string]string{allowUnsecureCommandsVar: "true"},
CurrentStep: "step",
StepResults: map[string]*model.StepResult{"step": {Outputs: map[string]string{}}},
}
for _, lines := range tt.steps {
writer := rc.commandLogWriter(ctx)
for _, line := range lines {
_, err := io.WriteString(writer, line+"\n")
require.NoError(t, err)
}
}
var log []string
for _, record := range hook.AllRecords() {
log = append(log, strings.TrimSuffix(record.Message, "\n"))
}
assert.Equal(t, tt.log, log)
assert.Equal(t, tt.masks, recorder.masks)
assert.Equal(t, tt.masks, rc.Masks)
})
}
}

View File

@@ -16,63 +16,59 @@ import (
var noopExecutor = func(context.Context) error { return nil }
type containerMock struct {
mock.Mock
container.Container
type linuxEnvironment struct {
container.LinuxContainerEnvironmentExtensions
}
func (linuxEnvironment) GetRunnerContext(context.Context) map[string]any {
return map[string]any{"os": "Linux", "arch": "X64", "temp": "/tmp", "tool_cache": container.DefaultToolCache}
}
type containerMock struct {
mock.Mock
container.Container
linuxEnvironment
}
func (cm *containerMock) Create(capAdd, capDrop []string) common.Executor {
args := cm.Called(capAdd, capDrop)
return args.Get(0).(func(context.Context) error)
return cm.Called(capAdd, capDrop).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Pull(forcePull bool) common.Executor {
args := cm.Called(forcePull)
return args.Get(0).(func(context.Context) error)
return cm.Called(forcePull).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Start(attach bool) common.Executor {
args := cm.Called(attach)
return args.Get(0).(func(context.Context) error)
return cm.Called(attach).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Remove() common.Executor {
args := cm.Called()
return args.Get(0).(func(context.Context) error)
return cm.Called().Get(0).(func(context.Context) error)
}
func (cm *containerMock) Close() common.Executor {
args := cm.Called()
return args.Get(0).(func(context.Context) error)
return cm.Called().Get(0).(func(context.Context) error)
}
func (cm *containerMock) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
args := cm.Called(srcPath, env)
return args.Get(0).(func(context.Context) error)
return cm.Called(srcPath, env).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Copy(destPath string, files ...*container.FileEntry) common.Executor {
args := cm.Called(destPath, files)
return args.Get(0).(func(context.Context) error)
return cm.Called(destPath, files).Get(0).(func(context.Context) error)
}
func (cm *containerMock) CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor {
args := cm.Called(destPath, srcPath, useGitIgnore, skipGitDir)
return args.Get(0).(func(context.Context) error)
return cm.Called(destPath, srcPath, useGitIgnore, skipGitDir).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Exec(command []string, env map[string]string, user, workdir string) common.Executor {
args := cm.Called(command, env, user, workdir)
return args.Get(0).(func(context.Context) error)
return cm.Called(command, env, user, workdir).Get(0).(func(context.Context) error)
}
func (cm *containerMock) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) {
args := cm.Called(ctx, srcPath)
err, hasErr := args.Get(1).(error)
if !hasErr {
err = nil
}
err, _ := args.Get(1).(error)
return args.Get(0).(io.ReadCloser), err
}

View File

@@ -7,6 +7,7 @@ package runner
import (
"bytes"
"context"
"errors"
"fmt"
"maps"
"path"
@@ -49,15 +50,15 @@ func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map
result := model.WorkflowCallResult{
Outputs: map[string]string{},
}
unlock := lockJob(job)
maps.Copy(result.Outputs, job.Outputs)
unlock()
workflowCallResult[jobName] = &result
}
}
}
ghc := rc.getGithubContext(ctx)
inputs := getEvaluatorInputs(rc, rc.actionInputs, ghc)
ee := &exprparser.EvaluationEnvironment{
Github: ghc,
Env: env,
@@ -65,13 +66,13 @@ func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map
Jobs: &workflowCallResult,
// todo: should be unavailable
// but required to interpolate/evaluate the step outputs on the job
Steps: rc.getStepsContext(),
Steps: rc.StepResults,
Secrets: getWorkflowSecrets(rc),
Vars: getWorkflowVars(ctx, rc),
Vars: rc.Config.Vars,
Strategy: strategy,
Matrix: rc.Matrix,
Needs: exprparser.NeedsContext(rc.Run),
Inputs: inputs,
Inputs: getEvaluatorInputs(rc, rc.actionInputs, ghc),
HashFiles: getHashFilesFunction(ctx, rc),
}
ee.Runner = rc.getRunnerContext(ctx)
@@ -103,9 +104,9 @@ func (rc *RunContext) newStepExpressionEvaluator(ctx context.Context, step step,
Github: step.getGithubContext(ctx),
Env: *step.getEnv(),
Job: jobContext,
Steps: rc.getStepsContext(),
Steps: rc.StepResults,
Secrets: getWorkflowSecrets(rc),
Vars: getWorkflowVars(ctx, rc),
Vars: rc.Config.Vars,
Strategy: exprparser.StrategyContext(rc.Run.Job().Strategy, rc.jobIndex, rc.jobTotal),
Matrix: rc.Matrix,
Needs: exprparser.NeedsContext(rc.Run),
@@ -125,9 +126,9 @@ func (rc *RunContext) newStepExpressionEvaluator(ctx context.Context, step step,
}
func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.Value) (any, error) {
hashFiles := func(v []reflect.Value) (any, error) {
return func(v []reflect.Value) (any, error) {
if rc.JobContainer != nil {
timeed, cancel := context.WithTimeout(ctx, time.Minute)
timeoutCtx, cancel := context.WithTimeout(ctx, 120*time.Second)
defer cancel()
name := "workflow/hashfiles/index.js"
hout := &bytes.Buffer{}
@@ -137,14 +138,12 @@ func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.
for i, p := range v {
s := p.String()
if i == 0 {
if strings.HasPrefix(s, "--") {
if strings.EqualFold(s, "--follow-symbolic-links") {
followSymlink = true
continue
}
return "", fmt.Errorf("invalid glob option %s, available option: '--follow-symbolic-links'", s)
if i == 0 && strings.HasPrefix(s, "--") {
if strings.EqualFold(s, "--follow-symbolic-links") {
followSymlink = true
continue
}
return "", fmt.Errorf("invalid glob option %s, available option: '--follow-symbolic-links'", s)
}
patterns = append(patterns, s)
}
@@ -156,31 +155,40 @@ func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.
}
stdout, stderr := rc.JobContainer.ReplaceLogWriter(hout, herr)
_ = rc.JobContainer.Copy(rc.JobContainer.GetActPath(), &container.FileEntry{
defer rc.JobContainer.ReplaceLogWriter(stdout, stderr)
err := rc.JobContainer.Copy(rc.JobContainer.GetActPath(), &container.FileEntry{
Name: name,
Mode: 0o644,
Body: hashfiles,
}).
Then(rc.JobContainer.Exec([]string{"node", path.Join(rc.JobContainer.GetActPath(), name)},
env, "", "")).
Finally(func(context.Context) error {
rc.JobContainer.ReplaceLogWriter(stdout, stderr)
return nil
})(timeed)
output := hout.String() + "\n" + herr.String()
guard := "__OUTPUT__"
outstart := strings.Index(output, guard)
if outstart != -1 {
outstart += len(guard)
outend := strings.Index(output[outstart:], guard)
if outend != -1 {
return output[outstart : outstart+outend], nil
}).Then(rc.JobContainer.Exec([]string{"node", path.Join(rc.JobContainer.GetActPath(), name)}, env, "", ""))(timeoutCtx)
quotedPatterns := expressionStringEscape(strings.Join(patterns, ", "))
switch {
case err == nil:
case ctx.Err() != nil:
return "", ctx.Err()
case timeoutCtx.Err() != nil || errors.Is(err, context.DeadlineExceeded):
return "", fmt.Errorf("hashFiles('%s') couldn't finish within 120 seconds", quotedPatterns)
case hashFilesNodeMissing(err):
common.Logger(ctx).With(rawOutputField, true).Warnf("##[warning]hashFiles could not run because node is unavailable in the job environment. Returning an empty string.")
return "", nil
default:
return "", fmt.Errorf("hashFiles('%s') failed. Fail to hash files under directory '%s'", quotedPatterns, rc.githubWorkspace())
}
for line := range strings.Lines(hout.String() + "\n" + herr.String()) {
if hash, ok := strings.CutPrefix(strings.TrimRight(line, "\r\n"), "__OUTPUT__"); ok {
if hash, ok = strings.CutSuffix(hash, "__OUTPUT__"); ok {
return hash, nil
}
}
}
}
return "", nil
}
return hashFiles
}
func hashFilesNodeMissing(err error) bool {
exitErr, ok := errors.AsType[container.ExitCodeError](err)
return ok && (exitErr == 126 || exitErr == 127) || err.Error() == "Cannot find: node in PATH"
}
type expressionEvaluator struct {
@@ -189,14 +197,15 @@ type expressionEvaluator struct {
type ExpressionEvaluator = expressionEvaluator
var addMaskExpression = regexp.MustCompile(`::add-mask::.*`)
func (ee expressionEvaluator) evaluate(ctx context.Context, in string, defaultStatusCheck exprparser.DefaultStatusCheck) (any, error) {
logger := common.Logger(ctx)
logger.Debugf("evaluating expression '%s'", in)
evaluated, err := ee.interpreter.Evaluate(in, defaultStatusCheck)
// evaluated is an any: %t renders everything but a bool as "%!t(string=...)"
printable := regexp.MustCompile(`::add-mask::.*`).ReplaceAllString(fmt.Sprintf("%v", evaluated), "::add-mask::***)")
logger.Debugf("expression '%s' evaluated to '%s'", in, printable)
logger.Debugf("expression '%s' evaluated to '%s'", in, addMaskExpression.ReplaceAllString(fmt.Sprintf("%v", evaluated), "::add-mask::***)"))
return evaluated, err
}
@@ -249,15 +258,13 @@ func getEvaluatorInputs(rc *RunContext, stepInputs map[string]any, ghc *model.Gi
if stepInputs != nil { // an action's own inputs, which workflow inputs must not shadow
return maps.Clone(stepInputs)
}
inputs := map[string]any{}
// only a called workflow's job itself takes the inputs Gitea resolved,
// composite actions copy its Config but have their own `inputs`
if rc.Config.WorkflowCallInputs != nil && rc.Parent == nil {
maps.Copy(inputs, rc.Config.WorkflowCallInputs)
return inputs
return maps.Clone(rc.Config.WorkflowCallInputs)
}
inputs := map[string]any{}
maps.Copy(inputs, rc.workflowCallInputs)
switch ghc.EventName {
@@ -327,7 +334,3 @@ func getWorkflowSecrets(rc *RunContext) map[string]string {
}
return rc.Config.Secrets
}
func getWorkflowVars(_ context.Context, rc *RunContext) map[string]string {
return rc.Config.Vars
}

View File

@@ -5,8 +5,20 @@
package runner
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"reflect"
"strings"
"sync"
"testing"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/exprparser"
"gitea.dev/actionslib/pkg/model"
@@ -21,10 +33,11 @@ func createRunContext(t *testing.T) *RunContext {
"os": {"Linux", "Windows"},
"foo": {"bar", "baz"},
})
assert.NoError(t, err)
require.NoError(t, err)
return &RunContext{
Config: &Config{
Env: testCheckoutEnv(),
Workdir: ".",
Secrets: map[string]string{
"CASE_INSENSITIVE_SECRET": "value",
@@ -84,70 +97,146 @@ func TestEvaluateRunContext(t *testing.T) {
ee := rc.NewExpressionEvaluator(t.Context())
tables := []struct {
in string
out any
errMesg string
in string
out any
}{
{" 1 ", 1.0, ""},
// {"1 + 3", "4", ""},
// {"(1 + 3) * -2", "-8", ""},
{"'my text'", "my text", ""},
{"contains('my text', 'te')", true, ""},
{"contains('my TEXT', 'te')", true, ""},
{"contains(fromJSON('[\"my text\"]'), 'te')", false, ""},
{"contains(fromJSON('[\"foo\",\"bar\"]'), 'bar')", true, ""},
{"startsWith('hello world', 'He')", true, ""},
{"endsWith('hello world', 'ld')", true, ""},
{"format('0:{0} 2:{2} 1:{1}', 'zero', 'one', 'two')", "0:zero 2:two 1:one", ""},
{"join(fromJSON('[\"hello\"]'),'octocat')", "hello", ""},
{"join(fromJSON('[\"hello\",\"mona\",\"the\"]'),'octocat')", "hellooctocatmonaoctocatthe", ""},
{"join('hello','mona')", "hello", ""},
{"toJSON(env)", "{\n \"ACT\": \"true\",\n \"ACT_SKIP_CHECKOUT\": \"true\",\n \"key\": \"value\"\n}", ""},
{"toJson(env)", "{\n \"ACT\": \"true\",\n \"ACT_SKIP_CHECKOUT\": \"true\",\n \"key\": \"value\"\n}", ""},
{"(fromJSON('{\"foo\":\"bar\"}')).foo", "bar", ""},
{"(fromJson('{\"foo\":\"bar\"}')).foo", "bar", ""},
{"(fromJson('[\"foo\",\"bar\"]'))[1]", "bar", ""},
// github does return an empty string for non-existent files
{"hashFiles('**/non-extant-files')", "", ""},
{"hashFiles('**/non-extant-files', '**/more-non-extant-files')", "", ""},
{"hashFiles('**/non.extant.files')", "", ""},
{"hashFiles('**/non''extant''files')", "", ""},
{"success()", true, ""},
{"failure()", false, ""},
{"always()", true, ""},
{"cancelled()", false, ""},
{"github.workflow", "test-workflow", ""},
{"github.actor", "nektos/act", ""},
{"github.run_id", "1", ""},
{"github.run_number", "1", ""},
{"job.status", "success", ""},
{"matrix.os", "Linux", ""},
{"matrix.foo", "bar", ""},
{"env.key", "value", ""},
{"secrets.CASE_INSENSITIVE_SECRET", "value", ""},
{"secrets.case_insensitive_secret", "value", ""},
{"vars.CASE_INSENSITIVE_VAR", "value", ""},
{"vars.case_insensitive_var", "value", ""},
{"format('{{0}}', 'test')", "{0}", ""},
{"format('{{{0}}}', 'test')", "{test}", ""},
{"format('}}')", "}", ""},
{"format('echo Hello {0} ${{Test}}', 'World')", "echo Hello World ${Test}", ""},
{"format('echo Hello {0} ${{Test}}', github.undefined_property)", "echo Hello ${Test}", ""},
{"format('echo Hello {0}{1} ${{Te{0}st}}', github.undefined_property, 'World')", "echo Hello World ${Test}", ""},
{"format('{0}', '{1}', 'World')", "{1}", ""},
{"format('{{{0}', '{1}', 'World')", "{{1}", ""},
{" 1 ", 1.0},
{"'my text'", "my text"},
{"contains('my text', 'te')", true},
{"contains('my TEXT', 'te')", true},
{"contains(fromJSON('[\"my text\"]'), 'te')", false},
{"contains(fromJSON('[\"foo\",\"bar\"]'), 'bar')", true},
{"startsWith('hello world', 'He')", true},
{"endsWith('hello world', 'ld')", true},
{"format('0:{0} 2:{2} 1:{1}', 'zero', 'one', 'two')", "0:zero 2:two 1:one"},
{"join(fromJSON('[\"hello\"]'),'octocat')", "hello"},
{"join(fromJSON('[\"hello\",\"mona\",\"the\"]'),'octocat')", "hellooctocatmonaoctocatthe"},
{"join('hello','mona')", "hello"},
{"toJSON(env)", "{\n \"ACT\": \"true\",\n \"ACT_SKIP_CHECKOUT\": \"true\",\n \"key\": \"value\"\n}"},
{"toJson(env)", "{\n \"ACT\": \"true\",\n \"ACT_SKIP_CHECKOUT\": \"true\",\n \"key\": \"value\"\n}"},
{"(fromJSON('{\"foo\":\"bar\"}')).foo", "bar"},
{"(fromJson('{\"foo\":\"bar\"}')).foo", "bar"},
{"(fromJson('[\"foo\",\"bar\"]'))[1]", "bar"},
{"hashFiles('**/non-extant-files')", ""},
{"hashFiles('**/non-extant-files', '**/more-non-extant-files')", ""},
{"hashFiles('**/non.extant.files')", ""},
{"hashFiles('**/non''extant''files')", ""},
{"success()", true},
{"failure()", false},
{"always()", true},
{"cancelled()", false},
{"github.workflow", "test-workflow"},
{"github.actor", "nektos/act"},
{"github.run_id", "1"},
{"github.run_number", "1"},
{"job.status", "success"},
{"matrix.os", "Linux"},
{"matrix.foo", "bar"},
{"env.key", "value"},
{"secrets.CASE_INSENSITIVE_SECRET", "value"},
{"secrets.case_insensitive_secret", "value"},
{"vars.CASE_INSENSITIVE_VAR", "value"},
{"vars.case_insensitive_var", "value"},
{"format('{{0}}', 'test')", "{0}"},
{"format('{{{0}}}', 'test')", "{test}"},
{"format('}}')", "}"},
{"format('echo Hello {0} ${{Test}}', 'World')", "echo Hello World ${Test}"},
{"format('echo Hello {0} ${{Test}}', github.undefined_property)", "echo Hello ${Test}"},
{"format('echo Hello {0}{1} ${{Te{0}st}}', github.undefined_property, 'World')", "echo Hello World ${Test}"},
{"format('{0}', '{1}', 'World')", "{1}"},
{"format('{{{0}', '{1}', 'World')", "{{1}"},
}
for _, table := range tables {
t.Run(table.in, func(t *testing.T) {
assertObject := assert.New(t)
out, err := ee.evaluate(t.Context(), table.in, exprparser.DefaultStatusCheckNone)
if table.errMesg == "" {
assertObject.NoError(err, table.in) //nolint:testifylint // pre-existing issue from nektos/act
assertObject.Equal(table.out, out, table.in)
require.NoError(t, err)
assert.Equal(t, table.out, out)
})
}
}
type hashFilesContainer struct {
fakeContainer
stdout, stderr io.Writer
copyErr error
execute func([]string, map[string]string, string, string) common.Executor
}
func (c *hashFilesContainer) Copy(_destPath string, _files ...*container.FileEntry) common.Executor {
return common.NewErrorExecutor(c.copyErr)
}
func (c *hashFilesContainer) Exec(command []string, env map[string]string, user, workdir string) common.Executor {
return c.execute(command, env, user, workdir)
}
func (c *hashFilesContainer) ReplaceLogWriter(stdout, stderr io.Writer) (io.Writer, io.Writer) {
previousOut, previousErr := c.stdout, c.stderr
c.stdout, c.stderr = stdout, stderr
return previousOut, previousErr
}
func TestGetHashFilesFunction(t *testing.T) {
const failureMessage = "hashFiles('dir''file, *.sum') failed. Fail to hash files under directory '/workspace'"
for _, test := range []struct {
name string
output string
execErr error
copyErr error
hash string
err string
warning bool
expired bool
}{
{name: "success", output: "patterns __OUTPUT__wrong__OUTPUT__ echoed\n__OUTPUT__digest__OUTPUT__", hash: "digest"},
{name: "no matches", output: "__OUTPUT____OUTPUT__"},
{name: "script failure", output: "Cannot find: node in PATH\n__OUTPUT__digest__OUTPUT__", execErr: container.ExitCodeError(1), err: failureMessage},
{name: "timeout", execErr: context.DeadlineExceeded, err: "hashFiles('dir''file, *.sum') couldn't finish within 120 seconds"},
{name: "job deadline", execErr: context.DeadlineExceeded, expired: true, err: context.DeadlineExceeded.Error()},
{name: "node missing", execErr: fmt.Errorf("wrapped: %w", container.ExitCodeError(127)), warning: true},
{name: "node not executable", execErr: container.ExitCodeError(126), warning: true},
{name: "host node missing", execErr: errors.New("Cannot find: node in PATH"), warning: true},
{name: "copy failure", copyErr: errors.New("open /runner/cache: permission denied"), err: failureMessage},
} {
t.Run(test.name, func(t *testing.T) {
logger, hook := logtest.NewLogger()
stdout, stderr := &bytes.Buffer{}, &bytes.Buffer{}
jobContainer := &hashFilesContainer{stdout: stdout, stderr: stderr, copyErr: test.copyErr}
jobContainer.execute = func(command []string, env map[string]string, user, workdir string) common.Executor {
assert.Equal(t, []string{"node", "/var/run/act/workflow/hashfiles/index.js"}, command)
assert.Equal(t, map[string]string{"patterns": "dir'file\n*.sum", "followSymbolicLinks": "true"}, env)
assert.Empty(t, user)
assert.Empty(t, workdir)
return func(_ctx context.Context) error {
_, err := io.WriteString(jobContainer.stderr, test.output)
require.NoError(t, err)
return test.execErr
}
}
ctx := common.WithLogger(t.Context(), logger)
if test.expired {
var cancel context.CancelFunc
ctx, cancel = context.WithDeadline(ctx, time.Now())
defer cancel()
}
result, err := getHashFilesFunction(ctx, &RunContext{
Config: &Config{Workdir: "/workspace"},
JobContainer: jobContainer,
})([]reflect.Value{reflect.ValueOf("--follow-symbolic-links"), reflect.ValueOf("dir'file"), reflect.ValueOf("*.sum")})
if test.err == "" {
require.NoError(t, err)
} else {
assertObject.Error(err, table.in) //nolint:testifylint // pre-existing issue from nektos/act
assertObject.Equal(table.errMesg, err.Error(), table.in)
require.EqualError(t, err, test.err)
}
assert.Equal(t, test.hash, result)
assert.Same(t, stdout, jobContainer.stdout)
assert.Same(t, stderr, jobContainer.stderr)
if test.warning {
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, "##[warning]hashFiles could not run because node is unavailable in the job environment. Returning an empty string.", hook.LastRecord().Message)
} else {
assert.Empty(t, hook.AllRecords())
}
})
}
@@ -164,33 +253,26 @@ func TestEvaluateStep(t *testing.T) {
ee := rc.NewStepExpressionEvaluator(t.Context(), step)
tables := []struct {
in string
out any
errMesg string
in string
out any
}{
{"steps.idwithnothing.conclusion", model.StepStatusSuccess.String(), ""},
{"steps.idwithnothing.outcome", model.StepStatusFailure.String(), ""},
{"steps.idwithnothing.outputs.foowithnothing", "barwithnothing", ""},
{"steps.id-with-hyphens.conclusion", model.StepStatusSuccess.String(), ""},
{"steps.id-with-hyphens.outcome", model.StepStatusFailure.String(), ""},
{"steps.id-with-hyphens.outputs.foo-with-hyphens", "bar-with-hyphens", ""},
{"steps.id_with_underscores.conclusion", model.StepStatusSuccess.String(), ""},
{"steps.id_with_underscores.outcome", model.StepStatusFailure.String(), ""},
{"steps.id_with_underscores.outputs.foo_with_underscores", "bar_with_underscores", ""},
{"inputs.forged", nil, ""}, // INPUT_* env is not an input
{"steps.idwithnothing.conclusion", model.StepStatusSuccess.String()},
{"steps.idwithnothing.outcome", model.StepStatusFailure.String()},
{"steps.idwithnothing.outputs.foowithnothing", "barwithnothing"},
{"steps.id-with-hyphens.conclusion", model.StepStatusSuccess.String()},
{"steps.id-with-hyphens.outcome", model.StepStatusFailure.String()},
{"steps.id-with-hyphens.outputs.foo-with-hyphens", "bar-with-hyphens"},
{"steps.id_with_underscores.conclusion", model.StepStatusSuccess.String()},
{"steps.id_with_underscores.outcome", model.StepStatusFailure.String()},
{"steps.id_with_underscores.outputs.foo_with_underscores", "bar_with_underscores"},
{"inputs.forged", nil},
}
for _, table := range tables {
t.Run(table.in, func(t *testing.T) {
assertObject := assert.New(t)
out, err := ee.evaluate(t.Context(), table.in, exprparser.DefaultStatusCheckNone)
if table.errMesg == "" {
assertObject.NoError(err, table.in) //nolint:testifylint // pre-existing issue from nektos/act
assertObject.Equal(table.out, out, table.in)
} else {
assertObject.Error(err, table.in) //nolint:testifylint // pre-existing issue from nektos/act
assertObject.Equal(table.errMesg, err.Error(), table.in)
}
require.NoError(t, err)
assert.Equal(t, table.out, out)
})
}
}
@@ -198,6 +280,7 @@ func TestEvaluateStep(t *testing.T) {
func TestInterpolate(t *testing.T) {
rc := &RunContext{
Config: &Config{
Env: testCheckoutEnv(),
Workdir: ".",
Secrets: map[string]string{
"CASE_INSENSITIVE_SECRET": "value",
@@ -282,10 +365,8 @@ func TestInterpolate(t *testing.T) {
}
func TestGetEvaluatorInputsBoolean(t *testing.T) {
workflows := map[string]string{
"workflow_call": `
on:
workflow_call:
workflow := func(event string) string {
return "on:\n " + event + `:
inputs:
flag:
type: boolean
@@ -295,20 +376,7 @@ on:
default: gitea
unset:
type: boolean
`,
"workflow_dispatch": `
on:
workflow_dispatch:
inputs:
flag:
type: boolean
default: true
name:
type: string
default: gitea
unset:
type: boolean
`,
`
}
tables := []struct {
@@ -317,7 +385,6 @@ on:
flag any
}{
{
// Gitea >= 1.27 resolves the inputs server-side and sends native JSON types
name: "native bool true",
event: map[string]any{"inputs": map[string]any{"flag": true}},
flag: true,
@@ -344,10 +411,10 @@ on:
},
}
for eventName, workflow := range workflows {
for _, eventName := range []string{"workflow_call", "workflow_dispatch"} {
for _, table := range tables {
t.Run(eventName+"/"+table.name, func(t *testing.T) {
wf, err := model.ReadWorkflow(strings.NewReader(workflow))
wf, err := model.ReadWorkflow(strings.NewReader(workflow(eventName)))
require.NoError(t, err)
rc := &RunContext{
@@ -365,7 +432,7 @@ on:
}
t.Run("deferred workflow call inputs", func(t *testing.T) {
workflow, err := model.ReadWorkflow(strings.NewReader(workflows["workflow_call"] + `
workflow, err := model.ReadWorkflow(strings.NewReader(workflow("workflow_call") + `
jobs:
test: {}
call:
@@ -379,9 +446,9 @@ jobs:
for _, value := range []string{"true", "false"} {
t.Run(value, func(t *testing.T) {
t.Parallel()
parent, err := (&runnerImpl{config: &Config{}}).newRunContext(t.Context(), &model.Run{Workflow: workflow, JobID: "call"}, map[string]any{"args": `{"flag":` + value + `,"name":"${{ 'runner' }}"}`})
parent, err := (&runnerImpl{config: &Config{PresetGitHubContext: &model.GithubContext{}}}).newRunContext(t.Context(), &model.Run{Workflow: workflow, JobID: "call"}, map[string]any{"args": `{"flag":` + value + `,"name":"${{ 'runner' }}"}`})
require.NoError(t, err)
child, err := (&runnerImpl{config: &Config{}, caller: &caller{runContext: parent}}).newRunContext(t.Context(), &model.Run{Workflow: workflow, JobID: "test"}, nil)
child, err := (&runnerImpl{config: &Config{PresetGitHubContext: &model.GithubContext{}}, caller: &caller{runContext: parent}}).newRunContext(t.Context(), &model.Run{Workflow: workflow, JobID: "test"}, nil)
require.NoError(t, err)
assert.Equal(t, map[string]any{"flag": value == "true", "name": "${{ 'runner' }}", "unset": false}, child.workflowCallInputs)
assert.Equal(t, rawWith, job.RawWith)
@@ -405,7 +472,6 @@ on:
config := &Config{Workdir: ".", WorkflowCallInputs: map[string]any{"name": "caller", "target": "dispatch"}}
ghc := &model.GithubContext{EventName: "workflow_dispatch", Event: map[string]any{"inputs": map[string]any{"name": "dispatch"}}}
// the event's own inputs must not override those Gitea resolved
job := &RunContext{Config: config, Run: &model.Run{JobID: "job1", Workflow: wf}}
assert.Equal(t, map[string]any{"name": "caller", "target": "dispatch"}, getEvaluatorInputs(job, nil, ghc))
@@ -461,7 +527,7 @@ jobs:
require.NoError(t, err)
notCallable, err := model.ReadWorkflow(strings.NewReader("on: push\njobs:\n test: {}\n"))
require.NoError(t, err)
config := &Config{Secrets: map[string]string{"token": "${{ github.token }}"}}
config := &Config{Secrets: map[string]string{"token": "${{ github.token }}"}, PresetGitHubContext: &model.GithubContext{}}
resolve := func(callerJobID string, workflow *model.Workflow) *RunContext {
parent, err := (&runnerImpl{config: config}).newRunContext(t.Context(), &model.Run{Workflow: callerWorkflow, JobID: callerJobID}, nil)
require.NoError(t, err)
@@ -488,7 +554,7 @@ jobs:
`))
require.NoError(t, err)
runner := &runnerImpl{config: &Config{Secrets: map[string]string{"A": "s3cr3t-a", "B": "s3cr3t-b"}}}
runner := &runnerImpl{config: &Config{Secrets: map[string]string{"A": "s3cr3t-a", "B": "s3cr3t-b"}, PresetGitHubContext: &model.GithubContext{}}}
containerName := func(jobID string) string {
rc, err := runner.newRunContext(t.Context(), &model.Run{JobID: jobID, Workflow: workflow}, nil)
require.NoError(t, err)
@@ -497,6 +563,25 @@ jobs:
}
a, b := containerName("a"), containerName("b")
assert.NotContains(t, a, "s3cr3t") // it reaches the container name, which no log masker covers
assert.NotEqual(t, a, b) // masking the name must not collapse two jobs onto one container
assert.NotContains(t, a, "s3cr3t")
assert.NotEqual(t, a, b)
}
func TestNewExpressionEvaluatorReadsJobOutputsWhileCombinationsWriteThem(t *testing.T) {
rc := createRunContext(t)
rc.caller = &caller{runContext: createRunContext(t)}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
job := rc.Run.Job()
var writes sync.WaitGroup
writes.Go(func() {
for range 100 {
unlock := lockJob(job)
job.Outputs = map[string]string{"out": "value"}
unlock()
}
})
for range 100 {
rc.NewExpressionEvaluator(t.Context())
}
writes.Wait()
}

View File

@@ -60,3 +60,7 @@ func requireHostTools(t *testing.T, tools ...string) {
}
}
}
func testCheckoutEnv() map[string]string {
return map[string]string{"GITHUB_REPOSITORY": "owner/repo", "GITHUB_REF": "refs/heads/main", "SHA_REF": "1234567890"}
}

View File

@@ -7,6 +7,7 @@ package runner
import (
"archive/tar"
"bytes"
"cmp"
"context"
"encoding/base64"
"encoding/json/v2"
@@ -19,10 +20,12 @@ import (
"slices"
"strconv"
"strings"
"sync"
"time"
"unicode"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/exprparser"
@@ -80,7 +83,7 @@ func reportStepError(ctx context.Context, rc *RunContext, err error) {
// executor can fetch all of them up front.
type actionPreparer interface {
prepareActionExecutor() common.Executor
actionDownloadInfo() (reference, sha string, ok bool)
actionDownloadInfo() (reference, sha string)
}
// printPrepareActions downloads every action the job uses before its first step runs and reports
@@ -95,21 +98,50 @@ func printPrepareActions(rc *RunContext, preparers []actionPreparer) common.Exec
rawLogger := common.Logger(ctx).With(rawOutputField, true)
rawLogger.Infof("Prepare all required actions")
fetchCtx, cancelFetches := context.WithCancel(ctx)
var fetches sync.WaitGroup
defer func() { cancelFetches(); fetches.Wait() }()
slots := make(chan struct{}, 4)
job := rc.topLevelRunContext()
if job.actionDownloads == nil {
job.actionDownloads = map[string]*actionDownload{}
}
for _, preparer := range preparers {
action, ok := preparer.(*stepActionRemote)
if !ok || strings.Contains(action.Step.Uses, "${{") || action.action != nil {
continue
}
github := action.getGithubContext(common.WithLogger(ctx, logging.New(logging.DiscardHandler)))
remote, err := newRemoteAction(action.Step.Uses, github, action.RunContext.Config.GitHubInstance)
if err != nil || action.skipsLocalCheckout(remote, github) {
continue
}
key := remote.downloadKey(action.RunContext.Config.DefaultActionURL())
if job.actionDownloads[key] != nil {
continue
}
download := &actionDownload{fetched: make(chan struct{})}
job.actionDownloads[key] = download
fetches.Go(func() {
defer close(download.fetched)
defer common.RecoverPanic(&download.err)
slots <- struct{}{}
defer func() { <-slots }()
download.dir, download.sha, download.err = action.fetchAction(fetchCtx, remote, github.Token)
})
}
for _, preparer := range preparers {
if err := preparer.prepareActionExecutor()(ctx); err != nil {
// No step has run yet, so the failure belongs to the job.
reportStepError(ctx, rc, err)
return setupError{err}
}
reference, sha, ok := preparer.actionDownloadInfo()
if !ok {
reference, sha := preparer.actionDownloadInfo()
if sha == "" {
continue
}
if sha == "" {
rawLogger.Infof("Download action repository '%s'", reference)
} else {
rawLogger.Infof("Download action repository '%s' (SHA:%s)", reference, sha)
}
rawLogger.Infof("Download action repository '%s' (SHA:%s)", reference, sha)
}
return nil
}
@@ -124,10 +156,7 @@ func (e setupError) Unwrap() error { return e.error }
func printCompleteJobName(rc *RunContext) common.Executor {
return func(ctx context.Context) error {
// Name holds a matrix combination; JobName is the shared name GitHub reports.
name := rc.JobName
if name == "" {
name = rc.Name
}
name := cmp.Or(rc.JobName, rc.Name)
if name == "" && rc.Run != nil {
name = rc.Run.JobID
}
@@ -256,7 +285,7 @@ func newJobExecutor(info jobInfo, sf stepFactory, rc *RunContext) common.Executo
jobError := common.JobError(ctx)
var err error
// always allow 1 min for stopping and removing the runner, even if we were cancelled
ctx, cancel := context.WithTimeout(common.WithLogger(context.Background(), common.Logger(ctx)), jobCleanupTimeout)
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), jobCleanupTimeout)
defer cancel()
logger := common.Logger(ctx)
@@ -327,10 +356,7 @@ func newStepsExecutor(rc *RunContext, preSteps, steps []common.Executor) common.
return preErr // as on GitHub, no step runs after the job setup failed
}
mainErr := mainExecutor(ctx)
if preErr != nil {
return preErr
}
return mainErr
return cmp.Or(preErr, mainErr)
}
}
@@ -374,6 +400,7 @@ func runMainStepsAfterInterrupt(ctx context.Context, rc *RunContext, steps []com
func setJobResult(ctx context.Context, info jobInfo, rc *RunContext, success bool) {
logger := common.Logger(ctx)
success = success && !rc.jobFailed
// Matrix combinations share one *model.Job and run in parallel; serialize the
// read-modify-write of the job result so a failing combination is not lost-updated by a
@@ -516,8 +543,7 @@ func tryUploadJobSummary(ctx context.Context, rc *RunContext) {
ctx, cancel := context.WithTimeout(ctx, jobSummaryUploadPhaseTimeout)
defer cancel()
env := rc.GetEnv()
caps := strings.TrimSpace(env["GITEA_ACTIONS_CAPABILITIES"])
if !hasJobSummaryCapability(caps) {
if !hasJobSummaryCapability(env["GITEA_ACTIONS_CAPABILITIES"]) {
// Server did not advertise support. Do not attempt upload.
return
}
@@ -650,19 +676,18 @@ func readSingleFileFromContainerArchive(ctx context.Context, env container.Execu
}
defer rc.Close()
requestedPath := path.Clean(strings.TrimPrefix(p, "/"))
tr := tar.NewReader(rc)
for {
header, err := tr.Next()
if err == io.EOF {
return nil, false
}
if err != nil {
return nil, false
}
if header.Typeflag != tar.TypeReg {
continue
}
if !archiveEntryMatchesPath(header.Name, p) {
entryName := path.Clean(strings.TrimPrefix(header.Name, "/"))
if entryName != requestedPath && entryName != path.Base(requestedPath) {
continue
}
// Summaries larger than the limit are truncated rather than dropped, so the
@@ -685,12 +710,6 @@ func readSingleFileFromContainerArchive(ctx context.Context, env container.Execu
}
}
func archiveEntryMatchesPath(entryName, requestedPath string) bool {
entryName = path.Clean(strings.TrimPrefix(entryName, "/"))
requestedPath = path.Clean(strings.TrimPrefix(requestedPath, "/"))
return entryName == requestedPath || entryName == path.Base(requestedPath)
}
// useJobStageLogger tags the log entries of a job level executor with the stage they belong to.
func useJobStageLogger(stage string, executor common.Executor) common.Executor {
return func(ctx context.Context) error {
@@ -701,17 +720,6 @@ func useJobStageLogger(stage string, executor common.Executor) common.Executor {
func useStepLogger(rc *RunContext, stepModel *model.Step, stage stepStage, executor common.Executor) common.Executor {
return func(ctx context.Context) error {
ctx = withStepLogger(ctx, stepModel.Number, stepModel.ID, rc.ExprEval.InterpolateName(ctx, stepModel.String()), stage.String())
logWriter := rc.commandLogWriter(ctx)
oldout, olderr := rc.JobContainer.ReplaceLogWriter(logWriter, logWriter)
defer rc.JobContainer.ReplaceLogWriter(oldout, olderr)
// Flush any buffered, not-yet-newline-terminated trailing line once the
// step has finished, so the final line of the step's output is not lost
// when it is not newline-terminated.
defer common.FlushWriter(logWriter)
return executor(ctx)
return rc.withCommandLogWriter(ctx, executor)
}
}

View File

@@ -14,13 +14,17 @@ import (
"io"
"net/http"
"net/http/httptest"
"path/filepath"
"slices"
"strconv"
"strings"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
@@ -98,7 +102,7 @@ func (jim *jobInfoMock) result(result string) {
type jobContainerMock struct {
container.Container
container.LinuxContainerEnvironmentExtensions
linuxEnvironment
}
func (jcm *jobContainerMock) ReplaceLogWriter(_, _ io.Writer) (io.Writer, io.Writer) {
@@ -118,43 +122,99 @@ func (sfm *stepFactoryMock) newStep(model *model.Step, rc *RunContext) (step, er
type actionPreparerMock struct {
reference string
sha string
ok bool
err error
prepared int
}
func (apm *actionPreparerMock) prepareActionExecutor() common.Executor {
return func(context.Context) error {
apm.prepared++
return apm.err
return nil
}
}
func (apm *actionPreparerMock) actionDownloadInfo() (string, string, bool) {
return apm.reference, apm.sha, apm.ok
func (apm *actionPreparerMock) actionDownloadInfo() (string, string) {
return apm.reference, apm.sha
}
func TestPrintPrepareActionsGolden(t *testing.T) {
buf := &bytes.Buffer{}
logger := logging.New(logging.NewLevelHandler(newJobLogHandler(buf, cyan), logging.LevelInfo))
ctx := common.WithLogger(t.Context(), logger.With("job", "j1"))
func TestPrintPrepareActionsFetchesConcurrently(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
buf := &bytes.Buffer{}
ctx := common.WithLogger(t.Context(), logging.New(logging.NewLevelHandler(newJobLogHandler(buf, cyan), logging.LevelInfo)).With("job", "j1"))
rc := newPrefetchRunContext(t)
started, release := make(chan string), make(chan struct{})
var fetched atomic.Int32
setFetch(t, func(_ctx context.Context, input git.NewGitCloneExecutorInput) (string, string, error) {
fetched.Add(1)
started <- input.Ref
<-release
return filepath.Join(input.Dir, "commit"), "sha-" + input.Ref, nil
})
preparers := append(newRemotePreparers(rc, "org/repo@v1", "org/repo/sub@v1", "org/repo@v2", "org/repo@v3", "org/repo@v4", "org/repo@v5"), &actionPreparerMock{})
finished := make(chan error)
go func() { finished <- printPrepareActions(rc, preparers)(ctx) }()
for range 4 {
<-started
}
synctest.Wait()
assert.EqualValues(t, 4, fetched.Load())
close(release)
<-started
require.NoError(t, <-finished)
assert.EqualValues(t, 5, fetched.Load())
lines := []string{"[j1] | Prepare all required actions"}
for _, ref := range []string{"v1", "v2", "v3", "v4", "v5"} {
lines = append(lines, fmt.Sprintf("[j1] | Download action repository 'org/repo@%s' (SHA:sha-%s)", ref, ref))
}
assert.Equal(t, strings.Join(append(lines, ""), "\n"), buf.String())
})
}
preparers := []actionPreparer{
&actionPreparerMock{reference: "actions/checkout@v7", sha: "9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0", ok: true},
// A resolved commit is best effort; the ref alone is reported when it is unknown.
&actionPreparerMock{reference: "actions/setup-go@v6", ok: true},
// A step that downloads nothing, such as the checkout of the workflow's own repository.
&actionPreparerMock{ok: false},
func TestPrintPrepareActionsFailsJobAndCancelsLaterDownloads(t *testing.T) {
for name, panicFetch := range map[string]bool{"error": false, "panic": true} {
t.Run(name, func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
rc := newPrefetchRunContext(t)
setFetch(t, func(ctx context.Context, input git.NewGitCloneExecutorInput) (string, string, error) {
if input.Ref == "v1" {
if panicFetch {
panic("broken archive")
}
return "", "", assert.AnError
}
<-ctx.Done()
return "", "", ctx.Err()
})
ctx := common.WithJobErrorContainer(common.WithLogger(t.Context(), logging.New(logging.DiscardHandler)))
remaining := &actionPreparerMock{}
err := printPrepareActions(rc, append(newRemotePreparers(rc, "org/repo@v1", "org/repo@v2"), remaining))(ctx)
if panicFetch {
require.ErrorContains(t, err, "panic: broken archive")
} else {
require.ErrorIs(t, err, assert.AnError)
}
require.ErrorIs(t, err, common.JobError(ctx))
assert.True(t, rc.jobFailed)
assert.Zero(t, remaining.prepared)
})
})
}
require.NoError(t, printPrepareActions(&RunContext{}, preparers)(ctx))
}
want := strings.Join([]string{
"[j1] | Prepare all required actions",
"[j1] | Download action repository 'actions/checkout@v7' (SHA:9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0)",
"[j1] | Download action repository 'actions/setup-go@v6'",
"",
}, "\n")
assert.Equal(t, want, buf.String())
func newPrefetchRunContext(t *testing.T) *RunContext {
return &RunContext{
Config: &Config{ActionCacheDir: t.TempDir(), PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
}
}
func newRemotePreparers(rc *RunContext, uses ...string) []actionPreparer {
preparers := make([]actionPreparer, 0, len(uses))
for _, use := range uses {
preparers = append(preparers, &stepActionRemote{Step: &model.Step{Uses: use}, RunContext: rc, readAction: func(_ctx context.Context, _step *model.Step, _actionDir, _actionPath string, _readFile actionYamlReader, _writeFile fileWriter) (*model.Action, error) {
return &model.Action{}, nil
}})
}
return preparers
}
func TestPrintPrepareActionsSkipsWithoutActions(t *testing.T) {
@@ -167,26 +227,6 @@ func TestPrintPrepareActionsSkipsWithoutActions(t *testing.T) {
assert.Empty(t, buf.String())
}
func TestPrintPrepareActionsFailsJobOnDownloadError(t *testing.T) {
logger := logging.New(logtest.New())
ctx := common.WithJobErrorContainer(common.WithLogger(t.Context(), logger.With("job", "j1")))
downloadErr := errors.New("failed to fetch \"actions/checkout\"")
rc := &RunContext{}
remaining := &actionPreparerMock{reference: "actions/setup-go@v6", ok: true}
err := printPrepareActions(rc, []actionPreparer{
&actionPreparerMock{err: downloadErr},
remaining,
})(ctx)
require.ErrorIs(t, err, downloadErr)
// No step has run yet, so the failure has to be recorded against the job itself.
assert.Equal(t, downloadErr, common.JobError(ctx))
assert.True(t, rc.jobFailed)
assert.Zero(t, remaining.prepared)
}
func TestNewStepsExecutorSkipsStepsAfterSetupFailure(t *testing.T) {
for name, tt := range map[string]struct {
preErr error
@@ -250,7 +290,7 @@ func TestNewJobExecutorDownloadsAllActionsBeforeTheFirstStep(t *testing.T) {
Jobs: map[string]*model.Job{"test": {}},
},
},
Config: &Config{},
Config: &Config{PresetGitHubContext: &model.GithubContext{}},
}
rc.ExprEval = rc.NewExpressionEvaluator(ctx)
@@ -267,7 +307,7 @@ func TestNewJobExecutorDownloadsAllActionsBeforeTheFirstStep(t *testing.T) {
for _, stepModel := range steps {
sm := &stepMock{}
apm := &actionPreparerMock{reference: "actions/checkout@v" + stepModel.ID, ok: true}
apm := &actionPreparerMock{reference: "actions/checkout@v" + stepModel.ID, sha: "sha" + stepModel.ID}
sfm.On("newStep", stepModel, rc).Return(&actionStepMock{stepMock: sm, actionPreparerMock: apm}, nil)
sm.On("pre").Return(func(ctx context.Context) error {
@@ -304,8 +344,8 @@ func TestNewJobExecutorDownloadsAllActionsBeforeTheFirstStep(t *testing.T) {
}
assert.Equal(t, []string{
"Prepare all required actions",
"Download action repository 'actions/checkout@v1'",
"Download action repository 'actions/checkout@v2'",
"Download action repository 'actions/checkout@v1' (SHA:sha1)",
"Download action repository 'actions/checkout@v2' (SHA:sha2)",
"Complete job name: test",
"Run Pre 1",
"Run Pre 2",
@@ -324,6 +364,7 @@ func TestNewJobExecutor(t *testing.T) {
output string
startError error
cancelOnStart bool
timeout bool
}{
{
name: "zeroSteps",
@@ -474,17 +515,28 @@ func TestNewJobExecutor(t *testing.T) {
result: "cancelled",
cancelOnStart: true,
},
}
contains := func(needle string, haystack []string) bool {
return slices.Contains(haystack, needle)
{
name: "post steps run after timeout",
steps: []*model.Step{{ID: "1"}},
preSteps: []bool{false},
postSteps: []bool{true},
executedSteps: []string{"startContainer", "step1", "post1", "interpolateOutputs", "stopContainer", "closeContainer"},
result: "failure",
timeout: true,
},
}
for _, tt := range table {
t.Run(tt.name, func(t *testing.T) {
fmt.Printf("::group::%s\n", tt.name) //nolint:forbidigo // pre-existing issue from nektos/act
ctx, cancel := context.WithCancel(common.WithJobErrorContainer(t.Context()))
ctx := common.WithJobErrorContainer(t.Context())
cancel := func() {}
var expire func()
if tt.timeout {
deadline := newControllableDeadlineContext(ctx)
ctx, expire = deadline, deadline.expire
} else {
ctx, cancel = context.WithCancel(ctx)
}
defer cancel()
jim := &jobInfoMock{}
sfm := &stepFactoryMock{}
@@ -498,7 +550,7 @@ func TestNewJobExecutor(t *testing.T) {
},
},
},
Config: &Config{},
Config: &Config{PresetGitHubContext: &model.GithubContext{}},
}
if tt.output != "" {
require.NoError(t, rc.Run.Job().RawOutputs.Encode(map[string]string{"bad": tt.output}))
@@ -523,7 +575,7 @@ func TestNewJobExecutor(t *testing.T) {
sfm.On("newStep", stepModel, rc).Return(sm, nil)
sm.On("pre").Return(func(ctx context.Context) error {
sm.On("pre").Return(func(_ctx context.Context) error {
if tt.preSteps[i] {
executorOrder = append(executorOrder, "pre"+stepModel.ID)
}
@@ -532,6 +584,10 @@ func TestNewJobExecutor(t *testing.T) {
sm.On("main").Return(func(ctx context.Context) error {
executorOrder = append(executorOrder, "step"+stepModel.ID)
if tt.timeout {
expire()
return ctx.Err()
}
if tt.hasError {
return errors.New("error")
}
@@ -542,6 +598,9 @@ func TestNewJobExecutor(t *testing.T) {
if tt.postSteps[i] {
executorOrder = append(executorOrder, "post"+stepModel.ID)
}
if tt.timeout {
require.NoError(t, ctx.Err(), "post step must run against a fresh, non-expired context")
}
return nil
})
@@ -559,7 +618,7 @@ func TestNewJobExecutor(t *testing.T) {
return nil
})
if contains("stopContainer", tt.executedSteps) {
if slices.Contains(tt.executedSteps, "stopContainer") {
jim.On("stopContainer").Return(func(ctx context.Context) error {
executorOrder = append(executorOrder, "stopContainer")
require.NoError(t, ctx.Err())
@@ -573,7 +632,7 @@ func TestNewJobExecutor(t *testing.T) {
}
if len(tt.steps) > 0 {
jim.On("closeContainer").Return(func(ctx context.Context) error {
jim.On("closeContainer").Return(func(_ctx context.Context) error {
executorOrder = append(executorOrder, "closeContainer")
return nil
})
@@ -594,8 +653,6 @@ func TestNewJobExecutor(t *testing.T) {
jim.AssertExpectations(t)
sfm.AssertExpectations(t)
fmt.Println("::endgroup::") //nolint:forbidigo // pre-existing issue from nektos/act
})
}
}
@@ -626,123 +683,32 @@ func (ctx *controllableDeadlineContext) expire() {
close(ctx.done)
}
// TestNewJobExecutorRunsPostStepsAfterTimeout guards the timeout-minutes cleanup
// path: when a job exceeds its timeout the job context is DeadlineExceeded, but
// the post steps (cleanup hooks like actions/checkout post and cache save) must
// still run against a fresh, non-expired context, and the job must still be
// reported as failed.
func TestNewJobExecutorRunsPostStepsAfterTimeout(t *testing.T) {
ctx := newControllableDeadlineContext(common.WithJobErrorContainer(t.Context()))
jim := &jobInfoMock{}
sfm := &stepFactoryMock{}
rc := &RunContext{
JobContainer: &jobContainerMock{},
Run: &model.Run{
JobID: "test",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"test": {},
},
},
},
Config: &Config{},
}
rc.ExprEval = rc.NewExpressionEvaluator(ctx)
stepModel := &model.Step{ID: "1"}
jim.On("steps").Return([]*model.Step{stepModel})
jim.On("matrix").Return(map[string]any{})
jim.On("startContainer").Return(func(ctx context.Context) error { return nil })
jim.On("interpolateOutputs").Return(func(ctx context.Context) error { return nil })
jim.On("closeContainer").Return(func(ctx context.Context) error { return nil })
// The job timed out, so it must be reported as failed and still cleaned up.
jim.On("stopContainer").Return(func(context.Context) error { return nil })
jim.On("result", "failure")
sm := &stepMock{}
sfm.On("newStep", stepModel, rc).Return(sm, nil)
sm.On("pre").Return(func(ctx context.Context) error { return nil })
sm.On("main").Return(func(stepCtx context.Context) error {
ctx.expire()
return stepCtx.Err()
})
var postRan bool
var postCtxErr error
sm.On("post").Return(func(ctx context.Context) error {
postRan = true
postCtxErr = ctx.Err()
return nil
})
executor := newJobExecutor(jim, sfm, rc)
// The executor itself returns nil on timeout: the failure is surfaced through
// the job result ("failure", asserted via the result mock below), not the
// return value.
require.NoError(t, executor(ctx))
assert.True(t, postRan, "post step must run after a job timeout")
require.NoError(t, postCtxErr, "post step must run against a fresh, non-expired context")
jim.AssertExpectations(t)
sfm.AssertExpectations(t)
sm.AssertExpectations(t)
}
// TestSetJobResultMatrixContinueOnError exercises the parallel-matrix path
// end-to-end: two combinations share one *model.Job and continue-on-error is
// keyed on matrix.experimental, so one combination tolerates its failure and the
// other does not. The job is reported as continue-on-error only when EVERY failing
// combination was tolerated; a single firm failure makes the whole job firm, and
// handleFailure then fails the run.
func TestSetJobResultMatrixContinueOnError(t *testing.T) {
const jobYAML = "continue-on-error: ${{ matrix.experimental }}\nruns-on: ubuntu-latest"
newSharedJob := func(t *testing.T) (*model.Job, *model.Workflow) {
t.Helper()
var job *model.Job
require.NoError(t, yaml.Unmarshal([]byte(jobYAML), &job))
return job, &model.Workflow{
Name: "workflow1",
Jobs: map[string]*model.Job{"job1": job},
}
}
planFor := func(wf *model.Workflow) *model.Plan {
return &model.Plan{Stages: []*model.Stage{{Runs: []*model.Run{{Workflow: wf, JobID: "job1"}}}}}
}
ctx := t.Context()
for name, tolerated := range map[string]bool{
"one tolerated and one firm failure fails the run": false,
"all tolerated failures do not fail the run": true,
} {
t.Run(name, func(t *testing.T) {
var job *model.Job
require.NoError(t, yaml.Unmarshal([]byte("continue-on-error: ${{ matrix.experimental }}\nruns-on: ubuntu-latest"), &job))
workflow := &model.Workflow{Name: "workflow1", Jobs: map[string]*model.Job{"job1": job}}
for _, experimental := range []bool{true, tolerated} {
rc := newTestRC(workflow, map[string]any{"experimental": experimental})
rc.Config.PresetGitHubContext = &model.GithubContext{}
setJobResult(ctx, rc, rc, false)
}
// fail drives a single matrix combination through the failure path; each
// RunContext is its own jobInfo (rc implements jobInfo) and shares the job.
fail := func(wf *model.Workflow, experimental bool) {
rc := newTestRC(wf, map[string]any{"experimental": experimental})
setJobResult(ctx, rc, rc, false)
assert.Equal(t, "failure", job.Result)
assert.Equal(t, tolerated, job.ContinueOnError)
err := handleFailure(&model.Plan{Stages: []*model.Stage{{Runs: []*model.Run{{Workflow: workflow, JobID: "job1"}}}}})(ctx)
if tolerated {
require.NoError(t, err)
} else {
require.Error(t, err)
}
})
}
t.Run("one tolerated and one firm failure fails the run", func(t *testing.T) {
job, wf := newSharedJob(t)
// Order is intentional: the tolerated combination finishes first, then the
// firm one. The firm-failure latch must still win regardless of order.
fail(wf, true)
fail(wf, false)
assert.Equal(t, "failure", job.Result)
assert.False(t, job.ContinueOnError, "a single firm failure must make the whole job firm")
assert.Error(t, handleFailure(planFor(wf))(ctx))
})
t.Run("all tolerated failures do not fail the run", func(t *testing.T) {
job, wf := newSharedJob(t)
fail(wf, true)
fail(wf, true)
assert.Equal(t, "failure", job.Result)
assert.True(t, job.ContinueOnError, "every failing combination was tolerated")
assert.NoError(t, handleFailure(planFor(wf))(ctx))
})
}
func TestHasJobSummaryCapability(t *testing.T) {
@@ -929,44 +895,34 @@ func TestTryUploadJobSummaryUploadsEachStepIndependentlyWithInstanceTransport(t
cm.AssertExpectations(t)
}
func TestTryUploadJobSummaryRequiresExactCapability(t *testing.T) {
requests := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests++
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
func TestTryUploadJobSummarySkipsUnsupportedUploads(t *testing.T) {
for name, tt := range map[string]struct {
capabilities string
runtimeToken string
}{
"requires exact capability": {"not-job-summary,job-summary-v2", fakeRuntimeToken(34)},
"requires job ID": {"job-summary", "not-a-jwt"},
} {
t.Run(name, func(t *testing.T) {
requests := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _r *http.Request) {
requests++
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
rc := newJobSummaryRC(map[string]string{
"GITEA_ACTIONS_CAPABILITIES": "not-job-summary,job-summary-v2",
"ACTIONS_RUNTIME_URL": server.URL,
"ACTIONS_RUNTIME_TOKEN": fakeRuntimeToken(34),
"GITEA_RUN_ID": "12",
}, &containerMock{}, 1)
rc := newJobSummaryRC(map[string]string{
"GITEA_ACTIONS_CAPABILITIES": tt.capabilities,
"ACTIONS_RUNTIME_URL": server.URL,
"ACTIONS_RUNTIME_TOKEN": tt.runtimeToken,
"GITEA_RUN_ID": "12",
}, &containerMock{}, 1)
tryUploadJobSummary(t.Context(), rc)
tryUploadJobSummary(t.Context(), rc)
assert.Equal(t, 0, requests)
}
func TestTryUploadJobSummarySkipsWhenJobIDMissingFromToken(t *testing.T) {
requests := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests++
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
rc := newJobSummaryRC(map[string]string{
"GITEA_ACTIONS_CAPABILITIES": "job-summary",
"ACTIONS_RUNTIME_URL": server.URL,
"ACTIONS_RUNTIME_TOKEN": "not-a-jwt",
"GITEA_RUN_ID": "12",
}, &containerMock{}, 1)
tryUploadJobSummary(t.Context(), rc)
assert.Equal(t, 0, requests)
assert.Zero(t, requests)
})
}
}
func TestExtractJobIDFromRuntimeToken(t *testing.T) {
@@ -995,47 +951,39 @@ func TestReadSingleFileFromContainerArchiveFindsMatchingRegularFile(t *testing.T
cm.AssertExpectations(t)
}
func TestReadSingleFileFromContainerArchiveTruncatesWhenTooLarge(t *testing.T) {
logger, hook := logtest.NewLogger()
ctx := common.WithLogger(t.Context(), logger)
cm := &containerMock{}
content := strings.Repeat("a", 300)
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/SUMMARY.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "SUMMARY.md", body: content}))),
nil,
).Once()
func TestReadSingleFileFromContainerArchiveSizeLimit(t *testing.T) {
for name, tt := range map[string]struct {
content string
maxBytes int64
want string
}{
"truncates oversized summary": {strings.Repeat("a", 300), 200, strings.Repeat("a", 200-len(jobSummaryTruncationMarker)) + jobSummaryTruncationMarker},
"keeps exact limit": {"abc", 3, "abc"},
} {
t.Run(name, func(t *testing.T) {
logger, hook := logtest.NewLogger()
ctx := common.WithLogger(t.Context(), logger)
cm := &containerMock{}
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/SUMMARY.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "SUMMARY.md", body: tt.content}))),
nil,
).Once()
const maxBytes = 200
body, ok := readSingleFileFromContainerArchive(ctx, cm, "/var/run/act/workflow/SUMMARY.md", maxBytes)
body, ok := readSingleFileFromContainerArchive(ctx, cm, "/var/run/act/workflow/SUMMARY.md", tt.maxBytes)
// Oversized summaries are truncated to the limit (reserving room for the marker)
// rather than dropped entirely, and the truncation marker is appended.
assert.True(t, ok)
assert.LessOrEqual(t, len(body), maxBytes)
keep := maxBytes - len(jobSummaryTruncationMarker)
assert.Equal(t, []byte(content[:keep]+jobSummaryTruncationMarker), body)
if assert.Len(t, hook.AllRecords(), 1) {
assert.Contains(t, hook.AllRecords()[0].Message, "job summary truncated")
assert.True(t, ok)
assert.Equal(t, []byte(tt.want), body)
assert.LessOrEqual(t, int64(len(body)), tt.maxBytes)
if int64(len(tt.content)) > tt.maxBytes {
if assert.Len(t, hook.AllRecords(), 1) {
assert.Contains(t, hook.LastRecord().Message, "job summary truncated")
}
} else {
assert.Empty(t, hook.AllRecords())
}
cm.AssertExpectations(t)
})
}
cm.AssertExpectations(t)
}
func TestReadSingleFileFromContainerArchiveKeepsExactLimitWithoutWarning(t *testing.T) {
logger, hook := logtest.NewLogger()
ctx := common.WithLogger(t.Context(), logger)
cm := &containerMock{}
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/SUMMARY.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "SUMMARY.md", body: "abc"}))),
nil,
).Once()
body, ok := readSingleFileFromContainerArchive(ctx, cm, "/var/run/act/workflow/SUMMARY.md", 3)
// A summary that is exactly at the limit is kept whole and not flagged as truncated.
assert.True(t, ok)
assert.Equal(t, []byte("abc"), body)
assert.Empty(t, hook.AllRecords())
cm.AssertExpectations(t)
}
type tarEntry struct {
@@ -1076,7 +1024,7 @@ func tarArchive(t *testing.T, entries ...tarEntry) []byte {
func newTestRC(wf *model.Workflow, matrix map[string]any) *RunContext {
return &RunContext{
Config: &Config{Workdir: ".", PlatformPicker: func([]string) string { return "ubuntu-latest" }},
Config: &Config{Workdir: ".", Env: testCheckoutEnv(), PlatformPicker: func([]string) string { return "ubuntu-latest" }},
StepResults: map[string]*model.StepResult{},
Env: map[string]string{},
Matrix: matrix,
@@ -1092,6 +1040,7 @@ func makeTestRC(t *testing.T, jobYAML string) *RunContext {
Name: "workflow1",
Jobs: map[string]*model.Job{"job1": job},
}, nil)
rc.Config.PresetGitHubContext = &model.GithubContext{}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
return rc
}
@@ -1141,34 +1090,24 @@ func TestEvaluateJobContinueOnError(t *testing.T) {
}
func TestJobSetContinueOnError(t *testing.T) {
t.Run("first call true", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(true)
assert.True(t, j.ContinueOnError)
})
t.Run("first call false", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(false)
assert.False(t, j.ContinueOnError)
})
t.Run("true then false locks to false", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(true)
j.SetContinueOnError(false)
assert.False(t, j.ContinueOnError)
})
t.Run("false then true stays false", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(false)
j.SetContinueOnError(true)
assert.False(t, j.ContinueOnError)
})
t.Run("true then true stays true", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(true)
j.SetContinueOnError(true)
assert.True(t, j.ContinueOnError)
})
for name, tt := range map[string]struct {
values []bool
want bool
}{
"first call true": {[]bool{true}, true},
"first call false": {[]bool{false}, false},
"true then false locks to false": {[]bool{true, false}, false},
"false then true stays false": {[]bool{false, true}, false},
"true then true stays true": {[]bool{true, true}, true},
} {
t.Run(name, func(t *testing.T) {
job := &model.Job{}
for _, value := range tt.values {
job.SetContinueOnError(value)
}
assert.Equal(t, tt.want, job.ContinueOnError)
})
}
}
func TestTryUploadJobSummaryMasksSecrets(t *testing.T) {

View File

@@ -40,15 +40,18 @@ func (rc *RunContext) runJobHook(ctx context.Context, hookPath, name string) err
if hookPath == "" {
return nil
}
previousStep := rc.CurrentStep
rc.CurrentStep = ""
defer func() { rc.CurrentStep = previousStep }()
cmd, shell := hookCommand(hookPath)
rawLogger := common.Logger(ctx).With(rawOutputField, true)
defer rawLogger.Infof("::endgroup::")
rawLogger.Infof("::group::Run '%s'", EscapeCommandData(hookPath))
rawLogger.Infof("A %s hook has been configured by the runner administrator", name)
rawLogger.Infof("::group::Run '%s'", EscapeCommandData(hookPath))
if shell != "" {
rawLogger.Infof("shell: %s", shell)
}
rawLogger.Infof("::endgroup::")
env := mergeMaps(rc.containerSpec.Env, rc.GetEnv())
rc.withGithubEnv(ctx, rc.getGithubContext(ctx), env)
@@ -56,12 +59,12 @@ func (rc *RunContext) runJobHook(ctx context.Context, hookPath, name string) err
err := rc.setupHookFileCommands(ctx, env)
if err == nil {
err = rc.JobContainer.Exec(cmd, env, "", "")(ctx)
}
// Processed even on failure, so a hook that exports what it managed to set up before
// failing still hands it to the job.
if processErr := rc.processHookFileCommands(ctx); err == nil {
err = processErr
err = rc.runWithCommands(ctx, env, func(ctx context.Context) error {
return rc.withCommandLogWriter(ctx, rc.JobContainer.Exec(cmd, env, "", ""))
})
if processErr := rc.processHookFileCommands(ctx); err == nil { // a failed hook still exports what it set up
err = processErr
}
}
if err == nil {
return nil

View File

@@ -14,6 +14,7 @@ import (
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
@@ -27,8 +28,11 @@ type hookContainer struct {
cmd []string
env map[string]string
err error
copyErr error
envFile map[string]string
pathTar []byte
output string
stdout io.Writer
}
func (c *hookContainer) ToContainerPath(path string) string { return path }
@@ -41,10 +45,25 @@ func (c *hookContainer) GetRunnerContext(context.Context) map[string]any {
func (c *hookContainer) Exec(command []string, env map[string]string, _, _ string) common.Executor {
return func(context.Context) error {
c.cmd, c.env = command, env
if c.output != "" {
if _, err := io.WriteString(c.stdout, c.output); err != nil {
return err
}
}
return c.err
}
}
func (c *hookContainer) ReplaceLogWriter(stdout, _ io.Writer) (io.Writer, io.Writer) {
old := c.stdout
c.stdout = stdout
return old, nil
}
func (c *hookContainer) Copy(_destPath string, _files ...*container.FileEntry) common.Executor {
return common.NewErrorExecutor(c.copyErr)
}
func (c *hookContainer) UpdateFromEnv(_ string, env *map[string]string) common.Executor {
return func(context.Context) error {
maps.Copy(*env, c.envFile)
@@ -59,7 +78,7 @@ func (c *hookContainer) GetContainerArchive(context.Context, string) (io.ReadClo
// newHookRunContext returns a RunContext and the context to run a hook with, whose logger is
// silenced so the hook's job-log output does not reach the test output.
func newHookRunContext(jobContainer *hookContainer, config *Config) (*RunContext, context.Context) {
// Env is left nil so that it is built from the config, as it is for a real job.
config.PresetGitHubContext = &model.GithubContext{}
rc := &RunContext{
Config: config,
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
@@ -72,15 +91,29 @@ func newHookRunContext(jobContainer *hookContainer, config *Config) (*RunContext
}
func TestRunJobHook(t *testing.T) {
t.Run("runs the hook with the job environment", func(t *testing.T) {
jobContainer := &hookContainer{}
t.Run("runs the hook with the job environment after its closed header group", func(t *testing.T) {
jobContainer := &hookContainer{output: "from the hook\n"}
rc, ctx := newHookRunContext(jobContainer, &Config{
JobStartedHook: "/hooks/started.sh",
Env: map[string]string{"A_VAR": "value", jobStartedHookEnv: "/from/env.sh"},
})
records := logtest.New()
ctx = common.WithLogger(ctx, logging.New(records))
require.NoError(t, rc.runJobStartedHook(ctx))
var lines []string
for _, record := range records.AllRecords() {
lines = append(lines, record.Message)
}
assert.Equal(t, []string{
"A job started hook has been configured by the runner administrator",
"::group::Run '/hooks/started.sh'",
"shell: bash -e {0}",
"::endgroup::",
"from the hook\n",
}, lines)
// The setting wins over the environment variable.
assert.Equal(t, []string{"bash", "-e", "/hooks/started.sh"}, jobContainer.cmd)
assert.Equal(t, "value", jobContainer.env["A_VAR"])
@@ -90,6 +123,57 @@ func TestRunJobHook(t *testing.T) {
assert.Equal(t, "/var/run/act/workflow/hook-path.txt", jobContainer.env["GITHUB_PATH"])
})
t.Run("each hook has its own command state", func(t *testing.T) {
jobContainer := &hookContainer{output: "::stop-commands::token77"}
rc, ctx := newHookRunContext(jobContainer, &Config{JobStartedHook: "/started.sh", JobCompletedHook: "/completed.sh"})
jobWriter := rc.commandLogWriter(ctx)
jobContainer.stdout = jobWriter
require.NoError(t, rc.runJobStartedHook(ctx))
jobContainer.output = "::add-mask::s3cret"
require.NoError(t, rc.runJobCompletedHook(ctx))
assert.Equal(t, []string{"token77", "s3cret"}, rc.Masks)
assert.Same(t, jobWriter, jobContainer.stdout)
})
t.Run("fails on refused commands and honors the container opt-in", func(t *testing.T) {
for _, allow := range []bool{false, true} {
jobContainer := &hookContainer{output: "::set-env name=FROM_HOOK::1\n::add-path::/opt/hook/bin"}
rc, ctx := newHookRunContext(jobContainer, &Config{JobStartedHook: "/started.sh"})
if allow {
rc.containerSpec.Env = map[string]string{allowUnsecureCommandsVar: "true"}
}
err := rc.runJobStartedHook(ctx)
if allow {
require.NoError(t, err)
assert.Equal(t, "1", rc.Env["FROM_HOOK"])
assert.Equal(t, []string{"/opt/hook/bin"}, rc.ExtraPath)
} else {
require.ErrorContains(t, err, "workflow command is disabled")
assert.Equal(t, "failure", rc.getJobContext().Status)
require.ErrorContains(t, common.JobError(ctx), "workflow command is disabled")
assert.NotContains(t, rc.Env, "FROM_HOOK")
assert.Empty(t, rc.ExtraPath)
}
assert.Nil(t, rc.currentStepEnv())
require.NoError(t, rc.takeUnsecureCommandError())
}
})
t.Run("the completed hook has no preceding step identity", func(t *testing.T) {
jobContainer := &hookContainer{output: "::set-output name=value::after\n::save-state name=value::after"}
rc, ctx := newHookRunContext(jobContainer, &Config{JobCompletedHook: "/completed.sh"})
rc.CurrentStep = "last-step"
rc.StepResults = map[string]*model.StepResult{"last-step": {Outputs: map[string]string{"value": "before"}}}
require.NoError(t, rc.runJobCompletedHook(ctx))
assert.Empty(t, jobContainer.env["GITHUB_ACTION"])
assert.Equal(t, "before", rc.StepResults["last-step"].Outputs["value"])
assert.Empty(t, rc.IntraActionState)
assert.Equal(t, "last-step", rc.CurrentStep)
})
// Each hook reads its own variable, so a swapped constant cannot pass.
t.Run("falls back to the GitHub environment variables", func(t *testing.T) {
for name, hook := range map[string]struct {
@@ -135,6 +219,15 @@ func TestRunJobHook(t *testing.T) {
require.ErrorContains(t, common.JobError(ctx), "boom")
})
t.Run("a hook whose file commands cannot be reset neither runs nor exports stale ones", func(t *testing.T) {
jobContainer := &hookContainer{copyErr: errors.New("boom"), envFile: map[string]string{"STALE": "1"}}
rc, ctx := newHookRunContext(jobContainer, &Config{JobStartedHook: "/hooks/started.sh"})
require.ErrorContains(t, rc.runJobStartedHook(ctx), "boom")
assert.Nil(t, jobContainer.cmd)
assert.NotContains(t, rc.Env, "STALE")
})
t.Run("is a no-op without a hook", func(t *testing.T) {
jobContainer := &hookContainer{}
rc, ctx := newHookRunContext(jobContainer, &Config{})

View File

@@ -51,29 +51,21 @@ const (
)
var (
colors []int
colors = []int{blue, yellow, green, magenta, red, gray, cyan}
nextColor int
mux sync.Mutex
)
func init() {
nextColor = 0
colors = []int{
blue, yellow, green, magenta, red, gray, cyan,
}
}
type masksContextKey string
const masksContextKeyVal = masksContextKey("logrus.FieldLogger")
type valueMaskerContextKey struct{}
// Logger returns the appropriate logger for current context
func Masks(ctx context.Context) *[]string {
val := ctx.Value(masksContextKeyVal)
if val != nil {
if masks, ok := val.(*[]string); ok {
return masks
}
if masks, ok := ctx.Value(masksContextKeyVal).(*[]string); ok {
return masks
}
return &[]string{}
}
@@ -98,6 +90,8 @@ func WithJobLoggerFactory(ctx context.Context, factory JobLoggerFactory) context
// WithJobLogger attaches a new logger to context that is aware of steps
func WithJobLogger(ctx context.Context, jobID, jobName string, config *Config, masks *[]string, matrix map[string]any) context.Context {
masker := valueMasker(config.InsecureSecrets, config.maskers())
ctx = context.WithValue(ctx, valueMaskerContextKey{}, masker)
ctx = WithMasks(ctx, masks)
var output logging.Handler
@@ -119,7 +113,7 @@ func WithJobLogger(ctx context.Context, jobID, jobName string, config *Config, m
}
handlers = append(handlers, &maskedHandler{
Handler: output,
masker: valueMasker(config.InsecureSecrets, config.maskers()),
masker: masker,
})
level := logging.LevelTrace
@@ -150,6 +144,9 @@ func WithCompositeStepLogger(ctx context.Context, stepID string) context.Context
}
func withStepLogger(ctx context.Context, stepNumber int, stepID, stepName, stageName string) context.Context {
if masker, ok := ctx.Value(valueMaskerContextKey{}).(entryProcessor); ok {
stepName = masker(ctx, logging.Record{Message: stepName}).Message
}
stepIDs := []string{stepID}
ctx = common.WithStepIDs(ctx, stepIDs)
@@ -178,7 +175,7 @@ var secretValueEncoders = []func(string) string{
uriDataEscape,
url.QueryEscape, // the form-encoded twin of uriDataEscape, which spells a space "+"
url.PathEscape,
xmlDataEscape,
xmlDataEscaper.Replace,
trimDoubleQuotes,
}
@@ -225,10 +222,6 @@ var xmlDataEscaper = strings.NewReplacer(
"'", "&apos;",
)
func xmlDataEscape(v string) string {
return xmlDataEscaper.Replace(v)
}
func trimDoubleQuotes(v string) string {
if len(v) > 8 && strings.HasPrefix(v, `"`) && strings.HasSuffix(v, `"`) {
return v[1 : len(v)-1]
@@ -407,7 +400,7 @@ func (f textFormatter) format(r logging.Record) ([]byte, error) {
b := &bytes.Buffer{}
// the web renderer decodes command data, so this local view has to as well
if _, _, _, ok := tryParseRawActionCommand(r.Message + "\n"); ok {
if _, _, _, ok := parseActionCommand(r.Message, ""); ok && (strings.HasPrefix(r.Message, "::") || strings.HasPrefix(r.Message, "##[")) {
r.Message = UnescapeCommandData(r.Message)
}
r.Message = strings.TrimSuffix(r.Message, "\n")
@@ -468,7 +461,8 @@ func (f textFormatter) print(b *bytes.Buffer, r logging.Record) {
}
func (f textFormatter) isColored() bool {
isColored := checkIfTerminal(f.w)
file, ok := f.w.(*os.File)
isColored := ok && term.IsTerminal(int(file.Fd()))
if force, ok := os.LookupEnv("CLICOLOR_FORCE"); ok && force != "0" {
isColored = true
@@ -481,15 +475,6 @@ func (f textFormatter) isColored() bool {
return isColored
}
func checkIfTerminal(w io.Writer) bool {
switch v := w.(type) {
case *os.File:
return term.IsTerminal(int(v.Fd()))
default:
return false
}
}
// maskSecrets hides this job's secrets in a value that reaches somewhere the log maskers cannot,
// such as a container name or a job summary. Masks added at runtime count, so a summary written
// after ::add-mask:: is covered too.

Some files were not shown because too many files have changed in this diff Show More