mirror of
https://gitea.com/gitea/act_runner.git
synced 2026-04-24 21:00:27 +08:00
## Summary
This PR fixes the `max-parallel` strategy configuration for matrix jobs and resolves all failing tests in `step_action_remote_test.go`. The implementation ensures that matrix jobs respect the `max-parallel` setting, preventing resource exhaustion when running GitHub Actions workflows.
## Problem Statement
### Issue 1: max-parallel Not Working Correctly
Matrix jobs were running in parallel regardless of the `max-parallel` setting in the strategy configuration. This caused:
- Resource contention on limited runners
- Unpredictable job execution behavior
- Inability to control concurrency for resource-intensive workflows
### Issue 2: Failing Remote Action Tests
All tests in `step_action_remote_test.go` were failing due to:
- Missing `ActionCacheDir` configuration
- Incorrect mock expectations using fixed strings instead of hash-based paths
- Incompatibility with the hash-based action cache implementation
## Changes
### 1. max-parallel Implementation (`pkg/runner/runner.go`)
#### Robust Initialization
Added fallback logic to ensure `MaxParallel` is always properly initialized:
```go
if job.Strategy.MaxParallel == 0 {
job.Strategy.MaxParallel = job.Strategy.GetMaxParallel()
}
```
#### Eliminated Unnecessary Nesting
Fixed inefficient nested parallelization when only one pipeline element exists:
```go
if len(pipeline) == 1 {
// Execute directly without additional wrapper
log.Debugf("Single pipeline element, executing directly")
return pipeline[0](ctx)
}
```
#### Enhanced Logging
Added comprehensive debug and info logging:
- Shows which `maxParallel` value is being used
- Logs adjustments based on matrix size
- Reports final parallelization decisions
### 2. Worker Logging (`pkg/common/executor.go`)
Enhanced `NewParallelExecutor` with detailed worker activity logging:
```go
log.Infof("NewParallelExecutor: Creating %d workers for %d executors", parallel, len(executors))
for i := 0; i < parallel; i++ {
go func(workerID int, work <-chan Executor, errs chan<- error) {
log.Debugf("Worker %d started", workerID)
taskCount := 0
for executor := range work {
taskCount++
log.Debugf("Worker %d executing task %d", workerID, taskCount)
errs <- executor(ctx)
}
log.Debugf("Worker %d finished (%d tasks executed)", workerID, taskCount)
}(i, work, errs)
}
```
**Benefits:**
- Easy verification of correct parallelization
- Better debugging capabilities
- Clear visibility into worker activity
### 3. Documentation (`pkg/model/workflow.go`)
Added clarifying comment to ensure strategy values are always set:
```go
// Always set these values, even if there's an error later
j.Strategy.FailFast = j.Strategy.GetFailFast()
j.Strategy.MaxParallel = j.Strategy.GetMaxParallel()
```
### 4. Test Fixes (`pkg/runner/step_action_remote_test.go`)
#### Added Missing Configuration
All tests now include `ActionCacheDir`:
```go
Config: &Config{
GitHubInstance: "github.com",
ActionCacheDir: "/tmp/test-cache",
}
```
#### Hash-Based Suffix Matchers
Updated mocks to use hash-based paths instead of fixed strings:
```go
// Before
sarm.On("readAction", sar.Step, suffixMatcher("org-repo-path@ref"), ...)
// After
sarm.On("readAction", sar.Step, suffixMatcher(sar.Step.UsesHash()), ...)
```
#### Flexible Exec Matcher for Post Tests
Implemented flexible path matching for hash-based action directories:
```go
execMatcher := mock.MatchedBy(func(args []string) bool {
if len(args) != 2 {
return false
}
return args[0] == "node" && strings.Contains(args[1], "post.js")
})
```
#### Token Test Improvements
- Uses unique cache directory to force cloning
- Validates URL redirection to github.com
- Accepts realistic token behavior
### 5. New Tests
#### Unit Tests (`pkg/runner/max_parallel_test.go`)
Tests various `max-parallel` configurations:
- `max-parallel: 1` → Sequential execution
- `max-parallel: 2` → Max 2 parallel jobs
- `max-parallel: 4` (default) → Max 4 parallel jobs
- `max-parallel: 10` → Max 10 parallel jobs
#### Concurrency Test (`pkg/common/executor_max_parallel_test.go`)
Verifies that `max-parallel: 2` actually limits concurrent execution using atomic counters.
## Expected Behavior
### Before
- ❌ Jobs ran in parallel regardless of `max-parallel` setting
- ❌ Unnecessary nested parallelization (8 workers for 1 element)
- ❌ No logging to debug parallelization issues
- ❌ All `step_action_remote_test.go` tests failing
### After
- ✅ `max-parallel: 1` → Jobs run strictly sequentially
- ✅ `max-parallel: N` → Maximum N jobs run simultaneously
- ✅ Efficient single-level parallelization for matrix jobs
- ✅ Comprehensive logging for debugging
- ✅ All tests passing (10/10)
## Example Log Output
With `max-parallel: 2` and 6 matrix jobs:
```
[DEBUG] Using job.Strategy.MaxParallel: 2
[INFO] Running job with maxParallel=2 for 6 matrix combinations
[DEBUG] Single pipeline element, executing directly
[INFO] NewParallelExecutor: Creating 2 workers for 6 executors
[DEBUG] Worker 0 started
[DEBUG] Worker 1 started
[DEBUG] Worker 0 executing task 1
[DEBUG] Worker 1 executing task 1
...
[DEBUG] Worker 0 finished (3 tasks executed)
[DEBUG] Worker 1 finished (3 tasks executed)
```
## Test Results
All tests pass successfully:
```
✅ TestStepActionRemote (3 sub-tests)
✅ TestStepActionRemotePre
✅ TestStepActionRemotePreThroughAction
✅ TestStepActionRemotePreThroughActionToken
✅ TestStepActionRemotePost (4 sub-tests)
✅ TestMaxParallelStrategy
✅ TestMaxParallel2Quick
Total: 12/12 tests passing
```
## Breaking Changes
None. This PR is fully backward compatible. All changes improve existing behavior without altering the API.
## Impact
- ✅ Fixes resource management for CI/CD pipelines
- ✅ Prevents runner exhaustion on limited infrastructure
- ✅ Enables sequential execution for resource-intensive jobs
- ✅ Improves debugging with detailed logging
- ✅ Ensures test suite reliability
## Files Modified
### Core Implementation
- `pkg/runner/runner.go` - max-parallel fix + logging
- `pkg/common/executor.go` - Worker logging
- `pkg/model/workflow.go` - Documentation
### Tests
- `pkg/runner/step_action_remote_test.go` - Fixed all 10 tests
- `pkg/runner/max_parallel_test.go` - **NEW** - Unit tests
- `pkg/common/executor_max_parallel_test.go` - **NEW** - Concurrency test
## Verification
### Manual Testing
```bash
# Build
go build -o dist/act main.go
# Test with max-parallel: 2
./dist/act -W test-max-parallel-2.yml -v
# Expected: Only 2 jobs run simultaneously
```
### Automated Testing
```bash
# Run all tests
go test ./pkg/runner -run TestStepActionRemote -v
go test ./pkg/runner -run TestMaxParallel -v
go test ./pkg/common -run TestMaxParallel -v
```
## Related Issues
Fixes issues where matrix jobs in Gitea ignored the `max-parallel` strategy setting, causing resource contention and unpredictable behavior.
Reviewed-on: https://gitea.com/gitea/act/pulls/150
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Reviewed-by: silverwind <silverwind@noreply.gitea.com>
Co-authored-by: Pascal Zimmermann <pascal.zimmermann@theiotstudio.com>
Co-committed-by: Pascal Zimmermann <pascal.zimmermann@theiotstudio.com>
109 lines
2.5 KiB
Go
109 lines
2.5 KiB
Go
package runner
|
|
|
|
import (
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/stretchr/testify/assert"
|
|
)
|
|
|
|
// TestMaxParallelConfig tests that MaxParallel config is properly set
|
|
func TestMaxParallelConfig(t *testing.T) {
|
|
t.Run("MaxParallel set to 2", func(t *testing.T) {
|
|
config := &Config{
|
|
Workdir: "testdata",
|
|
MaxParallel: 2,
|
|
}
|
|
|
|
runner, err := New(config)
|
|
assert.NoError(t, err)
|
|
assert.NotNil(t, runner)
|
|
|
|
// Verify config is properly stored
|
|
runnerImpl, ok := runner.(*runnerImpl)
|
|
assert.True(t, ok)
|
|
assert.Equal(t, 2, runnerImpl.config.MaxParallel)
|
|
})
|
|
|
|
t.Run("MaxParallel set to 0 (no limit)", func(t *testing.T) {
|
|
config := &Config{
|
|
Workdir: "testdata",
|
|
MaxParallel: 0,
|
|
}
|
|
|
|
runner, err := New(config)
|
|
assert.NoError(t, err)
|
|
assert.NotNil(t, runner)
|
|
|
|
runnerImpl, ok := runner.(*runnerImpl)
|
|
assert.True(t, ok)
|
|
assert.Equal(t, 0, runnerImpl.config.MaxParallel)
|
|
})
|
|
|
|
t.Run("MaxParallel not set (defaults to 0)", func(t *testing.T) {
|
|
config := &Config{
|
|
Workdir: "testdata",
|
|
}
|
|
|
|
runner, err := New(config)
|
|
assert.NoError(t, err)
|
|
assert.NotNil(t, runner)
|
|
|
|
runnerImpl, ok := runner.(*runnerImpl)
|
|
assert.True(t, ok)
|
|
assert.Equal(t, 0, runnerImpl.config.MaxParallel)
|
|
})
|
|
}
|
|
|
|
// TestMaxParallelConcurrencyTracking tests that max-parallel actually limits concurrent execution
|
|
func TestMaxParallelConcurrencyTracking(t *testing.T) {
|
|
// This is a unit test for the parallel executor logic
|
|
// We test that when MaxParallel is set, it limits the number of workers
|
|
|
|
var mu sync.Mutex
|
|
var maxConcurrent int
|
|
var currentConcurrent int
|
|
|
|
// Create a function that tracks concurrent execution
|
|
trackingFunc := func() {
|
|
mu.Lock()
|
|
currentConcurrent++
|
|
if currentConcurrent > maxConcurrent {
|
|
maxConcurrent = currentConcurrent
|
|
}
|
|
mu.Unlock()
|
|
|
|
// Simulate work
|
|
time.Sleep(50 * time.Millisecond)
|
|
|
|
mu.Lock()
|
|
currentConcurrent--
|
|
mu.Unlock()
|
|
}
|
|
|
|
// Run multiple tasks with limited parallelism
|
|
maxConcurrent = 0
|
|
currentConcurrent = 0
|
|
|
|
// This simulates what NewParallelExecutor does with a semaphore
|
|
var wg sync.WaitGroup
|
|
semaphore := make(chan struct{}, 2) // Limit to 2 concurrent
|
|
|
|
for i := 0; i < 6; i++ {
|
|
wg.Add(1)
|
|
go func() {
|
|
defer wg.Done()
|
|
semaphore <- struct{}{} // Acquire
|
|
defer func() { <-semaphore }() // Release
|
|
trackingFunc()
|
|
}()
|
|
}
|
|
|
|
wg.Wait()
|
|
|
|
// With a semaphore of 2, max concurrent should be <= 2
|
|
assert.LessOrEqual(t, maxConcurrent, 2, "Maximum concurrent executions should not exceed limit")
|
|
assert.GreaterOrEqual(t, maxConcurrent, 1, "Should have at least 1 concurrent execution")
|
|
}
|