mirror of
https://gitea.com/gitea/act_runner.git
synced 2026-04-25 13:20:32 +08:00
## Summary
This PR fixes the `max-parallel` strategy configuration for matrix jobs and resolves all failing tests in `step_action_remote_test.go`. The implementation ensures that matrix jobs respect the `max-parallel` setting, preventing resource exhaustion when running GitHub Actions workflows.
## Problem Statement
### Issue 1: max-parallel Not Working Correctly
Matrix jobs were running in parallel regardless of the `max-parallel` setting in the strategy configuration. This caused:
- Resource contention on limited runners
- Unpredictable job execution behavior
- Inability to control concurrency for resource-intensive workflows
### Issue 2: Failing Remote Action Tests
All tests in `step_action_remote_test.go` were failing due to:
- Missing `ActionCacheDir` configuration
- Incorrect mock expectations using fixed strings instead of hash-based paths
- Incompatibility with the hash-based action cache implementation
## Changes
### 1. max-parallel Implementation (`pkg/runner/runner.go`)
#### Robust Initialization
Added fallback logic to ensure `MaxParallel` is always properly initialized:
```go
if job.Strategy.MaxParallel == 0 {
job.Strategy.MaxParallel = job.Strategy.GetMaxParallel()
}
```
#### Eliminated Unnecessary Nesting
Fixed inefficient nested parallelization when only one pipeline element exists:
```go
if len(pipeline) == 1 {
// Execute directly without additional wrapper
log.Debugf("Single pipeline element, executing directly")
return pipeline[0](ctx)
}
```
#### Enhanced Logging
Added comprehensive debug and info logging:
- Shows which `maxParallel` value is being used
- Logs adjustments based on matrix size
- Reports final parallelization decisions
### 2. Worker Logging (`pkg/common/executor.go`)
Enhanced `NewParallelExecutor` with detailed worker activity logging:
```go
log.Infof("NewParallelExecutor: Creating %d workers for %d executors", parallel, len(executors))
for i := 0; i < parallel; i++ {
go func(workerID int, work <-chan Executor, errs chan<- error) {
log.Debugf("Worker %d started", workerID)
taskCount := 0
for executor := range work {
taskCount++
log.Debugf("Worker %d executing task %d", workerID, taskCount)
errs <- executor(ctx)
}
log.Debugf("Worker %d finished (%d tasks executed)", workerID, taskCount)
}(i, work, errs)
}
```
**Benefits:**
- Easy verification of correct parallelization
- Better debugging capabilities
- Clear visibility into worker activity
### 3. Documentation (`pkg/model/workflow.go`)
Added clarifying comment to ensure strategy values are always set:
```go
// Always set these values, even if there's an error later
j.Strategy.FailFast = j.Strategy.GetFailFast()
j.Strategy.MaxParallel = j.Strategy.GetMaxParallel()
```
### 4. Test Fixes (`pkg/runner/step_action_remote_test.go`)
#### Added Missing Configuration
All tests now include `ActionCacheDir`:
```go
Config: &Config{
GitHubInstance: "github.com",
ActionCacheDir: "/tmp/test-cache",
}
```
#### Hash-Based Suffix Matchers
Updated mocks to use hash-based paths instead of fixed strings:
```go
// Before
sarm.On("readAction", sar.Step, suffixMatcher("org-repo-path@ref"), ...)
// After
sarm.On("readAction", sar.Step, suffixMatcher(sar.Step.UsesHash()), ...)
```
#### Flexible Exec Matcher for Post Tests
Implemented flexible path matching for hash-based action directories:
```go
execMatcher := mock.MatchedBy(func(args []string) bool {
if len(args) != 2 {
return false
}
return args[0] == "node" && strings.Contains(args[1], "post.js")
})
```
#### Token Test Improvements
- Uses unique cache directory to force cloning
- Validates URL redirection to github.com
- Accepts realistic token behavior
### 5. New Tests
#### Unit Tests (`pkg/runner/max_parallel_test.go`)
Tests various `max-parallel` configurations:
- `max-parallel: 1` → Sequential execution
- `max-parallel: 2` → Max 2 parallel jobs
- `max-parallel: 4` (default) → Max 4 parallel jobs
- `max-parallel: 10` → Max 10 parallel jobs
#### Concurrency Test (`pkg/common/executor_max_parallel_test.go`)
Verifies that `max-parallel: 2` actually limits concurrent execution using atomic counters.
## Expected Behavior
### Before
- ❌ Jobs ran in parallel regardless of `max-parallel` setting
- ❌ Unnecessary nested parallelization (8 workers for 1 element)
- ❌ No logging to debug parallelization issues
- ❌ All `step_action_remote_test.go` tests failing
### After
- ✅ `max-parallel: 1` → Jobs run strictly sequentially
- ✅ `max-parallel: N` → Maximum N jobs run simultaneously
- ✅ Efficient single-level parallelization for matrix jobs
- ✅ Comprehensive logging for debugging
- ✅ All tests passing (10/10)
## Example Log Output
With `max-parallel: 2` and 6 matrix jobs:
```
[DEBUG] Using job.Strategy.MaxParallel: 2
[INFO] Running job with maxParallel=2 for 6 matrix combinations
[DEBUG] Single pipeline element, executing directly
[INFO] NewParallelExecutor: Creating 2 workers for 6 executors
[DEBUG] Worker 0 started
[DEBUG] Worker 1 started
[DEBUG] Worker 0 executing task 1
[DEBUG] Worker 1 executing task 1
...
[DEBUG] Worker 0 finished (3 tasks executed)
[DEBUG] Worker 1 finished (3 tasks executed)
```
## Test Results
All tests pass successfully:
```
✅ TestStepActionRemote (3 sub-tests)
✅ TestStepActionRemotePre
✅ TestStepActionRemotePreThroughAction
✅ TestStepActionRemotePreThroughActionToken
✅ TestStepActionRemotePost (4 sub-tests)
✅ TestMaxParallelStrategy
✅ TestMaxParallel2Quick
Total: 12/12 tests passing
```
## Breaking Changes
None. This PR is fully backward compatible. All changes improve existing behavior without altering the API.
## Impact
- ✅ Fixes resource management for CI/CD pipelines
- ✅ Prevents runner exhaustion on limited infrastructure
- ✅ Enables sequential execution for resource-intensive jobs
- ✅ Improves debugging with detailed logging
- ✅ Ensures test suite reliability
## Files Modified
### Core Implementation
- `pkg/runner/runner.go` - max-parallel fix + logging
- `pkg/common/executor.go` - Worker logging
- `pkg/model/workflow.go` - Documentation
### Tests
- `pkg/runner/step_action_remote_test.go` - Fixed all 10 tests
- `pkg/runner/max_parallel_test.go` - **NEW** - Unit tests
- `pkg/common/executor_max_parallel_test.go` - **NEW** - Concurrency test
## Verification
### Manual Testing
```bash
# Build
go build -o dist/act main.go
# Test with max-parallel: 2
./dist/act -W test-max-parallel-2.yml -v
# Expected: Only 2 jobs run simultaneously
```
### Automated Testing
```bash
# Run all tests
go test ./pkg/runner -run TestStepActionRemote -v
go test ./pkg/runner -run TestMaxParallel -v
go test ./pkg/common -run TestMaxParallel -v
```
## Related Issues
Fixes issues where matrix jobs in Gitea ignored the `max-parallel` strategy setting, causing resource contention and unpredictable behavior.
Reviewed-on: https://gitea.com/gitea/act/pulls/150
Reviewed-by: Lunny Xiao <xiaolunwen@gmail.com>
Reviewed-by: silverwind <silverwind@noreply.gitea.com>
Co-authored-by: Pascal Zimmermann <pascal.zimmermann@theiotstudio.com>
Co-committed-by: Pascal Zimmermann <pascal.zimmermann@theiotstudio.com>
281 lines
7.1 KiB
Go
281 lines
7.1 KiB
Go
package common
|
|
|
|
import (
|
|
"context"
|
|
"sync"
|
|
"sync/atomic"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/stretchr/testify/assert"
|
|
)
|
|
|
|
// TestMaxParallelJobExecution tests actual job execution with max-parallel
|
|
func TestMaxParallelJobExecution(t *testing.T) {
|
|
t.Run("MaxParallel=1 Sequential", func(t *testing.T) {
|
|
var currentRunning int32
|
|
var maxConcurrent int32
|
|
var executionOrder []int
|
|
var mu sync.Mutex
|
|
|
|
executors := make([]Executor, 5)
|
|
for i := 0; i < 5; i++ {
|
|
taskID := i
|
|
executors[i] = func(ctx context.Context) error {
|
|
current := atomic.AddInt32(¤tRunning, 1)
|
|
|
|
// Track max concurrent
|
|
for {
|
|
max := atomic.LoadInt32(&maxConcurrent)
|
|
if current <= max || atomic.CompareAndSwapInt32(&maxConcurrent, max, current) {
|
|
break
|
|
}
|
|
}
|
|
|
|
mu.Lock()
|
|
executionOrder = append(executionOrder, taskID)
|
|
mu.Unlock()
|
|
|
|
time.Sleep(10 * time.Millisecond)
|
|
atomic.AddInt32(¤tRunning, -1)
|
|
return nil
|
|
}
|
|
}
|
|
|
|
ctx := context.Background()
|
|
err := NewParallelExecutor(1, executors...)(ctx)
|
|
assert.NoError(t, err)
|
|
|
|
assert.Equal(t, int32(1), maxConcurrent, "Should never exceed 1 concurrent execution")
|
|
assert.Len(t, executionOrder, 5, "All tasks should execute")
|
|
})
|
|
|
|
t.Run("MaxParallel=3 Limited", func(t *testing.T) {
|
|
var currentRunning int32
|
|
var maxConcurrent int32
|
|
|
|
executors := make([]Executor, 10)
|
|
for i := 0; i < 10; i++ {
|
|
executors[i] = func(ctx context.Context) error {
|
|
current := atomic.AddInt32(¤tRunning, 1)
|
|
|
|
for {
|
|
max := atomic.LoadInt32(&maxConcurrent)
|
|
if current <= max || atomic.CompareAndSwapInt32(&maxConcurrent, max, current) {
|
|
break
|
|
}
|
|
}
|
|
|
|
time.Sleep(20 * time.Millisecond)
|
|
atomic.AddInt32(¤tRunning, -1)
|
|
return nil
|
|
}
|
|
}
|
|
|
|
ctx := context.Background()
|
|
err := NewParallelExecutor(3, executors...)(ctx)
|
|
assert.NoError(t, err)
|
|
|
|
assert.LessOrEqual(t, int(maxConcurrent), 3, "Should never exceed 3 concurrent executions")
|
|
assert.GreaterOrEqual(t, int(maxConcurrent), 1, "Should have at least 1 concurrent execution")
|
|
})
|
|
|
|
t.Run("MaxParallel=0 Uses1Worker", func(t *testing.T) {
|
|
var maxConcurrent int32
|
|
var currentRunning int32
|
|
|
|
executors := make([]Executor, 5)
|
|
for i := 0; i < 5; i++ {
|
|
executors[i] = func(ctx context.Context) error {
|
|
current := atomic.AddInt32(¤tRunning, 1)
|
|
|
|
for {
|
|
max := atomic.LoadInt32(&maxConcurrent)
|
|
if current <= max || atomic.CompareAndSwapInt32(&maxConcurrent, max, current) {
|
|
break
|
|
}
|
|
}
|
|
|
|
time.Sleep(10 * time.Millisecond)
|
|
atomic.AddInt32(¤tRunning, -1)
|
|
return nil
|
|
}
|
|
}
|
|
|
|
ctx := context.Background()
|
|
// When maxParallel is 0 or negative, it defaults to 1
|
|
err := NewParallelExecutor(0, executors...)(ctx)
|
|
assert.NoError(t, err)
|
|
|
|
assert.Equal(t, int32(1), maxConcurrent, "Should use 1 worker when max-parallel is 0")
|
|
})
|
|
}
|
|
|
|
// TestMaxParallelWithErrors tests error handling with max-parallel
|
|
func TestMaxParallelWithErrors(t *testing.T) {
|
|
t.Run("OneTaskFailsOthersContinue", func(t *testing.T) {
|
|
var successCount int32
|
|
|
|
executors := make([]Executor, 5)
|
|
for i := 0; i < 5; i++ {
|
|
taskID := i
|
|
executors[i] = func(ctx context.Context) error {
|
|
if taskID == 2 {
|
|
return assert.AnError
|
|
}
|
|
atomic.AddInt32(&successCount, 1)
|
|
return nil
|
|
}
|
|
}
|
|
|
|
ctx := context.Background()
|
|
err := NewParallelExecutor(2, executors...)(ctx)
|
|
|
|
// Should return the error from task 2
|
|
assert.Error(t, err)
|
|
|
|
// Other tasks should still execute
|
|
assert.Equal(t, int32(4), successCount, "4 tasks should succeed")
|
|
})
|
|
|
|
t.Run("ContextCancellation", func(t *testing.T) {
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
|
|
var startedCount int32
|
|
executors := make([]Executor, 10)
|
|
for i := 0; i < 10; i++ {
|
|
executors[i] = func(ctx context.Context) error {
|
|
atomic.AddInt32(&startedCount, 1)
|
|
time.Sleep(100 * time.Millisecond)
|
|
return nil
|
|
}
|
|
}
|
|
|
|
// Cancel after a short delay
|
|
go func() {
|
|
time.Sleep(30 * time.Millisecond)
|
|
cancel()
|
|
}()
|
|
|
|
err := NewParallelExecutor(3, executors...)(ctx)
|
|
assert.Error(t, err)
|
|
assert.ErrorIs(t, err, context.Canceled)
|
|
|
|
// Not all tasks should start due to cancellation (but timing may vary)
|
|
// Just verify cancellation occurred
|
|
t.Logf("Started %d tasks before cancellation", startedCount)
|
|
})
|
|
}
|
|
|
|
// TestMaxParallelPerformance tests performance characteristics
|
|
func TestMaxParallelPerformance(t *testing.T) {
|
|
if testing.Short() {
|
|
t.Skip("Skipping performance test in short mode")
|
|
}
|
|
|
|
t.Run("ParallelFasterThanSequential", func(t *testing.T) {
|
|
executors := make([]Executor, 10)
|
|
for i := 0; i < 10; i++ {
|
|
executors[i] = func(ctx context.Context) error {
|
|
time.Sleep(50 * time.Millisecond)
|
|
return nil
|
|
}
|
|
}
|
|
|
|
ctx := context.Background()
|
|
|
|
// Sequential (max-parallel=1)
|
|
start := time.Now()
|
|
err := NewParallelExecutor(1, executors...)(ctx)
|
|
sequentialDuration := time.Since(start)
|
|
assert.NoError(t, err)
|
|
|
|
// Parallel (max-parallel=5)
|
|
start = time.Now()
|
|
err = NewParallelExecutor(5, executors...)(ctx)
|
|
parallelDuration := time.Since(start)
|
|
assert.NoError(t, err)
|
|
|
|
// Parallel should be significantly faster
|
|
assert.Less(t, parallelDuration, sequentialDuration/2,
|
|
"Parallel execution should be at least 2x faster")
|
|
})
|
|
|
|
t.Run("OptimalWorkerCount", func(t *testing.T) {
|
|
executors := make([]Executor, 20)
|
|
for i := 0; i < 20; i++ {
|
|
executors[i] = func(ctx context.Context) error {
|
|
time.Sleep(10 * time.Millisecond)
|
|
return nil
|
|
}
|
|
}
|
|
|
|
ctx := context.Background()
|
|
|
|
// Test with different worker counts
|
|
workerCounts := []int{1, 2, 5, 10, 20}
|
|
durations := make(map[int]time.Duration)
|
|
|
|
for _, count := range workerCounts {
|
|
start := time.Now()
|
|
err := NewParallelExecutor(count, executors...)(ctx)
|
|
durations[count] = time.Since(start)
|
|
assert.NoError(t, err)
|
|
}
|
|
|
|
// More workers should generally be faster (up to a point)
|
|
assert.Less(t, durations[5], durations[1], "5 workers should be faster than 1")
|
|
assert.Less(t, durations[10], durations[2], "10 workers should be faster than 2")
|
|
})
|
|
}
|
|
|
|
// TestMaxParallelResourceSharing tests resource sharing scenarios
|
|
func TestMaxParallelResourceSharing(t *testing.T) {
|
|
t.Run("SharedResourceWithMutex", func(t *testing.T) {
|
|
var sharedCounter int
|
|
var mu sync.Mutex
|
|
|
|
executors := make([]Executor, 100)
|
|
for i := 0; i < 100; i++ {
|
|
executors[i] = func(ctx context.Context) error {
|
|
mu.Lock()
|
|
sharedCounter++
|
|
mu.Unlock()
|
|
return nil
|
|
}
|
|
}
|
|
|
|
ctx := context.Background()
|
|
err := NewParallelExecutor(10, executors...)(ctx)
|
|
assert.NoError(t, err)
|
|
|
|
assert.Equal(t, 100, sharedCounter, "All tasks should increment counter")
|
|
})
|
|
|
|
t.Run("ChannelCommunication", func(t *testing.T) {
|
|
resultChan := make(chan int, 50)
|
|
|
|
executors := make([]Executor, 50)
|
|
for i := 0; i < 50; i++ {
|
|
taskID := i
|
|
executors[i] = func(ctx context.Context) error {
|
|
resultChan <- taskID
|
|
return nil
|
|
}
|
|
}
|
|
|
|
ctx := context.Background()
|
|
err := NewParallelExecutor(5, executors...)(ctx)
|
|
assert.NoError(t, err)
|
|
|
|
close(resultChan)
|
|
|
|
results := make(map[int]bool)
|
|
for result := range resultChan {
|
|
results[result] = true
|
|
}
|
|
|
|
assert.Len(t, results, 50, "All task IDs should be received")
|
|
})
|
|
}
|