mirror of
https://gitea.com/gitea/act_runner.git
synced 2026-04-24 21:00:27 +08:00
feat: Add support for Dynamic Matrix Strategies with Job Outputs (#149)
# Matrix Strategy: Support for Scalar Values and Template Expressions ## 🎯 Overview This Pull Request implements support for **scalar values and template expressions** in matrix strategies. Previously, every matrix value had to be declared as a YAML array — a bare scalar like `os: ubuntu-latest` or `version: ${{ fromJSON(...) }}` caused silent failures. Now, scalars are automatically wrapped into single-element arrays, enabling the `${{ fromJSON(...) }}` pattern that is commonly used in GitHub Actions workflows for dynamic matrix generation. ## ⚠️ Semantic Changes Two subtle behavior changes are introduced that are worth understanding: ### 1. Scalar matrix values now produce a 1-element matrix expansion **Before this PR:** A scalar value like ```yaml strategy: matrix: os: ubuntu-latest # no brackets ``` failed to decode as `map[string][]interface{}`, so `Matrix()` returned `nil`. The job ran **once, without any matrix context** (`matrix.os` was undefined). **After this PR:** The same input is wrapped to `["ubuntu-latest"]`. The job runs **one matrix iteration** with `matrix.os = "ubuntu-latest"` set. > **Impact on existing workflows:** Workflows that accidentally used bare scalars instead of arrays will now run with a matrix context where they previously did not. The functional result is usually the same, but `matrix.*` variables are now populated. This is considered an improvement, but is a semantic change. --- ### 2. Unevaluated template expressions fall back to a literal string value The actual resolution of `${{ fromJSON(needs.job1.outputs.versions) }}` into an array happens **before** `Matrix()` is called — in `EvaluateYamlNode` inside `pkg/runner/runner.go`. This PR does not change that evaluation path. If evaluation succeeds → the YAML node already contains a proper array → `Matrix()` decodes it normally. If evaluation **fails or is skipped** (e.g., expression not yet resolved) → the scalar string is now wrapped as `["${{ fromJSON(...) }}"]` → the job runs **one iteration with the literal unexpanded string** as the matrix value. > **Note:** The test `TestJobMatrix/matrix_with_template_expression` covers this fallback path, not the successful resolution path. The literal-wrap behavior is the fallback, not the primary mechanism for template expression support. --- ## 🔑 Key Changes ### 1. Flexible Matrix Decoding (`pkg/model/workflow.go`) #### New Helper Function: `normalizeMatrixValue` ```go func normalizeMatrixValue(key string, val interface{}) ([]interface{}, error) ``` - Converts a single matrix value to the expected array format - Handles three cases: 1. **Arrays**: Pass through unchanged 2. **Scalars**: Wrap in single-element array (including template expressions) 3. **Invalid types** (maps, etc.): Return error to detect misconfigurations early **Supported scalar types:** - `string` — including unevaluated template expressions - `int`, `float64` — numeric values - `bool` — boolean values - `nil` — null values #### Enhanced `Matrix()` Method **Before:** ```go func (j *Job) Matrix() map[string][]interface{} { if j.Strategy.RawMatrix.Kind == yaml.MappingNode { var val map[string][]interface{} if !decodeNode(j.Strategy.RawMatrix, &val) { return nil } return val } return nil } ``` **After:** ```go func (j *Job) Matrix() map[string][]interface{} { if j.Strategy == nil || j.Strategy.RawMatrix.Kind != yaml.MappingNode { return nil } // First decode to flexible map[string]interface{} to handle scalars and arrays var flexVal map[string]interface{} err := j.Strategy.RawMatrix.Decode(&flexVal) if err != nil { // If that fails, try the strict array format var val map[string][]interface{} if !decodeNode(j.Strategy.RawMatrix, &val) { return nil } return val } // Convert flexible format to expected format with validation val := make(map[string][]interface{}) for k, v := range flexVal { normalized, err := normalizeMatrixValue(k, v) if err != nil { log.Errorf("matrix validation error: %v", err) return nil } val[k] = normalized } return val } ``` **Key improvements:** - Handles unevaluated template expressions as scalar strings - Automatic conversion to array format for consistent processing - Validation to catch nested maps and invalid configurations - Fallback to strict array format decoding for backward compatibility - Early nil check for strategy to prevent nil pointer panics --- ## 🎁 Use Cases ### Example 1: Template Expression in Matrix (primary motivation) ```yaml jobs: setup: runs-on: ubuntu-latest outputs: versions: ${{ steps.version-map.outputs.versions }} steps: - id: version-map run: echo "versions=[\"1.17\",\"1.18\",\"1.19\"]" >> $GITHUB_OUTPUT test: needs: setup strategy: matrix: # EvaluateYamlNode resolves this to an array before Matrix() is called version: ${{ fromJSON(needs.setup.outputs.versions) }} runs-on: ubuntu-latest steps: - run: echo "Testing version ${{ matrix.version }}" ``` ### Example 2: Mixed Scalar and Array Values ```yaml jobs: build: strategy: matrix: os: [ubuntu-latest, windows-latest] # Array — unchanged go-version: 1.19 # Scalar — wrapped to [1.19] node: [14, 16, 18] # Array — unchanged runs-on: ${{ matrix.os }} ``` ### Example 3: Single Value Matrix ```yaml jobs: deploy: strategy: matrix: environment: production # Scalar — wrapped to ["production"] runs-on: ubuntu-latest ``` --- ## ✅ Benefits - ✨ **Feature Parity with GitHub Actions**: Supports template expressions in matrix strategies - 🔒 **Robust Validation**: Detects misconfigured matrices (nested maps) early - 🔄 **Backward Compatible**: Existing array-based workflows continue to work - 📝 **Clear Error Messages**: Helpful validation messages for debugging - 🛡️ **Type Safety**: Handles all YAML scalar types correctly --- ## 🚀 Breaking Changes **Technically none in a strictly backward-incompatible sense**, but two semantic changes exist (see [⚠️ Semantic Changes](#️-semantic-changes-read-before-merging) above): 1. Scalar matrix values now iterate (previously they caused a `nil` matrix / no iteration) 2. Unevaluated template strings now produce a literal iteration instead of skipping Both changes are improvements in practice, but downstream users should be aware. --- ## 📚 Related Work - Related Gitea Server PR: [go-gitea/gitea#36564](https://github.com/go-gitea/gitea/pull/36564) - Enables dynamic matrix generation from job outputs - Supports advanced CI/CD patterns used in GitHub Actions ## 🔗 References - [GitHub Actions: Matrix strategies](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs) - [GitHub Actions: Expression syntax](https://docs.github.com/en/actions/learn-github-actions/expressions) - [GitHub Actions: `fromJSON` in matrix context](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix) Reviewed-on: https://gitea.com/gitea/act/pulls/149 Reviewed-by: silverwind <2021+silverwind@noreply.gitea.com> Co-authored-by: Pascal Zimmermann <pascal.zimmermann@theiotstudio.com> Co-committed-by: Pascal Zimmermann <pascal.zimmermann@theiotstudio.com>
This commit is contained in:
committed by
silverwind
parent
f923badec7
commit
15dd63a839
@@ -377,16 +377,61 @@ func (j *Job) Environment() map[string]string {
|
||||
return environment(j.Env)
|
||||
}
|
||||
|
||||
// Matrix decodes RawMatrix YAML node
|
||||
// normalizeMatrixValue converts a matrix value to []interface{}.
|
||||
// Arrays pass through unchanged; scalars are wrapped in a single-element array.
|
||||
// Unevaluated template expressions are wrapped as a fallback — proper resolution
|
||||
// happens via EvaluateYamlNode before Matrix() is called. Nested maps are rejected.
|
||||
func normalizeMatrixValue(key string, val any) ([]any, error) {
|
||||
switch t := val.(type) {
|
||||
case []any:
|
||||
// Already an array - use as-is
|
||||
return t, nil
|
||||
case string, int, float64, bool, nil:
|
||||
// Valid scalar types that can appear in YAML
|
||||
// These can be unevaluated template expressions (strings) or literal values
|
||||
return []any{t}, nil
|
||||
case map[string]any:
|
||||
// Nested map indicates misconfiguration - likely user error
|
||||
return nil, fmt.Errorf("matrix key %q has invalid nested object value - expected scalar or array, got map", key)
|
||||
default:
|
||||
// Unknown types might indicate parsing issues
|
||||
log.Warnf("matrix key %q has unexpected type %T, wrapping as single value", key, t)
|
||||
return []any{t}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Matrix decodes the RawMatrix YAML node into a map[string][]interface{}.
|
||||
// Scalar values are wrapped into single-element arrays automatically.
|
||||
// Template expressions are resolved by EvaluateYamlNode before this method is
|
||||
// called; if unresolved, the literal string is wrapped as a one-element fallback.
|
||||
func (j *Job) Matrix() map[string][]any {
|
||||
if j.Strategy.RawMatrix.Kind == yaml.MappingNode {
|
||||
if j.Strategy == nil || j.Strategy.RawMatrix.Kind != yaml.MappingNode {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Decode to flexible map first so that scalar values don't cause a type error.
|
||||
var flexVal map[string]any
|
||||
err := j.Strategy.RawMatrix.Decode(&flexVal)
|
||||
if err != nil {
|
||||
// Fall back to the strict array-only format for backward compatibility.
|
||||
var val map[string][]any
|
||||
if !decodeNode(j.Strategy.RawMatrix, &val) {
|
||||
return nil
|
||||
}
|
||||
return val
|
||||
}
|
||||
return nil
|
||||
|
||||
// Convert flexible format to expected format with validation
|
||||
val := make(map[string][]any)
|
||||
for k, v := range flexVal {
|
||||
normalized, err := normalizeMatrixValue(k, v)
|
||||
if err != nil {
|
||||
log.Errorf("matrix validation error: %v", err)
|
||||
return nil
|
||||
}
|
||||
val[k] = normalized
|
||||
}
|
||||
return val
|
||||
}
|
||||
|
||||
// GetMatrixes returns the matrix cross product
|
||||
|
||||
Reference in New Issue
Block a user