Skip to content

feat: introduce scheduler-based engine with goroutine-per-node execution#803

Merged
wolo-lab merged 2 commits into
wolo/workflowsfrom
wolo/workflow_engine
May 13, 2026
Merged

feat: introduce scheduler-based engine with goroutine-per-node execution#803
wolo-lab merged 2 commits into
wolo/workflowsfrom
wolo/workflow_engine

Conversation

@wolo-lab
Copy link
Copy Markdown

@wolo-lab wolo-lab commented May 7, 2026

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

2. Or, if no issue exists, describe the change:

Problem:
The legacy in-process BFS dispatcher in Workflow.Run cannot run sibling nodes
concurrently, lacks per-node timeouts and cancellation, and provides no place
to hang HITL pauses, retries, or dynamic children — all of which are upcoming
workflow milestones.

Solution:
Replace the BFS with a goroutine-per-node scheduler. Each scheduled node runs
in its own goroutine, pushes events into a buffered channel, and a single
consumer goroutine drains the channel, applies state-side effects, yields
events to the caller, and schedules successors when nodes complete.

Sibling cancellation on first error, per-node Config().Timeout, and
caller-break draining (break outside the range loop stops further
scheduling, not just yielding) are included. Producer-consumer model means no
mutexes are needed for scheduler state; race detector is clean.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

workflow/scheduler_test.go covers linear chains, fan-out concurrency
(barrier-based, verifies parallelism), sibling cancellation on first error,
caller-break stops scheduling (regression), per-node timeout, and the
single-output / single-routing-event accumulator contract.

go test -race ./workflow/...               # ok 1.24s
go test ./...                              # all packages pass
go vet ./...                               # clean
go build ./...                             # clean

Manual End-to-End (E2E) Tests:

  • examples/workflow/basic runs end-to-end on the new engine.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

@wolo-lab wolo-lab force-pushed the wolo/workflow_engine branch from 8f0e518 to 2dad321 Compare May 8, 2026 12:35
@wolo-lab wolo-lab force-pushed the wolo/workflow_engine branch 25 times, most recently from a674860 to 4c2fba9 Compare May 12, 2026 10:41
…ecution

Squashes the workflow engine WIP work onto the latest wolo/workflows
base. The branch had 8 incremental commits; this single commit
captures the cumulative state on top of upstream PR #795 (EdgeBuilder),
PR #796 (NodeConfig), and PR #797 (node name validation).

Architecture:
  * Goroutine-per-node execution model: each scheduled node runs in
    its own goroutine, pushing events into a buffered channel.
  * Single consumer goroutine (runState.run) drains the channel,
    applies state-side effects, yields events to the caller, and
    schedules successors when nodes complete.
  * Replaces the legacy in-process BFS in Workflow.Run with the new
    scheduler, removing findNextNodes and the inline event loop.

New types and constructors:
  * BaseNode (and NewBaseNode) for shared Name/Description/Config
    bookkeeping; FunctionNode and toolNode now embed it.
  * Graph helpers in graph.go: indexed adjacency for O(1) successor
    lookup.
  * RunState (persistable) and runState (in-process scheduler bag)
    in state.go; node lifecycle map (NodeStatus + per-node accumulators).
  * NodeContext wraps InvocationContext with a per-node TriggeredBy
    accessor; agent.InvocationContext gains TriggeredBy() returning
    "" for non-workflow contexts (mocks updated accordingly).
  * Scheduler (scheduler.go): runNode goroutine wrapper, eventQueue,
    cancelAll, and the run consumer loop with sibling cancellation
    and per-node timeout.

Routing and validation:
  * findSuccessors honours unconditional edges, concrete Routes, and
    the Default fallback. Silent dead-ends remain intentional per
    adk-python parity.
  * Workflow.New now returns (*Workflow, error) — picks up the name
    validation introduced upstream by PR #797.

NodeConfig timeout shape:
  * Timeout is time.Duration (not *time.Duration), with zero meaning
    "inherit parent context". Matches net.Dialer.Timeout and the
    http.Server.*Timeout convention; keeps call sites free of pointer
    boilerplate.
  * Adds RerunOnResumeOr / WaitForOutputOr accessor helpers for
    pointer-typed pointer-typed tri-state fields.
  * Adds TestDefaultRetryConfig from upstream PR #796.

Examples and tests:
  * examples/workflow/basic uses NodeConfig{RetryConfig: DefaultRetryConfig()}
    to demo the helper.
  * 7 New(edges) callers updated for the new (*Workflow, error)
    signature.

Tests verified: go build ./..., go vet ./..., go test -race
./workflow/... ./agent/workflowagent/... all pass.
@wolo-lab wolo-lab force-pushed the wolo/workflow_engine branch from 4c2fba9 to 0481149 Compare May 12, 2026 10:57
@wolo-lab wolo-lab changed the title Wolo/workflow engine feat: introduce scheduler-based engine with goroutine-per-node execution May 12, 2026
@wolo-lab wolo-lab self-assigned this May 12, 2026
@wolo-lab wolo-lab requested review from anFatum and hanorik May 12, 2026 11:17
@wolo-lab wolo-lab marked this pull request as ready for review May 12, 2026 11:18
Comment thread workflow/config.go Outdated
Comment thread workflow/scheduler.go Outdated
@wolo-lab wolo-lab merged commit bf32409 into wolo/workflows May 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants