Job lifecycle
| Status | Description |
|---|---|
PENDING | Job queued, waiting for a worker to claim it |
RUNNING | Worker is actively executing steps |
PAUSED | Waiting for a wait step or scheduled continuation |
RESUMING | Resuming from a checkpoint after a pause or recovery |
COMPLETED | All steps finished successfully |
FAILED | A step failed after exhausting retries |
CANCELLED | Manually cancelled |
Worker architecture
Job claiming
Workers claim jobs atomically from the queue using a claim epoch. If a job is reclaimed (e.g. after a heartbeat timeout), the original worker detects the epoch mismatch and stops processing. This prevents duplicate execution in multi-worker deployments.Heartbeats
Workers send heartbeats while processing jobs. If a worker stops sending heartbeats (crash, network failure) after the specified timeout, the job becomes eligible for another worker to claim and resume from the last checkpoint.Concurrency
Workflow execution has multiple levels of concurrency:- Jobs per worker — each worker processes multiple jobs concurrently. Configured via the max concurrency setting in workflow settings.
- Parallel steps per job — independent steps within a job execute simultaneously. Configured via the max parallel steps setting in workflow settings.
- forEach parallelism — iterations within a forEach step can execute concurrently. Configured per step in the forEach configuration.
Guarantees
| Guarantee | Description |
|---|---|
| At-least-once execution | Each step executes at least once. If a worker crashes after executing a step but before checkpointing the result, the step may re-execute on recovery. Use idempotency patterns for operations with real-world consequences. |
| No concurrent execution | The claim epoch mechanism prevents multiple workers from making progress on the same job simultaneously. |
| Resumable progress | Checkpointing at step, page, and item levels means recovery continues from where it left off, not from the beginning. |
| Data isolation | Each job run uses a unique table prefix, preventing cross-job data interference. |
Data processing & storage
DuckDB
Workflow steps execute on DuckDB, a high-performance columnar database. DuckDB runs in-process on the worker and handles SQL transformations, bulk data transfer to and from the app database, and intermediate storage operations. No external database connection is required for data processing. Step resource limits scale with worker configuration — see configuration for details.Step results
Step outputs are stored as Parquet files in the app’s configured workflow storage (i.e. S3 on AWS) or on the local filesystem during development. Each job run’s data is logically isolated from other concurrent runs. Step results are available for querying and debugging, and are automatically cleaned up after the configured TTL (default 48 hours).Monitoring
Track workflow execution through the platform’s monitoring tools:- Job status: Current state, start time, duration, step progress
- Step details: Per-step execution time, row counts, errors
- Logs: Structured logs for each step execution
- History: Past runs with outcomes and timing