Distributing and Scaling Jobs
Split a file or folder coming from an upstream node into many parallel jobs, so the destination node runs once per line, per file, or per batch.
On this page7
Overview
When a node receives a file or folder from an upstream node, you can distribute that input so the destination node runs as many parallel jobs instead of a single job. Each job receives one slice of the input (a line, a file, or a batch of lines), and the platform runs the jobs in parallel across the run's machines.
Distribution is configured per node from the destination node's modal. The available modes depend on the input type and the destination's parameter type.
Where Distribution Applies
Distribution is available when:
- An upstream node produces file or folder output, and
- That output is connected to an input on the destination node, and
- The destination node has at least one input that can accept the distributed form.
When these conditions hold, a Distribute button is available on the destination node's modal.
If they do not hold, opening the dialog shows "No inputs available to distribute."
If an upstream node is already configured to distribute the same data, the dialog blocks new configuration with "Already distributed by an upstream node."
The Three Modes
Distribution has three user-facing modes. Which one applies depends on the type of the upstream output and the type of the destination input:
| Mode | When it's offered | What each job receives |
|---|---|---|
| One job per line | Upstream is a file; destination has a string parameter | One line of the file as a string value |
| One job per batch | Upstream is a file; destination has a file parameter | A chunk of the file (lines grouped into roughly equal batches) |
| One job per file | Upstream is a folder; destination has a file or folder parameter | One file from the folder |
When the input is a file and both One job per line and One job per batch apply, the dialog asks you to pick. In all other cases the mode is selected automatically.
Note Folder distribution always runs in the "one job per file" form. There is no separate folder-batch mode.
Opening the Distribute Dialog
Double-click the destination node on the canvas to open its modal.
Click Distribute in the modal action bar. The Distribute dialog opens.
Filling Out the Wizard
The dialog has up to three steps: Input, How to split, and Parameter. Steps with only one valid choice are resolved automatically and shown as a read-only summary with an Auto pill. Steps with more than one option always require a selection.
Pick which connected input to distribute. The input picker is a dropdown listing every distributable connection on the node, including multiple connections that feed the same input port (each entry shows the source node name). If only one input is distributable, this step is auto-resolved.
Pick a mode (One job per line, One job per batch, or One job per file). The two file modes are presented as two cards separated by an OR divider, since the choice is binary. Modes that do not apply to the chosen input are not offered. If only one mode applies, this step is auto-resolved.
Only shown when the mode is One job per line. Pick the string parameter on the destination node that will receive each line. If the destination has only one matching string parameter, this step is auto-resolved.
After you pick an input, a Preview appears at the bottom of the dialog showing the first few lines from the source so you can sanity-check what will be split.
When all steps are complete, an outcome summary explains what will happen (for example, "your file input will fan out one line per job, spawning one parallel job per line").
Click Distribute to apply, or Cancel to close without changes.
After Distribution
Once distribution is applied, the destination node is marked as distributed on the canvas. When the workflow runs, the platform creates one job per slice of the input and runs them in parallel up to the run's machine cap (see Resource Allocation on Managed Fleets).
In the run view, each distributed task shows up as its own row with its own inputs, console output, and produced files.
To remove distribution, open the destination node's modal and use the undistribute action.