Summary

Processors do not always run continuously. When errors occur, they may retry work, yield, or penalize individual FlowFiles. These behaviors directly affect queue drain rates, processor scheduling, and overall flow performance. Understanding how these mechanics work is essential for diagnosing stalled queues and designing reliable retry logic.

1. Yielding (Processor-Level Pause)

When a processor yields, it stops running for a defined duration. This is a processor-level delay, not a FlowFile delay.

When processors yield

• API or DB connection failures

• External system unavailable

• Exceptions unrelated to a specific FlowFile

• Processor code explicitly calls yield

What yielding does

• Processor does not pull FlowFiles from its incoming queue

• Incoming queue may grow or appear stuck

• Yield ends automatically after the configured duration

Impact

A yielded processor makes its inbound queue appear frozen even though nothing is wrong with the queue. Yielding protects external systems from being hammered.

2. Penalization (FlowFile-Level Delay)

Penalization delays a specific FlowFile. The FlowFile is moved to the back of the queue and cannot be processed again until the penalty expires.

When processors penalize

• Bad or malformed FlowFile

• Missing values needed for the operation

• Temporary FlowFile-specific errors

What penalization does

• FlowFile waits in the queue

• Other FlowFiles continue processing

• Queue drains more slowly when many FlowFiles are penalized

Impact

Penalization prevents tight error loops where the same FlowFile is retried immediately and repeatedly.

3. Processor-Level Retry Engine (The Retry Checkbox System)

Every relationship in a processor's configuration has two options:

• terminate

• retry

Selecting retry activates Clockspring’s built-in retry engine.

How processor-level retry actually works

If a FlowFile would normally be routed to a relationship that has retry enabled:

1. The FlowFile is not routed yet

2. The processor retries the operation internally

3. Retry attempts follow exponential backoff

4. The FlowFile remains in the processor’s incoming queue

5. If any retry attempt succeeds → FlowFile goes to success or another normal relationship

6. Only after retry attempts are exhausted does the FlowFile get routed to that relationship

This applies to any relationship, including the retry relationship.

Common misunderstanding

Checking retry on the retry relationship does not send FlowFiles there immediately.
It still performs internal retries first.

4. Retry Backoff Configuration (Right-Side Panel)

When the retry checkbox is selected, this panel appears:

Number of Retry Attempts

Defines how many additional attempts will be made after the first failure.

Retry Backoff Policy

Controls how delays occur between retries:

Retry Maximum Backoff Period

Caps the exponential backoff duration.

After backoff reaches this value, all remaining retries use this maximum.

5. Processor-Level Penalty and Yield Durations (Settings Tab)

Each processor has base timing settings under the Settings tab:

• Penalty Duration

• Yield Duration

These values are often overlooked but extremely important.

These settings define the base timing for:

• FlowFile penalization

• Processor yielding

• Retry exponential backoff calculations

How they interact with retries

If retry backoff is set to penalize, Clockspring multiplies the Penalty Duration:

Example: 30 sec penalty
→ Retry delays will be 30s → 60s → 120s → 240s → capped by max backoff

If retry backoff is set to yield, Clockspring multiplies the Yield Duration:

Example: 1 sec yield
→ Retry delays will be 1s → 2s → 4s → 8s → capped by max backoff

Defaults vs overrides

• Clockspring defines system-wide defaults

• Each processor can override them

• If a processor behaves “too slow” or “too fast,” check these values first

6. How Exponential Backoff Works

Clockspring doubles the delay before each retry:

• 1x (base duration)

• 2x

• 4x

• 8x

• … until max backoff reached

Once the max is reached, the wait time stays constant for remaining retries.

This prevents overwhelming external systems.

7. How Retry, Yield, and Penalization Affect Queue Draining

Yield → queue does not drain

Processor is paused → no FlowFiles pulled → queue appears stuck.

Penalized FlowFiles → queue drains slowly

The processor will mark a single FlowFile as penalized and will not attempt to process that FlowFile again until the penalty duration lapses.   Other FlowFiles in the queue will continue to process.

Backpressure + yield → cascading stall

A very common pattern:

1. Downstream processor fails

2. It retries internally with backoff

3. It stops consuming from its inbound queue

4. Queue fills → backpressure triggers

5. Upstream processor stops

6. Entire flow appears frozen

Root cause is retry behavior, not the queue itself.

8. When to Adjust Retry Settings

Increase retry attempts when

• Downstream systems recover reliably

• Errors are likely temporary

Decrease retry attempts when

• You prefer explicit error handling

• Long retries cause upstream backpressure

Adjust backoff when

• Penalize = FlowFile waits

• Yield = processor waits

9. Quick Rules to Remember

• Yield = processor pause

• Penalize = FlowFile pause

• Retry checkbox = internal retry engine

• FlowFiles only get routed to retry/failure after retries are exhausted

• Penalty/Yield durations come from the processor’s Settings tab

• Exponential backoff doubles each attempt until capped

• Retry behavior explains most “stuck” queue situations

Related Articles

• Queues and Backpressure

• Managing Relationship Connections

• Terminating Relationships

• FlowFile Content vs Attributes