Summary

Queues hold FlowFiles between processors. Backpressure controls how much work can accumulate in a queue before Clockspring slows or stops upstream processing. Understanding both is essential for building flows that run reliably under load.

What a queue is

A queue is automatically created when you connect a processor’s relationship to another component. It acts as a buffer that stores FlowFiles waiting to be processed.

A queue exists only when there is a connection:

Processor A (success) → Processor B

This creates a queue between A and B.

Queues allow:

• Flow decoupling between components

• Different processors to run at different speeds

• Controlled buffering during spikes

• Safe recovery after outages (FlowFiles are checkpointed at every queue)

Multiple connections create multiple queues

If a relationship has multiple connections:

Processor A (success)   
↳ Processor B   
↳ Processor C

Each connection has its own independent queue:

• Queue A→B

• Queue A→C

One incoming FlowFile routed to success is duplicated into both queues.

Inspecting a queue

Click on a queue to see:

• FlowFile count

• Total size

• Oldest FlowFile

• Latest processor activity

• Prioritizer (if any)

• Whether backpressure is active

This view is essential during troubleshooting and load analysis.

Emptying queues

Clockspring gives two ways to clear queued data safely.

1. Empty a single queue

Right-click the queue → Empty Queue

Removes all FlowFiles in that queue only.

2. Empty all queues in a Process Group

Right-click the process group background → Empty all queues

This clears:

• All queues in the current process group

• All queues in child groups

• All relationships connected within the group

This is extremely helpful during development or after a flow failure.

Backpressure controls queue size

Every connection has configurable backpressure thresholds:

• Maximum object count

• Maximum data size

When either limit is exceeded, the queue becomes full.

What happens when a queue is backpressured

When a queue is full:

1. The upstream processor sees the queue is unable to accept more FlowFiles

2. The upstream processor stops running

3. Upstream queues may fill

4. Flow slows or halts until the downstream processor drains the queue

Backpressure prevents Clockspring from overwhelming downstream systems or exhausting memory.

Backpressure is a flow-control mechanism, not an error

Many users think backpressure means the flow is failing. It does not.

Backpressure means:

• Clockspring is protecting itself

• Downstream processing is slower than upstream production

• The flow is behaving correctly under load

You adjust thresholds to match real workloads.

Configuring backpressure

Right-click the connection → Configure

You can edit:

• Max FlowFiles in queue (object count)

• Max queue size (in bytes)

• Prioritizers (optional)

Best practices:

• Start with reasonable defaults

• Increase thresholds for high-throughput flows

• Decrease thresholds to detect system bottlenecks sooner

Backpressure and processor scheduling

Backpressure interacts with processor scheduling:

• If downstream is slow, upstream processors stop triggering

• If downstream is fast, upstream runs freely

• Backpressure prevents infinite buildup

• Yielding or penalization may also interact with how quickly the queue drains

Queues and scheduling together define flow performance.

When queues grow unexpectedly

Common causes:

• Downstream processor stopped

• Processor is running but failing/penalizing repeatedly

• External system latency (DB/api slow)

• Record processing slower than ingestion

• Misconfigured backpressure

• Dynamic route added with no connection (upstream stalls)

Queue inspection helps isolate which component is the bottleneck.

Quick rules to remember

• Every connection creates a queue.

• Each connection has its own backpressure limits.

• Backpressure affects upstream processors immediately.

• You can empty a single queue or all queues in a process group.

• Growing queues indicate downstream slowness, not upstream failure.

• Duplicated FlowFiles produce multiple queues with identical items.

Related Articles

• Relationships: What They Are and How They Work

• Managing Relationship Connections

• Retries, Yielding, and Penalization

• Terminating Relationships