Optimizing the workflow and job performance

This article describes how to configure parallel task execution in Silverstack using volume performance settings. It focuses on practical configuration and observable execution behavior when offloading camera media, running cascading backups, and scheduling subsequent jobs such as verification, transcoding, or uploads.The content assumes familiarity with Silverstack’s workflow concept and targets users who need to adapt execution behavior to specific hardware characteristics.

1. Core concepts#

Silverstack executes workflows by scheduling jobs, which are split into file-level tasks. Tasks access one or more volumes for reading and writing. Task execution is controlled through volume performance settings that define how many tasks may access a volume concurrently.

Parallel execution can improve throughput when hardware resources allow it. However, excessive task concurrency can reduce performance or reliability due to resource contention, device limitations, or interface constraints. Appropriate settings depend on the characteristics of the involved source and destination volumes.

Terminology and definitions#

• Workflow: A configured process that schedules one or more jobs for a set of input files
• Job: A logical operation such as offload, backup, verification, or transcoding
• Task: A file-level execution unit belonging to a job
• Source volume: A volume that tasks read data from
• Destination volume: A volume that tasks write data to
• Reading task: A task that consumes read capacity on a volume
• Writing task: A task that consumes write capacity on a volume
• Concurrency: The number of tasks accessing a volume at the same time
• Volume performance limits: Settings that restrict concurrent reading and writing access per volume

The required settings for performance optimization are located at two places in Silverstack:

• The performance settings for each volume can be found via the jobs view
• The default max reading task count for new sources is located in the “Copy&Jobs” tab of Silverstack’s settings

It is important to note that the performance settings must be configured for each destination volume. This is why the workflow configuration window in Silverstack highlights unconfigured volumes and provides quick access to presets with common task configuration options, as shown below.

2. Offloading multiple cards in parallel#

2.1 Max writing#

To enable concurrent offloads, destination volumes must be enabled to receive data from more than one task at a time. For example, one task for the backup job of card A and one task for card B.In this case, you should set the “Max Writing” to the desired concurrency level of 2, or higher. If a backup job uses multiple destinations, each destination volume must accept this number of writing tasks, otherwise, the lowest value will limit the operation.

2.2 Max reading#

The “Max Reading” defines the concurrency degree when reading from a specific volume. It is intended to control parallel execution from destination drives (e.g., should a cascading backup from a RAID happen in parallel to a transcoding job?).

Be cautious with this setting for source cards. Trying to read multiple files in parallel from the same camera card, usually results in very bad throughput as it generates process overhead and can overwhelm the card reader – which could result in unmounting of the cards or failure due to reading errors. Additionally, each file read from the source can block a writing task on the destination, which may prevent multiple cards from being offloaded in parallel.

In most scenarios, reading only one file after the other from each camera card is the most reliable and fastest setting. We strongly recommend keeping the source cards’ max reading task count at 1, which is the default for new source cards in Silverstack. In the gallery below, you can see how different read and write task counts influence the copy process for a setup with two source cards and two destination volumes.

3. Cascading backups and other subsequent jobs#

A cascading backup uses one destination of the initial offload as the source for a subsequent backup. A common scenario is to offload camera cards to a fast RAID first, then create a backup from the RAID to slower shuttle drives. This approach prioritizes early release of camera cards while deferring slower operations. The most important performance setting for cascading backups is the “Read/Write” option.It defines whether reading and writing tasks may access a volume simultaneously:

• Force Exclusive Reading or Writing: This setting delays reading tasks until all writing tasks have finished. Jobs will run sequentially
• Parallel Reading and Writing: It permits interleaved execution. Reading tasks may start as soon as individual files become available

With exclusive access enabled, cascading backups start only after the initial offload completes. With parallel access enabled, cascading backups may start while the offload is still in progress, once individual offload tasks finish. The same mechanism specifies whether transcodes (or other subsequent jobs) are allowed to start running from the RAID while an offload is still accessing it.

HDDs typically perform best when either reading or writing exclusively. In contrast, SSDs and RAIDs are better in handling parallel reading and writing. But even for SSDs or RAIDs, you might want to consider the “Exclusive Reading or Writing” option: When connecting highly performant drives with your computer, the thunderbolt bus can become the bottleneck. In such cases, this option avoids frequent switching of the data flow direction on the bus, leading to better overall throughput.

4. Limits for verification, transcoding, and upload tasks#

Transcoding tasks

Transcoding tasks read from a source and write results to a destination. The writing component is not considered when evaluating volume writing limits, as it typically produces minimal load. Transcoding tasks are internally limited to one task at a time to ensure optimal transcoding speed.

Backup task types

When the option “Verification Included in Copy Job” is enabled, the backup task combines copy and verification into a single task. The task reads from the source and writes to the destination during the copy phase, then switches internally to verification (read only on all drives). Throughout the entire execution, the task occupies one reading slot on the source and one writing slot on each destination.When “Separate Verification Job” is enabled, Silverstack creates distinct copy and verification jobs/tasks. Copy tasks consume reading and writing capacity. Verification tasks consume reading capacity only. This generally allows more precise control over execution order and concurrency.

5. Job priorities#

In addition to volume limits and task-type constraints, job priority affects task scheduling. Higher-priority jobs are scheduled before lower-priority jobs when multiple eligible tasks are available. If multiple tasks meet all execution criteria and share the same priority, tasks are started in the order in which they were scheduled.

6. Conclusion: Get to know your hardware#

Finding the optimal concurrency depends on your individual hardware setup and personal priorities on what you want to get done first. Silverstack gives you granular control to tailor workflow execution to your requirements. By configuring each volume individually, it is now easier to combine fast and slow drives while managing fast throughput.

Concurrency should be increased in small increments while observing throughput and reliability over time. Hardware performance can change due to thermal throttling or depleted write buffers, particularly on SSD-based storage. For this reason, it’s best to adjust parallel task execution in small increments, monitoring both reliability and overall throughput over a longer period.