Parallelism, Scheduling and Sync Speeds

You can configure the maximum number of connections to a Remote Server in its settings.

Files will run one sync job at a time per configured sync. Sync jobs run until they are complete. The larger the folder being synced, the longer sync jobs will take to complete. In practice, the majority of sync jobs complete very quickly (on the order of seconds), but we have some customers whose sync jobs take hours.

A sync job runs one control process, which runs in a single thread (i.e. not parallelized) to determine which files need to be transferred. The actual file transfers are parallelized to achieve faster sync throughput.

Said another way, you can expect that we will send multiple file upload and download requests at once, but we will not send multiple folder list requests at once.

The overall time that a sync job takes depends on how many files and folders are being inspected, the quantity of files requiring transfer, the size of the files, and the transfer speed of the remote server. Consumer-oriented remote servers, such as Dropbox and Google Drive, implement stricter rate limiting which results in slower syncs. Enterprise-oriented remote servers such as Azure, AWS, and GCP tend to have better throughput capabilities and therefore complete much more quickly.

Remote servers that run FTP or SFTP have performance that is widely variable depending on who is operating the remote and what software it runs. Where possible, we recommend using FTPS instead of SFTP, as it is usually many times faster than SFTP.