Best Practices for Integrating with Files.com

The Files.com platform provides built-in capabilities to connect to external systems, including built-in connectors to external storage solutions, identity solutions, and notification services. The platform also provides built-in workflow tasks, such as Automations, Remote Syncs, and Folder Settings, that can be used to automate encryption and data movement between systems. We also provide multiple ways for you to integrate with us.

We have always been developer friendly and have many customers that integrate with our platform in a variety of ways.

We have SDKs, RESTful APIs, and a Command Line Integration (CLI) App available for use. Each of these provides different methods for integration, allowing you to choose the methods that best fit your needs.

We also have customers who use iPaaS platforms, such as Boomi and MuleSoft, to integrate with Files.com using standard API and SFTP connectors from those platforms.

In this article we provide some best practices to consider when designing your integration processes with Files.com.

Whenever there are multiple possible methods available to integrate by, we recommend the following order of preference, from most recommended to least recommended: SDKs, CLI, APIs, and Protocols.

We provide SDKs for 7 programming languages; Go, Java, JavaScript, Microsoft .NET, PHP, Python, and Ruby.

An SDK is our most recommended choice for customers working with any of the supported programming languages.

Our SDKs use our secure APIs and include developer-friendly features such as built-in automatic retries, fast performance, automatic pagination handling, and automatic transfer chunking where available. The SDKs provide you with the tightest integration between our APIs and your programming language of choice.

We have invested time into optimizing file operations, including parallelization of uploads and downloads, in some SDKs.

As we add new platform features, our SDKs are automatically updated and optimized to support those features.

The Command Line Integration (CLI) App is cross-platform and available for Windows, Linux, and Mac.

The Command Line Integration (CLI) App is our most recommended choice for customers requiring integration from a scripting language that doesn't already have a Files.com SDK available for it, such as PowerShell, shell scripts (bash, csh, zsh, etc.), and DOS batch files (BAT).

The Command Line Integration (CLI) App also uses our secure APIs and includes features such as built-in automatic retries, fast performance, automatic pagination handling, and automatic transfer chunking. The CLI App provides you with the tightest integration between our APIs and your scripting language of choice.

Output format can be specified (json, csv, or table) for easier post-processing within your script.

As we add new platform features, the CLI App is automatically updated and optimized to support those features.

We provide RESTful web service APIs. These can be invoked from any HTTP client, including those contained within any programming language, programming script, or iPaaS platform.

The APIs are our recommended choice for customers requiring integration from a system which cannot use our SDKs or CLI App.

Using these "raw" APIs provide you with micro services which you'll need to build your own additional logic around to handle response outcomes, retries, and for exception handling.

As we add new platform features, the APIs are automatically updated to support those features.

Our platform provides support for file transfer protocols, including FTP(S), SFTP, and WebDAV. These protocols provide the ability to upload, download, move, copy, rename, and delete files and folders. These file and folder actions can be programmatically executed from batch command scripts for the protocol or from external scripts.

The protocols are our recommended choice for customers who need to transfer files and folders from a system that cannot use our SDKs, CLI App, or APIs, such as from embedded systems with built-in file transfer protocol support.

The protocols are our least recommended choice for performing programmatic move, copy, and rename operations.

These protocols were designed prior to cloud based platforms and assume that all operations are executed synchronously. They are unaware of asynchronous background operations and cannot provide accurate responses about the current state of a file or folder that has an asynchronous operation being performed on it.

The Files.com platform performs some operations asynchronously and these operations will run in the background in parallel.

For example, moving a file from one Files.com storage region to another, or from one Remote Server Mount to another, will asynchronously create a background process to move the file. Control is immediately returned even though the move is still occurring in the background.

Whenever you are programmatically integrating with Files.com, such as from a script or iPaaS process, make sure to account for asynchronous operations.

Our SDKs, CLI App, and APIs will provide responses that indicate when an asynchronous operation occurs and provides methods which you can use to query the current status, and final outcome, of an asynchronous operation.

Integrating via a protocol will not provide any indication about whether an operation is asynchronous. Make sure to implement checks and validations in your script, or process, to verify that an operation has completed prior to any further processing steps.

Our SDKs and CLI App include built-in retry logic. Some protocol clients provide built-in retries for uploads and downloads but this is typically implemented for interactive user sessions rather than for programmatic scripts.

Our SDKs, CLI App, and APIs will provide responses that indicate if an action fails which you can use in your programs, scripts, and processes to trigger your own retry logic.

Protocol clients may, or may not, provide exits codes that indicate the outcome of batch commands.

Bear in mind that too many retries, performed too quickly, will look like a cyberattack to network security devices. We recommend retry logic that introduces increasing durations between retry attempts.

The Files.com platform, as a Software-as-a-Service (SaaS) cloud-based solution, provides elastic scalability which allows for a practically unlimited number of parallel connections.

When using parallel threads or processes to interact with Files.com, take into consideration that operations may be asynchronous and that other processes, or users, may be interacting with files and folders at the same time.

Operations performed on the Files.com platform will take an identical amount of time no matter how the operation was triggered.

For example, moving a file from one folder to another will take the same amount of time no matter whether the copy operation was started via SDK, API, CLI App, Automation, Folder Setting, FTP, SFTP, WebDAV, or web portal.

There are caveats to the above example. Some protocol clients, for example, will "copy" a file by downloading it from its source folder and re-uploading it to its destination folder. In these kinds of scenario, the operation will take longer due to the round-trip route from server to client and back again.

Moves and copies from one Files.com storage region to another, or from one Remote Server Mount to another, will always take longer than an identical operation whose source and destination reside within the same storage region or storage location.

For uploads and downloads, the fastest transfer method is via HTTPS using our SDKs, CLI App, APIs, and web portal.

Our SDKs, CLI App, and web portal will automatically apply parallelism and chunking to transfers to give you the best possible transfer performance.

When using our APIs, your application, program, or script will need to manage any parallelism and chunking.

Protocol based transfers are slower due to the underlying protocol itself. Various factors affect the transmission speed of a protocol including its encryption methods and how "chatty" it is. A chatty protocol is one that requires a lot of back-and-forth acknowledgement to be sent between the client and the server. For example, even though WebDAV uses the HTTPS protocol, it is a very chatty protocol and is very affected by network latency.

Traditionally, on-premise SFTP has been a slower transmission protocol than FTP(S), due to its encryption methods and its use of a single data transmission port. However, cloud based SFTP benefits from SaaS elasticity and is often faster than FTP(S).

WebDAV is a convenient connection protocol for desktop users, as it's built into almost every Windows, Mac, and Linux desktop, but only performs well over a local area network (LAN). Over the internet, WebDAV is a slow transmission protocol due to its network chattiness.

Whenever there are multiple possible file transfer options available, we recommend the following order for transfer speeds, from fastest to slowest:

1. SDKs, CLI App, APIs, or web portal
2. SFTP or FTP(S)
3. WebDAV

We're developer-friendly and here to help. Please feel free to contact us with any questions.

Our Customer Success team is available to assist you with guidance and recommendations on the best ways to integrate your business processes with Files.com.