Downloading and uploading resources
The resource API allows users to download and upload resources through the use of presigned URLs. Presigned URLs have a specified lifetime, after which, they expire and can no longer be used to transfer data.
If expiration happens prior to the transfer beginning, a new URL has to be retrieved but if a URL expires in the middle of a transfer that transfer will continue.
Downloading resources
When downloading a resource:
- Retrieve a presigned download URL from the resource API, using the get download URL endpoint
- Use the provided presigned URL, optionally using ranges to download individual parts of the resource.
The presigned download URL can be used to retrieve a specific portion of a resource using the Range HTTP header to specify which bytes to retrieve. The same URL can be used for any range in the resource so only a single URL is needed even when doing multipart downloads.
It is also possible to request download URLs in batches for multiple resources within one storage space using the batch get download URLs, which is recommended to use when possible.
Note: Storage providers may set a maximum number number of URLs that can be retrieved in one request, please see the API method documentation.
Uploading a resource
When uploading a resource, the API client must determine how many parts they want to transfer the resource in. Each individual part is uploaded using a specific upload URL.
When uploading a resource:
- Ask the resource API for one or more upload URLs corresponding to the parts of the resource to upload. In addition to the upload URLs, the resource API responds with an upload ID. This is done using the get upload URLs endpoint.
- Call the provided presigned URL(s) and send the corresponding parts of the resource.
- When all the uploads for all the parts have finished, call the resource API to complete the upload, using the upload ID that was returned at the start of the transfer. This is done using the complete upload endpoint.
If individual presigned URLs expire before being used, or if more URLs are needed, the client can request more upload URLs using the upload ID that was given at the start of the transfer. For a large resource, multiple requests to retrieve upload URLs are typically needed.
It is also possible to retrieve upload URLs in batches, not only within one resource, but for multiple resources as well. This is done using the complete uploads endpoint, and similarly completion can be done using the batch complete uploads endpoint.
Uploads can be explicitly created using the create upload endpoint or batch create upload endpoint.
Multipart uploads and downloads
The use of multipart uploads and downloads is recommended when the size of the resource becomes larger. It can also be used to achieve a higher throughput, as well as better error recovery when transferring.
There is a limit on the number of parts that can be requested that may differ depending on the storage provider. For more information see the Flow Graph Engine APIs limits section