Resource handling
In order to make use of the Flow Graph Engine APIs, some limited resource handling is required. In particular, jobs typically need access to one or more input resources to operate on, and typically produce some form of output resources as well as log files.
Inputs and outputs can be fairly large. Therefore, jobs submitted to the Job API only contains references to the input resources. It is expected that the data has been uploaded using the resource API prior to submitting jobs. Similarly, the Job API provides the list of outputs and logs produced by tasks and jobs, but let’s the user selectively transfer data as needed using the Resource API.
Outputs from the job API can be downloaded by using the Resource API. The API provide presigned URLs to resources, which can be used to download the data.
Furthermore, the resource API allows upload of data to user storage. This is done using one or more presigned URLs retrieved from the Resource API.
General resource handling flow
- Use the Resource API to upload data
- Add references to the uploaded data into a job submitted to the Job API
- Use the Job API to retrieve references to results or logs produced by the job
- Use the Resource API to resolve the references to the results or logs and download the data
Storage providers and storage spaces
The Resource API does not store data by itself, instead this is delegated to different storage providers. A storage provider is responsible for storing the data and enforce access permissions. Support for other storage providers might be added in the future.
Each storage provider may have different storage spaces for resources to be stored in, where each storage space may have different access permissions. Within each storage space, each unique resource has a unique key or resource ID.
The resource API is used to retrieve presigned urls from references to resources. Theses references identify the provider, space, and resource ID. The presigned urls can be used to upload or download resources.
Different storage providers may have different permissions for users. For instance, the provider that allows users to download results of jobs does not allow for user data upload.
Current storage providers include
- Outputs - this storage space is used to handle outputs and logs produced by jobs. Only read access is allowed, and only users that have read permissions for a particular job have access to read the outputs and logs from that job. Resources are automatically deleted after a certain period of time.
- Scratch - this is a minimalistic storage space that provides temporary personal storage. There is a single logical space per user or app and resources are automatically deleted after a certain period of time.
Storage space ID
Multiple endpoints in the Job and Resource API may call for the usage of a storage space ID. A storage space ID indicate which storage provider and which storage space to use. It should be formatted like this:
<storage provider>:<storage space>
For example, scratch:@default would use the default storage space on the scratch storage provider.
Resource references
Resources can be referred to by using URNs that encode the storage provider, storage space, and resource ID. The provider, space, and ID can also be used directly. The URN of a resource can be found by querying the resource API or found in the resource API response when uploading a resource.
The general format for a resource reference URN is:
urn:adsk:<service>:<environment>:resource:<entity ID>
where:
- <service> is a short string that identifies the storage provider (and the service) - fc.outputs: Outputs provider string - fc.scratch: Scratch provider string
- environment is in the form {region}[.{server}] - region is us or another valid region - server is dev`|`stg`|`prd or another string representing a separate deployment
- resource is the entity type
- <entity ID> is in the form <storage space>/<resource ID>
Please see Submitting a job for an example of how to use references when submitting a job.
Uploading and downloading resources
More detailed information on how to transfer resources to and from the system can be found in the Uploading and downloading resources guide.
Resource rentention policy
To learn more on the retention policy consult the data retention policy section of the documentation.