Data Management OSS (Object Storage Service) migrating to Direct-to-S3 approach

As part of our commitment to innovate and improve customer services, Autodesk released a new set of Data Management API endpoints to allow direct upload and download from the underlying AWS S3 bucket. The primary impact of this direct-to-S3 approach is performance.

Our testing and validation continue to show a significant increase in upload and download speeds, especially for large files. 

How? When an object is uploaded or downloaded from an OSS bucket, Autodesk uses AWS S3 storage.

The current binary transfer approach moves the data through our proxy, while the new direct-to-S3 approach enables you to directly access the underlying S3 bucket and object.   

 Direct binary streaming data management endpoints to be deprecated December 31, 2022 

 Autodesk will deprecate the current binary transfer streaming endpoints approach of uploading and downloading files directly to and from our proxy (developer.api.autodesk.com) on December 31, 2022, and such transfer will no longer be allowed after that date.

You will be required to move your existing application(s) utilizing impacted Autodesk APIs to the new direct-to-S3 approach by December 31, 2022. 

See below for current information on impacted API endpoints and stay tuned for additional details to be released soon. 

The following APIs are affected: 

• Data Management – Buckets & Objects: The following endpoints will be removed: PUT object, PUT resumable, and GET object. Applications should move to signeds3upload and signeds3download, which is explained in detail in the next section (“Direct-to-S3 approach for Data Management OSS”). 
   
• Model Derivative: The following endpoints will be removed: GET Derivatives Applications should move to the new endpoint GET Fetch Derivative Download
   
• Viewer: If you are using any version prior to V7.68 you will need Add or change the endpoint option when initializing the Viewer. No changes to other options are required:

var options = {
    endpoint: 'https://cdn.derivative.autodesk.com'
    // keep other options unchanged
}; 

• BIM 360 Docs PDF Export -  GET export endpoint is updated to provide additional S3 signed url, (i.e., signedUrl field in the response). We can call this signed url to download the actual content. The current field, link, still functions at a moment for compatibility reason during migration period. Read more about it here 

The following APIs are not affected, but your application may have the following cross dependencies: 

• Design Automation: OSS is commonly used as storage for the Design Automation workitems input and output. Design Automation now improved the way to support the input/output url with the format like "urn:adsk.objects:os.object:<bucket_name>/<object_key>", developers are not required to generate the signed S3 url, what developer needs to prepare is just preparing the objectId and the access token, everything else will be automatically handled by Design Automation, and this new improvement is supported by all the Design Automation engines including Revit, 3ds Max, Inventor and AutoCAD. Read more about it here.

• Autodesk Construction Cloud & BIM360: OSS is commonly used as storage for these services. No change is happening with them. As explained referred to the steps below.

• Data Management – Projects, Folders & Files: Applications that upload and download files will need to use signed S3 URLs.  

• Download: Step 6 will change and require generating a signed S3 URL, then download using it. 
• Upload: Step 4 will require 3 actions: generate signed S3 URL for upload, upload using the S3 URL, then call upload complete. 
• BIM 360/Autodesk Build Cost API - As far as Cost API itself is concerned, there is no change in Cost API. This update is about instruction for upload/download workflows that have dependency to Data Management API, which is affected by the change. If you use Data Management API to upload/download files in Docs, you are most likely aware about the change already.  Read more about it here   

The following APIs are NOT affected: 

• Authentication 
• Data Visualization 
• Token Flex 
• Webhooks 
• Assets 

To upload and download files, applications must generate a signed URL, then upload or download the binary. Here are the steps (pseudo code):   

Upload  

1. Calculate the number of parts of the file to upload  

   1. Note: Each uploaded part except for the last one must be at least 5MB  (1024 * 5)

2. Generate up to 25 URLs for uploading specific parts of the file using the GET buckets/:bucketKey/objects/:objectKey/signeds3upload?firstPart=<index of first part>&parts=<number of parts> endpoint  

   1. The part numbers start with 1  

   2. For example, to generate upload URLs for parts 10 through 15, set firstPart to 10 and parts to 6  

   3. This endpoint also returns an uploadKey that is used later to request additional URLs or to finalize the upload  

3. Upload remaining parts of the file to their corresponding upload URLs  

   1. Consider retrying (for example, with an exponential backoff) individual uploads when the response code is 100-199, 429, or 500-599  

   2. If the response code is 403, the upload URLs have expired; go back to step #2  

   3. If you have used up all the upload URLs and there are still parts that must be uploaded, go back to step #2  

4. Finalize the upload using the POST buckets/:bucketKey/objects/:objectKey/signeds3upload endpoint, using the uploadKey value from step #2    

Download  

1. Generate a download URL using the GET buckets/:bucketKey/objects/:objectName/signeds3download endpoint  

2. Use the new URL to download the OSS object directly from AWS S3  

   1. Consider retrying (for example, with an exponential backoff) the download when the response code is 100-199, 429, or 500-599  

The signed URLs have a default expiration time of 2 minutes (*longer expiration times can be set using the minutesExpiration param up to 60 minutes). It is important that the download or upload process starts before the expiration. In case of single or multi-part upload, you may request signed URLs again. A set of steps for uploading and downloading can be found in our Step-by-Step Documentation. 

In our next series of blog posts, we will be sharing with all of you a couple of samples the Developer Advocates are currently working on, keep an eye for the Node.js, .Netcore and .NET ones coming very soon.

Thank you again for all your business and please feel free to reach out to us through forge.help@autodesk.com. 

*DISCLAIMER When generating signed URLs, it’s important to use the smallest possible expiration time, in order to avoid longer access in case of exposure of the URL.