2 legged workflow for model translation - part 2

Follow @XiaodongLiang

In the last post, I introduced my practice to make a whole workflow of 2legged with  AWS Lambda and API Gateway. This is the logic:

1. A Lambda function generates S3 signed uploading URL, sends back to client. The client uploads local file to S3.
2. The uploading to S3 will trigger another Lambda function. This function downloads the file from S3 to the space of Lambda.
3. Next, the Lambda codes invokes the process of Forge, upload file to Forge, and request translating. 
4. The last function of Lambda is to check the translating status, and return it to the client.

The complete project and test steps are available at: https://github.com/xiaodongliang/Forge-workflow-Lambda-APIGateway

In this article, I am sharing some key points regarding with #3 and #4. 

As said last time, the section with Forge is based on Forge SDK . Since it is a process by promise, we only need to throw the callback of Lambda to tell the status, either translating has started successfully, or something has failed in the middle way.

//send file to Forge and request to translate
function callForgeProcess(s3FilePath,s3FileName,callback)
{
	/**
	 * Create an access token and run the API calls.
	 */
	oAuth2TwoLegged.authenticate().then(function(credentials){
		
			console.log("**** Got Credentials",credentials);
		
			createBucketIfNotExist(BUCKET_KEY).then(
		
				function(createBucketRes){ 
		
					uploadFile(BUCKET_KEY, s3FilePath, s3FileName).then(function(uploadRes){
 						
						console.log("**** Upload file response:", uploadRes.body); 
						//delete the local file in Lambda space
                        fs.unlink(s3FilePath); 
                         
						var objectId = uploadRes.body.objectId;  
						
						translateFile(objectId).then(function(transRes){
                            console.log("**** Translate file response:", transRes.body);
                            callback(null,transRes.body) ; 
							
						}, defaultHandleError);
		
					}, defaultHandleError);
		
				}, defaultHandleError);
		
		}, defaultHandleError);
  }

When the uploading to S3 triggers Lambda function, the code will send the file to Forge. However, it always failed without any response from Forge Data Management : {{base_domain}}/oss/v2/buckets/{{bucketKey}}/objects/{{objectKey}}. It took me much time until I realized it was just because of timeout. Why? my test file is only 3M...Finally, I got it. 

Lambda function is designed to perform frequently batch jobs. It has default timeout for each HTTP response (3 seconds). So, if one call exceeds the threshold, it will exit. With this tip, I simply increased the timeout to 90 seconds. Then it worked now. Of course, if it is a large file, we should use resumable uploading of Forge, instead of directly uploading, otherwise, it would also hit the timeout.

It is relatively simpler to create the API function for checking translating status. Again, to let Lambda identify this is a request about checking status, the query string sets a special flag. On Lambda side, it will get token of Forge and invoke the call of checking status of manifest of a URN.

function checkForgeTransStatus(urn,callback)
{
	  //get token
	  oAuth2TwoLegged.authenticate().then(function(credentials){
		  //get translating status of Forge
		 derivativesApi.getManifest(urn,{},
               oAuth2TwoLegged, oAuth2TwoLegged.getCredentials())
						.then(function(transStRes){
								//callback for Lambda 
										callback(null, transStRes);
				});
		
	  }, defaultHandleError); 

}

After testing Lambda and API Gateway by CIL, we can deploy API Gateway for test in production. AWS provides us with the option to export to Postman script. Then we can import the collection to Postman.

If IAM is set, you will need to input the parameters of AWS Signature. In my current test, I skipped IAM, just for a simple demo.

Firstly, get S3 signed URL:

click the URL, select 'PUT' method, input the objectKey (the file name), select the local file, and Send. verify if the file has been uploaded to S3 bucket.

Finally, call the endpoint of checking translating status, input the based 64 encoded URN:

I will try to find time to make a web page application to consume the APIs. With the sample of the other blog, we can create complete scenario of 2legged by AWS Lambda and API Gateway.