7 Mar 2018

The Hitchhiker's Guide to ... Reality Capture API

Real to virtual

A couple of months ago, the Reality Capture API came out “silently” from “by invitation only” beta and made available to all.

In what follows, I will try to present tips and tricks when working with Reality Capture API and this should not be considered a replacement of the nice documentation, but merely a “The Hitchhiker’s Guide to” Reality Capture API, aiming to complement the documentation with observations and some undocumented features that might help in your endeavour to use the Reality Capture API.

The use of the Reality Capture API is pretty straightforward and can be seen as a 4-step workflow:

  1. Create a photoscene;
  2. Upload images to the created photoscene;
  3. Start the scene processing;
  4. Get results;

There are other steps in between, like getting scene processing progress, cancelling the scene progress, deleting the scene etc., but the main workflow is given by those 4 steps, upon which we will take a closer look:

1. Create a photoscene.

One thing must be clear when creating a photoscene - after a photoscene is created with a given parameter, the parameters cannot be changed. 

Knowing this, take great care when defining the parameters, as the poor choice of paramters, especially the format and the scenetypeparameters, will be noticed only at step 4 or while inspecting the final result and at that step it is too late to change something, so you will have to start over again.

The format parameter (rcm, rcs, obj etc.) is important because what format you will choose at step 1, will define what you can get at step 4. Another important aspect is the processing speed, which will depend on number and type of desired formats. More formats might require more time to process.
The scenetype parameter (aerial, object) will affect the algorithm used for the scene processing and consequently the result. Thus, in case you are receiving a somehow skewed result, most probably your forgot to properly set this very parameter, for which the default choice is aerial.

Don't forget the scenetype

Example of results with aerial vs object scenetype parameter, having the same set of images:

scenetype set to aerial


scenetype set to object

Another important aspect is that any further calls to a photoscene is made using the photosceneid. Thus, if the photosceneid is lost, you will have hard times to get it, as the Reality Capture API.

3documentation doesn’t specify a way of inspecting what scenes were created using tokens obtained with a certain pair of Forge Secrets - in other words, list all scenes I created with my Client ID and Client secret.
Nevertheless, there is an unsupported way to retrieve this information by accessing: 

GET https://developer.api.autodesk.com/photo-to-3d/v1/photoscene/foruser

curl like call would look like:

export TOKEN="put-your-token-here"

curl -X GET https://developer.api.autodesk.com/photo-to-3d/v1/photoscene/foruser \
  -H 'Authorization: Bearer '$TOKEN

You should expect an output like the following:

listing the scenes


2. Upload images to created photoscene

The input to photogrammetry is photographs, thus the quality of the photographs directly affect not only the quality of the result, but the availability of the results too.
If after your photoscene was processed you get results in form of several Kb zip file, most probably the engine could not make since out of your images. 

The reasons could vary:

  • blurry/out-of-focus images; 
  • shots made with too large aperture, making hard to identify the edges of the object;
  • too dark images,
  • too few images etc.

In general, any recommendations that were given for ReCap Photo app are more than applicable if you want good results from Reality Capture API. After all, the ReCap Photo app is using the very same Reality Capture API under the hood.

The “too few images” reason is the one that might be double tricky. There might be not enought photos not only because you uploaded too few, but also there might have been problems with uploading some of the images.
Reality Capture API allows two ways of adding images to a scene: by uploading a local file, or by specifying a link to a remote server. The task for uploading remote located files might have been accepted, but this doesn’t mean that there could not be problems with accessing some of them (timeouts, permissions etc.).

The Reality Capture API documentation doesn’t specify a way of inspecting a scene for uploaded images. However, as in case with listing scenes, there exists an unsupported/undocumented way of doing it by accessing the following endpoint:

GET https://developer.api.autodesk.com/photo-to-3d/v1/photoscene/:photosceneid/properties

curl like call would look like:

export TOKEN="put-your-token-here"
export PHOTOSCENEID=Imq4somephotosceneidJb0

curl -X GET https://developer.api.autodesk.com/photo-to-3d/v1/photoscene/$PHOTOSCENEID/properties \
  -H 'Authorization: Bearer '$TOKEN

The given results will help you double check what you want to upload versus what was really uploaded into a photoscene and you should get a lot of interesting information there:

Listing scene content


Also, keep in mind before uploading the images, that the processing cost depends on processed Megapixels. Thus, in case of an object processing, instead of uploading full images of an object and its surrounding, consider cropping the image to contain only the needed object. This might increase the processing speed, lower your cost and sometimes even improve the result.

Worth mentioning another aspect related to the order of uploading images, that might seem obvious to you, but I’ve noticed that some you had doubts. The order of uploading images does not affecting the final result.
Roughly speaking, the engine will “stitch” the images based on common pixels/regions/patches. Thus, don’t worry about uploading your images in the right order. If you missed some images, just add the missing ones to the scene, the order will not impact the quality of result.

3. Start the scene processing.

After a scene is created and images were uploaded there is not much left to do but to start the scene processing. One thing that is sometimes skipping to some developers is that Reality Capture API offers two ways of knowing when the photoscene finished processing:

  1. The “classic” polling of a dedicated endpoint GET photoscene/:photosceneid/progress to get the status of the scene and the processing progress. 

  2. The “webhook approach”, available as a callback parameter when creating the photoscene. This parameter can be an url or an email, on which you will receive notification upon photoscene processing completion. Obviously, as mentioned before, you should take care of the correctness of this parameter too, as there is no way of changing once the photoscene is created.

4. Get results.

Once the photoscene has status “Done”, the results can be requested by specifying the needed format. At this step, as mentioned before, if you haven’t specified the (now) needed format, you are out of luck and there is no other solution, but to start over again.

Thus, after the scene was processed, there is nothing to be done, but to ask for results and delete the scene. One important observation here is that according to documentation “The link will expire 7 days after the date of processing completion”, thus you can safely delete the scene, as it will not affect the “result links”.

In end, I would like to point to idea that the best way of learning an API is to write a client for it in one of your favorite languages, and I know that some of you already do that. In my case, the language of choice was Go and the unofficial library for Reality Capture API and others can be found at github.com/apprentice3d/forge-api-go-client.
For a very simple illustration, both for the mentioned Go library and for Reality Capture API in general, I would suggest to have a look at reality.capture-go-cli.sample

Nevertheless, if you don’t care writing a client for Reality Capture API and just want to easily check it, or you are writing one and need a way to double check/debug your client, I highly recommend you have a look at my RECAP.postman_collection, for which you obviously will need Postman and where you will find all official and unofficial requests:

good old postman

Related Article