SceneBuilder Extension for Forge Viewer ... in Action

One of our mighty engineers pointed well that each time when somebody builds a model viewer, there will be always attempts to transform it into an editor. Forge Viewer is not an exception and we always get questions in this context, where the most frequent one is about adding custom geometries.

Adding custom geometry to the Viewer was addressed previously in some of our blog posts and there is even a recommended (read “official way”) of doing it.

It is not a secret that Forge Viewer is based on three.js and so far, all approaches of adding custom geometry to the Viewer, exploit three.js layer (a layer below Forge Viewer). This explains why Forge Viewer is not aware of existence of such geometry and there is no interaction with it. To be able to select a custom geometry added this way, you have to work at three.js level and implement selection by yourself and we illustrated in some of the blogposts how to solve this using ray casting.

Nevertheless, with all this work, any custom geometry still feels like a foster child and tools like Measuring or Sectioning will ignore their pretty face.

For this very reason, the engineering team developed SceneBuilder extension, which facilitates integration of custom geometry into Forge Viewer and make it play well with tools like Measuring and Sectioning.

There is an official Developer Guide on adding custom geometry using the scene builder and a comprehensive blog post named Custom models in Forge Viewer from my colleague Petr Broz, both being a good starting point with this relatively new and powerful Extension.

In this post I would like to put this knowledge in action and build something practical, with focus on particularities and some tips & tricks when using SceneBuilder extension and not only.

Thus, I invite you to start this quest with me and explore SceneBuilder features by going through building a customizable corner shelve, having as a starting point just a simple model of a board:

During this quest we will review and understand how to:

• create an empty custom model and add geometry to it;
• replace geometry and look at ways of changing geometry position in custom model;
• grab geometry from existing/imported model and copy it to custom model 

If you want to follow along, here is the simple Fusion360 model we will start with: CornerBoard. All my work I will do in an extension, so if you are not familiar how to create an extension with a docking panel, check first the Viewer extension tutorial.

Step I: Adding a side board as custom geometry

The most simplest way of adding a custom geometry is the following:

viewer.loadExtension("Autodesk.Viewing.SceneBuilder").then(() => {
    sceneBuilder = viewer.getExtension("Autodesk.Viewing.SceneBuilder");

    sceneBuilder.addNewModel({})
        .then((modelBuilder) => {
            let geom = new THREE.BufferGeometry().fromGeometry(
                                        new THREE.BoxGeometry(210,160,10));
            let phongMaterial = new THREE.MeshPhongMaterial({
                color: new THREE.Color(1, 0, 0)
            });
            mesh = new THREE.Mesh(geom, phongMaterial);
            modelBuilder.addMesh(mesh);

        });

Having our simple model, if we add the above code into a our SimpleCustomGeometry extension we will get a custom model set at position [0, 0, 0]: 

With just this, we already can start working with the new geometry, it is selectable using Measuring tool we check the offset of new custom geometry to original component, for further position adjustments:

From the measuring tool we see that we should move the newly added mesh to the right by 100 units (here I am not referring to Z-axis and Y-axis on purpose).

At this point, there are three approaches of adjusting the position of newly added custom geometry:

1. Set position before adding mesh to the model builder:

   
   ...
   mesh = new THREE.Mesh(geom, phongMaterial);
   mesh.matrix = new THREE.Matrix4().compose(
                   new THREE.Vector3(0, 0, -100),
                   new THREE.Quaternion(0, 0, 0, 1),
                   new THREE.Vector3(1, 1, 1)
       );
   modelBuilder.addMesh(mesh);
   
   ...

   If we apply this approach in our extension, we will get the following result: 

    

   

   and we still see that we should have moved up our mesh by 85 units, but let us do it using another approach.
    

2. Update the mesh position and then update the mesh:

   This approach is ok in small doses, but if you will try to build an animation using this approach, this is wrong way to do it. The idea is to to use ModelBuilder in-build method to replace the mesh and it works both, for changing mesh geometry, as well as changing mesh transforms:

   
       ...
       mesh = new THREE.Mesh(geom, phongMaterial);
       modelBuilder.addMesh(mesh);
   
       mesh.matrix.setPosition(new THREE.Vector3(0,85,-100));
       modelBuilder.updateMesh(mesh);
   
       ...
   

   using this new position adjustments, everything starts to look better:

   

   As mentioned above, this method is very useful for changing the geometry, all that is need is to be taken care of is the transforms for the new geometry:

   
   ...
   mesh = new THREE.Mesh(geom, phongMaterial);
   modelBuilder.addMesh(mesh);
   
   mesh.geometry = new THREE.BufferGeometry().fromGeometry(
                                   new THREE.CylinderGeometry( 105, 105, 10, 128 ));
   mesh.geometry.computeBoundingBox();
   mesh.matrix.makeRotationFromEuler(new THREE.Euler(Math.PI/2,0,0));
   mesh.matrix.setPosition(new THREE.Vector3(0,0,-100));               
   modelBuilder.updateMesh(mesh);
   
   ...

   and here you have a new geometry for the side panel:

   

3. Set position after adding the mesh by transforming the related fragments:

   This approach will look more complicated, but it is very powerful and its complexity can be easily abstracted. To explain better this, let us understand first what data the ModelBuilder instance holds when we create it and add a mesh to it:

   

   There is no mention about any mesh here. The mesh object can be found “deep” inside, but Forge Viewer works differently (for performance reasons). When adding a mesh, the information is extracted and needed parts filled with it, out of which fragments are the most important to us now. If we want to transform our custom component, we will have to work with fragments. 

   Before that, if we check geomList.geoms from ModelBuilder instance, we can notice that this is the place were we can find all geometries added either through mesh creation or through fragments: 

   

This is important for this step, because one of the ways of getting the fragment ids associated with our component is through having the geometry. Since in our case we have just one geometry, this is simple:

Now, knowing the fragmentID, transforming our component cannot be easier:

let new_matrix = new THREE.Matrix4().compose(
                    new THREE.Vector3(0, 85, -100),
                    new THREE.Quaternion(0, 0, 0, 1),
                    new THREE.Vector3(1, 1, 1)
                    );

modelBuilder.changeFragmentTransform(1, new_matrix);

where in my case 1 is the fragmentID.

Compared with second approach, this should be the approach when experimenting with custom component animation, but … I never told you that.

One final touch before we go further is to quickly change the material to one already available in the scene. In our case, it would be good to replace this basic red materials with the one that the initial component is using - a nice Mahogany material.

For that we will have to first find where it is stored, and many of you might already be familiar with matman:

We can notice the materials for both our models, but we are interested in materials of the original model, in our case model:1|mat:0, which we can then assign to our custom component, again through use of mighty fragments:

giving as a nicer look:

Nicer, but not nice enough, because the geometry from our new components lacks proper UV data, but we will take care of it in another post, dedicated ways of to changing materials, replacing materials and other fun stuff with materials.

Nevertheless, we have now a nice mix of “native” and “custom” model in our scene. A live illustration of the project at this stage can be found here.
 

Step II: Add middle shelve as custom geometry from geometry of original shelve

Bring custom geometry is useful, but not always enough. In many cases, the needed geometry is already available in the scene as another “native” component and the challenge is to extract that geometry and bring it again as a custom component for further “manipulations”.

In our case, it would be nice to “clone” the original shelve component and put it above our side panel. This way, this is the first step for a customizable shelve, where a potential customer can set the needed number of corner shelves.

To achieve this, we would need to master the renderProxy:

From above, we see that the workflow of getting the render proxy of a fragment is to:

• get the id of needed component, 
• get the associated fragment ids (could be more than one), 
• get the render proxy for that very fragment. 

The RenderProxy by itself is actually a Mesh, and for now, we are interested only in geometry it stores:

a bit unusual geometry structure, and now our mission (should you choose to accept it) is to translate the data from this geometry format into THREE.BufferGeometry format, which we can then use ModelBuilder extension to add custom component.

We have everything we need in vb, which stores position, normal, uv and index, in attributes we can see the itemOffset and itemSize and vbstride shows us the “repeating chunk size”.

All what we need to do at this step is to extract this data (or at least vertex positions and the indices for face formation) and organize into way THREE.BufferGeometry structures it:

This is a quite daunting task, mainly because the original data is interleaved into a single array and you have to know how to carve from it the needed info.

Fortunately, there is a hidden shortcut created by the engineering team, kept in total secret, yet widely available to anyone, with code name VertexEnumerator and it can be found in the namespace with a name that should never draw nobody’s attention: Autodesk.Viewing.Private

So, to illustrate the power of this tool, we will use it to extract the minimal needed data from renderProxy geometry and push it into THREE.Geometry:

let geom = new THREE.Geometry();
let renderProxy = this.viewer.impl.getRenderProxy(this.viewer.model, fragmentId);

let VE = Autodesk.Viewing.Private.VertexEnumerator;

VE.enumMeshVertices(renderProxy.geometry, (v, i) => {
            geom.vertices.push(new THREE.Vector3(v.x, v.y, v.z));
        });

VE.enumMeshIndices(renderProxy.geometry, (a, b, c) => {
                    geom.faces.push(new THREE.Face3(a, b, c))
            });

geom.computeFaceNormals();

Having a THREE.Geometry we can easily create a BufferGeometry out of it, put into a mesh along with default material and add this “clone” to the scene:

let mesh = new THREE.Mesh(
        new THREE.BufferGeometry().fromGeometry(geom),
        new THREE.MeshPhongMaterial({
            color: new THREE.Color(1, 0, 0)
        }));
        
modelBuilder.addMesh(mesh);

and we succeeded … sort of …:

The “cloned” mesh is there (I’ve “ghosted” the original component), but the scale doesn’t look right, which means that the final look is the result of taking the geometry and scaling it up, all this being done at renderProxy level, easy to confirm and identify the scale ratio by looking at it’s world matrix:

thus, the simplest way of fixing the scale, potential position and rotation, is to assign it to our newly created mesh and adjust position if needed:

mesh.matrix = renderProxy.matrixWorld.clone();
mesh.matrix.setPosition(new THREE.Vector3(0,140,0));

change the material as we did before and here we go:

Having all these ingredients, it is now a matter of combining them into an extension where ui controls trigger addition, removal and movement of geometries in our custom component:

You can play with the full scene here and inspect the extension behind it here.

I hope this simple example will inspire you to create more useful things.