Revit Cloud Worksharing - fast extraction of Revit zip files with partials

The Problem

When a model is published from "Revit Cloud Worksharing", a single zip file is saved to BIM 360 docs.  That zip file contains the master Revit file, along with all linked Revit files.

These zip files are big, and they get renamed from .ZIP to a .RVT file in the webpage.

How do I know it's a zip or a RVT file?

How do I only download the RVT file I want, contained inside the zip file ?

I don't want to download an entire 4.2GB zip file, just to pull out a small 35MB Revit file.  It's too slow, and the Revit file changes daily.

If only there was a better way ... ?

Maybe I can take advantage of the zip file structure and only download the bits I need ?

Yes, you can !

The Solution: Partial Transfer

• a ZIP file has a magic header
• It has a fixed data structure - the header at the start and a footer at the end
• I can transfer the 8kb header and footer, using two range GET requests

Finally:  I only transferred 35MB instead of 4.2GB.  Much faster, much less bandwidth needed.

Results

To test this out, I built an node.js API that goes through these steps via 4 HTTP calls. 

They are a little complicated, so I added a very basic UI, this one...

BIM360 Source Project on the left, Destination Project on the right.

Steps:

1. add a BIM360 access token
2. add a BIM360 source project URL and destination project URL
3. click LOGIN
4. Double click your zip file to show the internal file listing contents
5. click a RVT file and finally
6. click the TRANSFER button

See the Revit file, contained inside the zip file, quickly transfer into the destination BIM360 project (and start processing).

Code Details:

Using a range GET function, like this...

async _fetchWrite( fd, offset, length ) { 
const res = await fetch( this.URL, { headers: {
    'range': `bytes=${offset}-${offset+length}`,
    'Authorization': `Bearer ${this.token}`
}});

we pull the 8k header and footer, to create a zip file, like this...

    async _createTempZip(offset, size) {
        const tmpfile = fs.openSync(this.tmpFn, 'w');
        const chunksize = 4 * 1024; // only need 16k bytes of data
        await this._fetchWrite(tmpfile, 0, chunksize); // fetch/write header            
        await this._fetchWrite(tmpfile, this.fileLength - chunksize, chunksize); // fetch/write footer
        const zipHeaderOffset = 128;
        if (size)
            await this._fetchWrite(tmpfile, offset, size + zipHeaderOffset); // fetch/write our filename within the zip
        fs.closeSync(tmpfile);        
    }

// snip snip

async extractFile( filename, destURL ) {
    ...
   zipObject = new StreamZip({ file: tmpfile, storeEntries: true });

and read the zip folder contents...

this.zip.on('ready', async () => { 
            this.entries = this.zip.entries();

and finally, unzip just one file, based on the zip structure's offset and length......

this.zip.extract( filename, filename, async err => {

Walkthrough video (YouTube): https://youtu.be/02ChjbwLjUY

Source: https://github.com/wallabyway/bim360-zip-extract

Demo: http://bim360-zip-extract.herokuapp.com

Saving everyone, a lot of bandwidth and speed.  Look out for Autodesk Consulting's Project Deckard, which offers a professional UI, and much more.

Follow me on Twitter: @micbeale