Create an ETL Process for BuildingConnected
This tutorial demonstrates how to perform ongoing data retrieval for an Extract, Transform, and Load (ETL) process, so you can store all of your BuildingConnected data in your own data warehouse. The steps include retrieving your company’s BuildingConnected projects, bid packages, and invites, retrieving bids that your company receives, and periodically checking for updates of this data.
You can learn more about the BuildingConnected data model in the BuildingConnected Field Guide. In this tutorial, we’ll assume that you know the basic relationships between the key objects in BuildingConnected Pro: projects, bid packages, invites, and bids.
Before You Begin
- Register an app (select either the Traditional Web App or the Desktop, Mobile, Single-Page App application type).
- Make sure to select the BuildingConnected API via the API Access dropdown after registering your app.
- Acquire a 3-legged OAuth token with
data:readscope.
Pagination
Each endpoint returns paginated results. As long as there are additional pages of results available, the response will include a nextURL field indicating the URL to call for the next page. If the nextURL field is present in the pagination object that is returned (e.g. bid.lineItems.pagination.nextUrl), you can call the provided URL to page through the remaining line items before continuing your extraction.
Keep in mind that some objects have sub-resource pagination. This might be relevant if you have very long bid forms (>100 questions).
Subsequent Extractions
Your response payloads will always include an updatedAt field. The API enables you to query each endpoint using this timestamp with the filter[updatedAt] query parameter so that you get only data that has been changed or added since your previous extraction. If an object is new but has not been updated yet, the updatedAt field will be equal to the createdAt field (i.e. it will not be null) so that you can be sure you’re capturing all changes since your previous extraction.
The following steps provide examples of how to retrieve each data object type, along with some notes.
Step 1: Retrieve BuildingConnected Projects
Projects are the top-level objects in BuildingConnected. They are not equivalent to Autodesk Construction Cloud projects, but multiple BC projects can be linked to a single ACC project. If they are linked, the BC project field currentAccLinkedProjectId contains the linked ACC project’s project ID.
To get your company’s projects, call GET projects. If you wish to include projects that have been closed, set the query parameter includeClosed=true.
Request
curl 'https://developer.api.autodesk.com/construction/buildingconnected/v2/projects?includeClosed=true' \
-H 'Authorization: Bearer nFRJxzCD8OOUr7hzBwbr06D76zAT'
Response
{
"pagination": {
"limit": 100,
"cursorState": null
},
"results": [
{
"id": "59d2bd7440b36a0da258f24d",
"name": "Office Renovation",
"number": "12345",
"client": "Name of client",
"description": "Short description of the project",
"notes": "Any notes about the project",
"value": 1000000,
"projectSize": 2000,
"awarded": "WON",
"isBiddingSealed": false,
"state": "PUBLISHED",
"isNdaRequired": false,
"isPublic": false,
"createdAt": "2014-09-22T16:27:37.437Z",
"updatedAt": "2014-09-23T16:28:37.437Z",
"publishedAt": "2014-09-22T21:34:44.511Z",
"bidsDueAt": "2014-10-03T19:00:00.000Z",
"endsAt": "2015-03-31T07:00:00.000Z",
"rfisDueAt": "2014-09-25T19:00:00.000Z",
"startsAt": "2014-11-03T08:00:00.000Z",
"dueAt": "2014-09-03T08:00:00.000Z",
"location": {
"country": "US",
"state": "CA",
"streetName": "California St.",
"streetNumber": "600",
"suite": "6th Floor",
"city": "San Francisco",
"zip": "94108",
"complete": "600 California St., 6th Floor, San Francisco, CA 94108",
"coords": {
"lat": -122.4069826,
"lng": 37.7929546
}
},
"isTemplate": false,
"accountManager": "Name of account manager",
"fee": "15.55",
"marketSector": "Sports Stadium Construction",
"creatorId": "5a1dc3cf5b54e06ca307d7af",
"defaultCurrency": "USD",
"officeId": "5fcdcd0b3db0876c6692d5bb",
"architect": "Browning Day Mullins Dierdorf Architects",
"isForBudgeting": false,
"currentAccLinkedProjectId": "00f1b796-293a-4c1e-9474-f09711011426",
"currentAccLinkedHubId": "28b0d8b3-4730-4f4f-8bad-4a2a5d75ed26",
"relevantCertificates": [
{
"id": "5d1dcd5a4a7196ca208ed623",
"agencyId": "600dcf56ffa3d03ac170f967",
"certificationTypeId": "610d4fe28a3719ec41c02596"
}
]
}
]
}
Step 2: Retrieve Bid Packages
To get your company’s bid packages, call GET bid-packages. The returned projectId field identifies the bid package’s associated project.
Request
curl 'https://developer.api.autodesk.com/construction/buildingconnected/v2/bid-packages' \
-H 'Authorization: Bearer nFRJxzCD8OOUr7hzBwbr06D76zAT'
Response
{
"pagination": {
"limit": 100,
"cursorState": null
},
"results": [
{
"id": "590dd127b359f408f190b5a7",
"projectId": "590dd127b319f408f190b3b8",
"name": "Windows",
"number": "123",
"keywords": [
"Glass"
],
"estimatedCost": 100000,
"state": "PUBLISHED",
"createdAt": "2014-09-25T19:00:00.000Z",
"updatedAt": "2014-09-26T16:00:00.000Z",
"publishedAt": "2014-09-25T19:00:00.000Z",
"bidsDueAt": "2014-09-25T19:00:00.000Z",
"endsAt": "2014-09-25T19:00:00.000Z",
"rfisDueAt": "2014-09-25T19:00:00.000Z",
"startsAt": "2014-09-25T19:00:00.000Z",
"jobWalkAt": "2014-09-25T19:00:00.000Z"
}
]
}
Step 3: Retrieve Invites
An invite represents a solicitation of an individual company to bid on a bid package. There is one invite object for each company that has been invited to bid on a given bid package.
To get invites sent by your company, call GET invites. The returned bidPackgeId field identifies the invite’s associated bid package.
An individual who has been invited to bid is listed in the invitees array. While there may be people from multiple companies in invitees, each invite is associated with a single company, indicated by the field bidderCompany. Use bidderCompany to organize bids by trade partner rather than company IDs in the invitees array.
Request
curl 'https://developer.api.autodesk.com/construction/buildingconnected/v2/bid-packages' \
-H 'Authorization: Bearer nFRJxzCD8OOUr7hzBwbr06D76zAT'
Response
{
"pagination": {
"limit": 100,
"cursorState": null
},
"results": [
{
"id": "590dd527b359f408f191b5b6",
"projectId": "59d2bd7440b36a0da258f24d",
"bidPackageId": "590dd127b359f408f190b5a7",
"companyName": "Building Group",
"bidderOfficeId": "57d2bd7450b37a0c52585d2a",
"bidderCompany": {
"id": "51bea397f5846f95994b1da7",
"name": "Gr8 Builders",
"businessType": [
"Subcontractor"
],
"website": "http://www.example.com",
"laborType": "Union",
"enterpriseType": [
"Other"
],
"companyTags": [],
"certificates": [
{
"id": "627074db25c97830ad3ea8c8",
"type": {
"id": "59d2bd7440b36a0da258f24d",
"name": "Disadvantaged Business Enterprise (DBE)"
},
"agency": {
"id": "611d794f7f063800a7274d46",
"name": "United States Department of Transportation",
"website": "https://www.transportation.gov/civil-rights/disadvantaged-business-enterprise"
},
"number": "12345",
"expiresAt": "2014-10-03T19:00:00.000Z",
"file": {
"name": "certificate.pdf",
"url": "..."
}
}
]
},
"state": "INVITED",
"createdAt": "2014-09-25T20:00:00.000Z",
"updatedAt": "2014-09-25T21:00:00.000Z",
"invitedAt": "2014-09-25T20:30:00.000Z",
"inviterType": "HOST",
"invitees": [
{
"userId": "5bff41c079b25e7ccebc202e",
"firstName": "First",
"lastName": "Last",
"companyId": "51bea397f5846f95994b1da7",
"phoneNumber": "555-555-5555",
"title": "Estimator",
"email": "someone@gmail.com"
}
]
}
]
}
Step 4: Retrieve Bids
Bids are associated with invites on a many-to-one basis. Each bid object represents a bid revision submitted either by your company or by the bidder’s company.
To get the bids that were submitted on your company’s projects, call GET bids.
The revision field indicates the revision number, which is tracked on a per-invite basis. The revision number starts at 0 for the initial bid, and increments by 1 for each new bid submitted for an invite. The most recent bid has the highest revision number. If you want only the most recent revisions, set the query parameter onlyLatestRevision=true.
The inviteId field identifies the bid’s associated invite. The bidPackgeId field identifies the bid’s associated bid package.
Each bid has summary information that expresses the total bid value. The field leveledTotal contains the net value of the bid including any plugs you make on the Bid Leveling tab.
If you intend to do more thorough cost analysis, consider creating a separate table to organize line items and using inviteId and bidPackageId to map line items back to a specific scope of work.
Request
curl 'https://developer.api.autodesk.com/construction/buildingconnected/v2/bids' \
-H 'Authorization: Bearer nFRJxzCD8OOUr7hzBwbr06D76zAT'
Response
{
"pagination": {
"limit": 100,
"cursorState": null
},
"results": [
{
"id": "611ee35b3935e500a8b18c8a",
"projectId": "59d2bd7440b36a0da258f24d",
"bidPackageId": "590dd127b359f408f190b5a7",
"inviteId": "590dd527b359f408f191b5b6",
"bidderCompanyId": "51bea397f5846f95994b1da7",
"creatorType": "BIDDER",
"notes": "Notes about bid",
"total": 5000,
"leveledTotal": 5200,
"createdAt": "2021-08-19T23:07:16.083Z",
"updatedAt": "2021-08-19T23:07:16.083Z",
"submittedAt": "2021-08-19T23:08:32.075Z",
"attachments": [],
"revision": 0,
"createdBy": "5bff41c079b25e7ccebc202e",
"submittedBy": "5bff41c079b25e7ccebc202e",
"lineItems": {
"pagination": {
"limit": 100,
"cursorState": null
},
"results": []
},
"plugs": {
"pagination": {
"limit": 100,
"cursorState": null
},
"results": []
}
}
]
}
Step 5: Periodically Check for Updates
On a periodic basis, perform the preceding steps using the query parameter filter filter[updatedAt]={Previous fetch time} to check for updates since the previous time you used that endpoint.
