23 Sep 2020

Splitting processing into multiple Work Items

Nothing is ever fast enough, and we all look for ways to improve processing speed.

When it comes to Design Automation, one way to speed things up could be to split up a WorkItem into multiple ones and run them in parallel.

WorkItems are fully independent: they do not know about each other, have no access to the same local files, input/output files are not cached between them.

Therefore, if you split a WorkItem into multiple independent ones, you would have to orchestrate them: i.e. if WorkItem3 needs the output of WorkItem1 and WorkItem2, you have to keep track of them and pass the input and output files around. 

The two main things to consider:
1) can the job be split into completely independent pieces
2) how does the overhead of upload/download of the inputs/outputs of the WorkItems affect overall speed

You can get an estimate for all these locally on your machine using the desktop version of the given product. Split the processing into separate parts, check how long each of those takes. Also, zip up the files necessary for each process and check their size, so that you can estimate the upload/download times as well - see Estimate Design Automation costs 

As an example, if you plan to split one WorkItem into two WorkItems that can work on a subset of all the files, but also need a third WorkItem to combine the results produced by the other two WorkItems running in parallel, then these are the times to compare:

Single WorkItem

Single Work Item

Multiple WorkItems

Multiple Work Items

Whether the above approach would improve performance depends on the kinds of models you're working with and the processing you're doing on them, but it's good to be aware of this option. 

Related Article