Parallel Builds in Azure DevOps

The build capabilities within Azure Pipelines (the Build and Deploy technologies within Azure DevOps) are extensive and powerful. If you can build it on a Windows, Linux or macOS machine, then you can build it within Azure DevOps. In this post, I’d like to highlight one of many powerful features of Azure Pipelines – the ability to run builds in parallel.

Caveat: To run multiple jobs in parallel, you must have the appropriate number of build pipelines licensed. For example, if you want to run three jobs in parallel, then you must have at least three pipelines licensed for use. You can read more about pipeline licensing here.

Getting Started

Whether you’ve realized it or not, all of your Azure DevOps builds come with at least one predefined job. A job is simply a set of one or more build tasks that run sequentially on the same target. To enable parallel builds, simply add additional jobs to your build definition.

An example of this might be a set of projects that consists of:

• An MVC Web App
• A set of Web APIs

One could argue that each of the above projects should equate to their own build definition. Rather than have that argument, let’s simply use it as one example for setting up parallel builds.

Each application type in the above list could equate to its own job within a build definition – each with its own respective build pool/agent. Assuming you have at least two licensed build pipelines at your disposal, each of the above applications (jobs) could build concurrently, likely decreasing the overall build time noticeably (assuming you were previously building them all sequentially within the same build definition).

Another example would be to split up tests across jobs so that chunks of tests can be executed across parallel builds (to reduce overall build times).

How Do I Add Additional Jobs?

This all sounds great, you say, so how do you go about setting them up? In Figure 1 below you can see that clicking on the ellipses (…) for the Pipeline will display a menu with options to create an agent job or an agentless job.

For this example, I will be creating three agent jobs. The first job will run on a Microsoft-hosted agent, the second job will run on a private agent (that I have hosted in Azure) and the third job will run on the Microsoft-hosted agent.

You can see how this might look in Figure 2 below.

By default, each job will default to the agent pool specified for the overall Pipeline as shown in Figure 3 below

To override the Agent Pool for a specific job, simply click on the job and edit the Agent Pool as shown in Figure 4 below.

At this point, the first and third jobs are using the Default Agent Pool as specified in the overall Pipeline settings and the second job has overridden the Agent Pool so it runs on a completely different server/agent.

By default, all jobs will execute concurrently (assuming you have enough licensed pipelines to do so). However, you might want a job to execute only after all other jobs have completed. You can do this by specifying one or more dependencies within the job settings. For example, Figure 5 below shows how the third job is dependent on the first two jobs which simply means that it will wait for those jobs to complete before executing the tasks in job three.

When the build is queued, you can watch as multiple jobs start executing simultaneously. In Figure 6 below, you can see that the first and second jobs are both running. The third job has not started yet because it’s dependent on the first two jobs completing before it starts.

Summary

While I’ve only given a few examples above as to when you might want to take advantage of parallel builds, you will likely come up with even more (better!) reasons. You might already be thinking about ways to optimize your builds to get even better build times by breaking them apart into parallel jobs.

For further information, check out Microsoft’s Jobs documentation.