Dip into Managed DevOps Pools
Managed DevOps Pools came generally available (GA) at the end of November 2024 but I didn’t have the time to dig into it right away. Let’s now explore what they are and what you can achieve with Managed DevOps Pools. We can also clarify here that agents in this context have nothing to do with secret service and to these pools you cannot jump in as you would into the pool you have, for example, in your backyard.
In 2021, 1ES inventoried Microsoft’s Azure DevOps agents infrastructure and estimated that engineers had created over 5,000 self-hosted Azure DevOps pools with hundreds of thousands of agents. In Azure DevOps, Microsoft-hosted agents are readily available for teams to use, but teams were unable to use them for all their scenarios.
https://devblogs.microsoft.com/engineering-at-microsoft/managed-devops-pools-the-origin-story/
Managed DevOps Pools emerged from Microsoft’s internal need to optimize Azure DevOps agent infrastructure. Many teams relied on self-hosted Agent Pools with Virtual Machine Scale Sets, leading to cost, scalability and compliance challenges. By adopting Managed DevOps Pools internally, Microsoft reduced CI/CD costs by 60%. They also were able to lower the amount of their self-hosted Agent Pools from over 5000 to just a few dozen, dramatically improving efficiency. In a way, Managed DevOps Pools combine some of the best features from Self-hosted agent pools and Microsoft-hosted agents. Let’s explore how they compare to other agent types in Azure DevOps.
You can read the full origin story HERE.
Understanding DevOps agents in Azure
At first, let’s look into what is a DevOps agent and what are the differences between self-hosted and Microsoft-hosted agents. An agent is, in Azure DevOps context, a machine with agent software installed that performs a pipeline job. So when you execute a pipeline in Azure DevOps you need at least one agent to build or deploy your code. As the team grows and there are more pipeline runs you likely need more agents.
Azure DevOps supports four types of agents:
- Microsoft-hosted agents are as the name states agents hosted and managed by Microsoft.
- Self-hosted agents are hosted on your own Virtual Machines that you configure and manage.
- Virtual Machine Scale Set agents are a form of self-hosted agents that can be autoscaled based on need.
- Managed DevOps Pool agents are fully managed agents where Microsoft manages the infrastructure (VMs) in their subscription.
Key features of Managed DevOps pools
As Microsoft-hosted agents have no proper customization and self-hosted agents need you to be the one in charge of maintenance, Managed DevOps Pools combine these two in a way that you are able to customize your agents, attach them into your Virtual networks etc. but the infrastructure is hosted by Microsoft. This takes away all the patching and figuring out what went wrong and where. Microsoft calls this HOBO which means ‘hosted on behalf of [you]’. You can see illustration about the setup below.
Microsoft has prepared great table where you can compare Managed DevOps Pools and Virtual Machine Scale Set agents. You can find it HERE. There are plenty of similarities but also differences. I have listed some key points here but feel free to look into full table.
- Pricing is exactly same for both. You pay for capacity used.
- Virtual network integration is possible with both.
- Stateful and stateless agents are supported by both.
- Managed DevOps Pools have better image support as it supports Azure Pipeline images, Azure Marketplace images as well as custom images from Azure Compute Gallery.
- Managed DevOps Pools support usage across multiple Azure DevOps organizations and can be restricted to specific projects, whereas VMSS pools only support one organization.
- Pool size can be thousands of agents with Managed DevOps Pools as with VMSS pools it can be hundreds.
- Multiple images within same pool is supported by Managed DevOps Pools. VMSS pools can only have one image per pool.
- Managed DevOps Pools have dedicated quota. VMSS pools use your tenant’s quota.
It is also worth to mention that Managed DevOps Pools will be getting Spot VM support as well as support for containers. Business Continuity Disaster Recovery is also on product roadmap. This means you can add backup pools for your primary pools and automatically switch to them if the primary fails.
Setting up Managed DevOps Pools
Creating a Managed DevOps Pool in Azure is a fairly simple task. Microsoft has you covered and tutorial can be found HERE and prerequisites HERE. I’ll cover the setup quickly below.
- Register Resource providers
- Check for quota (optional)
- Managed DevOps Pools have their own separate CPU quota so it may be a good idea to check that you have enough for planned agents.
- Create DevCenter and DevCenter Project
- Managed DevOps pools are created under Azure DevCenter project so creating one is pretty necessary.
- Link your Azure DevOps organization to your Entra ID
- Verify Azure DevOps permissions
- You need to be Organization-level Agent pool administrator or Project Collection Administrator.
- You need to be Organization-level Agent pool administrator or Project Collection Administrator.
After these steps you can create your first Managed DevOps pool. To do this you can just search ‘Managed DevOps pool’ in Azure Portal and click ‘Create’. Follow the steps and you’ll have your pool running in no time.
If you want to test your new pool, John Lokerse has written a helpful demo in his blog.
Conclusions
Managed DevOps pools are a great way for hosting your own agent pools without the need for you to handle everything related to infrastructure. As it takes away the burden to do that it may be able to give you more time to concentrate more meaningful things (for example getting your deployments done).
You are also able to use same images as Microsoft-hosted agents which makes it pretty hassle-free if you are moving from those. As you also can utilize Azure marketplace images as well as your own custom images you have really good flexibility across various projects.
With Virtual network integration you can connect to your private endpoints as well as other resources living inside your Virtual network.
There are some caveats also. I’ve heard and read that sometimes scaling and scheduling doesn’t work as it should but for me, they have worked fine. I have also marked myself that sometimes even if you have standby agent waiting it still takes quite the same amount of time for pipeline to start as if you wouldn’t have any. Microsoft states that support for containers that they have in the roadmap can help with this. Third thing to mention is that as you need to have admin rights in Azure DevOps organization it limits the personnel that can create new pools for teams.
I think that this may be your best option if you need to go outside Microsoft-hosted pools when working with Azure DevOps.
Useful links:
Managed DevOps Pools documentation (Microsoft Learn)
Managed DevOps Pools Module (Bicep) (GitHub)
