Azure DevOps has a great REST API, but every now and then there are some things you can do in the user-interface that you can’t do from the REST API.
But did you know Azure DevOps has a hidden API?
Wait, Hidden API?
Technically, it’s not really a hidden API. Like most modern web applications, Azure DevOps exposes a set of REST endpoints to their user-interface that is different than the official REST API. These endpoints follow a REST convention and use the same authentication scheme, so you can use them interchangeably from a non-interactive script.
While the Azure DevOps REST API is versioned, documented and supported by Microsoft, these endpoints are not so there’s no guarantee that they won’t change in the future, so buyer beware. Here be dragons, etc.
If you’re okay with this and you’re open to some experimentation, here’s how you can go about finding these endpoints and including them in your scripts.
Use the Inspector
It should come as no surprise that you need to use your browser’s Developer Tools to inspect the DOM and the Network Traffic while using the feature you’re interested in.
As an example, there isn’t a great API that can give you the list of builds and their status by stage. You can query the list of Builds and then use Timeline API for each build to get the outcome of the stages, but it’s really heavy and chatty. However, the user-interface for build history however easily shows us the build history and the stage status. So let’s use see how the user-interface gets its data:
- Navigate to the pipeline and open the Developer Tools (CTRL + SHIFT + I)
- Switch to the Network panel
- In the Filter, select Fetch/XHR
- Slowly scroll down until you see new requests being submitted from the page
Few things to note:
- Requests are POST
- Endpoint is
/<organization>/_apis/Contribution/HierarchyQuery/project/<project-id> - The payload of the request contains a
contributionIdsand details that represent the current context
{
"contributionIds": [ "ms.vss-build-web.runs-data-provider"],
"dataProviderContext": {
"properties": {
"continuationToken":"2021-10-08T15:53:40.2073297Z",
"definitionId":"1324",
"sourcePage": {
"url":"https://dev.azure.com/myOrganization/myProject/_build?definitionId=1234",
"routeId":"ms.vss-build-web.pipeline-details-route",
"routeValues":{
"project":"myProject",
"viewname":"details",
"controller":"ContributedPage",
"action":"Execute",
"serviceHost":"<guid>"
}
}
}
}
}
- The response object has
data-providersobject that corresponds to thecontributionIdsvalue specified above.
{
"data-providers": {
"ms.vss-build-web.runs-data-provider" : {
"continuationToken": "<date-time>",
"runs": [
{
"run": {},
"stages": []
}
]
}
}
}
Using the above syntax, you could in theory replicate any UI-based function by sniffing the contributionId and payload.
Magic Parameters
Now, here’s the fun part. I haven’t tested this trick for all endpoints, but it works for the majority that I’ve tried. Take any endpoint in Azure DevOps and tack on the following querystring parameters: __rt=fps&__ver=2.0
If we stick with our example of viewing Build History for a YAML Pipeline. We can take this url:
https://dev.azure.com/<organization>/<project>/_build?definition=<build-defintion>
Then tack on our magic parameters:
https://dev.azure.com/<organization>/<project>/_build?definition=<build-defintion>&__rt=fps&__ver=2.0
And immediately, you’ll notice that JSON is rendered instead of the UI. This JSON has a lot of data in it, but if we narrow on the details of the ms.vss-build-web.runs-data-provider you’ll notice the same data as what’s returned from the POST above:
{
"fps": {
"data-providers": {
"data": {
"ms.vss-build-web.runs-data-provider": {
"continuationToken": "<date-time>",
"runs": [
{
"run": [],
"stages": []
}
]
}
}
}
}
}
}
Paging Results
The Azure DevOps UI automatically limits the amount of data to the roughly the first 50 records, so if you need to fetch all the data you need to combine the two approaches:
- Query the endpoint with
GET __rt=fpsvalue to obtain the first set of data and the continuationToken - Subsequent requests use
POSTand pass in the continuationToken with the body.
A wrapper for Invoke-RestMethod
Just like the title of this post says, we’re going to access this API from PowerShell.
So let’s start with the following wrapper around Invoke-RestMethod that contains the logic of communicating with either of the two approaches listed above:
function Invoke-AzureDevOpsRestMethod
{
param(
[Parameter(Mandatory)]
[string]$Organization,
[Parameter(Mandatory)]
[string]$ProjectName,
[Parameter()]
[ValidateSet("GET","POST","PATCH","DELETE")]
[string]$Method = "GET",
[Parameter(Mandatory)]
[string]$AccessToken,
[Parameter()]
[string]$ApiPath = "",
[Parameter()]
[hashtable]$QueryStringParameters = [hashtable]@{},
[Parameter()]
[string]ApiVersion = "6.0-preview",
[Parameter()]
[string]$ContentType = "application/json",
[Parameter()]
[psobject]$Body
)
$QueryStringParameters.Add("api-version", $ApiVersion)
$queryParams = $QueryStringParameters.GetEnumerator() |
ForEach-Object { "{0}={1}" -f $_.Name, $_.Value} |
Join-String -Separator "&"
if ([string]::IsNullOrEmpty($ApiPath)) {
$uriFormat = "https://dev.azure.com/{0}/_apis/Contribution/HierarchyQuery/project/{1}"
} else {
$uriFormat = "https://dev.azure.com/{0}/{1}/{2}?{3}"
}
$uri = $uriFormat -f $Organization, $ProjectName, $ApiPath, $queryParams
$uri = [Uri]::EscapeUriString($uri)
$header = @{ Authorization = "Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCI.GetBytes(":$($AccessToken)")) }
$invokeArgs = @{
Method = $Method
Uri = $uri
Headers = $header
}
if ($Method -ne "GET" -and $null -ne $Body) {
$json = ConvertTo-Json $Body -Depth 10 -Compress
$invokeArgs.Add("Body", $json)
$invokeArgs.Add("ContentType", $ContentType)
}
Invoke-RestMethod @invokeArgs
}
Error Handling and Retry Logic
No REST API is guaranteed to return results 100% of the time. You’ll want to accomodate for intermittent network issues, minor service interruptions or request throttling.
Let’s further improve upon this implementation by adding some error handling and retry-logic by leveraging some retry logic that is built into the Invoke-RestMethod cmdlet. When the MaximumRetryCount parameter is specified, Invoke-RestMethod will retry the request if the status code is 304, or between 400-599.
This should work for the majority of throttling issues but it also means that unauthorized requests (403) will be attempted multiple times. In my opinion, if your credentials are wrong then it won’t matter how many times you try, so repeating these requests will ultimately just make your script slower. For the purposes of this post, I’m willing to live with that but if that’s not for you, you might want to roll your own retry loop.
$maxRetries = 3
$retryInterval = 10
try
{
Invoke-RestMethod @invokeArgs -RetryIntervalSec $retryInterval -MaximumRetryCount $maxRetries
}
catch
{
if ($null -ne $_.Exception -and $null -ne $_.Exception.Response) {
$statusCode = $_.Exception.Response.StatusCode
$message = $_.Exception.Response.Message
Write-Information "Error invoking REST method. Status Code: $statusCode : $message"
Write-Verbose "Exception Type: $($_.Exception.GetType())"
Write-Verbose "Exception StackTrace: $(_$.Exception.StackTrace)"
}
Write-Information "Failed to complete REST method after $maxRetries attempts."
throw
}
Putting it all Together
Let’s flush this out by continuing with a psuedo-example of getting YAML Build History. The initial request fetches the most recent builds with a GET request, and the loops using the POST method on the PageContribution endpoint until the continuation token is empty.
function Get-AzurePipelineHistory
{
param(
[string]$Organization,
[string]$Project,
[string]$PipelineId,
[string]$AccessToken
)
$commonArgs = @{ Organization=$Organization; Project=$Project; AccessToken=$AccessToken }
$apiPath = "_build"
$pipelineArgs = @{
definitionId = $PipelineId
"__rt" = "fps"
"__ver" = "2.0"
}
$results = @()
$continuationToken = $null
do {
if ($null -eq $continuationToken) {
# Fetch initial result
$apiResult = Invoke-AzDevOpsRestMethod -Method "GET" @commonArgs -ApiPath $apiPath -QueryParameters $queryArgs
$pipelineRuns = $apiResult.fps.dataProviders.data."ms.vss-build-web.runs-data-provider"
} else {
$requestBody = @{
contributionIds = @("ms.vss-build-web.runs-data-provider")
dataProviderContext =
@{
properties = @{
continuationToken = $continuationToken
definitionId = $PipelineId
sourcePage = @{
routeId = "ms.vss-build-web.pipeline-details-route"
routeValues = @{
project = $ProjectName
viewname = "details"
controller = "ContributedPage"
action = "Execute"
}
}
}
}
}
# fetch additional results
$apiResult = Invoke-AzDevOpsRestMethod -Method "POST" @commonArgs -Body $requestBody
$pipelineRuns = $apiResult.dataProviders."ms.vss-build-web.runs-data-provider"
}
# hang onto the continuation token to fetch additional data
$continuationToken = $pipelineRuns.continuationToken
# append results
$results += $pipelineRuns.runs
} until ($null -eq $continuationToken)
return $results
}
The end result is a concatenated collection of all the runs for this pipeline. It’s worth noting that if you have a very long list of pipeline runs, you’re going to have a lot of data to sift through. This can definitely be optimized and my next post will dig deeper into that.
That’s it for now. Happy coding.
