It might be love...

It's been a very long time since I've posted on my blog. There are many reasons, but the largest blocker was support for OpenLiveWriter disappeared abruptly. OpenLiveWriter was one of the only tools that supported my blog platform.

I played with other tools.

OneNote is kinda garbage as a blogging tool, because storing my notes in rich text meant I had to spend a lot of time reformatting the content when I wanted to publish it. I also found that once I started writing something I could never remember where I put it.

Notepad++ was years ahead of everyone else by introducing a feature that didn't require that you actually save your files. This was my authoring tool of choice for markdown files, but without the publishing capabilities, I might as well be using blogspot's online editor.

Typora is a gorgeous interface that let's you write in Markdown (my preferred way of getting thoughts into digital form), but the tool lacked external integrations. I was motivated to create my own PowerShell library for publishing markdown files to blogger, which worked but was very clunky.

Last week, I started with Obsidian and it's a game changer. I'm in love.

Obsidian isn't a blogging tool. It's a Personal Knowledge Management (PKM) tool that stores its data in markdown files. As a PKM, it's all about "linking your thinking": creating small notes with connections to other notes. This interconnectedness between notes creates a web of knowledge with easy retrieval, and Obsidian has this magical indexing feature that doesn't just index the content, but the meta data about the content itself: YAML front-matter, todo items, incoming and outgoing links between pages. There's also a wealth of community plugins that allow you to write dynamic JavaScript code-blocks to query and display the content. Each note is an object, so I'm not just taking notes, I'm capturing thoughts while using the tool as a Journaling / CRM / Task tracker. Every day, I have new ideas on how to use the tool differently.

Today, I discovered that there's a community plugin for publishing directly from Obsidian to my blog platform. Hello!

In the past, my blog has always been a mechanism of sharing knowledge with others, but it's also been a place to store things that I want to remember. The intersection between PKM and my blog has always been there, but now there's no separation between the idea and the tool.

Invoking Azure DevOps Hidden REST API

Azure DevOps has a great REST API, but every now and then there are some things you can do in the user-interface that you can’t do from the REST API.

But did you know Azure DevOps has a hidden API?

Wait, Hidden API?

Technically, it’s not really a hidden API. Like most modern web applications, Azure DevOps exposes a set of REST endpoints to their user-interface that is different than the official REST API. These endpoints follow a REST convention and use the same authentication scheme, so you can use them interchangeably from a non-interactive script.

While the Azure DevOps REST API is versioned, documented and supported by Microsoft, these endpoints are not so there’s no guarantee that they won’t change in the future, so buyer beware. Here be dragons, etc.

If you’re okay with this and you’re open to some experimentation, here’s how you can go about finding these endpoints and including them in your scripts.

Use the Inspector

It should come as no surprise that you need to use your browser’s Developer Tools to inspect the DOM and the Network Traffic while using the feature you’re interested in.

As an example, there isn’t a great API that can give you the list of builds and their status by stage. You can query the list of Builds and then use Timeline API for each build to get the outcome of the stages, but it’s really heavy and chatty. However, the user-interface for build history however easily shows us the build history and the stage status. So let’s use see how the user-interface gets its data:

1. Navigate to the pipeline and open the Developer Tools (CTRL + SHIFT + I)
2. Switch to the Network panel
3. In the Filter, select Fetch/XHR
4. Slowly scroll down until you see new requests being submitted from the page

Few things to note:

• Requests are POST
• Endpoint is /<organization>/_apis/Contribution/HierarchyQuery/project/<project-id>
• The payload of the request contains a contributionIds and details that represent the current context

{
  "contributionIds": [ "ms.vss-build-web.runs-data-provider"],
  "dataProviderContext": {
    "properties": {
      "continuationToken":"2021-10-08T15:53:40.2073297Z",
      "definitionId":"1324",
      "sourcePage": {
        "url":"https://dev.azure.com/myOrganization/myProject/_build?definitionId=1234",
        "routeId":"ms.vss-build-web.pipeline-details-route",
        "routeValues":{
          "project":"myProject",
          "viewname":"details",
          "controller":"ContributedPage",
          "action":"Execute",
          "serviceHost":"<guid>"
        }
      }
    }
  }
}

• The response object has data-providers object that corresponds to the contributionIds value specified above.

{
  "data-providers": {
    "ms.vss-build-web.runs-data-provider" : {
      "continuationToken": "<date-time>",
      "runs": [
        {
          "run": {},
          "stages": []
        }
      ]
    }
  }
}

Using the above syntax, you could in theory replicate any UI-based function by sniffing the contributionId and payload.

Magic Parameters

Now, here’s the fun part. I haven’t tested this trick for all endpoints, but it works for the majority that I’ve tried. Take any endpoint in Azure DevOps and tack on the following querystring parameters: __rt=fps&__ver=2.0

If we stick with our example of viewing Build History for a YAML Pipeline. We can take this url:

• https://dev.azure.com/<organization>/<project>/_build?definition=<build-defintion>

Then tack on our magic parameters:

• https://dev.azure.com/<organization>/<project>/_build?definition=<build-defintion>&__rt=fps&__ver=2.0

And immediately, you’ll notice that JSON is rendered instead of the UI. This JSON has a lot of data in it, but if we narrow on the details of the ms.vss-build-web.runs-data-provider you’ll notice the same data as what’s returned from the POST above:

{
  "fps": {
    "data-providers": {
      "data": {
        "ms.vss-build-web.runs-data-provider": {
          "continuationToken": "<date-time>",
            "runs": [
              {
                "run": [],
                "stages": []
              }
            ]
          }
        }
      }
    }
  }
}

Paging Results

The Azure DevOps UI automatically limits the amount of data to the roughly the first 50 records, so if you need to fetch all the data you need to combine the two approaches:

• Query the endpoint with GET __rt=fps value to obtain the first set of data and the continuationToken
• Subsequent requests use POST and pass in the continuationToken with the body.

A wrapper for Invoke-RestMethod

Just like the title of this post says, we’re going to access this API from PowerShell.

So let’s start with the following wrapper around Invoke-RestMethod that contains the logic of communicating with either of the two approaches listed above:

function Invoke-AzureDevOpsRestMethod
{
  param(
    [Parameter(Mandatory)]
    [string]$Organization,
    
    [Parameter(Mandatory)]
    [string]$ProjectName,
    
    [Parameter()]
    [ValidateSet("GET","POST","PATCH","DELETE")]
    [string]$Method = "GET",
    
    [Parameter(Mandatory)]
    [string]$AccessToken,
    
    [Parameter()]
    [string]$ApiPath = "",
    
    [Parameter()]
    [hashtable]$QueryStringParameters = [hashtable]@{},
    
    [Parameter()]
    [string]ApiVersion = "6.0-preview",
    
    [Parameter()]
    [string]$ContentType = "application/json",
    
    [Parameter()]
    [psobject]$Body
  )
  
  $QueryStringParameters.Add("api-version", $ApiVersion)
  $queryParams = $QueryStringParameters.GetEnumerator() |
                 ForEach-Object { "{0}={1}" -f $_.Name, $_.Value} |
                 Join-String -Separator "&"
  
  if ([string]::IsNullOrEmpty($ApiPath)) {
    $uriFormat = "https://dev.azure.com/{0}/_apis/Contribution/HierarchyQuery/project/{1}"
  } else {
    $uriFormat = "https://dev.azure.com/{0}/{1}/{2}?{3}"
  }
  
  $uri = $uriFormat -f $Organization, $ProjectName, $ApiPath, $queryParams
  $uri = [Uri]::EscapeUriString($uri)
  
  $header = @{ Authorization = "Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCI.GetBytes(":$($AccessToken)")) }
  
  $invokeArgs = @{
     Method = $Method
     Uri = $uri
     Headers = $header
  }
  
  if ($Method -ne "GET" -and $null -ne $Body) {
    $json = ConvertTo-Json $Body -Depth 10 -Compress
    $invokeArgs.Add("Body", $json)
    $invokeArgs.Add("ContentType", $ContentType)
  }
  
  Invoke-RestMethod @invokeArgs
}

Error Handling and Retry Logic

No REST API is guaranteed to return results 100% of the time. You’ll want to accomodate for intermittent network issues, minor service interruptions or request throttling.

Let’s further improve upon this implementation by adding some error handling and retry-logic by leveraging some retry logic that is built into the Invoke-RestMethod cmdlet. When the MaximumRetryCount parameter is specified, Invoke-RestMethod will retry the request if the status code is 304, or between 400-599.

This should work for the majority of throttling issues but it also means that unauthorized requests (403) will be attempted multiple times. In my opinion, if your credentials are wrong then it won’t matter how many times you try, so repeating these requests will ultimately just make your script slower. For the purposes of this post, I’m willing to live with that but if that’s not for you, you might want to roll your own retry loop.

$maxRetries = 3
$retryInterval = 10

try
{
  Invoke-RestMethod @invokeArgs -RetryIntervalSec $retryInterval -MaximumRetryCount $maxRetries
}
catch
{
  if ($null -ne $_.Exception -and $null -ne $_.Exception.Response) {
    $statusCode = $_.Exception.Response.StatusCode
    $message    = $_.Exception.Response.Message
    Write-Information "Error invoking REST method. Status Code: $statusCode : $message"
    Write-Verbose "Exception Type: $($_.Exception.GetType())"
    Write-Verbose "Exception StackTrace: $(_$.Exception.StackTrace)"
  }
   
  Write-Information "Failed to complete REST method after $maxRetries attempts."
  throw
}

Putting it all Together

Let’s flush this out by continuing with a psuedo-example of getting YAML Build History. The initial request fetches the most recent builds with a GET request, and the loops using the POST method on the PageContribution endpoint until the continuation token is empty.

function Get-AzurePipelineHistory
{
  param(
    [string]$Organization,
    [string]$Project,
    [string]$PipelineId,
    [string]$AccessToken
  )
  
  $commonArgs = @{ Organization=$Organization; Project=$Project; AccessToken=$AccessToken }
  $apiPath = "_build"
  $pipelineArgs = @{
     definitionId = $PipelineId
     "__rt" = "fps"
     "__ver" = "2.0"
  }
  
  $results = @()
  
  $continuationToken = $null
  
  do {
  
    if ($null -eq $continuationToken) {
        
      # Fetch initial result
      $apiResult = Invoke-AzDevOpsRestMethod -Method "GET" @commonArgs -ApiPath $apiPath -QueryParameters $queryArgs
        
      $pipelineRuns = $apiResult.fps.dataProviders.data."ms.vss-build-web.runs-data-provider"
        
    } else {
     
      $requestBody = @{
        contributionIds = @("ms.vss-build-web.runs-data-provider")
        dataProviderContext =
          @{
            properties = @{
            continuationToken = $continuationToken
            definitionId = $PipelineId
            sourcePage = @{
              routeId = "ms.vss-build-web.pipeline-details-route"
              routeValues = @{
                project = $ProjectName
                viewname = "details"
                controller = "ContributedPage"
                action = "Execute"
              }
            }
          }
        }
      }
       
      # fetch additional results
      $apiResult = Invoke-AzDevOpsRestMethod -Method "POST" @commonArgs -Body $requestBody
       
      $pipelineRuns = $apiResult.dataProviders."ms.vss-build-web.runs-data-provider"
    }
     
    # hang onto the continuation token to fetch additional data
    $continuationToken = $pipelineRuns.continuationToken
     
    # append results
    $results += $pipelineRuns.runs
  
  } until ($null -eq $continuationToken)
  
  return $results
}

The end result is a concatenated collection of all the runs for this pipeline. It’s worth noting that if you have a very long list of pipeline runs, you’re going to have a lot of data to sift through. This can definitely be optimized and my next post will dig deeper into that.

That’s it for now. Happy coding.

PowerShell Module Quick Start

PowerShell modules are a great way to organize related functions into reusable unit. Today's post will provide a simple walk-through that creates a bare-bones module that you can start to work with.

Table of Contents

• Why create a Module?
• 1. Create the Manifest Module
• 2. Public vs Private Folders
• 3. Create the Module Script (psm1)
• 4. Using the Module
• What's Next?
• Wrap Up

Why create a Module?

First of all, let's look at some of the motivations for creating a module - after all, you could simply move all your common utility methods into a 'helper' script and then import it into the session. Why go through the overhead of creating a module?

Not every script or utility function needs to be a module, but if you're creating a series of scripts that rely on some common helper functions, I can think of two primary reasons why you should consider creating a module:

• Pathing: when you import a script using "dot sourcing", the "." means the current working folder. If you're import individual script files, you need to specify the relative path of the file you're importing. This strategy starts to fall apart if the file you're importing references additional imports. However, when you create a module, you import all the scripts into the session so pathing no longer becomes an issue.
• Discoverability: PowerShell scripts can get fairly complex so if your helper utility contains dozens of smaller methods then developers consuming your script will need to 'grok' all of those functions, too. By creating a module, you can expose only the functions you want, which will make it easier for developers to understand and use.

1. Create the Manifest Module

A module is comprised of:

• <module-name>.psd1: a manifest file that describes important details about the module (name, version, license, dependencies, etc)
• <module-name>.psm1: the main entry point when the module is imported.

Important Note: The folder and module manifest name must match!

To simplify this process, Microsoft has provided the New-ModuleManifest cmdlet for us. They have some good guidance listed here on some additional settings you may want to provide, but here's the bare-bones nitty-gritty:

mkdir MyModule cd MyModule New-ModuleManifest -Path .\MyModule.psd1 -RootModule MyModule.psm1

2. Public vs Private Folders

When you're creating a module, it's best to think about what functions and features that you're exposing to your module consumers. This is very similar to the access-modifiers we put on classes in our .NET assemblies.

The less you expose, the easier it is for consumers to understand what your module does. Limiting what you expose can also protect you from accidentally introducing breaking changes to consumers - if all of your functions are public you won't know which methods that external team members might be using; keeping this list small can help you focus where version compatibility is required.

A good practice is to put our functions into two folders: public and private:

mkdir Private mkdir Public

3. Create the Module Script (psm1)

The last piece is the psm1 script. This simple script finds all the files and imports them into the session:

# MyModule.psm1  
# Get Functions 
$private = Get-ChildItem -Path (Join-Path $PSScriptRoot Private) -Include *.ps1 -File -Recurse
$public = Get-ChildItem -Path (Join-Path $PSScriptRoot Public) -Include *.ps1 -File -Recurse  

# Dot source to scope 
# load private scripts first 
($private + $public) | ForEach-Object {
     try {
         Write-Verbose "Loading $($_.FullName)"
         . $_.FullName
     }
     catch {
         Write-Warning $_.Exception.Message
     }
}  

# Expose public functions. Assumes that function name and file name match 
$publicFunctions = $public | Select-Object -ExpandProperty BaseName Export-ModuleMember -Function $publicFunctions

The last section of the script exposes the functions in your public folder as part of your module. This makes the assumption that the function name and file name are the same. An alternative to this approach is to set the Functions to export in the module manifest. I find this approach easier.

4. Using the Module

At this point, you're ready to start adding PowerShell scripts into your module. When you're ready to try it out, simply import the module by it's definition:

Import-Module .\MyModule.psd1 -Verbose

To verify that your commands are exposed correctly:

(Get-Module MyModule).ExportedCommands

During development of your module, it's important to realize that changes that are made to your scripts won't be visible until you reload the module:

Remove-Module MyModule -ErrorAction Silent Import-Module .\MyModule.psm1

What's Next?

There's lots more to creating a PowerShell module, such as setting the minimum supported PowerShell version, declaring dependencies, including .NET code assemblies, exposing types, etc. There are likely some additional considerations for publishing the module to a gallery, but unfortunately I'm not going to get into that for the purposes of this post. This article is helpful for creating the basic shell of a module, which you can use locally or on build servers.

Wrap Up

Creating a PowerShell module is a fairly quick process that helps to promote reuse between related scripts. With Microsoft's New-ModuleManifest cmdlet and the basic psm1 file provided above, you can fast track creating a Module.

Happy codin'

Using the Azure CLI to Call Azure DevOps REST API

Suppose the Azure DevOps REST API that you want to call isn't in the list of az cli supported commands. Does this mean your script needs to toggle between az cli and invoking REST endpoints? Fear not, there's actually a built in az devops command "az devops invoke" that can call any Azure DevOps REST API endpoint.

The az devops invoke command is fairly easy to use, but the trick is discovering the command-line arguments you need to provide to pull it off. This post will walk you through that.

Table of Contents

• Obtaining a List of Available Endpoints
• Finding the right endpoint
• Invoking endpoints
• Adding Query-string Parameters
• Specifying the API version
• Providing a JSON Body
• Known Issues
• Wrapping Up

Obtain a List of Available Endpoints

The az devops invoke command is neat alternative to using the REST API, but understanding what command-line arguments you'll need isn't obvious.

Let's start by finding out which endpoints are available by calling az devops invoke with no arguments and pipe this to a file for reference:

az devops invoke > az_devops_invoke.json 

This will take a few moments to produce. I've got a full listing of endpoints located here.

The list of endpoints are grouped by 'Area' and have a unique 'resourceName' and 'routeTemplate'. Here's an snippet:

 ... 
{ 
   "area": "boards", 
   "id": 
   "7f9949a0-95c2-4c29-9efd-c7f73fb27a63", 
   "maxVersion": 5.1, 
   "minVersion": 5.0, 
   "releasedVersion": "0.0", 
   "resourceName": "items", 
   "resourceVersion": 1, 
   "routeTemplate": "{project}/_apis/{area}/boards/{board}/{resource}/{*id}" 
},
{ 
   "area": "build", 
   "id": "5a21f5d2-5642-47e4-a0bd-1356e6731bee", 
   "maxVersion": 6.0, 
   "minVersion": 2.0, 
   "releasedVersion": "5.1", 
   "resourceName": "workitems", 
   "resourceVersion": 2, 
   "routeTemplate": "{project}/_apis/{area}/builds/{buildId}/{resource}" 
}
...

You can also use the JMESPath query syntax to reduce the list:

az devops invoke --query "[?area == 'build']"

Interesting note: If you study the source code for the az devops cli extension, you'll notice that all commands in the devops extension are using this same list as the underlying communication mechanism.

Finding the right endpoint

Finding the desired API in the list of endpoints might take a bit of research. All of the endpoints are grouped by 'area' and then 'resourceName'. I find that the 'area' keyword lines up fairly close with the API documentation, but you'll have to hunt through the endpoint list until you find the 'routeTemplate' that matches the API you're interested in.

Let's use the Get Latest Build REST API as an example. It's REST endpoint is defined as:

GET https://dev.azure.com/{organization}/{project}/_apis/build/latest/{definition}?api-version=6.0-preview.1

The routeTemplate is parameterized such that area and resource parameters correspond to the area and resourceName in the object definition.

From this, we hunt through all the 'build' endpoints until we find this matching endpoint:

{
  "area": "build",
  "id": "54481611-01f4-47f3-998f-160da0f0c229",
  "maxVersion": 6.0,
  "minVersion": 5.0,
  "releasedVersion": "0.0",
  "resourceName": "latest",
  "resourceVersion": 1,
  "routeTemplate": "{project}/_apis/{area}/{resource}/{definition}"
}

Invoking endpoints

Once you've identified the endpoint from the endpoint list, next you need to map the values from the route template to the command-line.

The mapping between command-line arguments and the routeTemplate should be fairly obvious. The values for "{area}" and "{resource}" are picked up from their corresponding command-line arguments, and the remaining arguments must be supplied as name-value pairs with the --route-parameters argument.

Using our Get Latest Build example, "{project}" and "{definition}" are provided on the command line like this:

az devops invoke `
     --area build `
     --resource latest `
     --organization https://dev.azure.com/myorgname `
     --route-parameters `
          project="MyProject" `
          definition=1234 `
     --api-version=6.0-preview

Adding query-string parameters

We can further extend this example by specifying query string parameters using the --query-parameters argument. In this example, we can get the latest build for a specific branch by specifying the branchName parameter:

az devops invoke `
     --area build --resource latest `
     --organization https://dev.azure.com/myorgname `
     --route-parameters project="MyProject" defintion=1234 `
     --query-parameters `
          branchName=develop `
     --api-version=6.0-preview

Note that while the CLI will validate route-parameters, it does not complain if you specify a query-string parameter that is misspelled or not supported.

Specifying the API Version

One of the challenges is knowing which API version to use. Frankly, I've had the most luck by specifying the latest version (eg 6.0-preview). As a general rule, the releasedVersion in the endpoint list should indicate which version to use, which is constrained by the 'maxVersion'.

If the releaseVersion is set to "0.0", then the preview flag is required.

Providing a JSON Body

To provide a JSON body for PUT and POST requests, you'll need to provide a JSON file using the --in-file and --httpMethod parameters.

For example, cancelling a build:

# Write the JSON body to disk
'{ "status": "Cancelling"}' | Out-File -FilePath .\body.json

# PATCH the build with a cancelling status
az devops invoke `
     --area build `
     --resource builds `
     --organization https://dev.azure.com/myorgname `
     --route-parameters `
          project="MyProject" `
          buildId=2345 `
     --in-file .\body.json `
     --http-method patch

Known Issues

By design, you would assume that the area and resourceNames in the list of endpoints are intended to be unique, but unfortunately this isn't the case. Again, referring to the source code of the extension, when trying to locate the endpoints by area + resource it appears to be a first-past-the-post scenario where only the first closest match is considered.

In this scenario, it would be helpful if we could specify the endpoint id from the command-line but this isn't supported yet.

To see the duplicates (it's not a small list):

$endpoints = az devops invoke | Select-Object -skip 1 | ConvertFrom-Json
$endpoints | Group-Object -Property area,resourceName | Where-Object { $_.Count -gt 1 }

The important thing to realize is that this list isn't unique to the az devops extension, it's actually a global list which is exposed from Azure DevOps. Perhaps how this list is obtained is something I'll blog about later.

Wrapping Up

While there are still somethings that are easier to do using the REST API, the Azure DevOps CLI offers a built-in capability to invoke the majority of the underlying APIs, though the biggest challenge is finding the right endpoint to use.

My personal preference is to start with the Azure DevOps CLI because I can jump in and start developing without having to worry about authentication headers, etc. I can also combine the results JMESPath filtering.

Quick Start with the JIRA REST API Browser

Looking to start programming against the JIRA API? Me too! This post walks through setting up JIRA locally on your computer and installing the JIRA API Explorer to start poking at the APIs.

Why use the REST API Browser?

Good question. Postman and other tools can poke at JIRA endpoints just as well, but the API Explorer comes equipped with help documentation and a lets you quickly pick out an endpoint without having to worry about authentication, etc. You can easily toggle between GET, POST, PUT, DELETE commands for endpoints that support it, and the explorer provides a simple form entering required and optional fields.

Setup a JIRA instance in Docker

Setting up JIRA in a docker container is pretty easy.

docker volume create --name jiraVolume
docker run -v jiraVolume:/var/atlassian/application-data/jira --name="jira" -d -p 8080:8080 atlassian/jira-software

After the docker image is downloaded, open your browser to http://localhost:8080

On first start, you will be prompted with a series of prompts:

• Set it up for me
• Create a trial license key
• Setup an admin user account
• Create an empty project, use a sample project or import from an existing project.

Quick Setup

When you first navigate to the docker instance, you'll be prompted with a choice:

Let's go with "Set it up for me", and then click on "Continue to MyAtlassian"

Create a Trial License

You'll be redirected to provide information for creating a trial license. Select "JIRA Server", provide a name for your organization and leave the default values for the server name.

When you click next, you'll be redirected to a screen that shows the license key and a Confirmation dialog.

Click Yes to confirm the installation.

Create an Admin User Account and default settings

Somewhat self-explanatory, provide the details for your admin account and click Next.

You should see a setup page run for a few minutes. When this completes, you'll need to:

• Login with the admin account you created
• Pick your default language
• Choose an Avatar

Create your First Project

The last step of the Quick Setup is to create your first project. Here you can create an empty project, import from an existing project or use a sample project. As we're interested in playing with the API, pick the "See it in action" option to create a JIRA project with a prepopulated backlog.

Selecting "See it in action", prompts for the style of project and the project details.

Great job. That was easy.

Install the API Browser

The API Explorer is available through the Atlassian Marketplace. To install the add-on:

1. In the top right, select Settings ⟶ Applications
2. Click the "Manage Apps" tab
3. Search for "API Browser"
   
   
   
4. Click "Install"

Super easy.

Using the REST API Browser

The hardest part of using the REST API Browser is locating it after it's been installed. Fortunately, that's easy, too.

In the top-right, select Settings ⟶ System and scroll all the way to the bottom to Advanced: REST API Browser

Try it out!

The REST API Browser is fairly intuitive, simply find the endpoint you're interested in on the right hand side and provide the necessary fields in the form.

Mind your verbs...

Pay special attention to the HTTP verbs being used: GET, POST, PUT, DELETE. Most of JIRA's endpoints differ only by the verb, items that are POST + PUT will either create or update records and not all endpoints will have a GET.

Wrap up

Now you have now excuse to start picking at the JIRA API.

Side note: I wrote this post using Typora, a super elegant markdown editor. I'm still experimenting on how to best integrate with my blog platform (Blogger), but I might look at some form of combination of static markdown.md files with a command-line option to publish. I will most likely post that setup when I get there.

Until then, happy coding.

Cleaning up stale git branches

If you have pull-requests, you likely have stale branches. What’s the best way to find which branches can be safely deleted? This post will explore some approaches to find and delete stale branches.

As there are many different ways we can approach this, I’m going to start with the most generic concepts and build up to a more programmatic solution. I want to use the Azure DevOps CLI and REST APIs for this instead of git-centric commands because I want to be able to run the scripts from any computer against the latest version of the repository. This also opens up the possibility of running these activities in a PowerShell script in an Azure Pipeline, as outlined in my previous post.

Table of Contents

• Who can Delete Branches?
• Finding the Author of a Branch
• Finding the Last Contributor to a Branch
• Check the Expiry Date
• Finding branches that have completed Pull-Requests
• Wrapping up

Who can delete Branches?

One of the reasons branches don’t get deleted might be a permissions problem. In Azure DevOps, the default permission settings set up the creator of the branch with the permission to delete it. If you don’t have permission, the option isn’t available to you in the user-interface, and this creates a missed opportunity to remediate the issue when someone other than the author completes the PR.

You can change the default setting by adding the appropriate user or group to the repository’s permissions and the existing branches will inherit. You’ll need the “Force push (rewrite history, delete branches and tags)” permission to delete the branch.  See my last post on ways to apply this policy programmatically.

If we want to run this from a pipeline, we would have to grant the Build Service the same permissions.

Finding the Author of a Branch

One approach to cleaning-up stale branches is the old fashion way: nagging. Simply track down the branch authors and ask them to determine if they’re done with them.

We can find the authors for the branches with the following PowerShell + Az DevOps CLI:

$project    = "<project-name>"
$repository = "<repository-name>"

$refs = az repos ref list `
           --query "[?starts_with(name, 'refs/heads')].{name:name, uniqueName:creator.uniqueName}" `
           --project $project --repository $repository |
        ConvertFrom-Json

$refs | Sort-Object -Property uniqueName

Finding the Last Contributor to a Branch

Sometimes, the author of the branch isn’t the person doing the work. If this is the case, you need to track down the last person to commit against the branch. This information is available in the Azure DevOps user interface (Repos –> Branches):

If you want to obtain this information programmatically, az repos list ref provides us with the objectId SHA-1 of the most recent commit. Although az repos doesn’t expose a command to retrieve the commit details, we can use the az devops invoke command to call the Get Commit REST endpoint.

When we fetch the detailed information on the branch, we want to get the author that created the commit and the date that they pushed it to the server. We want the push details because a developer may have made the commit a long time ago but only recently updated the branch.

$project    = "<project-name>"
$repository = "<repository-name>"

$refs = az repos ref list –p $project –r $repository --filter heads | ConvertFrom-Json

$results = @()

foreach($ref in $refs) {

    $objectId = $ref.objectId

    # fetch individual commit details
    $commit = az devops invoke `
                 --area git `
                 --resource commits `
                 --route-parameters `
                     project=$project `
                     repositoryId=$repository `
                     commitId=$objectId |
              ConvertFrom-Json

    $result = [PSCustomObject]@{ 
        name = $ref.name
        creator = $ref.creator.uniqueName
        lastAuthor = $commit.committer.email
        lastModified = $commit.push.date
    }

    $results += ,$result
}

$results | Sort-Object -Property lastAuthor

This gives us a lot of details for the last commit in each branch, but if you’ve got a lot of branches, fetching each commit individually could be really slow. So, instead we can use the same Get Commit endpoint to fetch the commit information in batches by providing a collection of objectIds in a comma-delimited format.

Note that there’s a limit to how many commits we can ask for at a time, so I’ll have to batch my batches. I could also use the Get Commits Batch endpoint that accepts the list of ids in the body of a POST message.

The following shows me batching 50 commits at a time. Your batch size may vary if the name of your server or organization name is a longer length:

$batchSize = 50
$batches = [Math]::Ceiling($refs.Length / $batchSize)

for( $x=0; $x -lt $batches; $x++ )
{
    # take a batch
    $batch = $refs | Select-Object -First $batchSize -Skip ($x * $batchSize)

    # grab the ids for the batch
    $ids = ($batch | ForEach-Object {$_.objectId}) -join ','

    # ask for the commit details for these items
    $commits = az devops invoke --area git --resource commits `
                  --route-parameters `
                    project=$project `
                    repositoryId=$repository `
                  --query-parameters `
                    searchCriteria.ids=$ids `
                    searchCriteria.includePushData=true `
                  --query "value[]" | ConvertFrom-Json

    # loop through this batch of commits
    for($i=0; $i -lt $commits.Length; $i++) {

        $ref = $refs[($x*$batchSize)+$i]
        $commit = $commits[$i]

        # add commit information to the batch
        $ref | Add-Member -Name "author" -Value $commit.author.email -MemberType NoteProperty
        $ref | Add-Member -Name "lastModified" -Value $commit.push.date -MemberType NoteProperty 

        # add the creator’s email on here for easier access in the select-object statement...
        $ref | Add-Member –Name "uniqueName” –Value $ref.creator.uniqueName –MemberType NoteProperty
    }
}

$refs | Select-Object -Property name,creator,author,lastModified

Caveat about this approach: If you've updated the source branch by merging from the target branch, the last author will be from that target branch – which isn’t what we want. Even worse, there's no way to infer this scenario from the git commit details alone. One way we can solve this problem is to fetch the commits in the branch and walk up the parents of the commit until we find a commit that has more than one parent – this would be our merge commit from the target branch, which should have been done by the last author on the branch. Note that the parent information is only available if you query these items one-by-one, so this approach could be painfully slow. (If you know a better approach, let me know)

Check the Expiry

Now that we have date information associated to our branches, we can start to filter out the branches that should be considered stale. In my opinion anything that’s older than 3 weeks is a good starting point.

$date = [DateTime]::Today.AddDays( -21 )
$refs = $refs | Where-Object { $_.lastModified -lt $date }

Your kilometrage will obviously vary based on the volume of work in your repository, but on a recent project 10-20% of the branches were created recently.

Finding Branches that have Completed Pull-Requests

If you’re squashing your commits when your merge, you’ll find that the ahead / behind feature in the Azure DevOps UI is completely unreliable. This is because a squash merge re-writes history, so your commits in the branch will never appear in the target-branch at all. Microsoft recommends deleting the source branch when using this strategy as there is little value in keeping these branches around after the PR is completed. Teams may argue that they want to cherry-pick individual commits from the source-branch, but the practicality of that requires pristine

Our best bet to find stale branches is to look at the Pull-Request history and consider all branches that are associated to completed Pull-Requests as candidates for deletion. This is super easy, barely an inconvenience.

$prs = az repos pr list `
          --project $project `
          --repository $repository `
          --target-branch develop `
          --status completed `
          --query "[].sourceRefName" |
       ConvertFrom-Json |

$refs | 
    Where-Object { $prs.Contains( $_.name ) } |
    ForEach-Object {
       $result = az repos ref delete `
                    --name $_.name `
                    --object-id $_.id `
                    --project $project `
                    --repository $repository |
                ConvertFrom-Json
       Write-Host ("Success Message: {0}" –f $result.updateStatus)
    }

At first glance, this would remove about 50% of the remaining branches in our repository, leaving us with 10-20% recent branches and an additional 30-40% of branches without PRs. This is roughly a 40% reduction, and I’ll take that for now. It’s important to recognize this only includes the completed PRs, not the active or abandoned.

Wrapping Up

Using a combination of these techniques we could easily reduce the amount of stale branches, and then provide the remaining list to the team to have them clean-up the dredges. The majority of old branches are likely abandoned work, but there's sometimes scenarios where partially completed work is waiting on some external dependency. In that scenario, encourage the team to keep these important branches up-to-date to retain the value of the invested effort.

The best overall strategy is to adopt a strategy that does not let this situation occur: give the individuals reviewing and completing PRs the permission to delete branches and encourage teams to squash and delete branches as they complete PRs. Good habits create good hygiene.

Happy coding.

Securing Git Branches through Azure DevOps CLI

I've been looking for an way to automate branch security in order to enforce branch naming conventions and to control who can create release branches. Although the Azure DevOps documentation illustrates how to do this using the tfssecurity.exe command, the documentation also suggests that the tfssecurity.exe command is now deprecated.

This post will walk through how to apply branch security to your Azure DevOps repository using the Azure DevOps CLI.

Table of Contents

• Understanding Azure DevOps Security
• Security Tokens for Repositories and Branches
• Converting Branch Names to git hex format
• Obtaining Branch Name from Git Hex Format
• Fetching Group Descriptors
• Apply Branch Security
• Putting it all together

Understanding Azure DevOps Security

The Azure DevOps Security API is quite interesting as security can be applied to various areas of the platform, including permissions for the project, build pipeline, service-connection, git repositories, etc. Each of these areas support the ability to assign permissions for groups or individuals to a security token. In some cases these tokens are hierarchical, so changes made at the root are inherited on children nodes. The areas that define the permissions are defined as Security Namespaces, and each token has a Security Access Control List that contains Access Control Entries.

We can obtain a complete list of security namespaces by querying https://dev.azure.com/<organization>/_apis/securitynamespaces, or by querying them using the az devops cli:

az devops security permission namespace list

Each security namespace contains a list of actions, which are defined as bit flags. The following shows the "Git Repositories" security namespace:

az devops security permission namespace list `
    --query "[?contains(name,'Git Repositories')] | [0]"

{
    "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87",
    "name": "Git Repositories",
    "displayName": "Git Repositories",
    "separatorValue": "/",
    "elementLength": -1,
    "writePermission": 8192,
    "readPermission": 2,
    "dataspaceCategory": "Git",
    "actions": [
        {
            "bit": 1,
            "name": "Administer",
            "displayName": "Administer",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 2,
            "name": "GenericRead",
            "displayName": "Read",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 4,
            "name": "GenericContribute",
            "displayName": "Contribute",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 8,
            "name": "ForcePush",
            "displayName": "Force push (rewrite history, delete branches and tags)",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 16,
            "name": "CreateBranch",
            "displayName": "Create branch",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 32,
            "name": "CreateTag",
            "displayName": "Create tag",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 64,
            "name": "ManageNote",
            "displayName": "Manage notes",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 128,
            "name": "PolicyExempt",
            "displayName": "Bypass policies when pushing",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 256,
            "name": "CreateRepository",
            "displayName": "Create repository",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 512,
            "name": "DeleteRepository",
            "displayName": "Delete repository",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 1024,
            "name": "RenameRepository",
            "displayName": "Rename repository",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 2048,
            "name": "EditPolicies",
            "displayName": "Edit policies",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 4096,
            "name": "RemoveOthersLocks",
            "displayName": "Remove others' locks",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 8192,
            "name": "ManagePermissions",
            "displayName": "Manage permissions",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 16384,
            "name": "PullRequestContribute",
            "displayName": "Contribute to pull requests",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        },
        {
            "bit": 32768,
            "name": "PullRequestBypassPolicy",
            "displayName": "Bypass policies when completing pull requests",
            "namespaceId": "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"
        }
    ],
    "structureValue": 1,
    "extensionType": "Microsoft.TeamFoundation.Git.Server.Plugins.GitSecurityNamespaceExtension",
    "isRemotable": true,
    "useTokenTranslator": true,
    "systemBitMask": 0
}

Security Tokens for Repositories and Branches

The tokens within the Git Repositories Security Namespace follow the naming convention repoV2/<projectId>/<repositoryId>/<branch>. As this is a hierarchical security namespace, you can target very specific and granular permissions by adding parameters from left to right.

Examples:

• All repositories within the organization: repoV2/
• All repositories within a project: repoV2/<projectId>
• All branches within a repository: repoV2/<projectId>/<repositoryId>
• Specific branch: repoV2/<projectId>/<repositoryId>/<branch>

As the tokens are hierarchial, a really cool feature is that we can define patterns for branches that do not exist yet.

While the project and repository elements are relatively self-explanatory, the git branch convention is case sensitive and expressed in a hex format. Both Jesse Houwing and the Azure DevOps blog have some good write-ups on understanding this format.

Converting Branch Names to git hex format

The following PowerShell script can produce a security token for your project, repository or branch.

function Get-RepoSecurityToken( [string]$projectId, [string]$repositoryId, [string]$branchName) {

   $builder = "repoV2/"

   if ( ![string]::IsNullOrEmpty($projectId) ) {

     $builder += $projectId
     $builder += "/"

     if ( ![string]::IsNullOrEmpty($repositoryId) ) {

        $builder += $repositoryId
        $builder += "/"

        if ( ![string]::IsNullOrEmpty( $branchName) ) {

            $builder += "refs/heads/"
            
            # remove extra values if provided
            if ( $branchName.StartsWith("/refs/heads/") ) {
                $branchName = $branchName.Replace("/refs/heads/", "")
            }
            if ( $branchName.EndsWith("/")) {
                $branchName = $branchName.TrimEnd("/")
            }

            $builder += (($branchName.Split('/')) | ForEach-Object { ConvertTo-HexFormat $_ }) -join '/'   
        }

     }
   }

   return $builder
}

function ConvertTo-HexFormat([string]$branchName) {
   return ($branchName | Format-Hex -Encoding Unicode | Select-Object -Expand Bytes | ForEach-Object { '{0:x2}' -f $_ }) -join ''
}

Obtaining Branch Name from Git Hex Format

If you're working with query results and would like to see the actual name of the branches you've assigned, this function can reverse the hex format into a human readable string.

function ConvertFrom-GitSecurityToken([string]$token) {
    $refHeads = "/refs/heads/"

    $normalized = $token

    if ($token.Contains($refHeads)) {

        $indexOf = $token.IndexOf($refHeads) + $refHeads.Length

        $firstHalf = $token.Substring(0, $indexOf)
        $secondHalf = $token.Substring($indexOf)

        $normalized = $firstHalf
        $normalized += (($secondHalf.Split('/')) | ForEach-Object { ConvertFrom-HexFormat $_ }) -join '/'  
    }

    return $normalized
}

function ConvertFrom-HexFormat([string]$hexString) {

    $bytes = [byte[]]::new($hexString.Length/2)
    
    for($i = 0; $i -lt $hexString.Length; $i += 2) {
        $bytes[$i/2] = [convert]::ToByte($hexString.Substring($i,2), 16)
    }

    return [Text.Encoding]::Unicode.GetString($bytes)
}

Fetching Group Descriptors

Although it's incredibly easy to fetch security permissions for individual users, obtaining the permissions for user groups requires a special descriptor. To make them easier to work with, I'll grab all the security groups in the organization and map them into a simple lookup table:

function Get-SecurityDescriptors() {

    $lookup = @{}
    
    $decriptors = az devops security group list --scope organization --query "graphGroups[]" | ConvertFrom-Json
    
    $descriptors | ForEach-Object { $lookup.Add( $_.principalName, $_.descriptor) }
    
    return $descriptors
}

Apply Branch Security

Given that we'll call the function several times, we'll wrap it in a method to make it easier to use.

$namespaceId = "2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87"

function Set-BranchSecurity( $descriptor, $token, $allow = 0 , $deny = 0) {

    $result = az devops security permission update `
                --id $namespaceId `
                --subject $descriptor `
                --token $token `
                --allow-bit $allow `
                --deny-bit $deny `
                --only-show-errors | ConvertFrom-Json
}

Putting it All Together

This crude example shows how to apply the guidance laid out in the Require branches to be created in folders article, to a specific repository:

$project   = "projectName"
$projectId = az devops project list --query "value[?name=='$project'].id | [0]" | ConvertFrom-Json

# grab the first repo in the project
$repoId    = az repos list --project $project --query [0].id | ConvertFrom-Json

$groups = Get-SecurityDescriptors
$administrators = $groups[ "[$project]\\Project Administrators" ]
$contributors   = $groups[ "[$project]\\Contributors" ]

# a simple array of tokens we can refer to
$tokens = @(
                Get-RepoSecurityToken( $projectId, $repoId ),             # repo - 0
                Get-RepoSecurityToken( $projectId, $repoId, "main" ),     # main - 1
                Get-RepoSecurityToken( $projectId, $repoId, "releases"),  # releases/* - 2
                Get-RepoSecurityToken( $projectId, $repoId, "feature"),   # feature/* - 3
                Get-RepoSecurityToken( $projectId, $repoId, "users")      # users/* - 4
           )
           
$CreateBranch = 16


# prevent contributors from creating branches at the root of the repository
Set-BranchSecurity $contributors, $tokens[0], -deny $CreateBranch

# limit users to only create feature and user branches
Set-BranchSecurity $contributors, $tokens[3], -allow $CreateBranch
Set-BranchSecurity $contributors, $tokens[4], -allow $CreateBranch

# restrict who can create a release
Set-BranchSecurity $admins, $token[2], -allow $CreateBranch

# allow admins to recreate master/main
Set-BranchSecurity $admins, $token[1], -allow $CreateBranch

To improve upon this, you could describe each of these expressions as a ruleset in a JSON format and apply them to all the repositories in a project.

Some considerations:

• Granting Build Service accounts permissions to create Tags
• Empowering the Project Collection Administrators with the ability to override branch policy
• Empowering certain Teams or Groups with the ability to delete feature or user branches

Happy coding!