Pain removal with BizTalk Deployment Automation

14 March, 2018 by Matt Corr

Introduction

Microsoft BizTalk has been around for a long time. It was first released at the end of 2000, with the latest version, Biztalk 2016 released at the end of 2016. It was already a well-established technology before there were even whispers of the words "devops" or "automation", now commonplace terms in 2018.

In the past, the deployment options for BizTalk applications have been limited, manual and time-consuming. It was not uncommon to be deploying from Visual Studio or have large checklist word documents or using limited MSIs that only did about 40% of the job. But thankfully there are now existing and battle tested tools available in the workplace that allows us to automate the entire process. 

This gives us the assurance that BizTalk applications can be deployed to any environment in working and tested order with full traceability to what has changed and when.

There are various tools available of course, but we will be discussing the ones that have worked well for us. These include:

One of the advantages of the tools above, is if you have a small number of machines to deploy to, there will be very little or no cost to use them.

In this article we are covering the deployment process for BizTalk 2013 R2. This is due to 2013 R2 being the primary version of BizTalk deployed at most of our clients. With the newer version of BizTalk 2016, there is a Feature pack which does add support for a BizTalk Application Projects to help with deployments. But that is still quite new and a “work in progress”, and not available for BizTalk 2013 R2 solutions.

The approach we use is one we have implemented several times for different clients. This has been consistently successful in removing the pain points for application deployment. We are not suggesting that this is the only way to do this, but by sharing our experiences we hope to at least give you and your company ideas on how to improve your deployment process.

BizTalk Application dependencies

Before we start, I want to mention one of the key reasons DevOps is needed for BizTalk.

One of the issues with BizTalk deployments is that there can be dependencies between applications. If an application is a dependency for other applications, they will need to be removed first before you can perform an upgrade. This is due to BizTalk not supporting the same schema in multiple applications so dependencies can be hard to avoid. Normally if there are schemas to be shared, they are put into a Schemas BizTalk application and Logic based applications (containing maps, orchestrations, pipelines) which have a dependency on the Schema only applications.

It is also advisable to group the schemas by a “domain” or an “area” so instead of having something like the image below, this means if you have to update the schema application on the bottom you will have to redeploy all logic applications which is not desirable.

many-dependencies.png

The “dependency by domain” approach is preferred in a scenario where you have to redeploy a schema app, and would only need to redeploy the impacted applications. Since you will most likely need to redeploy them anyway to cater for the updated Schema, this should have less of an impact and restrict how far the “redeploy the world” would go.

domain-dependencies.png

It is not advisable to create many layers of dependencies; instead it is recommended to keep the dependencies as shallow as possible. More layers of dependencies will exponentially increase the complexity overall and will introduce more deployment issues.

This is not a problem when you have a few applications that can be easily managed. See below image: 

simple-apps.png

But when you start deploying many applications with dependencies and even have different client projects with different groups of applications, it can quickly become very complex and the need for effective and automated deployments becomes critical.

So lets get started, or as they say “let's rub some DevOps on it!”.

BizTalk Application in Visual Studio

There is no 'magic trick' required for the BizTalk artefacts themselves. Create them as per your normal process. Mexia prefers to break up BizTalk artefacts into separate DLLs. This helps with granularity and also works well with the BTDF configuration file.

solution-explorer.png

BTDF to encapsulate application deployment

When you add a Deployment Framework for BizTalk Project to your solution, this by default will store some files in the <solution folder>\Deployment folder.

btdf-solution.png

BTDF encapsulates the deployment of all BizTalk artefacts into a single MSI executable. Other external artefacts like BAM, Business Rules, SQL queries, IIS websites and PowerShell commands can be added to BTDF as well to ensure that everything you need for the BizTalk application is there. No more install a BizTalk project, then follow another checklist of additional steps to ensure you have everything.

There are three main files in the Deployment folder:

Deployment.btdfproj

The main configuration file in XML format. It contains many switches for what the deployment will do and the list of artefacts to deploy and any external components in a readable format. Example snippet of part of the config file:

2018-02-27_16-33-30.png

SettingsFileGenerator.xml

This is an XML file that is opened in Microsoft Excel in order to edit the values in an easy to view table format. Normally you would put all of the application variables that would be different in each deployed environment in this file. This is a good start, but it is very static.

There are also issues with passwords for accounts in plain text and if any settings were changed, then you would have to check in new code changes for them to be applied.

This is where Octopus Deploy comes in. When we use Octopus, we only need two columns for all environments. One for Local Development and one for Octopus. Octopus itself will handle the values of variables. The format of their variables is #{<variable name>} . More on that later. Below is a sample of how a SettingsFileGenerator.xml file could look.

SettingsFileGenerator.png

When the application is deployed locally, the Exported_LocalSettings.xml is used. When the application is deployed via Octopus, the Exported_OctopusSettings.xml file is used.

PortBindingsMaster.xml

This contains the binding files for the BizTalk application. Once all the values required are in here (configured in your local environment), you would then substitute any values that would be different in deployed environments. An example would be:

Before modification:

2018-02-27_12-19-49.png

After modification:

2018-02-27_12-20-11.png

During deployment, these tokens (along with the values in the SettingsFileGenerator.xml file) will get updated with the correct values for that environment.

Utilise a build server and agent

We can utilise either TeamCity and VSTS Build for building and creating the BTDF MSI on a build agent. Both applications handle the build process well, but Mexia has a preference for TeamCity as its feature list is still more comprehensive than VSTS Build (but the gap is closing).

The basic process to use is as follows:

  • Code check in detected - this triggers the build
  • Increment the version count
  • Application is built in debug mode
  • Unit test cases are run against the debug build DLLs
  • Application is built in release mode
  • Application BTDF MSI package is created
  • Generate Release Notes from check in change details
  • Create an Octopus Package containing at least the MSI executable, deployed test files and the Exported_OctopusSettings.xml file
  • Push the package to the Octopus Deploy server
  • Create a release for that version with that package on the Octopus Server
    • release creation will contain the release notes as generated earlier

Release Notes

Release notes are an important part of the process. They tie the release object in Octopus to the build process and the list of what has changed. When a developer checks in code, they can add a detailed comment about the change and a reference to a issue or bug number.

The release notes are generated in markdown so they can include the comment, the link to the issue and the list of the files modified. This can include hyperlinks to the build server showing the details. So when this is viewed in Octopus, it is a single click to the issues related to a change, and also to see exactly what the differences are. This can be very useful for tracking down what is different in a release that has introduced a bug.

Octopus Deploy

For each BizTalk application there is to be a single Octopus project. This will provide useful information on the dashboard showing the version of each application deployed into each environment and when and if it was a success.

Octopus-dashboard.png

Variables

How variables are utilised in Octopus projects is one of the primary features that makes this tool so powerful. Each of the steps in the process above can have one or more parameters defined. These will all need to be set in the steps or set as variables (the preferred option).

The Octopus project will need to have variables defined that provide values for all the tokens in the Exported_OctopusSettings.xml and PortBindingsMaster.xml files.

Variables can either be defined in the project context or via variable sets which are global variable lists that can be inherited by projects. The advantage of this is common variables for many projects can be defined once and referenced by all projects.

Variables can also be nested. So instead of having a url defined as https://biztalk01svr/customer/list.svc
it can be
https://#{BiztalkServer.Name}/customer/list.svc
and the BiztalkServer.Name variable can be defined in a variable set.

Variables can be scoped to various attributes such as environment, roles, step names or channels. They provide a lot of control over what the values of each variable will be for each environment, or for all environments if there is no scope.

Deployment process

Once the projects are created and the variables defined and scoped, we need to focus on what will be performed as part of the deployment into each environment. Each Octopus project has a process containing steps to execute when deployed.

BizTalk based projects can follow this general step structure:

  • Deploy the package generated from the build server to the BizTalk Server
    • this will replace all the #{tokens} in the files
  • Validate the Exported_OctopusSettings.xml and PortBindingsMaster.xml files to ensure all tokens have been replaced
  • Ensure a newer version is not installed, if so, then remove it
  • Install the MSI on the server
  • Run the application installation
    • If the application with the same version or earlier is there, it will be replaced
    • If there are any applications listed as dependancies to this application, they will be removed.
  • Update any un-deployed applications
    • This will query to see if any other installed applications were removed. If so, their status in Octopus is updated so it is visible.
  • Set the tracking level for application
  • Run any integration or system testing for this application (if any defined)
  • Optionally send a notification to a group saying the application has been deployed

Application removal

Octopus has recently added features that makes it much easier to handle the notification of application removal. This is important so I have created a section about this here.

When a BizTalk application is installed, it's possible that some applications that are dependent on it will also be removed. However this is all contained within the BTDF process, so Octopus won’t know that this has occurred. We get around this by having a step in the deployment process that checks for removed applications and alters the state of installed apps in Octopus if they have been altered.

octopus-removal-before.png

With the list of applications above, The Customer and Insurance applications are both dependant on the Schemas application. If we were to deploy 1.0.3 from DEV into Test, it would remove both Customer and Insurance.

octopus-removal-after.png

Here we can see 1.0.3 has been successfully installed, with the Octopus UI updated to also reflect the two applications that were removed.

The user can then choose to redeploy the applications to return the dashboard to all green. The important thing is the user now knows they have been removed, so they can take the appropriate action.

Solution Projects

For a client project, it is usually rare to just have a single BizTalk application deployed in Octopus. There are likely to be a group of applications and even applications in different technologies (like websites or gateway policies). There may also be the need to perform some pre or post steps after the applications are deployed (ie check a change request for approval before deployment, running external system tests after deployment or sending notifications after deployment). If this is the case then it is worth exploring another recently added Octopus feature called “Deploy a Release” step.

This involves creating an Octopus project that contains one or more child projects in the process definition. Additional steps can be included like checking change management systems, running system tests across all deployed applications or sending notifications.

octopus-process.png

We call these sorts of projects a “solution project”. These solution projects would be what is migrated through controlled environments and will ensure that everything that is needed for the entire solution is present. So instead of having a list of many Octopus projects to deploy and having to remember to send notifications afterwards, this can all be automated and the order of applications getting deployed will ensure dependency conflicts can be avoided.

Octopus REST API

Octopus Deploy has a well documented and very detailed REST API available. This can be heavily utilised either by our own applications or scripts to make our own functionality or to provide more detail and comparisons to project data. This way if there are any gaps in the process or “wouldn’t it be nice if you could do x” moments, there is a high chance it can be addressed by using the REST API.

Summary

This has been a quick summary of some of the different tools that can be utilised in order to make BizTalk application deployment pain a thing of the past. No longer do you need to worry about issues like:

  • creating a large word/pdf document with massive checklists for manual deployments
  • wondering if a deployed application is actually tested and working
  • wondering what versions of applications are running in which environment
  • wondering what the changes are for an application release
  • being worried if applications have been built consistently

With a little know-how and passion these issues can all be automated away and the entire BizTalk deployment process can become well tested, reported, transparent and painless!

Questions? Get in touch with one of our integration experts. 

Latest Posts

Tags