In my previous post, I talked about the messages conveyed by the Microsoft product team at the INTEGRATE 2017 event in London a few weeks ago. In this post, I’ll discuss the sessions that were presented by the Integration MVP community, which made up almost half of the 3-day agenda.
BizTalk Server 2016 featured in a number of presentations by the Microsoft team, with sessions led by Tord Glad Nordahl and Saffieldin Ali focusing on the Feature Pack 1 and health check practices. But there were also three sessions offered by veteran MVP Sandro Pereira and MVP Wagner Silveira. Also, Saravana Kumar talked about BizTalk Server indirectly in his presentation on BizTalk360.
Sandro gave a dynamic presentation on increasing the performance of BizTalk Server, appropriately named “BizTalk Server Fast & Loud”. Most of his tips centred on the importance of managing the underlying SQL Server databases. He expressed his amazement that even though BizTalk is seventeen years old, many organisations still fail to correctly configure and enable the SQL Agent Jobs! (Apparently they are not reading some of the numerous blog posts out there like this one.) And of course, you need to monitor the jobs and ensure the jobs are running successfully; Saravana would have some great tips about this using BizTalk360 alerts!
Whilst many might consider scaling out the BizTalk Server tier to solve performance bottlenecks, they really should be considering how to optimise the configuration of the SQL Server; for example, ensuring the separation of logical disks for data, transactions logs, and the tempdb. Other tips include pre-allocating disk space to the BizTalkMsgBoxDb and the BizTalkDTADb databases, splitting the tempdb database into separate data files, and (for serious speed demands) provisioning up to eight dedicated drives for each database.
In addition to these tips for the SQL Server, Sandro also advocated using multiple host instances, including one dedicated host for tracking (a best practice that Mexia has been following for years). And his clever analogy of BizTalk Server to a racing car was particular effective in a presentation that concentrates on improving performance!
Wagner Silveira also featured BizTalk Server quite heavily in his presentation about Hybrid Integration, including a number of demos using the Logic Apps adapter, the REST adapter, and the Service Bus adapter to connect Azure to on-prem and vice-versa.
This was the most heavily featured technology within the event, with four Microsoft sessions and as many MVP sessions devoted to it.
First up was a session by Kent Weare on using Logic Apps to establish connectivity with BOTS. While the majority of his presentation focussed on Cognitive Services and the value they can add to digital transformation, Kent’s demo of a “Crypto BOT” service of doing trades with Bitcoin and Ethereum was wired together using Logic Apps as the connectivity vehicle to QuadrigaCX and Azure SQL:
(More about Kent’s presentation later in this post.)
Immediately following Kent, Steef-Jan Wiggers used his session to talk about empowering the business with Logic Apps. After explaining his green jersey (having lost a bet with Sandro Pereira around the European football championships), Steef-Jan launched into a business case for using the fully iPaaS solution offered by Logic Apps:
- Cloud First
- Microsoft unless (Azure)
- Minimal customisation (no code)
- Unified IT landscape
After showing a couple of testimonials for organisations which rely on Logic Apps (My Tailored Energy, Phidiax, and also BizTalk360), Steef-Jan questioned why Gartner had not yet put Microsoft into the Leaders quadrant for iPaaS integration. Unfortunately this seems due to a perception that Logic Apps lack functional depth, are too Microsoft centric, and do not offer a migration path for BizTalk workloads. Most of these justifications almost seem misinformed given the advances the product has made in the last six to twelve months.
Steef-Jan then went on to reveal the results of a survey he conducted in which over 30 integration experts (including a few from Mexia) had participated in. Two of the questions asked were “What is your view on Logic Apps and its evolution”, and “What is the developer experience?”. The results were overwhelmingly favourable, although some of the quotes expressed a desire to reduce the necessity for code manipulation as well as concern about competing with other integration platforms. Before launching into a fascinating demo that uses Logic Apps, Cognitive Services, and Azure Functions to conduct sentiment analysis on #Integrate2017 tweets, Steef-Jan discussed some common patterns that Logic Apps can easily implement: Content-Based Routing, Aggregator, Splitter, and Process Manager. As tweeted by Tomasso Groenendijk, Steef-Jan showed “the best examples of how you can use Logic Apps”.
Next up was Johan Hedberg who talked about how to perform continuous integration and deployment (CI/CD) with Logic Apps and Visual Studio Team Services (VSTS). Johan used humorous personas to depict three different roles and approaches to ALM:
- “Pete” who likes to work in the portal and quickly deploy changes in production,
- “Charlotte” who works in Visual Studio and prefers a slow carefully managed process of migration using source controls, and
- “Bruce” who represents IT Ops approver and is the vigilant guardian of production.
Using gated deployments, “Bruce” can choose to approve or deny a release into production that has already been built, deployed and tested in the CI/CD pipeline.
Johan talked about some of the common concerns that an ALM pipeline needs to address:
- Development standards
- Resource groups
- Process standards
- Changes in production?
- Approval in test environment
- Source control? Master branch?
- Security Standards
- Who can see data in production
- Credentials management
- Deployment standards
- Touches on other things
- Through Visual Studio, or the Portal?
- Team communication & culture
Richard Seroter also featured a Logic App in his demo of building a pipeline to call an enrichment service. The point of this was to illustrate the concept of “composable systems” that:
- Are loosely coupled
- Choreograph services
- Keep logic in the endpoints (“dump pipes, smart endpoints”)
- Offer targeted updates.
(More about Richard’s talk later in this post.)
Last in the Logic Apps category was Stephen W. Thomas who based his talk on some real-world experience with using Logic Apps at a major client. Providing an absolute ton of useful information from the field (my notes cover three pages!), Stephen divided his talk into the following phases: 1) Decisions 2) Right & Wrong 3) Tips. The first phase was about justifying the decision to use Logic Apps other the other alternatives of BizTalk Server and custom .NET. In the end, it came down to the customer not having BizTalk skills or infrastructure in place, an aggressive timeline, and poor vendor integration requirements (which resulted in every single schema changing late in the game). The 2nd phase included a discussion about varied techniques and practices, some which proved to be very helpful and others…. not so much. Stephen mentioned difficulties with the platform always moving (2-week sprints by the product team) and sometimes introducing breaking features. He also alluded to difficulties with deploying across regions. A decided best practice from this experience was to use one subscription for each environment and not to cross regions with Logic Apps. The last phase was chocker full of great tips, too many to list here… but some notable ones include:
- Don’t be afraid to work with the underlying JSON – but always take a backup first as you can easily break the Logic App
- Name the action correctly upon creation as it is reference later
- Confine to one Microsoft account on the dev box (multiple accounts can cause headaches)
- Access the Azure Portal within an Incognito (In Private) browser window
- Define your connector parameters within a single Logic App
- Use one connection per destination per Resource Group and share it across many Logic Apps
Internet of Things (IoT)
Sam’s session focussed on the importance of integration in an IoT world. He talked about the different types of connectivity when it comes to IoT: Direct Connectivity, Cloud Gateways, and Field Gateways. The last option brings all of the complexity of IoT into a single place, and “feels” more secure. It also allows for “edge processing”, which can result in up to a 60% cost reduction over processing in the cloud. Some of the data processing activities of a field gateway include filtering, batching, compression, and aggregation. Sam believes that “bringing cloud capabilities to the edge is key in most enterprise solutions”. This led in naturally to a mention about NebulusTM, an IoT gateway product made by Sam’s company Codit.
Sam also talked about the difference between a Proof of Concept (where the emphasis is on quick value creation) and Real World Deployments (where robustness and security are primary concerns). He recommended spending less time/effort on generic infrastructure tasks (message handling/persistence, credential management, health monitoring, etc) and more on value generators (edge analytics, backend automation, business logic modules, rich reporting, etc). He also introduced Microsoft IoT Edge, an upcoming offering that is not even in preview yet. IoT Edge is a device that runs on premises to perform stream analytics and other cognitive functions. It can use models that are trained in the cloud before being run on-prem.
Sam finished up by stressing the importance of the integration professional’s role in the IoT industry, pointing out all of the integration points in the IoT chain as an example. Some new skills that are required in this area include data analytics / data science, multi-tenancy, resource deployment and automation, specific protocol knowledge, and custom development.
While Sam’s presentation was theoretical, Mikael’s session was extremely hands-on with no less than four demos showing bidirectional communication with IoT devices and IoT Hub. Mikael spent some time introducing IoT Hub and its features – but not before he was interrupted by Sandro who mounted the stage and insisted that Mikael wear the same green jersey that Steef-Jan wore, on account of losing the same bet. An unimpressed Mikael reluctantly complied with the request despite protests that he never was part of the bet. None of this impacted his ability to perform flawless demos based on temperature telemetry using IoT Hub, Stream Analytics, Function Apps, Service Bus queues, and Power BI. Awesome work, Mikael, as always!
Prior to executing the demos, Mikael talked about the IoT architecture landscape in general, with some excellent diagrams that communicate very clearly the various communication scenarios (telemetry, inquiries, commands & notifications) as well as the information exchange patterns (events, messaging, methods & state) that form the basic of IoT communication. Mikael has made his slide deck available here.
My session, entitled “The Hitchhiker’s Guide to Hybrid Connectivity”, presented a tour of the various options for connecting cloud and on-premises resources using Microsoft Azure technologies. After a brief mention of the network options (VPNs, ExpressRoute, and VNET integration), I concentrated mostly on the non-network options, all underpinned in some way by the relay principle. Of these, WCF Relay offered the cheapest option and provided options for leveraging features in the WCF application stack. The On-Premises Data Gateway is an add-on feature for iPaaS solutions built on Logic Apps, Power Apps, Flow or Power BI. There is no extra cost for this above the services using it. Hybrid Connections is perhaps the most flexible option since it works at the web socket level and therefore is not dependent on any application framework or even a specific operating system. And the new version of Hybrid Connections that is now part of Azure Relay exposes an API that enables you to connect more than just Web Apps and Mobile Apps like it’s BizTalk Services hosted predecessor. However, it does mean that everything other than delivering TCP packets (including application level security, reliable messaging, etc.) needs to be built into your solution. You also need to watch the amount of data transmitted as the $1/GB charge (beyond the first free 5GB) can trigger a hefty monthly bill if you’re talking about a moving terabytes of information.
I finished up by running through five common hybrid scenarios and discussing the various options for addressing connectivity. My slide deck is available for download here.
Wagner directly followed my talk with a session focussed on the role that BizTalk Server plays in hybrid integration (“to the cloud and back again”). Using some very creative and engaging images and clear, concise explanations, Wagner reflected on what BizTalk brings to Hybrid Integration (On-Prem and Azure adapters, Separation of Concerns, Availability, and leverage of existing resources) before discussing the various options that BizTalk affords for cloud connectivity:
- Service Bus (SB-Messaging adapter)
- Azure WCF Relay (WCF-BasicHttpRelay and WCF-WebHTTP adapters)
- App Services / API Management (WCF adapters)
- Logic Apps (Logic Apps adapter)
Wagner then proceeded to perform a series of demos to illustrate the scenarios he discussed, including:
- Connecting Service Bus with BizTalk (cloud to on-prem) using the SB-Messaging adapter and asynchronous messaging
- Connecting Service Bus with BizTalk (cloud to on-prem) using WCF Relay - showing the real time side
- Connecting on-prem to Logic Apps in the cloud using the Logic Apps adapter
Wagner finished up with a slide which encapsulates one of the key messages of INTEGRATE 2017:
The full slide deck is available for download here.
It would be remiss to not mention the session delivered by Saravana Kumar, Founder of BizTalk360 and the chief organiser of this event! For more than six years, BizTalk360 has been the premier product for managing and monitoring BizTalk Server environments. With more than 48 releases since its inception, the product has grown to include not only on-prem systems but has extended to the cloud as well, offering monitoring and management of Logic Apps, Azure Integration Accounts, and Service Bus. The capabilities of this product and the critical need that it fills is the reason why Mexia is a proud BizTalk360 Partner, and why yours truly is a BizTalk360 Product Specialist.
More recently, Kovai has launched a new product – ServiceBus360. Azure Service Bus forms a critical part of integration solutions today, but there are significant challenges in the operational management side:
- It is a raw platform
- Limited management capability within the Azure portal
- Lack of productivity tools
- Limited monitoring & notification capability
- Limited analytics and reporting capability
ServiceBus360 addresses all of these concerns by providing not only the ability to create and manage queues and topics, but also provide auditing, alerting, scheduled activities, and analytics. Saravana also demonstrated the monitoring capabilities for relays, as well as discussing the product’s ability to manage dead lettering and message deferral. Utilities like Service Bus Explorer have become a staple in the integrator’s toolbox due to a void in management tools capabilities for Service Bus, but ServiceBus360 goes far beyond these features in terms of alerting, auditing and analytics.
Miscellaneous Integration Topics
Integration in General
Starting off the third and final day of the event with Nino Crudele. If anyone is going to wake people up at 8:30 in the morning after a long night’s partying, it would be Nino! Known for his charisma, he is often referred to as the “Brad Pitt of the Integration Community”. Nino was in top form this day as he discussed “Rethinking Integration” and the need for us to consider how to leverage the tools at hand to provide the best customer value. Integration needs to be fast, because business is fast. Having identified his “top stack” of integration technologies (BizTalk is top of the list!), Nino went on to talk about various approaches, including the “chicken way” (with cards to give out to boot!). Nino also went on to talk about solving performance problems at a client with a custom framework that extends BizTalk.
Nino was followed by Richard Seroter who was introduced with music by the Spice Girls. I am always impressed with Richard’s talks, as they are always thoughtful, well-researched, and convey ideas that extend beyond just the Microsoft space. This session about moving to “Cloud Native Integration” was no different. With a number of quotes and charts from Gartner, Richard highlighted how cloud service integration is becoming increasingly important, especially in terms of hybrid integration. “Cloud Native” is an approach that leverages cloud computing to produce software that is “built for scale, built for continuous change, [and] built to tolerate failure”. Richard describes what he’d like to see in modern integration solutions:
- More “always on”
- More scale
- More self-service
- More endpoints
- More automation
Richard supported his ideas with no less than three demos, including the Logic App pipeline mentioned earlier in this post. The 2nd demo was particularly impressive in that is showed a custom “BizTalk Pipes” application that made managing a BizTalk environment as easy as picking up your mobile phone; it leveraged the REST API in Feature Pack 1. I sense a potential product here! His third demo was around automating Azure via Service Broker.
Richard’s messages were clear:
- Don’t allow integration to be the bottleneck.
- Start delivering integration as a service at scale.
Azure Cognitive Services
It is worth revisiting Kent Weare’s presentation from the perspective of business intelligence. In his keynote, Jim Harrer spoke about the inclusion of insights tools like Text Analytics, Face API and Computer Vision in the “better together” assembly of Azure integration services. Kent gave this vision real life by citing real world applications of this at clients like TransAlta, a global energy organisation. Kent talked about the changing face of IT in the context of Digital Transformation, including emphasis on Big Data, Cloud and Intelligence. Azure Cognitive Services provides a platform for “giving your apps a human side” through vision, speech, language, knowledge, and search capabilities. The BOT framework is one aspect of this platform which is gaining popularity due to the value BOTS offer in the way of automated, problem-solving conversations. Kent talked through the process of building and deploying a BOT:
- Download a BOT application template in Visual Studio
- Build out controllers, models and prompts
- Publish the BOT service to Azure App Service
- Register the BOT with Azure BOT Service
- Generate an App ID and password
- Update the Web.config with the credentials and re-publish
- Connect to different channels
- Enable side loading of application in Teams
Kent also talked about Language Understanding Intelligence Service (LUIS), another component of Cognitive Services which supports creating trainable models for the interpretation of user commands. All of this was illustrated in his Crypto BOT demo which accepted and responded to commands typed into a Skype conversation with the BOT:
One of the key takeaways from Kent’s talk is that “the future is not as far away as you think” and that we should “look for where the puck is headed, not where it currently is”. Azure Cognitive Services is a great way for integrators to help organisations realise the power of insights and automated problem solving.
Power Apps and Flow
Perth-based MVP Martin Abbott (the last member of the APAC trio along with Wagner and myself) had the honour of giving the final presentation of the INTEGRATE event. With only three slides in his deck, Martin gets the bravery award for basing his entire presentation on a live demo, building out a full end-to-end customer form application in less than 45 minutes. It was amazing to see how easy it was (or appeared to be) to create a mobile application user interface for entering customer data with Power Apps and managing the workflow and connectivity via Microsoft Flow. Seeing these tools in action within a live demo makes a great case for touting them as “citizen integrator” friendly, empowering business users to create their own apps without writing code.
Martin, who is also a global organiser for both the Global Azure Bootcamp and the Global Integration Bootcamp, also took the opportunity before leaving the stage to introduce fellow organisers Eldert Grootenboer and Tomasso Groenendijk and officially announce the next Global Integration Bootcamp, which will take place on Saturday, 24th March 2018. I can assure you that Brisbane will again be participating in this event, so please stay tuned for announcements early next year by following the hashtag #gib2018!