A Pace-Layered Integration Architecture

05 July, 2018 by Dan Toomey

Integration at Pace Within the Adaptive Enterprise

In the modern enterprise, it is rare to see a single monolith application that rules the entire landscape. While there still may be a large mainframe or another system which holds the principal data and source of truth (SoT) for an organisation, most landscapes today feature a medium to large set of applications that fulfill various business capabilities. These can range from a handful of apps to literally hundreds depending on the size and complexity of the enterprise. 

Whilst it is clear that the cost of integration increases along with the number of applications, one can also argue that the cost of change decreases as you move away from the “one app does it all” model. The reasoning behind this argument is that every single change, no matter how small, typically results in a full redeployment of the monolith application and necessarily incurs the cost of regression testing for the entire system.

 

But aside from cost and the numbers of applications, there is an additional dimension to be considered – time.  Not all apps are created equal. The problem with any architecture diagram is that is represents a single point in history – it is essentially a snapshot. The reality is that applications change over time; some are upgraded, other are modified or expanded, and others may be dropped or replaced. The rate of change varies across these applications, as some systems change more slowly than others. This can be represented in a layered view:

The concept of layers in an application architecture is not new; Gartner created the Pace Layered Application Strategy almost ten years ago to address the common disconnect between business leaders (who want their systems to be flexible and adapt to changes in the business landscape) and IT owners (who typically want to keep systems consistent as long as they are running smoothly). Identifying these varying rates of change and grouping the applications accordingly helps to apply an appropriate level of governance, change control, testing and DevOps – enabling the business to innovate where it needs to whilst protecting its critical data and core processes.

 

Understanding the Pace-Layered Architecture

Gartner’s Pace-Layered model consists of three layers:

  • Systems of Record (SOR) – These systems enable the core capabilities of an organisation, without which the business simply could not run. Because these are typically standard across the industry, these capabilities are not unique to a given brand or business (for example, every bank needs to manage accounts, transactions, customers, etc.). Consequently, these systems often are vendor supplied Commercial Off-the-Shelf (COTS) products. Because an organisation’s core capabilities don’t change very often, neither to these systems – change is incremental and at a slow pace.
  • Systems of Differentiation – While core capabilities don’t vary much from one organisation to another in the same industry, business processes do. For example, your bank and my bank may both offer loans, but the way those two banks process the loans is likely to differ. Applications in this layer represent the processes that make an organisation unique, and typically are not accommodated out of the box by vendor provided Systems of Record platforms. Business processes may not change every day, but they do tend to change faster than core capabilities, for example streamlining processes and/or incorporating new technologies.
  • Systems of Innovation – This layer moves the fastest of all. This is the “sandpit” where new ideas and technologies are tested. Experimentation here will likely include ad hoc proof-of-concept (PoC) applications which are rapidly developed and then manually deployed and tested.

 Image based on presentation by Michael Guay (Gartner)

Let’s look at an example enterprise, for example a banking institution, and see how some of their applications might map to the pace-layered model.

LAYER SYSTEMS / APPLICATIONS CHARACTERISTICS

Systems of Record

ABC Bank has several key systems including a core banking system, a loan management system, and a document repository.

  • These systems are vendor supplied and installed.
  • They are expected to have a long lifetime (e.g. 7-10 years)
  • Change control is very strict and the data is heavily protected.
  • Systems are subject to auditing by legislative bodies.

Systems of Differentiation

The automated loan processing capability is managed by a custom-built integration solution which integrates with several externally SaaS services for real property valuations, title searches, credit scoring, and an online web forms provider.

  • The solution was delivered through multiple phases of a large project.
  • It is expected to have a medium lifetime (3-4 years).
  • Fixes and change requests are managed via a BAU team.

Systems of Innovation

ABC Bank wishes to expose a chat-based interface for lodging loan applications powered by an intelligent bot.

  • A proof-of-concept solution is built with minimal features and manually deployed
  • It is tested by a small sample group of customers.
  • Adjustments and bug fixes are made quickly without a formal change control process until the solution is stabilised.
  • After a few months of trial, a decision is made whether to progress the solution to a full-grown application or to scrap the initiative.

 

Integration Within a Pace-Layered Architecture

So now that we understand the pace-layered model, how do we implement integration within it?  Let’s look at how a logical model of APIs/Services to see how they combine into applications across the layers:

Starting from the bottom layer, we see that each System of Record is typically a package which may contain several services/APIs. However, these APIs may not be directly consumable by the business due to misalignment with the logical data model, outdated protocols, or other reasons.  In these cases, it is a good idea to introduce a “sub-layer” of APIs that bridge the SoRs with other APIs within the organisation. These abstraction APIs (called Product Adapters) are tightly coupled with the underlying SORs yet expose functionality in a much more palatable format. They may also introduce stricter access control, validation and security than the SORs themselves. The APIs generally represent the core data entities (Customer, Product, Order, etc.), therefore they are granular and designed for reusability. Because they are so tightly linked with the core systems, they move at the same pace and therefore are considered part of the Systems of Record layer. Governance and change control will normally be very strict at this level because of the criticality of the data and the high dependency of services and processes that consume these APIs.

In the Systems of Differentiation layer, we see applications that are composed of the granular services/APIs sourced from the System of Record layer as well as possibly external APIs.  Here is where the business logic of the organisation lives, for example Loan Processing or User Provisioning. Functions that an application may perform in this layer include data aggregation, routing, filtering and often choreography/orchestration. Since they are specific to a process, they are less reusable than the underlying SOR APIs that they may consume. It is in this layer that the bulk of the integration within an organisation occurs. And because business processes can (and will) change over time, these applications will need to adapt as well, most certainly at a faster pace than the SOR applications. Governance should also be applied at this level, although perhaps not quite a rigidly as within the SOR layer; the organisation will want their business processes to be flexible enough to accommodate improvements in efficiency and expanding functionality.

The Systems of Innovation layer also features applications that consume both the SOR APIs and external APIs, as well as possibly business processes in the Systems of Differentiation layer. As the fastest moving layer, it will have even lighter governance in order to facilitate the experimentation of new apps and technologies. The functionality enabled in this layer is often peripheral to the core capabilities of the business, and therefore presents less risk to the organisation in case of failure. Also apps that are created quickly to prove a concept will rarely feature automated testing or a mature CI/CD pipeline, as they would be manually deployed and tested.

Finally, we use a Message Bus in order to facilitate both inter-layer and intra-layer communication. Asynchronous messaging patterns such as publish-subscribe keep systems loosely coupled and promote both extensibility and flexibility. The publisher doesn’t need to know anything about the subscriber, and you can always add or subtract subscribers without breaking existing integrations. A message bus is a critical element for lubricating the friction between applications that change at different paces.

This diagram provides a cross-section view of several attributes and how they vary across the layers:

 

How Does Microsoft Help with Pace-Layered Integration?

Microsoft offers a range of services and products, both on-premises and in the cloud, to help build powerful integration solutions that can cope with the different paces of enterprise application layers. Here we will discuss only some of these offerings and how they might fit into a pace-layered architecture (note there are many possible solutions; these suggestions are only one possible way to look at this):

 

System of Record Layer

The following products lend themselves very well to build the API layer on top of the SOR applications:

TECHNOLOGY USE WHEN CONSIDERATIONS

Product APIs

  • Product has granular APIs with a modern interface
  • APIs align with business needs
  • Vendor support is available

+ Tight integration with System of Record

–  Difficult or expensive to change or customize

–  May not suit business data model

Web Service / REST APIs
(IIS hosted on-prem or
App Service Plan hosted in Azure)

  • Exposing REST or SOAP interfaces
  • Implementing custom validation/security
  • Mapping to a canonical model

+ Inexpensive to host

+ Easy to consume

+ Can be hosted on-prem or in Azure (IaaS)

–  Development effort required

API Management

  • Exposing APIs in the cloud
  • Implementing policy based security & access control
  • Leveraging caching/auditing/analytics/etc.

+ Customizable façade

+ Developer portal facilitates new app creation

–  Requires VNet Integration – No on-prem option

–  Expensive option if not using additional features

Service Fabric

  • Aligning to a microservices architecture
  • Catering for multiple programming languages
  • Automated redundancy, load balancing, and
    no-downtime deployments are required

+ Can host anywhere

+ Supports containers

–  Significant development effort required (no OOTB software components supplied)

BizTalk Server

  • OOTB adapters are available/suitable
  • Robust platform is required

+ BAM tracking available

+ Single platform integration

–  Expensive option

–  Specialised dev skills required

–  Future support model in question

 

System of Differentiation Layer

These products are capable of implementing business logic, providing connectivity to internal and external applications and services, and supporting hybrid connectivity for applications that span both the cloud and on-prem:

TECHNOLOGY USE WHEN CONSIDERATIONS

Logic Apps

  • Business logic can be cloud hosted
  • Connecting with SaaS systems or other Azure services

+ Rapid development

+ More than 200 built-in connectors!

–  No VNet support  (until ISE becomes available)

–  No on-prem option (yet)

Azure Functions

  • Need to run discrete pieces of stateless arbitrary code on demand
  • Integrating with other Azure services
  • Visual Studio development is preferred
  • Automated unit testing is a must

+ Good CI/CD support

+ VNet support

+ Can run on-prem

–  Not as many connectors as Logic Apps

Web/Mobile Apps

  • Cloud hosting is desired
  • Supporting multiple devices
  • Need a flexible programming model
  • Need exposure to external clients
  • Desire Blue/Green deployment slots

+ Good CI/CD support

+ Numerous deployment options

+ Azure Relay / VNet Integration supported

–  Not ideal in themselves for long running processes

–  Consider security layer for hybrid apps

Service Fabric

  • Aligning to a microservices architecture
  • Catering for multiple programming languages
  • Automated redundancy, load balancing, and no-downtime deployments are required

+ Can host anywhere

+ Supports containers

–  Significant development effort required

–  Infrastructure investments (on-prem only)

BizTalk Server

  • OOTB adapters are available/suitable
  • Robust platform is required
  • Wanting a reliable/durable workflow capability
  • Leveraging Business Rules Engine and/or BAM

+ Single platform for integration

+ Can be hosted on-prem or in Azure (IaaS)

–  Expensive option

–  Specialised dev skills required

–  Future support model in question

 

System of Innovation Layer

Here we need the ability to expand the range of technology into the areas of artificial intelligence, predictive analytics, and business insight whilst enabling rapid (even ad hoc) development. There are numerous possibilities here, as it is less about which technology you use as opposed to how you use it. But these products all lend themselves well to innovative solutions:

TECHNOLOGY USE WHEN CONSIDERATIONS

Microsoft Flow

  • Automating simple processes and tasks
  • Empowering business users to create their own integrations
  • Existing connectors are fit for purpose

+ Rapid development

+ Can be easily migrated to Logic Apps

*  Requires Office365

Power Apps

  • Developing in-house apps for devices
  • Leveraging built-in connectors

+ Easy integration with Flow / SharePoint /
   Dynamics 365 / Teams / etc.

+ Multi-platform

*  Requires Office365

Power BI

  • Need to quickly build custom charts and visuals
  • Integrate with multiple data sources

* Relies on access to data sources

Cognitive Services

  • Seeking advanced insights and analytic capabilities

+ Multiple services/APIs available (Vision, Knowledge,
    Language, Speech, Search)

*  Programming skills required

Machine Learning

  • Seeking insights through predictive analytics

*  Data science skills required

Bots

  • Seeking more human interaction with customers
  • Automating routine information retrieval or routing to appropriate support personnel

*  Programming skills required

*  Bots need to be trained well to function as expected

 

Message Bus

If you are integrating mainly on-premises systems, BizTalk Server is at its heart a powerful messaging engine which can not only support the full gamut of messaging patterns, but also provide several out-of-the-box adapters for connectivity as well as a robust business process automation capability. This is why it features within so many layers. When integrating within the cloud however, Azure Service Bus provides a number of offerings for enterprise messaging, big-data streaming, event handling, and hybrid connectivity:

TECHNOLOGY USE WHEN CONSIDERATIONS

Event Grid

  • Building event-driven applications
  • Managing notifications
  • High scalability and throughput required
  • Handling events within Azure (or anywhere)

+ Resilient (retries up to 24 hours)

+ Push-push model

+ Easy to integrate

*  Small message size

 

Event Hubs

  • Ingesting big data / streaming data
  • Replay / archiving is desired

* Needs at least one downstream processor

–  No on-prem option

Relays

  • Need hybrid connectivity without firewall changes

+ Can use Hybrid Connection or WCF Relay

On-Prem Data Gateway

  • Connecting Logic Apps to on-premises systems
  • Bridging SaaS applications to LOB systems

+ Good alternative to a VNet if using Logic Apps

–  Only supported by a handful of Logic App connectors

Service Bus Queues

  • Decoupling sender / receiver processes
  • Each message should only be processed once
  • Data can flow through the cloud

+ Extremely resilient and full-featured

–  No on-prem option

Service Bus Topics

  • Decoupling systems / processes via pub / sub
  • Supporting multiple subscribers to a message
  • Data can flow through the cloud

+ Extremely resilient and full-featured

–  No on-prem option

BizTalk Server

  • Requiring robust pub/sub messaging
  • Leveraging BAM for tracking
  • Using OOTB adapters
  • Confined to an on-prem solution

+ Single platform for integration

–  Expensive option

–  Specialised dev skills required

–  Future support model in question

 

Tips and Best Practices

Here are some tips on how to maintain adaptive integration within a pace-layered enterprise architecture.

  1. Consider how the applications you are integrating will be used.
    • Is the integration mission critical? Then Logic Apps would be a better bet than Microsoft Flow.
    • What are the security risks to consider? An API Management layer can provide policy-based governance, threat protections, and access control.
    • How quickly might the solution change? This will influence how much investment is made in automated testing, CI/CD pipelines, etc.
  2. Make sure your System of Record layer is solid.
    • Are your APIs granular enough and well-defined? Remember that these will form the composable units for applications in the other layers.
    • Is security and data validation strongly enforced? Don’t rely on consumers for this; protect your critical data close to the source!
  3. Limit customisation within each System of Record.
    • If you customise the SOR, what will happen with the next vendor upgrade?
    • Keep customisation with the System of Differentiation layer as much as possible, or at least in the API layer over each SOR.
  4. Consider using canonical data models to avoid tight-coupling with vendor systems.
    • This typically requires a sound information architecture to define the business data entities.
    • An information architect can build a logical model independent of software implementation; invest in this.
  5. Loosely couple inter-layer communications.
    • Vertical dependencies are difficult to maintain – unless you are the owner of the whole stack.
    • Choose a publish-subscribe messaging model wherever possible to maximise loose coupling and extensibility.
  6. Allow room for innovation!
    • Employ the right level of governance at each layer. Avoid crippling change control policies where experimentation is both necessary and safe.
    • Empower business leaders to create their own solutions (e.g. using Microsoft Flow to automate mundane processes).
    • Encourage experimentation! Use Microsoft iPaaS features to achieve “integration at the speed of business” (Jim Harrer, Microsoft).

If you’d like to learn more about how Microsoft integration services can reduce the friction between application layers that move at different speeds, please contact Mexia and talk with one of our integration experts!

Latest Posts

Tags