Digital transformation creates new possibilities for organizations by offering innovative products and services. As part of this journey, enterprises develop their digital transformation blueprints that provide the capabilities needed to achieve their business goals. An Integration of digital technologies and services is a key component of this blueprint that they can leverage to their competitive advantage.

The areas we’ll cover as part of this blog on digital transformation include the API-based integration strategy across the application landscape – SaaS, PaaS, IaaS, and LCNC platforms. We will also look into how the integration landscape is evolving with an API-first strategy coupled with integration patterns and thoughts on API governance for the future paired with a real-world example. Finally, we will discuss the Microsoft Azure Integration Platform as a Service (iPaaS) to build integration solutions.

When you attempt to modernize your legacy systems, you need to integrate new systems with existing databases. But organizations should consider moving away from point-to-point integrations and embracing API-based integration strategy for digital transformation. API-led connectivity is also advised for other digital technologies/data sources like SaaS, Big data, IoT, and mobile workloads.

Organizations should consider moving away from point-to-point integrations and embracing API-based integration strategy for digital transformation.

APIs in the Application Landscape: Organizations are expanding their capabilities by implementing IaaS through Cloud(s) using SaaS applications to upgrade their legacy systems. In both scenarios, the use of API for integrations has increased tremendously – through REST, which is a perfect fit for these various scenarios.

With Innovation as a driver for large enterprises, they are leaning more and more towards the ‘No code’

According to Gartner, such no code Integration users are ‘Citizen Integrators’ who can create automated workflows for integration without any single point of failure.

and ‘Low code’ platforms. These platforms and digital technologies also often leverage enterprise-scale APIs and web service catalogs to help integrate existing business systems while adding a layer of user functionality. No Code platforms enable enterprises to build and deploy integrations at a much faster rate without IT infrastructure friction.

According to Gartner, such no code Integration users are ‘Citizen Integrators’ who can create automated workflows for integration without any single point of failure.

Enterprises are increasingly adopting an API First approach, and it has become a de facto standard for API development, offering well-designed, managed, and secure APIs for consumption. Swagger defines an API-first approach for any given development project as APIs treated as “first-class citizens.” That everything about a project revolves around the idea that the end product will be consumed by mobile devices and that APIs will be consumed by client applications.

The rising value of APIs has created an API Economy. It allows a business to monetize a portion of its data and services by making software platforms that are easier to integrate. Consequently, many large monolithic software systems currently supported on premises will decompose into highly organized sets of microservices available in the cloud

A Real-World Example: Twilio is an example of the successful implementors of the API Economy. And Uber’s underlying technology isn’t created by its own company but instead relies on external APIs. Both are hailed for their API integration strategy.

API Integration - Twilio

API Integration - UBER

API’s – Integration Patterns, Architecture, and Governance: The momentum behind containers, serverless, multi-cloud, and APIs is ever increasing. It has also created all sorts of new problems surrounding the management and integration of APIs too. We need some standard solutions to fix common and recurring integration problems in the digital transformation process. Enter Integration Patterns.

Integration Patterns refer to the implementation of useful design patterns that provide technology-independent design guidance to developers and architects by describing and aiding the development of robust integration solutions. Following are some of the key patterns falling into six major categories for message-based enterprise integration and a few widely used patterns (not a comprehensive list though):

1. Message Construction

• In the Correlation Identifier pattern, each message has a unique Correlation ID attached with the request message to match it with a response.
• The request-Reply pattern is based on asynchronous messaging between the service provider and consumer.

2. Message Routing

• Content-based routing routes the message based on the content of the message rather than by an explicitly specified destination. 
• The Splitter pattern splits the received message into multiple messages. Each of the individual messages is processed separately after being.
• The Scatter-Gather pattern is a method for broadcasting and processing messages in parallel.

3. Message Transformation

• Content enrichment pattern enriches the content of received messages.
• Claim Check pattern splits a large message into a claim check and a payload. It allows large messages to be processed while preventing the message bus and the client from being overwhelmed or slowed down.

4. Message Endpoints

• In the Event-Driven Consumer pattern, the receiver acts like the message delivery is an event that triggers the receiver into action.
• The Messaging Gateway encapsulates messaging-specific code (e.g., the code required to send or receive a message) and separates it from the rest of the application code.

5. Messaging Channels

• Invalid Message Channel - The receiver should move the improper message to an Invalid Message Channel, a special channel for messages that could not be processed by their receivers.
• Message Bus - Structure the connecting middleware between these applications as a Message Bus that enables them to work together using messaging.

6. Platform Management

• Control Bus uses the same messaging mechanism used by the application data but uses separate channels to transmit data that is relevant to the management of components involved in the message flow.
• The Smart Proxy pattern describes an implementation of the Request-Reply pattern where multiple requestors can send requests and expect responses to their own independent queues, while the replier expects to have a single queue for all requests and responses.

All these Messaging Channels patterns focus on solving the challenges of transmitting messages from a sender application to the intended receiver applications.

Gartner defines iPaas as “An Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.” 

The following sections discuss how we can implement and manage these integration patterns through Microsoft’s iPaaS offering - Azure Integration Services:

1. Orchestrating Workflows: Azure Logic Apps is the core iPaaS offering from Microsoft. It allows us to manage our integration workflows using a visual designer (using Azure Portal or Visual Studio). You can use Azure Logic Apps as the platform to host the business logic in a serverless manner using the LCNC approach.
2. API Mediation and Management: For Microservices architectures, HTTP-based APIs have become the preferred method for synchronous interaction. These APIs often expose an application’s data and functionality to third parties, leading the way for powerful integrations. All requests from clients first go through the API Gateway. It then routes requests to the appropriate microservice.
   Organizations now need specialized API management software to make sure the APIs are used in compliance with corporate policies and allow governance by appropriate levels of security. API first approach can be easily accommodated with such tools for serving the full lifecycle of API Management - from API design to serving the API developer community consuming the APIs.
   Microsoft's Azure API Management is a hybrid, multi-cloud management platform for APIs across all environments. It’s an Azure service that functions as an API gateway for backend APIs in the cloud or on-premises. It plays a key role in creating API Economy discussed earlier by allowing you to monetize APIs. You can charge your customers a fee to access your APIs to support your Azure API management.
3. Messaging & Events: Event-driven applications require robust integration solutions to react to events and to push events. Azure Service Bus (ASB) Messaging provides reliable message queuing and publish-subscribe messaging capabilities for improved Azure API management. ASB provides rich and robust messaging capabilities, like transactions, ordering, sessions, dead-lettering, deduplication, among others. In contrast, Azure Event Grid is a fully managed event-routing platform enabling event-based programing with pub/sub semantics in legacy systems. Logic Apps can react to events on Event Grid that are published from the connected resources. 
4. Serverless computing: For certain integration scenarios, you may need to write custom connectors for Logic Apps or even require writing some custom code to fix complex orchestration problems. With Azure Functions, we can write custom code in C#, F#, Javascript, or any other of the supported languages. Azure Functions are an event-driven serverless compute platform allowing you to invoke them from Logic Apps via HTTP (synchronously or asynchronously) or via Azure Service Bus or from Azure Event Grid.

To wrap up, digital enablement leveraging significant incorporation of ‘API-first’ driven integration solutions are inevitable, and the architecture community needs to be cognizant of both the API development approaches, technology perspectives and governance aspects, in an increasingly ‘API Economy’ driven application landscape in the future.

In this blog, I have covered the integration and API development approaches, patterns, and the problems they address. I have also talked about leveraging Azure-based integration tools for architecting enterprise integration solutions. This will keep evolving in the future.