The pandemic has wired global enterprises to become thriftier and value focused with a view to get their technological ecosystem more efficient, lean, and meaningful that can cater to the larger strategic goal of being effective in addressing user and customer experience. Basically, the world has moved notches towards being an “Experience Economy” than a “Feature Driven Economy”.
This movement is being felt in every stratum of an enterprise’s architecture, including the Integration layer. With more and more enterprises migrating their workloads to cloud, not just one, but multiple of them and with each of the cloud “ecosystems” having their own native integration solutions (Azure Integration Solutions, Amazon API Gateway, GCP Integration, SAP PI/PO etc.), it becomes imperative for enterprises to not only utilize the native integrations for enhanced service efficiency but to also have an Enterprise Integration Platform (EiPaaS) in the centre that can be the conduit for cross-ecosystem interoperability.
Likewise, as the strategic prerogatives tend more and more towards “experience”, reusability and scalability of existing APIs and integrations assume greater importance. This apart, the ability of an enterprise to create a Center for Enablement (C4E) that can be business and domain focused, as opposed to being technology focused (as was the situation earlier) is gaining a lot of currency in these times. The whole idea is to keep things simple and if any API is working in a certain functional area, the same should be made available to other similar functional areas across geos to drive larger RoI for the enterprises.
The third trend being observed is the ability to bake in native “observability” into the Integrations so that real-time operational monitoring and reporting can take place at the level of integrations themselves. This will entail capturing key metrics around API requests, API status, Integration Compliance, API latency etc. and the ability to draw thresholds to get proactive alerts that can enable the Site Reliability Engineers to take pre-emptive measures towards service continuance and resiliency.
These trends capture the spirit of co-existence (use the best of breed solutions), the intention to scale and replicate than to build and build more and finally the ability to track, measure, alert and act based on insights derived from native observability functionalities. Thus efficiency, scalability and observability are the three key decision markers for enterprises today while evaluating Integration solutions.
Let’s look at each of these three trends in detail.
Co-Existence of Ecosystem Integration with EiPaaS
In a multi-cloud world, where enterprises are parking their workloads in multiple cloud environments while maintaining their most-essential, confidential, and risky workloads on premises, it is a scenario that calls for various types of integrations – between on premises and cloud, between cloud and on premises, between cloud and cloud and between applications on premises. Several components, applications, systems need to interact across these environments to ensure seamless data integration, due process orchestration and rightful end user experience. Often, the decision makers in an enterprise become spoiled for choice and hence confused on what should be the “go to” Integration solution that can orchestrate value across the many environments.
Well, there isn’t any silver bullet solution to this quandary. But what should guide the enterprises is using the “best of the breed” approach where they leverage the native integrations of a given ecosystem (Azure Integration Solutions, Amazon API Gateway, GCP Integration, SAP PI/PO etc.) for integrating all applications located in that ecosystem. Therefore, all applications stored and run out of Azure ecosystem can leverage Azure APIs and Azure Integrations. Likewise, clients leveraging SAP or Oracle in a heavy way for their ERP needs will do well to leverage SAP PI/PO or Oracle Cloud Integrations for their respective applications. This will not only drive better runtime efficiency but also allow enterprises to derive maximum value of that ecosystem.
The above solves one side of the problem. So, each of our ecosystem applications are well integrated within that ecosystem by using the native integration functionalities of that ecosystem and within those ecosystems siloes, we have well oiled, perfectly synced machinery. But come out of these ecosystem bubbles and you may see distinct, siloed, nonintegrated ecosystems that are struggling to interact with each other. While most ecosystem platforms have “connectors” these days to connect to another ecosystem but imagine a customer with 10-12 different ecosystems within the enterprise and there could be 120+ connector interactions outside of these ecosystems that would make the enterprise stare at a spaghetti set up, drawing them back to the same problem that they want to get out of.
Enter Enterprise Integration Platform that becomes a one stop orchestrator of all cross-ecosystem interactions. All connectors from each ecosystem meets the common EiPaaS hub which will be capable of reading, translating, converting, and exposing messages in the target ecosystem friendly language and hence becomes a common point of interaction as shown below.
The above solves the second problem. Therefore, we have a “best of the breed” approach where an enterprise can use all “native” integration capabilities that the ecosystem provides while we have an EiPaaS Integration Hub which connects the dots between the ecosystems.
Below is an illustration that shows how “MuleSoft”, a leading Enterprise Integration platform becomes a central integrating hub that brings together all ecosystems and on-premises systems while the ecosystems themselves integrate the applications in them using their native functionalities.
Enterprises should respect each platform for its own benefits and leverage native integrations for application eco-system and leverage enterprise integration tool for integrations outside the application eco-system. The technologies shown in the figure above are for illustration only.
Scalability of Integration best practices with an “all encompassing” C4E (Centre for Enablement)
“All encompassing” is the operative term here. For a reason. Enterprises today use different integration technologies for different use cases. Different units and functions use various integration technologies too, as per their need, expertise, and comfort factor. With the empowerment Citizen Technologists, low code, and no code integration tools like Workato, Tray, Zapier etc. are also sitting in the overall enterprise landscape, alongside more strategic platforms like MuleSoft, Boomi, TIBCO etc. which also co-exist, as seen in the trend above, alongside ecosystem integration functionalities. Thus, suddenly, an enterprise looks at a smorgasbord of technologies, each with its niche requirement and each being used by a separate team or a group of teams!
The challenge is how to govern is wide swathe of technologies and teams using them; more importantly, how to make the integrations reusable, replicable and scalable across geos and units for similar use cases. Therein lies the value.
Many questions and options arise. Should there be a “decentralized model” where we have one CoE governing each business unit? Or one CoE governing all the work being done using an integration technology? Or should there be a “centralized model” where one common CoE orchestrates and governs the work done across all business units or technologies? Or should there be a “hybrid model” where one Hub CoE interacts with a few “Spoke” CoEs in a hub and spoke” model and the “spoke CoEs” in turn support a group of business units.
There are pros and cons to each of these models. For example, the “decentralized model” is more suited for large enterprises with independent business units. But lack of consistency across the units can breed a shadow IT culture. Likewise, the “centralized model” is more suitable for the small and medium enterprises and start-ups which have a small central IT running the show. The hybrid model is more suited for large enterprises that encourage autonomous business units, which run in a product organization model and want to be self-sufficient with a centralized, horizontal CoE, for any governance and utilities support.
While each of these options are germane, one factor that can truly accommodate for all these options to co-exist is the provisioning of an “all encompassing” C4E (Center for Enablement) that is truly “extended” and “accommodates” all teams, technologies and practices under its ambit, governs them, supplies the enterprises with common standards of working, best practices enabling business divisions to build and drive the consumption of assets successfully, enabling speed and agility. It allows the business and IT teams to shift from a production-based to a production-and-consumption-based delivery model. This production and consumption model fuels reusability and scalability.
The C4E is a broader concept than CoE. While the CoEs develop, run, and manage APIs, their operational elements and are a container of functional expertise in a centralized manner, C4Es are enablers, capability builders, decentralized mechanisms of governance that bring in best practices, reusable templates, evangelism, and community building. They are domain focused and make the business units self-sufficient, independent, and scalable.
Observability baked into integrations for reactive knowhow and proactive alerts
Observability is a comprehensive set of capabilities around logging, analytics, monitoring, troubleshooting, and measurement of platform, runtimes, and applications. Observability plays a decisive role in providing the real-time performance of a system. Lacking observability in a system would be like flying an aircraft without a fuel indicator.
Observability is not only shifting left, but also diving deep into each layer of the enterprise architecture. From the systems to applications to the processes to the integrations between them, enterprises expect observability and monitoring mechanisms to be baked into each of these elements so that not only is the performance reported but with the application of AI and by drawing clear performance thresholds, proactive alerts can be sounded out to the SREs, thus ensuring service resiliency and continuance.
It is in this regard that most Integration platforms today come laced with out of the box (OOB) support for many of the key observables. What can be observed and monitored also depend on the deployment option- Cloud, On Prem or Hybrid.
Some of the key observables around Integration that the OOB functionalities can observe, and reporting are:
There are various third-party observability tools in the market that fill the vacuum left by the OOB functionalities of the Integration platforms. Splunk, ELK, SumoLogic are commonly used for log aggregation and analysis while the likes Jaeger, Prometheus, Datadog etc. are popular for their tracing capabilities.
However, tools alone do not solve the problem of observability. Having an experienced team of reliability engineers who can interpret metrics and act proactively will be key to success. As they say it is not data, but the action taken on the insights derived from the data which determines true success. Therefore, partnering with the right service partner will be key to Enterprise Integration success.
Conclusion
The Enterprise Integration market is in a flux post pandemic as effectiveness, efficiency, cost optimization and performance precision are all expected by enterprises in their Integration solution. These are seemingly contrarian demands but a balanced solution that can ensure co-existence of diverse solutions for diverse use cases, with an ability to scale and a capability to observe and report key metrics are key to enterprise integration success. While many technologies are available in the wider market, a capable service partner with experience of deploying, managing, orchestrating, scaling, and bringing about governance support will be key to Integration strategy success.
