Integration is one of those concepts that is easy to “know,” but becomes less obvious that more you try to define it. A basic, casual definition is making different things work together. The complexity, though, comes from the fact that every single part of that has to be broken down: what are the “things,” what are they doing that makes them “work together,” how are they working, and what is the goal or purpose of them working together. All of those elements can be answered differently for different organizations, or even within the same organization at different times.
An understanding of integration comes from looking at the different potential patterns that you can integrate and then defining the logic behind the integration so you can select the right patterns for your environment.
Integration itself is an architectural structure within your infrastructure, rather than an action or specific process. While getting various systems to work together has long been an IT (and organizational) responsibility, integration as a practice became more of a focus in the early 2000s. With emerging large-scale enterprise applications, there became a growing need to get those applications working together without having to redesign or redeploy the applications themselves. That push became integration.
Integration is subdefined by what is being integrated; these are the integration patterns.
There are different types of patterns, depending on perspective. There are patterns based on what is being integrated and then there are patterns based on the topology or design of the integration. Basically, it’s the what and the how.
Application integration (also called enterprise application integration or EAI) is a way of coupling applications so that they were together. This can be related to information sharing, but it could also be for other reasons or to support modular service architectures.
There has been an evolution in application integration. Early on, it was to unify large-scale, monolithic applications running in traditional, usually physical architectures. Since the mid-2000s, that has been changing into more and more modular, interactive architectures, from ESBs to service-oriented architectures, cloud-based integration platforms (iPaaS), and now microservices.
This type of integration works at the business logic layer of the architecture.
Data integration is also called enterprise information integration. Application integration may not, necessarily, require information sharing among all of its services; the goal is to get the services working together. With data integration, the applications may or may not be working together, but they all require access to the same shared data.
Data integration requires an understanding of both the underlying data formats in each application and the required information for each application. Trying to transfer information between applications may require converting it into different formats, while trying to use a shared database or information system may limit how applications can interact or access the data.
Certain architectural approaches can benefit data integration as well as application integration. For example, using a services or microservices architecture usually means that there is a common messaging framework unifying the involved applications. Rather than having to convert data from and into multiple formats, there is a shared data protocol such as REST that is used across all applications.
Systems architectures can be thought of as layers, with data at the bottom, then business logic / applications in the middle. There is also a top layer, the user interface or presentation layer where the user actually interacts with the system. It is possible (though generally not strategic) to integrate at that presentation layer. It requires pulling the information from the interfaces for the applications or data sources and creating a single, unified user interface.
However, because Uis can change so frequently and can be limited in what data they display, this is a more limited and brittle approach to integration.
The “how” integration classifications are architectural decisions. This is related to application integration because it relates to the topology of services and how they relate to each other. While not reflective of performance or scalability, there are some parallels between architectural integration patterns and scalability patterns.
Vertical integration is related to application stacks or silos. The structure is that required services are daisy-chained together into a logical group for a fully functional stack. However, because the stack is designed vertically, it can be very difficult to scale or to change, because there can be cascade effects throughout the silo when changing one application within it.
Horizontal and ESBs
Horizontal integration, like horizontal scale, refers to the ability to add and extend services within the environment. Horizontal integration uses loose coupling between related services. While using ESBs are not required, ESBs can be an important component in horizontal integration patterns because they allow services to be added, removed, or updated within the topology with less of an impact on other services. When using an ESB, the topology is relatively simple, since everything communicates with the ESBs (which can also handle data conversions or transfers) without having to communicate directly with the other services.
Messaging and Microservices
ESBs and service-based architectures have been evolving over the last couple of years as microservices have emerged as an architectural design. Over the past 15 years, infrastructures have been moving from tightly centralized to federated, dispersed service structures, as physical infrastructures have been moving to virtual and now cloud-based environments.
Integration is likewise evolving, with a growing emphasis on loosely-coupled services, strong messaging frameworks, and standardized formats like REST. Rather than using ESBs to transform data, data can be supported in multiple formats within the integration environment. As this loosely-coupled, messaging-centric environment matures, it is increasingly more critical to have a strong middleware layer for things like business policy and process automation to help manage the data flows for mobile and Internet of Things initiatives.
Integration Reflects a Business Purpose
Integration is going to be unique to every organization. Defining an integration strategy is inherently a business strategy decision.
There are several key questions where the business goal defines the integration:
- What applications are being linked
- What data is being integrated
- What data formats are involved
- How that data will be associated and presented
- What the integrated data will be used for
As organizations across industries are trying to evolve into digital businesses (digital transformation), there are a couple of primary challenges. According to Enterprise Apps Today, almost half (48%) of businesses are focusing on integration related to data analytics; and 43% are trying to integrated cloud-based environments with on-premise applications. The situation is new, but these are basic integration use cases, data and application integration.
The Enterprise Integration Patterns blog had a post way back in 2004 which defined three different, overlapping ways of looking at integration:
- What is the intent or purpose for the integration?
- What technologies will you use to connect the underlying applications?
- How is the information shared between the applications?
The first classification, as Gregor Hohpe defined it, is the goal of the integration. This could be to reuse code, to simplify a given service, to unify data gathered from different sources. The intent is the problem that is going to be solved by the integration.
Particularly when integration is part of a larger, more complex strategy like digital transformation, there may be multiple intents. But defining that overarching goal (and subgoals) gives clarity to the integration plan. It provides a focus that can make the other issues – like identifying required data and the standards required to support it – easier to address.
Some of the chatter around integration today can give the impression that this is an entirely new imperative for IT organizations, and it really isn’t. Enterprise Apps Today had an article in the spring with a great quote from Dan Huberty, a vice president at Unisys: “Digital is not new, cloud is not new, analytics is not new, Big Data is not new. What is new is the need for speed and scale and the need for everything to work together.”
What’s different about integration as a practice now is twofold: its scope and scalability.
The first wave of integration in the early 2000s was driven by a need to integrate separate applications for strategic alignment, particularly after an acquisition (which was the point of a CIO article highlighting several merger-related integration projects).
While aligning separate groups is still a goal of some integration planning, the scope of integration is very different. With strategic alignments, it was largely necessary only to get the systems talking together for data sharing or (possibly) migration preparation. The integration only went as far as was necessary to get those systems talking.
Now, the big drivers for integration are not simply to get separate systems running in parallel; integration is a foundational requirement for major IT strategic initiatives, especially digital transformation initiatives.
Integration patterns can allow for very limited integration – only certain types of data or only certain applications or silos. As data streams become the main business driver across industries, and not just one advantage of many, then the ability to integrate all of these data streams becomes crucial. It’s a bigger scope within the enterprise and a bigger scale than previously required.
The scalability required is related to both the amount of data and applications that need to be integrated and the need for everything – applications, integration platforms, services – to be located in cloud and container based environments for dynamic scaling.
As Huberty noted, integration isn’t a new thing. At its core, it is a systems architecture challenge. The complexity can be broken down effectively:
- Assess the current architecture.
- Identify the data types and formats involved.
- Identify what applications must be involved in the integration.
- Define the business objectives that the integration will meet.
- Find standards-based technologies that can support the broadest base of underlying applications and data.