Friday, October 13, 2017

ESB vs EAI: "Universal Service", What is Wrong with This Pattern

Some technical people do understand the Enterprise Service Bus (ESB) concept as a universal channel designed just to enable some XML messages encoded as plain string transmission among enterprise applications. The channel should provide no validation/enrichment/monitoring capabilities, the channel is considered only as a dumb message router that also provides message transformation into an accessible for the enterprise applications format. A powerful and expensive integration middleware, like Oracle Service Bus, Oracle SOA Suite, IBM Integration Bus, or SAP PI/XI, is chosen as a platform for the integration solution. Usually, it's required that the IT team should be able to configure new or existing routes just by edit a few records in the configuration database.

The developers of such "universal solution" believe that a new application can be connected to the solution just by design an appropriate adapter and insert a few records into the configuration database.

In fact, the developers have to implement a number of integration patterns and, optionally, a canonical data model using a small subset of the capabilities provided by the integration platform.

The focus of the article is to explain why the above approach is not effective and why developers have to leverage as many capabilities of their preferable middleware platform as possible.



The diagram above presents a typical schema of a "universal service bus". A source application sends messages presented in an application-specific format. A message usually contains a number of headers which are containers for the information about the message route: the source system code, the destination system code(s), the type of the message and so on. The processing may be implemented in two ways: either the adapter transforms the message into the canonical data model and sends the new message to the "core" or the adapter sends the message to the "core" as is. The "core" retrieves a route for the message from the configuration database using metadata held in the message header. If there is no canonical data model, the "core" transforms the message to formats valid for each destination application (so, the 1 -> N topology is allowed). A destination system adapter either transforms the message from the canonical data model to a valid format or just sends the message to the input of the destination application. It is expected that in order to connect a new enterprise application to the bus, it's enough just to develop an adapter for the application, implement all required transformations from/to the application-specific formats, and put the metadata about routing into the configuration database.

Well, in my humble opinion, it's not enough. There are some limitations for every "most-universal solution".

First of all, even if all messages are communicated by the bus presented in the canonical data model, every adapter just does transform those messages from application-specific formats toward the canonical data model and vice versa. However, the canonical data model can't represent all possible information within the business ecumene. In order to connect a new application which operates with totally new business information, the following activities should be undertaken: to expand the canonical data model: add new data types and fields to the existing messages, update all transformations related to the modified messages, implement new adapters for the destination applications, and sometimes refine the destination application itself. As we can see, one not does simply walk into Mordor connect a new abstract application to an Enterprise Service Bus even if the canonical data model is established otherwise, the situation is becoming more and more complex: up to N(N-1) transformations must be developed for N new connected applications.

Secondly, each solution which ends up in production, especially an integration solution, must be tested before. An adequate architect won't approve to connect an enterprise application to the integration solution without full end-to-end testing. But if there is a "universal solution", one is tempted to approve the connection without an adequate testing (in other words - ad hoc integration): the adapter is developed, the configuration database is updated too, what else you're looking for? Nevertheless, actually, if even messages are transmitted well amongst the connected to the bus applications, it doesn't ensure the automated business processes, which depends on the messages, are performed properly. The overall integration must be tested as a complex.

Thirdly, if you have a hammer, everything looks like a nail. When a new enterprise application can be connected to the integration solution very simple, more and more interchanges which can be implemented in the scope of an application is being implemented on the service bus. For example, instead of to develop a method who returns a claim along with all comments to this one inside a source application, the developers implement a service who requests a claim from the endpoint (the source application), then requests the related comments. The problem is the bus can be installed in Moscow while the application is installed in Vladivostok (it's about 9 000 km). On the other hand, the concept of universal transport means the transport is dumb, therefore such capabilities as message aggregation, enrichment and more complex manipulation can't be implemented on the bus so these capabilities must be provided by the destination application or, if the message flow has numerous destination, by every such application.

Worst of all, complex and very expensive software is considered to implement the controversial concept of universal transport, at the same time, less than half of provided capabilities is leveraged. I think the problem has two aspects:

Firstly, customers ask for a "universal solution" since they have negative experiences of enterprise application implementation and customization. In particular, the concern is the implementation takes so much time. And this concern is true while the application is being developed using general-purpose programming languages: Java, SQL, PL/SQL, etc. A few months might be spent on solving the classical integration problems, e.g. message transformation, especially if collections should be processed. A number of well known frameworks e.g., Apache Camel and Spring Integration, can help here.

Secondly, very often developers just don't know how to use the provided capabilities. They consider a proprietary integration platform as an implementation of some enterprise application integration patterns, and a "universal solution" should be quickly built from the patterns like from lego cubicles. On the Russian market, for example, some developers have religious hatred to enterprise solutions, they don't understand why there are WebLogic, WebSphere, JBoss, and so on, also why there are Oracle Service Bus and IBM Integration Bus, if Apache Tomcat and Spring Framework exist and any application can be cooked using these technologies. The application might be able to work but no one cares about who will provide support for the platform and what about a time frame for development: the implementation buit on pure Java may take 6 months vs one month for the same project implemented on Oracle Service Bus.

Just to list the main problems is not enough, though. A solution must be proposed. Let's consider to implement an integration solution using an integration platform as it was designed. The modern integration platforms were developed not as a bunch of lego cubicles for building adapters and a core for a "universal solution", these products are the integration solutions themselve. All the more, Oracle's products as well as IBM's ones, Tibco's, Software AG's, and products from other distinguished vendors. In fact, the development of the integration solution based upon these products looks like drawing information flows, services and processes. The process isn't more complicates than working with the configuration database but provides the following benefits:


- Agility. The enterprise service bus is considered not only like a dumb message transport. Modern integration platforms also provide the message validation, enrichment, and split-aggregation capabilities. Even composite applications can be deployed on the platforms.

- Visibility. Metadata about the integration aren't persisted in tables inside the configuration database; instead, the metadata is embedded in the message flows. The flows can be easy opened in the IDE and presented in graphic form. A flow might be very complicated and a large number of applications is involved in the integration, also all applications and their collaboration will be shown on the flow diagram. Their assistance is invaluable for IT operational stuff as well as business and system analysts during system lifecycle.

- Performance. There isn't a configuration database at all, so the solution doesn't wast any time on metadata (in particular, routing tables) retrieving. The code of an integration flow itself is this kind of metadata. The overhead is decreasing. Some modern integration platforms also provide the zero downtime upgrade capability. The code of a message flow can be upgraded without any impact on running other flows. So, not only the performance but the availability of the integration solution is also increasing.

- No single point of failure. On the first diagram in the article, two points of failure can be found: the core and configuration database. If one of these components fails, the overall solution ceases to function. The concept of Enterprise Service Bus doesn't mean a core at all: all operations on a message are just coded in corresponding services. If one service fails, other ones keep working as designed. The configuration database usually isn't required.

- Testing and monitoring. Some modern integration solution, like Oracle SOA Suite, provide tools for unit and integration testing for developed integration flows, services and business processes. These platforms armed with rich and powerful runtime monitoring capabilities, which can display performance metrics for every service as well as send alerts by e-mail or SMS when an SLA is violated. Developers of "universal solutions" usually don't take these capabilities into account.

What conclusions could be drawn from the story? Developers should convince to their customers that the development process when the modern integration platforms are taken into account is not the same as programming on Java. The integration solution won't be ready in months, the solution is already on the table. Customization of message flows in response to changes in business requirements takes days if not hours including testing and release management. Also, the size of the development team is significantly decreasing.

For example, an Oracle Service Bus based integration solution, which handles more than 5 000 000 messages in a day and interconnects about 60 applications, was developed by a 3-member team. A 3-member team! For comparison, at the same time, a Java application connected to the bus was customized by 20 developers.

Since the integration team usually shows better velocity, up to several times, than other development teams, the management is tempting deploy on the bus as much business logic as possible. Please notice, this business logic usually isn't relevant to integration.

Would you like to give a 'Like'? Please follow me on Twitter!