New styles of integration are the hallmark of Digital Transformation

New Styles of Integration 2

Shakeup your integration strategy to enable digital transformation, says VP & Gartner Fellow Massimo Pezzini. Pezzini asserts that it is not just about transforming and modernizing the infrastructure and the applications concerned.  Some of the fundamental concepts of integration need to be revisited and transformed as well.  Such systemic transformation punctuate the migration of  legacy environments to microservices and the cloud.  What may have worked in the past will no longer be applicable going forward.  “Integration is dead.  Long live integration,” screamed the title of one of the sessions at the Red Hat Summit 2016.  The session was making a point.  Integration, as we knew it a few years back, is dead.  Integration in the digital world has a long life in the decades ahead.  Join me as I walk through the new styles of integration that are the hallmark of digital transformation.

Continue reading “New styles of integration are the hallmark of Digital Transformation”

Visualizing Integration Applications

Since I’ve changed roles and started performing architect duties, I have to draw more boxes and arrows than write code. There are ways to fight that, like contributing to open source projects during sleepless nights, POCs, demos, but drawing boxes to express architectures and designs is still big part of it. This post is about visualizing distributed messaging, SOA, microservices applications in agile environments (this term has lost its meaning, but there is not better one in this case). What I like about the software industry in recent years is that the majority of organizations I’ve worked with value the principles behind lean and agile software development methodologies. As long as it is practical, everyone strives to deliver working software (rather than documentation), deliver fast (rather than plan for a long time), eliminate waste, and respond to change. And there are management practices such as scrum and kanban, and technical practices from extreme programming (XP) methodology such as unit testing, pair programing, and other practices such as CI/CD and DevOps to help implement those principles. In this line of thinking, I decided to put together a summary of the design tools and diagrams I find useful in my day to day job while working with distributed systems.

Issues with 4+1 View Model and Death by UML

Every project kicks off with big ambitions, but there is never enough time to do things perfectly, and at the end we have to deliver whatever works. And that is a good thing, it is the way the environment helps us avoid gold plating and supports principles like YAGNI and KISS, so we do just enough and adapt to changes.

Looking back, I can say that most of the diagrams I’ve seen around are inspired by the 4+1 view model of Philippe Kruchten which has logical, development, process and physical views.

4+1_Architectural_View_Model
4+1 View Model.  From A Practical Guide to Enterprise Architecture by James McGovern, Scott W. Ambler, Michael E. Stevens, James Linn, Vikas Sharan, Elias K. Jo, 2003.

I quite like the ideas and the motivation behind this framework: using separate views and perspectives to address specific set of constraints and targeting the different stakeholders. That is a great way of describing complex software architectures. But I have two issues with using this model for integration applications.

Diagram Applicability

Typically these views are expressed through a unified modeling language (UML), and for each view, you have to use one or more UML diagrams. If I have to use 15 types of UML diagrams to communicate and express a system architecture in an accessible way, it defeats the purpose of UML.

Death by UML
Death by UML. From Wikipedia, derived from a diagram by Paulo Merson.

With such a complexity, the chances are that there are only one or two people in the whole organization who have the tools to create and ability to understand and maintain these diagrams. And having hard-to-interpret, out-of-date diagrams is as useful as having out-of-date documentation. These diagrams are too complex and with limited value, and very quickly they turn into a liability that you have to maintain rather than an asset expressing the state of a constantly changing system.
Another big drawback is that the existing UML diagram types are primarily focused on describing object-oriented architectures rather than pipes and filters architectures. The essence of messaging applications is around interaction styles, routing, and data flow rather than structure. Class, object, component, package, and other diagrams are of less value for describing pipes and filters based processing flows. Behavioral UML diagrams such as activity and sequence get closer, but still cannot express easily concepts such filtering and content based routing, which are a fundamental part of integration applications.

View Applicability

Having a different set of views for a system to address different concerns is a great way of expressing intent. But the existing views of the 4+1 model don’t reflect the way we develop and deploy software nowadays. The idea of that directional flow — that you have a logical view first, which then leads to development and process views, and those lead to a physical view — is not always the case. The systems development life cycle is not following the traditional (waterfall) sequence of requirement gathering, designing, implementing, and maintaining.

2000px-CPT-SystemLifeSycle.svg
Software Development Lifecycle. Derived from an image by Web Serv

Instead other development methodologies such as agile, prototyping, synchronize and stabilize, and spike and stabilize are used too. In addition to the process, the stakeholders are changing too. With practices such as DevOps, developers have to know about the final physical deployment model and the operations team have to know about the application processing flows.

Modern architectures such as microservices affect the views too. Knowing one microservice in a plethora of services is not very useful. Knowing too much about all the services is not practical either. Having the right abstraction level to have a system wide view with just enough details becomes vital.

Practical Visualization for Integration Applications

The closest thing that has been working for me is described by Simon Brown as the C4 model. (You should also get a free copy of Simon’s awesome The Art of Visualising Software Architecture book). In his model, Simon is talking about the importance of a common set of abstractions rather than common notation (such as UML) and then using simple set of diagrams for different level of abstractions: system context, container, componentand class. I quite like this “outside-in” approach, where you first have 10000 foot view and with each next level, going deeper with more detailed views.
C4 is also not an exact match for middleware/integration applications either, but it is getting closer. If we were to use the C4 model, then the system context diagram would be one box that says ESB (or middleware, MOM, or microservices) with tens of arrows from north to south. Not very useful. The container diagram is quite close, but the term container is so overloaded (VM, application container, docker container) which makes it less useful for communication. Component and class diagrams are also not a good fit as pipes and filter architectures are focused around enterprise integration patterns, rather than classes and packages.
So at the end, what is it that worked for me? It is the following three types of diagrams which abbreviate as SSD (not as cool as C4):  system context, service design, and deployment.

System Context Diagram

The aim of this model is to show all the services (whether they are SOA or microservices) with their inputs and outputs, ideally having the external systems on the north, the services in the middle section, and internal services in the south. Or you could use both external and internal services on both side of the middleware layer as shown below. Also having the protocol (such as HTTP, JMS, file) on the arrows, with the data format (XML, JSON, CSV) gives useful context too, but it is not mandatory. If there are too many services, you can leave the protocol and the data format for the service level diagrams. I use the direction of the arrow to indicate which service is initiating the call rather than the data flow direction.

System Context Diagram
System Context Diagram

Having such a diagram gives a good overview of the scope of a distributed system. We can see all the services, the internal and external dependencies, the types of interaction (with protocol and data format), and the call initiator.

Service Design Diagram

The aim of this diagram is to show what is going on in each box representing a middleware service from the system context diagram. And the best diagram for this is to use EIP icons and connect those as message flows. A service may have a number of flows, support a number of protocols, implement real time, or batch behaviour.

Service Design Diagram
Service Design Diagram

At this level, we want to show all possible data flows implemented by a specific service, from any source to any destination.

Deployment Diagram

The previous two diagrams are the logical views of the system as a whole and each service separately. With the deployment diagram, we want to show where each service is going to be deployed. Maybe there will be multiple instances of the same service running on multiple hosts. Maybe some services will be active on one host, and passive on the other. Maybe there will be a load balancer fronting the services.

Deployment Diagram
Deployment Diagram

The deployment diagram is supposed to show how individual services and the system as a whole relates to the host systems (regardless whether that is physical or virtual).

What Tools Do I Use?

The system context and the deployment diagrams are composed only of boxes and arrows and do not require any special tools. For the service design diagram, you will need a tool that has the enterprise integration pattern icons installed. So far, I have seen the following tools with EIP icon support:

Other development tools that could be also used for creating EIP diagrams are:

A system context diagram is useful to show the system wide scope and reach of the services, a service design diagram is good for describing what a service does, and a deployment diagram is useful mapping all that into something physical.

In IT, we can expand work and fill up all the available time with things to do. I’m sure given more time, we can invent ten more useful views. But without those basic three, I cannot imagine describing an integration application. As Antoine de Saint-Exupery put it long ago: “Perfection is finally attained not when there is no longer anything to add but when there is no longer anything to take away.

How To Import Any JBoss BRMS Example Project

This tips & tricks comes to you after I have been asked the following repeatedly over the last few weeks by users of the JBoss BRMS demos:

“How can I import the projects associated with the various JBoss BRMS demo projects into my own existing installation?”

What this means is that users want to have an example project in their personal installation of the product without using the projects installation process. This is certainly possible but not totally obvious to everyone.

Below I will walk you through how the various example projects for JBoss BRMS are setup, how the actual rules projects are loaded into JBoss BRMS when you set them up and why. After this I will show you how to extract any of the available rules projects for importing in to any previously installed JBoss BRMS server.

Figure 1: In JBoss BRMS open the Administration
perspective with menu options, Authoring -> Administration.

Background on how it works

The normal installation of a JBoss BRMS demo project that I have provided uses a template. This template ensures that the process is always the same; download, unzip, add products and run the installation script. After doing this, you are done, just fire up the JBoss BRMS for the adjusted experience where you open up the Authoring perspective to a pretty process designer with the demo project displayed for you to kick off a demo run.

These projects have a demo template that provides some consistency and you can read about how it works in a previous article.  For the initial installation run of any of these demo projects, a folder is copied from support/brms-demo-niogit to the installation at the location target/jboss-eap-{version}/bin/.niogit. 

Figure 2: To import a new project, open the Clone repository
from the menu Repositories. This will allow you to bring
in any rules project to your JBoss BRMS.

This folder contains all of the project and system Git repositories that are formatted for the version of the project you have downloaded. By installing this directory or complete repository, when JBoss BRMS starts up the first time, it will pick up the state I left it in when designing the experience around you using this demo project.

Get your hands on a specific rules project

The problem I want to help you with in this article is to show you how to extract only the rules project from one of these examples and import this into your own installation of JBoss BRMS.

Figure 3: Cloning a repository is how you import an
existing project, which requires the 
information shown.

The following list is the order you do the tasks, after which I will explain each one:

  1. Download any JBoss BRMS demo project and unzip (or clone it if you like).
  2. Log in to your own JBoss BRMS and open Administration perspective via menu: Authoring -> Administration.
  3. Setup the new rules project you want to import: Repositories -> Clone repository -> fill in details including import project URL
  4. Explore the new project in the Authoring perspective: Authoring -> Project Authoring
I am going to assume you can find a JBoss BRMS demo project of your liking from the link provided in step 1 and download or clone to your local machine.

I will be using the JBoss BRMS Cool Store Demo as the example project you want to import into your current JBoss BRMS installation instead of leveraging the standalone demo project.

In your current installation where you are logged in,  open the Administration perspective as shown in figure 1 by menu options Authoring -> Administration. This allows you to start importing any existing rules project. We will be importing the Cool Store rules project by using the feature to clone existing projects found in menu options, Repositories -> Clone repository as shown in figure 2.

Figure 4: Once the project has been imported (cloned), you
will receive this message in a pop-up.
This will produce a pop-up that asks for some information about the project to be imported, which you can fill in as listed below and shown in figure 3:
  • Repository Name: retail
  • Organizational Unit: Demos    (select whatever org you want to use from your system)
  • Git URL:  file:///[path-to-project-you-downloaded]/brms-coolstore-demo/support/brms-demo-niogit/coolstore-demo.git
Figure 5: Explore your newly imported rules project in the
authoring perspective within your JBoss BRMS installation.

The most interesting bit here is the Git URL, which is normally something hosted online, but this project we want to import is positioned locally in our filesystem, so we use a file based URL to point to it. Click on Clone button to import the project and you should see a pop-up that looks like figure 4 stating that you have successfully imported your project.

Now you can explore the new imported project in your authoring perspective and proceed as you desire with this project as shown in figure 5. This will work for any project I have put together for the field that is based on the standard template I use.

I hope this tips & tricks helps you to explore and enjoy as many of the existing rules examples offered in the current collection of demo projects.

 

See more by Eric D. Schabell, contact him on Twitter for comments or visit his home site.

Upcoming: Webinar on Design Approaches for Business Automation

Justin Holmes is a business automation practice lead with Red Hat. It is essentially his job to come up with practical solutions to business problems. He is conducting a webinar next week to go over design practices to more effectively develop and deliver software products, with a heavy emphasis using business rules and process automation to make testing and deploying software more controlled and easier. This webinar will look at two historically separate development processes — engineering-driven development and business rules-driven automation — and how it is possible to develop a design model using the strengths of both.

He has more details on the Services Speaks blog.

register_now

This event is free. It will be Tuesday, July 12, at 11am Eastern time.

Mike Piech and Rich Sharples on Facebook LIVE from Red Hat Summit

Rich Sharples, senior director of product management, and Mike Piech, vice president of marketing, got together for a half hour at the end of the Summit day today to discuss some of the major issues that have come out related to middleware this week. There have been some major announcements: the new microprofile project, the release of Red Hat JBoss EAP 7, the growth of microservices, and the recent acquisition of 3scale and what that means for API management in Red Hat Middleware.

As a quick summary, two of the major themes underscoring a lot of the announcements around JBoss, middleware, and Java this week relate to things that are micro: microservices and microprofile.

Microservices has been a subtext in many of the JBoss EAP 7 sessions and in the OpenShift sessions because this containerized, immutable, consistent environment is what makes microservices possible.Containers fundamentally enable microservices. You have an underlying runtime that is commensurate with the idea of “micro.” You can scale elastically, add instances to scale up and down. The opportunity to change things as an application travels from the desktop to the data center is much less. These are communicating systems, and that’s what container orchestration is. It coordinates these complex webs. we’re The application is the only thing that matters. Operations is there to support the application. I hit a build button and it goes through my CI/CD system, and it’s the same configuration in the environment.

However, like any application or project architecture, it’s more than “JBoss + OpenShift  = awesome microservices.” There has to be consideration and weight given to the application and the underlying technology to find a structure that fits. Microservices architecture isn’t about taking everything you’ve got and decomposing it into atomic services. It’s about having a range of sizes and services, depending on what you need. It is important to be conscious of the trade-offs that come from the increased complexity of the system. It really depends on the organization and the technology platforms they have what architecture is appropriate.

That need to understand and define the underlying framework to do microservices effectively is the theme of the second topic: the microprofile. There are defined specifications for different Java platforms (Standard and Enterprise) but both have the assumption of large-scale, full server architectures. New wave development, though, is increasingly small, with small services in those larger complex systems. What Java EE introduced to development was consistency and dependability. As we move into a new containerized world, we must do it responsibly, preserving the consistency and stability of previous environments. The microprofile project was created because a lot of vendors – Red Hat, IBM. Tomitribe, Payara – were just on a Slack chat, discussing what they needed to do for microservices and ways they could implement it. And then there was a lightbulb: maybe there’s something here. This is a chance to bring the whole Java community around a new architecture, with the strengths and discipline they’ve already developed.

Watch the whole video. For microprofile, you can join the Google group or check out the microprofile site for more information and emerging discussions.

Red Hat Summit Preview – Discovery session series

When we go to the Red Hat Summit this year in San Francisco, we have planned to attend sessions, labs, evening events and even maybe a few good seafood restaurants. Little did you know that there is a gem you might want to fit into your busy schedule, as it is a chance to meet some of the rock stars that are backing the  Red Hat Open Innovation Labs.

There will be a series of sessions hosted by experts to showcase use of Red Hat technologies and demonstrate the best practices with interactive white boarding. That is a personal touch session where you can interact with the storytellers and will be taking place in the West Lobby of M0scone Center on level 2.

Continue reading “Red Hat Summit Preview – Discovery session series”

App Dev Cloud Stack – Open interoperability critical to success

This series started with the statement, what do you mean by “Can’t ignore the stack anymore?”

When your background is application development, you have spent many hours, days and years perfecting your craft. You have not only mastered languages and concepts, you have made it a point to learn to make good architectural decisions when pulling together the applications you develop.

The problem is, we tend to ignore the stack we are working on as much as we can. Well it’s time that we as application developers broadened our horizons a bit, expanding our understanding of the stack we work on with the introduction of Cloud, Platform As A Service (PaaS) and containers to our toolboxes.

Our tour of your Cloud stack continues, from our previous article in this series where we talked about our PaaS interface for our application delivery, onto how open interoperability is critical to the success of our Cloud stack.

Continue reading “App Dev Cloud Stack – Open interoperability critical to success”

Building an API-Based Connected Healthcare Solution: Q&A Followup

Christina Lin (a technology evangelist for Red Hat) and Sameer Parulkar (middleware product marketing manager for Red Hat) conducted a webinar earlier this week about data integration challenges which specifically face healthcare providers. As promised, this is a brief roundup of the major questions that came out of the webinar and pointers to more detailed information about the demo. (If you would like more background on integration challenges in healthcare, we do have posts on integration architecture for healthcare and another on how to overcome integration challenges.)

A Quick Summary

The recording of the full webinar is available here, but I’ll summarize it briefly if you can’t watch it yet.

Continue reading “Building an API-Based Connected Healthcare Solution: Q&A Followup”

Upcoming: Architecture Designs for Camel Developers

Bilgin Ibryam, a senior architect with Red Hat, will be conducting a webinar about design patterns for new architectures like microservices, IoT, and SOA — which Apache Camel developers can use to be more effective in their coding.

Apache Camel itself is based on defined set of design patterns for messaging and integration. This makes Apache Camel a natural framework for designing microservices and IoT applications, which are inherently distributed computing systems. However, developing applications in Camel requires layers of design decisions, because effectively isolating computing components requires a clear understanding of how they will be interacting. This webinar will call out commonly used patterns and design principles for Camel application development, based on real-world examples. This covers a variety of principles, from error handling to complex, multi-route applications, scalability, and high availability.

Registration is open. The webinar is Tuesday, June 7, at 11:00am Eastern Time (US).

register_now

Fun Follow Up: Webinar Q&A

I will collect any questions asked during the webinar, and I’ll do a follow-up post on Friday, June 10, to try to capture the most interesting (or confounding) questions that arise.

Defining A New Value for IT

One trending phrase for CIOs is digital transformation. While the phrase itself has an easily-discerned meaning (digital technologies are changing the way that businesses operate), it is a superficial simplicity. Since every organization has a unique culture, product, and customer set, the ways and means that those organizations will digitally transform is also unique. In a real sense, digital transformation is less about technology and more about culture change.

CIO_ITAccomplishment_4

Although it steers clear of the trendy buzzwords, this kind of culture change is at the heart of the whitepaper and Society for Information Management presentation by Jason Daube and Matt Lyteson of Red Hat IT.

The practical effect of digital transformation is that IT is no longer a back-office department. IT priorities — and IT challenges — now have a strategic impact on business priorities. WHat Daube and Lyteson outline is a high-level approach to aligning IT objectives with business objectives.

Of course, the whole thing is worth reading. For this post, I just want to touch on the foundational layer that they identify: creating an IT-business partnership.

Continue reading “Defining A New Value for IT”