Portfolio Management: Balancing the Portfolio

One of the challenges of IT management is to balance the enterprise portfolio with initiatives that deliver on objectives and outcomes with varying timeframes and differing investment categories. Yet this balance is key to run, grow, and transform the business now and over time.

Balancing the enterprise portfolio is important to deliver on initiatives within short (within the fiscal year), medium (1 to 2 years) and long (over 2 years) timeframes. This is part of the advice for a lean startup.

epmo-ppm-investment-planning

Source: Gartner PPM & IT Governance Summit 2016 – Secrets of Prioritizing IT Demand – Audrey Apfel

Continue reading “Portfolio Management: Balancing the Portfolio”

Data and Architecture Simplified, pt. 3: Business Architecture – The Core Diagram

To effectively plan and execute a technology-driven service or product offering, IT and business leaders should start with business architecture. Business architecture is the essential building block for mapping an organization’s business vision of what they want to accomplish. Business architecture is one of the four enterprise architecture domains – including data, applications and technology.

Continue reading “Data and Architecture Simplified, pt. 3: Business Architecture – The Core Diagram”

It’s a great Red Hat day in Minneapolis — Go Microservices !

Cross posted from the Red Hat Events Blog.

It was a great day in Minneapolis! The Microservices with Apache Camel was held at Target Field (inside the ballpark, overlooking the field of play). “Takes a lot to put together an event like this but can certainly be a lot of fun! Go microservices!,” says Red Hat associate Jen Fissel.

1-jen-fissel
Jen Fissel

I had the privilege of hosting the event and kicked off the event with a reference to the connected world we live in that requires enterprises to be agile while being integrated across the systems of yesterday with the evolving applications of the future. The future of Enterprise IT, containers, are here today and microservices are the stars of the show. Welcome to Minneapolis!

Continue reading “It’s a great Red Hat day in Minneapolis — Go Microservices !”

Data and Architecture, pt 2: Process Improvement

Does your organization need to reduce costs and improve efficiencies? Start with a process-first approach. Before you dive into what software tool to implement or select a new solution to address a business challenge, understand your existing business processes. What steps does your organization take within the business processes? Are things manual? Can you automate and improve the way you do business?

Continue reading “Data and Architecture, pt 2: Process Improvement”

How to integrate business logic in processes with JBoss BPM

In June 2016 the Manning Early Access Program (MEAP) started for the book Effective Business Process Management with JBoss BPM.

What is a MEAP?

The Effective Business Process Management with JBoss BPM MEAP gives you full access to read chapters as they are written, get the finished eBook as soon as it’s ready, and receive the paper book long before it’s in bookstores.

You can also interact with the author, that’s me, on the forums to provided feedback as the book is being written. So come on over and get started today with Effective Business Process Management with JBoss BPM.

The way the MEAP works is that every month or so Manning puts a new chapter online. As of this week chapter 5 is available and those already in the MEAP will have access to start reading the chapter.

This is a large chapter and it is one of the harder topics to confine to a single chapter. I do expect to split this chapter up in the future so that you have the basics and then more advanced topics regarding learning to effectively implement your business logic with JBoss BPM.

To give you an idea of what’s available so far:

You can read this excerpt online before you decide, but I look forward to hearing from you on the content and stay tuned for more.

 

See more by Eric D. Schabell, contact him on Twitter for comments or visit his home site.

Data and Architecture, pt. 1: Ask the Right Questions

Your business needs to better use its data — but what does that mean? Context matters. Data governance, reporting and analytics, business intelligence. When you approach your data architecture, first start with asking the right questions that solve business challenges. What data does the sales team need to increase sales by x %? What data does the engineering team need to work on and innovate products that provide competitive advantage?

Similar questions have inspired companies to disrupt markets. Uber started with asking questions like, how can we optimize drivers at the right locations with the most customer demand? How can we give consumers the ability to call a cab with a click of a button? Tomasz Tunguz highlights similar examples in his book Winning with Data, where he states that “the best data-driven companies operationalize data.” To operationalize data is where companies can use the right data to rapidly change the way they operate.

In order to use the right data, ask the right questions. Gartner states exactly this, “Ask the right questions” in the 2015 article Big Data Analytics Failures and How to Prevent Them. Simply, what problem or toughest business challenge is your company trying to solve with data?

At the foundation and beginning practice of enterprise architecture, dating back to the 70’s with the Zachman Framework, the principle question related to data was “what data is needed to list the things most important to the business?” The framework focuses on the “what”, what is needed, in order to produce the right enterprise, data and technology models for the best use of data.

In order to harness the “Power of Data”, HBR suggests that companies start with the business problem in mind, and then “seek to gain insights from vast amounts of data.” With stating the specific business problem, companies can narrow the search and refine how they are going to find data-driven answers to their most challenging business problems.

Intro to In-Memory Data Grids

Some of the biggest technology trends aren’t necessarily about doing something new. Things like cloud computing (as an environment) and design patterns for the Internet of Things and mobile applications (as business drivers) are building on existing conceptual foundations — virtualization, centralized databases, client-based applications. What is new is the scale of these applications and the performance expected from them.

That demand for performance and scalability has inspired an architectural design called distributed computing. Technologies within that larger umbrella used distributed physical resources to create a shared pool for that service.

One of those technologies is the purpose of this post — in-memory data grids. It takes the concept of a centralized, single database and breaks it into numerous individual nodes, working together to create a grid. Gartner defines an in-memory data grid as “a distributed, reliable, scalable and … consistent in-memory NoSQL data store[,] shareable across multiple and distributed applications.” That nails the purpose of distributed computing services: scalable, reliable, and shareable across multiple applications.

Continue reading “Intro to In-Memory Data Grids”

Intro to Integration

Integration is one of those concepts that is easy to “know,” but becomes less obvious that more you try to define it. A basic, casual definition is making different things work together. The complexity, though, comes from the fact that every single part of that has to be broken down: what are the “things,” what are they doing that makes them “work together,” how are they working, and what is the goal or purpose of them working together. All of those elements can be answered differently for different organizations, or even within the same organization at different times.

An understanding of integration comes from looking at the different potential patterns that you can integrate and then defining the logic behind the integration so you can select the right patterns for your environment.

Integration Patterns

Integration itself is an architectural structure within your infrastructure, rather than an action or specific process. While getting various systems to work together has long been an IT (and organizational) responsibility, integration as a practice became more of a focus in the early 2000s. With emerging large-scale enterprise applications, there became a growing need to get those applications working together without having to redesign or redeploy the applications themselves. That push became integration.

Integration is subdefined by what is being integrated; these are the integration patterns.

There are different types of patterns, depending on perspective. There are patterns based on what is being integrated and then there are patterns based on the topology or design of the integration. Basically, it’s the what and the how.

Continue reading “Intro to Integration”

New styles of integration are the hallmark of Digital Transformation

New Styles of Integration 2

Shakeup your integration strategy to enable digital transformation, says VP & Gartner Fellow Massimo Pezzini. Pezzini asserts that it is not just about transforming and modernizing the infrastructure and the applications concerned.  Some of the fundamental concepts of integration need to be revisited and transformed as well.  Such systemic transformation punctuate the migration of  legacy environments to microservices and the cloud.  What may have worked in the past will no longer be applicable going forward.  “Integration is dead.  Long live integration,” screamed the title of one of the sessions at the Red Hat Summit 2016.  The session was making a point.  Integration, as we knew it a few years back, is dead.  Integration in the digital world has a long life in the decades ahead.  Join me as I walk through the new styles of integration that are the hallmark of digital transformation.

Continue reading “New styles of integration are the hallmark of Digital Transformation”

Visualizing Integration Applications

Since I’ve changed roles and started performing architect duties, I have to draw more boxes and arrows than write code. There are ways to fight that, like contributing to open source projects during sleepless nights, POCs, demos, but drawing boxes to express architectures and designs is still big part of it. This post is about visualizing distributed messaging, SOA, microservices applications in agile environments (this term has lost its meaning, but there is not better one in this case). What I like about the software industry in recent years is that the majority of organizations I’ve worked with value the principles behind lean and agile software development methodologies. As long as it is practical, everyone strives to deliver working software (rather than documentation), deliver fast (rather than plan for a long time), eliminate waste, and respond to change. And there are management practices such as scrum and kanban, and technical practices from extreme programming (XP) methodology such as unit testing, pair programing, and other practices such as CI/CD and DevOps to help implement those principles. In this line of thinking, I decided to put together a summary of the design tools and diagrams I find useful in my day to day job while working with distributed systems.

Issues with 4+1 View Model and Death by UML

Every project kicks off with big ambitions, but there is never enough time to do things perfectly, and at the end we have to deliver whatever works. And that is a good thing, it is the way the environment helps us avoid gold plating and supports principles like YAGNI and KISS, so we do just enough and adapt to changes.

Looking back, I can say that most of the diagrams I’ve seen around are inspired by the 4+1 view model of Philippe Kruchten which has logical, development, process and physical views.

4+1_Architectural_View_Model
4+1 View Model.  From A Practical Guide to Enterprise Architecture by James McGovern, Scott W. Ambler, Michael E. Stevens, James Linn, Vikas Sharan, Elias K. Jo, 2003.

I quite like the ideas and the motivation behind this framework: using separate views and perspectives to address specific set of constraints and targeting the different stakeholders. That is a great way of describing complex software architectures. But I have two issues with using this model for integration applications.

Diagram Applicability

Typically these views are expressed through a unified modeling language (UML), and for each view, you have to use one or more UML diagrams. If I have to use 15 types of UML diagrams to communicate and express a system architecture in an accessible way, it defeats the purpose of UML.

Death by UML
Death by UML. From Wikipedia, derived from a diagram by Paulo Merson.

With such a complexity, the chances are that there are only one or two people in the whole organization who have the tools to create and ability to understand and maintain these diagrams. And having hard-to-interpret, out-of-date diagrams is as useful as having out-of-date documentation. These diagrams are too complex and with limited value, and very quickly they turn into a liability that you have to maintain rather than an asset expressing the state of a constantly changing system.
Another big drawback is that the existing UML diagram types are primarily focused on describing object-oriented architectures rather than pipes and filters architectures. The essence of messaging applications is around interaction styles, routing, and data flow rather than structure. Class, object, component, package, and other diagrams are of less value for describing pipes and filters based processing flows. Behavioral UML diagrams such as activity and sequence get closer, but still cannot express easily concepts such filtering and content based routing, which are a fundamental part of integration applications.

View Applicability

Having a different set of views for a system to address different concerns is a great way of expressing intent. But the existing views of the 4+1 model don’t reflect the way we develop and deploy software nowadays. The idea of that directional flow — that you have a logical view first, which then leads to development and process views, and those lead to a physical view — is not always the case. The systems development life cycle is not following the traditional (waterfall) sequence of requirement gathering, designing, implementing, and maintaining.

2000px-CPT-SystemLifeSycle.svg
Software Development Lifecycle. Derived from an image by Web Serv

Instead other development methodologies such as agile, prototyping, synchronize and stabilize, and spike and stabilize are used too. In addition to the process, the stakeholders are changing too. With practices such as DevOps, developers have to know about the final physical deployment model and the operations team have to know about the application processing flows.

Modern architectures such as microservices affect the views too. Knowing one microservice in a plethora of services is not very useful. Knowing too much about all the services is not practical either. Having the right abstraction level to have a system wide view with just enough details becomes vital.

Practical Visualization for Integration Applications

The closest thing that has been working for me is described by Simon Brown as the C4 model. (You should also get a free copy of Simon’s awesome The Art of Visualising Software Architecture book). In his model, Simon is talking about the importance of a common set of abstractions rather than common notation (such as UML) and then using simple set of diagrams for different level of abstractions: system context, container, componentand class. I quite like this “outside-in” approach, where you first have 10000 foot view and with each next level, going deeper with more detailed views.
C4 is also not an exact match for middleware/integration applications either, but it is getting closer. If we were to use the C4 model, then the system context diagram would be one box that says ESB (or middleware, MOM, or microservices) with tens of arrows from north to south. Not very useful. The container diagram is quite close, but the term container is so overloaded (VM, application container, docker container) which makes it less useful for communication. Component and class diagrams are also not a good fit as pipes and filter architectures are focused around enterprise integration patterns, rather than classes and packages.
So at the end, what is it that worked for me? It is the following three types of diagrams which abbreviate as SSD (not as cool as C4):  system context, service design, and deployment.

System Context Diagram

The aim of this model is to show all the services (whether they are SOA or microservices) with their inputs and outputs, ideally having the external systems on the north, the services in the middle section, and internal services in the south. Or you could use both external and internal services on both side of the middleware layer as shown below. Also having the protocol (such as HTTP, JMS, file) on the arrows, with the data format (XML, JSON, CSV) gives useful context too, but it is not mandatory. If there are too many services, you can leave the protocol and the data format for the service level diagrams. I use the direction of the arrow to indicate which service is initiating the call rather than the data flow direction.

System Context Diagram
System Context Diagram

Having such a diagram gives a good overview of the scope of a distributed system. We can see all the services, the internal and external dependencies, the types of interaction (with protocol and data format), and the call initiator.

Service Design Diagram

The aim of this diagram is to show what is going on in each box representing a middleware service from the system context diagram. And the best diagram for this is to use EIP icons and connect those as message flows. A service may have a number of flows, support a number of protocols, implement real time, or batch behaviour.

Service Design Diagram
Service Design Diagram

At this level, we want to show all possible data flows implemented by a specific service, from any source to any destination.

Deployment Diagram

The previous two diagrams are the logical views of the system as a whole and each service separately. With the deployment diagram, we want to show where each service is going to be deployed. Maybe there will be multiple instances of the same service running on multiple hosts. Maybe some services will be active on one host, and passive on the other. Maybe there will be a load balancer fronting the services.

Deployment Diagram
Deployment Diagram

The deployment diagram is supposed to show how individual services and the system as a whole relates to the host systems (regardless whether that is physical or virtual).

What Tools Do I Use?

The system context and the deployment diagrams are composed only of boxes and arrows and do not require any special tools. For the service design diagram, you will need a tool that has the enterprise integration pattern icons installed. So far, I have seen the following tools with EIP icon support:

Other development tools that could be also used for creating EIP diagrams are:

A system context diagram is useful to show the system wide scope and reach of the services, a service design diagram is good for describing what a service does, and a deployment diagram is useful mapping all that into something physical.

In IT, we can expand work and fill up all the available time with things to do. I’m sure given more time, we can invent ten more useful views. But without those basic three, I cannot imagine describing an integration application. As Antoine de Saint-Exupery put it long ago: “Perfection is finally attained not when there is no longer anything to add but when there is no longer anything to take away.