Announcing AMQ Streams: Apache Kafka on OpenShift

Cross-posted from the Developers Blog. See the session at Red Hat Summit on Apache Kafka and AMQ data streams on Thursday, May 10, at 11:15.

We are excited to announce a Developer Preview of Red Hat AMQ Streams, a new addition to Red Hat AMQ, focused on running Apache Kafka on OpenShift.

Apache Kafka is a leading real-time, distributed messaging platform for building data pipelines and streaming applications.

Using Kafka, applications can:

  • Publish and subscribe to streams of records.
  • Store streams of records.
  • Process records as they occur.

Kafka makes all of this is possible while being fast, horizontally scalable and fault tolerant. This makes Kafka suitable for a large range of use cases, including website activity tracking, metrics and log aggregation, stream processing, event sourcing, and IoT telemetry. The forthcoming AMQ Streams product will provide Red Hat customers with a supported offering for running Apache Kafka on Red Hat Enterprise Linux and on Red Hat OpenShift Container Platform.

As more and more applications move to Kubernetes and OpenShift, it is increasingly important to be able to run the communication infrastructure on the same platform. OpenShift as a highly scalable platform is a natural fit for messaging technologies such as Kafka. But with AMQ Streams, our target is not to just run Apache Kafka on OpenShift, but rather AMQ Streams makes running and managing Apache Kafka “OpenShift native.”

Uniting the massive scalability of Kafka on an elastic platform like OpenShift involves resolving a number of technology challenges:

  • Kafka brokers are inherently stateful, because each has its own identity and data logs that must be preserved in the event of restarts.
  • Updating and scaling a Kafka cluster requires careful orchestration to ensure that messaging clients are unaffected and no records are lost.
  • By design, Kafka clients connect to all the brokers in the cluster. This is part of what gives Kafka its horizontal scaling and high availability, but when running on OpenShift, this means the Kafka cluster cannot simply be put behind a load-balanced service like other services. Instead services have to be orchestrated in parallel with cluster scaling.
  • Running Kafka also requires running a Zookeeper cluster, which has many of the same challenges as running the Kafka cluster.

AMQ Streams simplifies the deployment, configuration, management and use of Apache Kafka on OpenShift using the Operator concept, thereby enabling the inherent benefits of OpenShift, such as elastic scaling. An Operator is an application-specific controller that extends the Kubernetes APIs and combines them with domain-specific knowledge and makes it easy to run and manage complex applications.  Developers and administrators used to OpenShift’s declarative approach to resource provisioning can now enjoy those same benefits when working with Kafka, Kafka Connect, and Kafka topics.

AMQ Streams makes it easy to:

  • Deploy a complete Kafka cluster, at the scale that suits you, with the click of a button or with a single oc create command.
  • Deploy the Kafka topic right alongside the microservice that uses it.
  • Scale up the partitions of that topic.
  • Trivially scale up and and down the Kafka cluster according to load.

Strimzi Logo

AMQ Streams is optimized for running on OpenShift (as opposed to regular Kubernetes). Not only does it benefit from Red Hat’s years of experience and in-depth knowledge gained from developing and running OpenShift, but there is, for example, special support for building Kafka Connect clusters with the user’s own Kafka Connect plugins. Further, OpenShift-specific features and Red Hat product integrations are anticipated, with the overall aim being a seamless experience with full support from the OpenShift fabric up.

Unsurprisingly, because Red hat is the world’s leading provider of open source technologies for the enterprise, AMQ Streams is fully open source and based on the Strimzi project.

The Developer Preview, which is being made available to interested customers this week, provides the foundation for running Kafka on OpenShift. Interested customers and other interested parties are invited try it out, give us their feedback and, if desired, collaborate on the open source Strimzi project to help shape the future direction of AMQ Streams on OpenShift.

If you’re lucky enough to be in San Francisco this week for Red Hat Summit, then you can hear a lot more about AMQ Streams (and the broader Red hat AMQ product) at the following sessions:

  • Running data-streaming applications with Kafka on OpenShift
    Tue May 8, 1:00 PM–3:00 PM, Moscone South 156
    Marius Bogoevici, Paolo Patierno, Gunnar Morling [L1099]
  • Red Hat AMQ overview and roadmap
    Wed May 9, 11:45 AM–12:30 PM, Moscone West 2011
    David Ingham, Jack Britton [S2802]
  • Introducing AMQ Streams—data streaming with Apache Kafka
    Thu May 10, 11:15 AM–12:00 PM, Moscone West 2014
    Paolo Patierno, David Ingham [S1775]
  • Red Hat AMQ Online—Messaging-as-a-Service
    Thu May 10, 1:45 PM–3:45 PM, Moscone South 214
    Ulf Lilleengen, Paolo Patierno [W1098]

We expect to release further previews as we iterate towards the general availability release, which is planned for later this year.

Please give it a try and let us know what you think.

 

Red Hat OpenShift Application Runtimes: Delivering new productivity, performance, and stronger standards support with its latest sprint release

Red Hat OpenShift Application Runtimes is a collection of cloud-native application runtimes that are optimized to run on OpenShift, including Eclipse Vert.x, Node.js, Spring Boot, and WildFly Swarm. In addition, OpenShift Application Runtimes includes the Launch Service, which helps developers get up and running quickly in the cloud through a number of ready-to-run examples — or missions — that streamline developer productivity.

New Cache Booster with JBoss Data Grid integration

In our latest continuous delivery release, we have added a new cache mission  that demonstrates how to use a cache to increase the response time of applications.  This mission shows you how to:

  1. Deploy a cache to OpenShift.
  2. Use a cache within an application.

The common use case for this booster is to cache service result sets to decrease latency associated with data access as well as reduce workload on backend service.  Another very common use case is to reduce the data volume of message send across in distributed system.

Continue reading “Red Hat OpenShift Application Runtimes: Delivering new productivity, performance, and stronger standards support with its latest sprint release”

Cloud Native Application Development – Adopt or Fail

In today’s digital world, software strategy is central to business strategy. To stay competitive, organizations need customized software applications to meet their unique needs — from customer engagements to new product and services development. Drawn-out development projects are no longer acceptable, given business demands. Therefore, the need to speed up application development, testing, delivery, and deployment is no longer optional but a must-have competency.   

At the same time that developers are confronting this challenge to deliver solutions more quickly, they are also facing the most diverse technology ecosystem in the history of computing.  To address this challenge, development teams must modernize architecture, infrastructure, and processes to deliver higher-quality applications with greater agility.

Cloud native development is an approach to building and running applications that fully exploits the advantages of the cloud computing model.  Cloud native development multidimensionality involves architecture, infrastructure, and processes based upon four key tenets:

  1. Services-based architecture: could be microservices or any modular loosely coupled model for independent scalability and flexibility of maintenance and polyglot language runtimes.
  2. Containers and Docker image: as the deployment unit and self-contained execution environment with consistency and portability across cloud infrastructures.
  3. DevOps automation: implementing processes and practices and instrumentation of development to test deployment of applications.     
  4. API-based design: The only communication allowed is via service interface calls over the network. No direct linking, no direct reads of another team’s data store, no shared-memory model with an outside- in perspective.

Continue reading “Cloud Native Application Development – Adopt or Fail”

“Micro-rules,” event-driven apps, and Red Hat Decision Manager

As we described in an earlier blog, microservices are mini-applications which are devoted to a single, specific function. They are discrete (independent of other services in the architecture), polyglot with a common messaging or API interface, and they have well-defined parameters.

As application development and IT operations teams have started streamlining and speeding up their processes with methodologies like Agile and DevOps, they have increasingly begun treating IT applications as microservices. This breaks up potential bottlenecks, reduces dependencies on services used by other teams, and can help make IT infrastructure less rigid and more distributed.

One area where we are seeing this looser, more distributed approach to service development is with business rules.

“Micro-rules”

Business rules and processes in a traditional structure tend to be centralized, with the complete set of functionality defined for all workflows. The problem with centralization is because there is a single, centralized collection of business rules, any changes to one set of rules can affect many other sets, even those for different business functions.

Micro-rules essentially treat each functional set of rules as its own service — well-defined, highly focused, and independent of other rules.

Figure – Function rule sets as micro-rules

Continue reading ““Micro-rules,” event-driven apps, and Red Hat Decision Manager”

Taking Control of your IoT APIs

At its core, IoT is all about data: data from devices, commands to devices, integrating IoT data with other data to gain insights. The data sources include devices, enterprise applications, vendor/partner systems, service providers and customers. The point-to-point integration between these various systems is not feasible; hence, APIs become the primary means of communication between these disparate systems. A clean architectural approach is the one suggested by the agile integration concept. APIs are central to this concept, which allow data to be shared securely between internal and external systems. The opening of APIs enables a company to provide uniform data and transaction interfaces to internal and external developers, partners, and customers, for improved data access and control of remote resources. By providing well-defined APIs, developers can use data in a programmatic manner; e.g., app developers can get access to IoT devices data without worrying about the underlying hardware interfaces. Considering the importance of APIs for IoT, it’s imperative for an organization to manage these APIs effectively. In fact, APIs have been called a fundamental enabler of IoT however, without an effective API Management solution, API sprawl can easily lead to catastrophe.

Continue reading “Taking Control of your IoT APIs”

The Role of Agile Integration in Open Banking

In the mid 90s, Bill Gates famously said that “banking is necessary, banks are not.” There is certainly a lot of truth in this statement. We all need banking services in some shape or form. But who delivers these services to us is secondary. In fact, Accenture concluded in a study conducted in 2016 asking over 30,000 people in 18 countries that if the tech titans like Google, Amazon, or Facebook would offer such services, 31% of the respondents would switch to them. This clearly imposes a significant threat on traditional banking institutions.

Another challenge that banks are facing worldwide are the increasing demands for regulatory compliance with respect to openness. Such regulations include, for instance, Payment Services Directive 2 (PSD2) in Europe, the Amendment Bill to Japanese Banking Law in Japan, the National Payments Corporation of India (NPCI) with the Unified Payment Interface, UK’s Open Banking standard by the Competition and Markets Authority (CMA), or the Open Banking Regime by Australia’s Federal Government. Banks approach these regulatory challenges in many different ways. Some see it as a serious business threat and only do the bare minimum for compliance; others see it as an opportunity and with smart investment start building banking platforms for the future.

Our suggestion for building the banking platform of the future resides on the principles of agile integration, which is an architectural approach centered around application programming interfaces (APIs) and API management. At its core, agile integration resides on the three pillars: distributed integration for greater flexibility, containers for the ability to scale better, and managed APIs for re-usability and hence speed. We described the details in an earlier post.  

Continue reading “The Role of Agile Integration in Open Banking”

How to Address the Challenges of a Pervasive Integration Strategy

Earlier this months at the Gartner ITxpo event, Massimo Pezzini presented the challenges that must be addressed by a pervasive enterprise integration strategy. In summary there are four types of hybrid challenges (see Massimo’s diagram below).

Gartner-HiP

Continue reading “How to Address the Challenges of a Pervasive Integration Strategy”

The State of Microservices Survey 2017 – Eight trends you need to know

During the fall of 2017, we conducted a microservices survey with our Red Hat Middleware and Red Hat OpenShift customers. Here are eight interesting trends discerned by the results:

1. Microservices are being used to re-architect existing applications as much as for brand new projects

There seems to be a strong emphasis in the market by technology vendors for positioning microservices as being only for new projects.  However, our survey reveals that organizations are also using microservices to re-architect existing and legacy applications.

Sixty-seven percent of Red Hat Middleware customers and 79 percent of Red Hat OpenShift customers indicated this. This data tells us that microservices offer value to users all along their IT transformation journey — whether they are just looking to update their current application portfolio or are gearing up new initiatives. So, if you are only focused on greenfield projects for microservices, it may be a good idea to also start evaluating your existing applications for a microservice re-architecture analysis. Microservices introduce a set of benefits that our customers have already started seeing, and they are applying these benefits not just to new projects but to existing ones as well.

Continue reading “The State of Microservices Survey 2017 – Eight trends you need to know”

Announcing Red Hat Fuse Online Technical Preview

On May 2, 2017, we announced a new open source project called Syndesis.io. Syndesis.io provides a low code environment for agile integration. We also demonstrated key capabilities at the Red Hat Summit 2017 keynote.

Building on our foundational work in Syndesis.io, we have expanded those capabilities into a new product and are happy to announce Red Hat Fuse Online as a technical preview.

Continue reading “Announcing Red Hat Fuse Online Technical Preview”

What is agile integration?

If you Google the term “agile integration,” you’ll come up with about 30 million results, but they focus heavily on one area: continuous integration within agile development. That definition of agile integration is based on the build environment.

However, it is possible to have another definition for “agile integration,” one that looks at the platform architecture.

In this definition, “agile” doesn’t relate to the process or the infrastructure, but to the flexibility and adaptability–the agility–of the application architecture. Integration within this context has a more strategic role, as the architectural framework that defines the interoperability of services and with a focus on the application functionality.

Traditional vs. agile as an architectural approach

There are functional similarities between traditional integration and agile integration – like routing, connectivity, and orchestration capabilities. The difference between traditional enterprise application integration and agile integration is not in the tasks performed, but in the strategic perspective of those tasks. Put simply, integration can be viewed as a necessary but often limited part of the infrastructure (traditional) or it could be viewed as the core framework of the application architecture (agile).

Continue reading “What is agile integration?”