Announcing AMQ Streams: Apache Kafka on OpenShift

Cross-posted from the Developers Blog. See the session at Red Hat Summit on Apache Kafka and AMQ data streams on Thursday, May 10, at 11:15.

We are excited to announce a Developer Preview of Red Hat AMQ Streams, a new addition to Red Hat AMQ, focused on running Apache Kafka on OpenShift.

Apache Kafka is a leading real-time, distributed messaging platform for building data pipelines and streaming applications.

Using Kafka, applications can:

  • Publish and subscribe to streams of records.
  • Store streams of records.
  • Process records as they occur.

Kafka makes all of this is possible while being fast, horizontally scalable and fault tolerant. This makes Kafka suitable for a large range of use cases, including website activity tracking, metrics and log aggregation, stream processing, event sourcing, and IoT telemetry. The forthcoming AMQ Streams product will provide Red Hat customers with a supported offering for running Apache Kafka on Red Hat Enterprise Linux and on Red Hat OpenShift Container Platform.

As more and more applications move to Kubernetes and OpenShift, it is increasingly important to be able to run the communication infrastructure on the same platform. OpenShift as a highly scalable platform is a natural fit for messaging technologies such as Kafka. But with AMQ Streams, our target is not to just run Apache Kafka on OpenShift, but rather AMQ Streams makes running and managing Apache Kafka “OpenShift native.”

Uniting the massive scalability of Kafka on an elastic platform like OpenShift involves resolving a number of technology challenges:

  • Kafka brokers are inherently stateful, because each has its own identity and data logs that must be preserved in the event of restarts.
  • Updating and scaling a Kafka cluster requires careful orchestration to ensure that messaging clients are unaffected and no records are lost.
  • By design, Kafka clients connect to all the brokers in the cluster. This is part of what gives Kafka its horizontal scaling and high availability, but when running on OpenShift, this means the Kafka cluster cannot simply be put behind a load-balanced service like other services. Instead services have to be orchestrated in parallel with cluster scaling.
  • Running Kafka also requires running a Zookeeper cluster, which has many of the same challenges as running the Kafka cluster.

AMQ Streams simplifies the deployment, configuration, management and use of Apache Kafka on OpenShift using the Operator concept, thereby enabling the inherent benefits of OpenShift, such as elastic scaling. An Operator is an application-specific controller that extends the Kubernetes APIs and combines them with domain-specific knowledge and makes it easy to run and manage complex applications.  Developers and administrators used to OpenShift’s declarative approach to resource provisioning can now enjoy those same benefits when working with Kafka, Kafka Connect, and Kafka topics.

AMQ Streams makes it easy to:

  • Deploy a complete Kafka cluster, at the scale that suits you, with the click of a button or with a single oc create command.
  • Deploy the Kafka topic right alongside the microservice that uses it.
  • Scale up the partitions of that topic.
  • Trivially scale up and and down the Kafka cluster according to load.

Strimzi Logo

AMQ Streams is optimized for running on OpenShift (as opposed to regular Kubernetes). Not only does it benefit from Red Hat’s years of experience and in-depth knowledge gained from developing and running OpenShift, but there is, for example, special support for building Kafka Connect clusters with the user’s own Kafka Connect plugins. Further, OpenShift-specific features and Red Hat product integrations are anticipated, with the overall aim being a seamless experience with full support from the OpenShift fabric up.

Unsurprisingly, because Red hat is the world’s leading provider of open source technologies for the enterprise, AMQ Streams is fully open source and based on the Strimzi project.

The Developer Preview, which is being made available to interested customers this week, provides the foundation for running Kafka on OpenShift. Interested customers and other interested parties are invited try it out, give us their feedback and, if desired, collaborate on the open source Strimzi project to help shape the future direction of AMQ Streams on OpenShift.

If you’re lucky enough to be in San Francisco this week for Red Hat Summit, then you can hear a lot more about AMQ Streams (and the broader Red hat AMQ product) at the following sessions:

  • Running data-streaming applications with Kafka on OpenShift
    Tue May 8, 1:00 PM–3:00 PM, Moscone South 156
    Marius Bogoevici, Paolo Patierno, Gunnar Morling [L1099]
  • Red Hat AMQ overview and roadmap
    Wed May 9, 11:45 AM–12:30 PM, Moscone West 2011
    David Ingham, Jack Britton [S2802]
  • Introducing AMQ Streams—data streaming with Apache Kafka
    Thu May 10, 11:15 AM–12:00 PM, Moscone West 2014
    Paolo Patierno, David Ingham [S1775]
  • Red Hat AMQ Online—Messaging-as-a-Service
    Thu May 10, 1:45 PM–3:45 PM, Moscone South 214
    Ulf Lilleengen, Paolo Patierno [W1098]

We expect to release further previews as we iterate towards the general availability release, which is planned for later this year.

Please give it a try and let us know what you think.

 

What is agile integration?

**This post was updated on September 26, 2018.**

If you Google the term “agile integration,” you’ll come up with about 30 million results, but they focus heavily on one area: continuous integration within agile development. That definition of agile integration is based on the build environment.

However, it is possible to have another definition for “agile integration,” one that looks at the platform architecture.

In this definition, “agile” doesn’t relate to the process or the infrastructure, but to the flexibility and adaptability–the agility–of the application architecture. Integration within this context has a more strategic role, as the architectural framework that defines the interoperability of services and with a focus on the application functionality.

Check out this e-book to learn more about Agile Integration: The Blueprint for enterprise architecture.

Traditional vs. agile as an architectural approach

There are functional similarities between traditional integration and agile integration – like routing, connectivity, and orchestration capabilities. The difference between traditional enterprise application integration and agile integration is not in the tasks performed, but in the strategic perspective of those tasks. Put simply, integration can be viewed as a necessary but often limited part of the infrastructure (traditional) or it could be viewed as the core framework of the application architecture (agile).

Continue reading “What is agile integration?”

Summit Notes: Tuesday Morning General Session

If you missed it, the keynote speeches are available on the Summit page or on YouTube.

“You don’t need to focus on technology. You need to empower your developers.”

There are certain patterns in the middleware / application development tracks for Red Hat Summit this year, and they revolve a lot around microservices. That makes a certain kind of sense (microservices are the new hotness in app development), but it’s also reflective of a larger current in technology, a continuing push toward … something.

In his opening keynote, Red Hat EVP Paul Cormier noted that one of the themes of Summit 2016 was “dev and ops coming together through common architectures, processes, and platforms.” This echoes major trends in technology — DevOps and architectures, process, and platform as a unifying IT strategy — and yet none of these concepts are really new. Two decades ago, there were developers and operations, there was enterprise architecture, application platforms, and internal processes. So what’s new and what is bringing the urgency now?

I think the difference comes down to speed (and eventually differences in degree become differences in kind). Twenty years ago, an application was released yearly, sometimes even every couple of years. A patch or security update could take a few months to move in the pipeline from development to testing to production.

Now customers expect patches for security vulnerabilities within hours of them being detected, and the expanding number of applications (from consumer mobile apps to internal systems to IoT devices) means that enterprises have potentially dozens of touchpoints and hundreds of services to maintain.

The “modern” part of modern application development isn’t in the app — it’s in the speed.

This year’s Summit kicked off with three interlocking demos, each showing the different paths and progressions that an IT environment will face as they juggle modernizing existing applications and creating new ones within a heterogeneous (and dynamically changing) ecosystem.

 Lifting and Shifting (Windup)

Continue reading “Summit Notes: Tuesday Morning General Session”

Reactive architecture for hybrid cloud environments: Red Hat JBoss AMQ 7 is now available

Red Hat JBoss AMQ 7, now available, introduces a new reactive architecture, with an enhanced broker, a new interconnect router, and expanded client support. This new architecture is more responsive and increases both throughput and performance for messaging services.

The JBoss AMQ broker, based on Apache ActiveMQ Artemis, manages connections, queues, topics, and subscriptions. Using innovations from Artemis, the broker has an asynchronous internal architecture, which can increase performance and scalability and enable it to handle more concurrent connections and achieve greater message throughput. Additionally, the high availability topology for AMQ has been redesigned for a “share nothing” architecture — this removes the need for a centralized database or storage location and uses a distributed, highly available topology instead.

The new interconnect router allows unrestricted redundancy. The router automatically reroutes messaging traffic between data centers, cloud services, and geographic locations. As with the broker’s distributed data topology, the interconnect router is the core for distributed messaging services, which allows operations to have redundant, secure, and reliable connectivity and to optimize messaging between services.

JBoss AMQ 7 expands its support of popular messaging APIs and protocols by adding new client libraries (on top of its existing MQTT and AMQP support):

  • Java Message Service (JMS) 2.0
  • JavaScript
  • C++
  • .NET
  • Python

By creating a more distributed topology and broad protocol and language support, JBoss AMQ is a more reactive messaging platform and can support dynamic microservices and other application architectures.

JBoss AMQ is a lightweight, standards-based open source messaging platform designed to enable real-time communication between different applications, services, devices, and the Internet of Things (IoT). It also serves as the messaging foundation for Red Hat Fuse, Red Hat’s lightweight, flexible integration platform, and is designed to provide the real-time, distributed messaging capabilities needed to support an agile integration approach for modern application development.

Additional resources

Messaging: The Underappreciated Element of Integration

Many people take integration messaging for granted, and many organizations assume that any messaging platform is as fully featured as any other. But in today’s increasingly connected world, with the emergence of major trends in consumer and enterprise technology like mobile, cloud computing, big data, and the Internet of Things, your organization needs to carefully review its messaging platforms and capabilities if you hope to continue to reliably serve your customers and deliver and maintain critical advantages over your competition.

To illustrate how vital and varied messaging platforms can be, let’s explore what exactly messaging is and compare several different approaches to meeting your organization’s messaging needs.

Continue reading “Messaging: The Underappreciated Element of Integration”