What Are You Getting from (Big) Data?

Gartner has a term for information which is routinely gathered, but not really used: dark data. This is information which is collected for a direct purpose (like processing an online transaction), but then never really used for anything else. By IDC estimates, dark data represent about 90% of the data collected and stored by organizations.

The Internet of Things (specifically) and digital transformation (more generally) are business initiatives that try to harness that dark data by incorporating new or previously untapped data streams into larger business processes.

Big data refers to that new influx of data. The “big” adjective can be a bit misleading — it doesn’t necessarily mean that these are massive amounts of data. Some organizations may be dealing with petabytes of data, but some may only be gigabytes. It’s not a given amount of data, but rather the scale of increase from previous data streams.

Continue reading “What Are You Getting from (Big) Data?”

Intro to In-Memory Data Grids

Some of the biggest technology trends aren’t necessarily about doing something new. Things like cloud computing (as an environment) and design patterns for the Internet of Things and mobile applications (as business drivers) are building on existing conceptual foundations — virtualization, centralized databases, client-based applications. What is new is the scale of these applications and the performance expected from them.

That demand for performance and scalability has inspired an architectural design called distributed computing. Technologies within that larger umbrella used distributed physical resources to create a shared pool for that service.

One of those technologies is the purpose of this post — in-memory data grids. It takes the concept of a centralized, single database and breaks it into numerous individual nodes, working together to create a grid. Gartner defines an in-memory data grid as “a distributed, reliable, scalable and … consistent in-memory NoSQL data store[,] shareable across multiple and distributed applications.” That nails the purpose of distributed computing services: scalable, reliable, and shareable across multiple applications.

Continue reading “Intro to In-Memory Data Grids”

Intro to Integration

Integration is one of those concepts that is easy to “know,” but becomes less obvious that more you try to define it. A basic, casual definition is making different things work together. The complexity, though, comes from the fact that every single part of that has to be broken down: what are the “things,” what are they doing that makes them “work together,” how are they working, and what is the goal or purpose of them working together. All of those elements can be answered differently for different organizations, or even within the same organization at different times.

An understanding of integration comes from looking at the different potential patterns that you can integrate and then defining the logic behind the integration so you can select the right patterns for your environment.

Integration Patterns

Integration itself is an architectural structure within your infrastructure, rather than an action or specific process. While getting various systems to work together has long been an IT (and organizational) responsibility, integration as a practice became more of a focus in the early 2000s. With emerging large-scale enterprise applications, there became a growing need to get those applications working together without having to redesign or redeploy the applications themselves. That push became integration.

Integration is subdefined by what is being integrated; these are the integration patterns.

There are different types of patterns, depending on perspective. There are patterns based on what is being integrated and then there are patterns based on the topology or design of the integration. Basically, it’s the what and the how.

Continue reading “Intro to Integration”

Intro to DevOps

DevOps is a portmanteau of development and operations. Simple definition, blog post over.

Of course, in practice, DevOps is much more nuanced. The signal there is in practice — while there are different schools of thought on whether DevOps is a set of tools or a culture or a process, one thing that is consistent is that it is a thing that is done. It is not a theory (which agile development kind of is) and it’s not a tooling change in that simply changing tools won’t change your outcomes. It really is a way of practicing how your applications are designed and delivered.

The Background: Agile and CI / CD

DevOps means a lot of things to a lot of people, and its meaning depends on the context. Because it is a rather vague term, DevOps is frequently confused with a couple of other critical concepts, namely agile development and continuous integration / continuous delivery (CI/CD). The associations are fair — it’s just important to understand what those associations are.

Continue reading “Intro to DevOps”

Intro to the Internet of Things

According to a 2014 Forbes article (actually, the autoplay video on the article), 87% of people had never heard the term “the Internet of things.” That has changed rapidly in the last two years (research firm 451 Research pegs 2016 as the year that the Internet of Things goes mainstream). Still, as with many cloud computing concepts, IoT is a vague term.

A Simple Description of the Internet of Things

Consumer-centric devices have emerged over the last forty years, from ATMs to inventory tracking in vending machines. Smart phones were a massive jolt, introducing a new means to connect to and interact with both consumers and physical objects. That networked, digitized environment of physical objects is the Internet of Things. 451 Research had a fantastic term for it: the Internet of things “virtualizes the physical world.”

Continue reading “Intro to the Internet of Things”

Intro to Scalability

Scalability is one of those words that can mean very different things to different people, even in the same context or the same project. It’s not so much nuanced as it is that the definition matters on perspective — scale can be different for different goals.

There will be upcoming posts on data virtualization, in-memory data grids, integration methods — all areas where an understanding of your current and future needs, resourcing, and loads are critical for planning. Going into those concepts, it helps to understand scale — not just “make it bigger,” but how you make it bigger and when and why.

Continue reading “Intro to Scalability”

Intro to Microservices

An increasingly common buzzword in cloud computing is microservices. Like a lot of things associated with cloud technologies, a precise definition is difficult to find — and it can mean a lot of things to a lot of different people, depending on the context. Since this is a blog devoted to middleware issues, I want to define microservices within the context of that middle layer in computing, for application development.

A Definition

Microservices is an architectural approach for a software system. Meaning, it defines how individual services fit together and how those services are constructed (like, general constraints or best practices). What sets microservices apart from other architectural approaches is that it treats each service as a discrete and independent part of the architecture. That means that services themselves (within that system) have very clear definitions:

Continue reading “Intro to Microservices”