As Systems Get More Complex, Programming Is Getting “Reactive”


Matt Asay

September 23, 2014

Hardware keeps getting smaller, more powerful and more distributed. To keep up with growing system complexity, there’s a growing software revolution—called “reactive” development—that defines how to architect applications that are going to participate in this new world of multicore, cloud, mobile and Web-scale systems.

Jonas Bonér

One of the leaders of the reactive-software movement is distributed computing expert and Typesafe co-founder and CTO Jonas Bonér, who published the original Reactive Manifesto in September 2013. 

Similar to the early days of the “agile” software development movement, reactive programming got early traction with a hardcore fan base (mostly functional programming, distributed computing and performance experts) but is starting to creep into more mainstream development conversations as high-profile organizations like Netflix adopt and evangelize the reactive model.

See also: Netflix’s Chaos Engineering Should Be Mandatory—Everywhere

I caught up with Bonér to ask him about reactive’s traction on the eve of publishing version 2.0 of the Reactive Manifesto. Beware: This stuff gets deep very quickly.

A Reactive Solution To Broken Development

ReadWrite: So what’s not reactive about software today, and what needs to change?

Jonas Bonér: Basically what’s “broken” ties back to software having synchronous call request chains and poor isolation, yielding single points of failure and too much contention. The problem exists in different parts of the application infrastructure.

At the database layer, most SQL/RDBMS databases still rely on a thread pool or connection pool accessing the database through blocking APIs. So if you exhaust the thread pool by blocking all available threads then everything stops. This problem goes all the way down to the native drivers that the vendors provide, and the JDBC standard specification (for accessing relational databases)—which doesn’t support non-blocking/asynchronous access. 

It will take years before it’s supported.

In the service layer, we usually see a tangled mix of highly contended, shared mutable state managed by strongly coupled deep request chains. This makes this layer immensely hard to scale and to make resilient. The problem is usually “addressed” by adding more tools and infrastructure; clustering products, data grids, etc. But unfortunately this won’t help much at all unless we address the fundamental underlying problem. 

This is where reactive can help; good solid principles and practices can make all the difference—in particular relying on share nothing designs and asynchronous message passing.

See also: How One Developer Set Out To Make The Internet Of Things Manageable

In the Web layer, we often see request chains executed in a completely serial fashion, meaning that the response time is the sum of the time it takes to do everything, which can sometimes be hundreds of different service calls. This means that keeping latency under control—bounded, within the SLAs and predictable—while allowing for scale, is both technically very challenging and requires a lot more hardware thanks to inefficient usage of resources. 

In a reactive application you would split up the work in many small composable chunks and run them in parallel—which will bound the latency to a max of the longest performing chunk and make very efficient use of the resources available.

There are many other ways that today’s software is not reactive, but those are a few of the big ones.

Defining Reactive

ReadWrite: What’s the goal of the reactive movement? What are you trying to accomplish?

JB: A lot of companies have been doing reactive without calling it “reactive” for quite some time, in the same way companies did agile software development before it was called “agile.” But giving an idea a name and defining a vocabulary around it makes it easier to talk about and communicate with people. It makes it easier to explain and bring to market a set of principles that are known to work well together.

Not everyone’s view of agile fits into the agile definition, but what has become agile—and the reason for all these experts to write up the Agile Manifesto—is that they knew which principles worked well together and completed each other in a cohesive story. 

This is what reactive is all about. We found these core principles to work well together in a cohesive story. People have used these approaches years before, but this grouping and this reactive story has meaning, in the same sense of agile. And it provides a baseline for solving problems against the wish-list of application behavior that everyone wants.

The Future Of Reactive Programming

ReadWrite: What are the next steps for the reactive movement?

JB: The reactive principles trace all the way back to the 1970s (e.g., Tandem Computers) and 1980s (e.g., Erlang), but scale challenges are for everybody today. You don’t have to be Facebook or Google anymore to have these types of problems. There’s more data being produced by individual users, who consume more data, and expect so much more, faster. There’s more data to shuffle around at the service layer; replication that needs to be done instantaneously, and the need to go to multiple nodes almost instantaneously. 

And the opportunities have changed, where virtualization and containerization make it easy to spin up nodes and cost almost nothing—but where it’s much harder for the software to keep up with those nodes in an efficient way.

So in many ways the next steps for reactive are to just keep refining its view of that problem set—and the application characteristics that developers should aspire to—to conquer them. Martin Thompson, Roland Kuhn, Dave Farley and I have in fact just rewritten the Reactive Manifesto, taking a lot of great feedback we have been getting from the community into account, distilling it into a much shorter and simpler document.

But the big next step for Reactive is expanding beyond principles, to also bring in more specific tools, techniques, patterns and best practices, thereby making it approachable for the masses. 

We are planning to write an appendix to the Reactive Manifesto in which we can dive in and provide more hands on practical advice on how to design and implement reactive systems. We’re also starting to see more vendors provide solutions and tools that support building reactive systems, more technical books being published about reactive, and more presentations (and and even full tracks) at events tied to reactive—so this is already happening. 

One good example of this is the React conference that I’m helping to organize, which will be a great place to learn how to build reactive systems from some of our top thought leaders in the industry and discuss its principles and practices.

ReadWrite: What are the green field areas that you think will drive the requirements for reactive in the future?

JB: There are several:

  1. One interesting area that has a lot of debate is around “microservices”—which basically is a conversation around what the smallest ideal isolation of a single “service” and its behavior looks like. 
  2. Another is the emerging need is to stream large volumes of potentially infinite data streams in real-time, while keeping latency predictable and without overloading the server. 
  3. This ties in to the rising need of reactive Big Data solutions (sometimes called “fast data”)—providing (close to) real-time analytics and data processing. 
  4. Internet of Things is another huge driver for new approaches to application infrastructure, where machines and devices are generating new challenges for managing and replicating bursts of data throughout distributed environments, and where individual nodes have new requirements for starting/stopping/dealing with failure based on events.   

Lead photo by nerovivo

Great ! Thanks for your subscription !

You will soon receive the first Content Loop Newsletter