Why Data First Development Rapidly Fills Application Whitespace


Woods, Dan

August 12, 2013

Technology leaders seeking a way to fill the yawning whitespace that exists in most companies should consider taking a data first approach to creating applications.

In his book Business Process Management: The SAP Roadmap, SAP co-CEO Jim Hagemann Snabe pointed out, based on substantial research, that packaged enterprise applications can address 20 percent of the processes in the enterprise. That leaves 80 percent of the processes defined as application whitespace.

By “whitespace,” I mean the gap between the automation and data provided by packaged applications and the work done by end-users to solve their own problems using productivity tools like spreadsheets. Let’s call extending the reach of packaged applications the “top down” approach to whitespace and the use of user-created solutions the “bottom up” approach.

Virtually every company suffers from the excesses of an uncontrolled bottom up approach: the spreadsheet gone wild. Spreadsheets that start out as a way for a talented end-user to organize individual work or collaborative work often end up becoming a management system. These spreadsheets are a nightmare for CIOs and CTOs. They are as complex and manageable as a bowl of spaghetti. They are full of errors and untestable. But damn it: they actually do what users want. (Check out a startup called DataNitro that links Python to Excel for a novel approach to allow such bottom up spreadsheets to become supportable.)

The pain at the top down end of the spectrum is represented by the bottleneck that surrounds enterprise applications and Business Intelligence systems as well. When an application has valuable data but the development tools can only be used by experts, it doesn’t take long for the list of requests to outstrip the ability to meet them. Frustration mounts as the long wait sets in, and then it’s back to the spreadsheets. This explains another weakness of spreadsheet applications. They often don’t make use of data from enterprise applications because it is not easy to get to. In addition, when they do use data, it typically gets stale quickly.

Is Better BI the Answer?

Part of the solution to this problem is better BI. QlikView, Tableau, and other similarly clever products have made good businesses allowing data to be gathered from many sources and presented to users. For simple applications, end-users can meet their own needs. For complex ones, the apps allow exploration. IBM, Oracle, and SAP have been hard at work following this lead and making their BI tools easier to use. But even if BI were perfect, we would still be missing an important class of applications: those that use information from lots of systems and also add automation and the ability to take action. These apps cannot be developed by end-users alone. They also cannot be developed as fast as needed using languages like Java, Python, Ruby, and C# because there are too few developers to do that work and many analysts (who effectively use spreadsheets) are experts in working with data but are not developers.

Companies that find a way to fill application whitespace to allow users to raise their hands about what they need and quickly get a solution that can do substantial work are able to race ahead. These types of apps raise the average level of performance and allow the secret sauce of the virtuosos to be captured and propagated. These apps also greatly expand access to data and help promote a culture of data-driven decision-making.

But how to get there?

Is BPM the Answer?

One of the ways companies are filling whitespace is through business process management (BPM) applications. Appian and Pegasystems are popular choices from independent companies. Most of the big players have one or more BPM systems in their product portfolios. In the right hands, a development team using BPM can run circles around one using Java or other such languages.

But the question of finding the right hands is a complex one. A book I helped write called Process First argued that the key to making BPM work was to involve a person called a “business process expert” in the creating of applications. The business process expert is defined as someone conversant with the details of both IT capabilities and the business problems that are attempting to be solved. (SAP had a community devoted to this role.)

BPM tools can be used in many different ways, but most of the time the focus is on the business process. The business process expert plays the role of the designer of the right process who has the big picture in mind and can crystalize a solution. When we think of BPM we often think of flowcharts that describe processes as the central artifact. After all, isn’t it called Business Process Management? Why shouldn’t the process be the focus?

But talk to end-users about how they do their work. Do they ever start explaining things by drawing flowcharts? No; they show you the first spreadsheet they’ve developed that tracks stuff. Then they show you the next sheet that tracks the next step in the process. Then they show you the incomprehensible macros that translate from one sheet to the next.

My argument is that it is time for technology leaders to stop fighting this trend. BPM’s focus on the process is putting the process before the data. The better approach is to put data first. (To be fair, it is possible to use BPM tools in a data first manner, but in my experience most often the process first approach is used.)

The Data First Approach

My case that the data first approach is worth trying is founded in the success of three related approaches.

Spreadsheet First

First, the way spreadsheets are used proves that users naturally think of data, often in a grid, as a way to organize their work. DataNitro’s approach is to keep the grid in the spreadsheet, but replace the logic layer with Python so that everything becomes more manageable. Ben Lerner, co-founder and CEO of DataNitro argues that because Python is an interpreted language it can more easily be used by normal humans, not just by developers. But as the logic becomes more gnarly, you can use object orientation, source code management, and other well-understood techniques to make the code testable and manageable. I call this approach the spreadsheet first approach to applications.

Unstructured Data First

Second, Peter Thoeny’s concept of the structured wiki has been successful in a variety of environments, most prominently Google, but many others as well. Structured wikis add data in records and grids to be added the shared pages at the foundation of a wiki. This approach, developed to its fullest extent in Twiki, the open-source project Thoeny founded, allows users to take an unstructured data first approach and add structured data as needed.

Structured Data First

The third approach, which crystallized for me after a fascinating conversation with MicroPact’s CTO and co-founder, Mike Cerniglia, can be called the structured data first approach. At MicroPact, using the term “case management,” but very much solving the same problems as BPM and spreadsheet-based apps, the process of creating apps to fill whitespace is far closer to the way business people think about how systems work.

MicroPact calls the collection of information related to an issue a “case”, which collects not only structured data, but also documents, links, videos, and any other related information. For this reason, MicroPact and other similar vendors are referred to as case management software, which is usually considered a related category to BPM.

This approach, in my view, has a real chance of accelerating the speed at which whitespace is filled because it brings that natural focus on data together with the ability to incrementally develop an application not bound by the grid of the spreadsheet. In addition, the application can start out simple, with the basic sheet to sheet transitions that you see in a spreadsheet app, but then grow in depth and complexity in a manageable way. In other words, end-users can get the ball rolling and the super-users or developers can arrive when needed to flesh out the applications.

If we look at the way that MicroPact creates solutions, there are some interesting lessons to be learned about how to build systems to meet business needs.

Bridging the Requirements Gap

First of all, a little theory. In a book I helped write in 2009 called Lost In Translation, Nigel Green argued that IT systems development projects go astray at the beginning when business people and IT people first meet. The business people don’t know how to describe what they want. (They know what they want but they don’t have the language and the tools to express it. MIT Professor Eric Von Hippel, pioneer of User Driven Innovation, calls this sticky information. He recommends giving users the ability to create solutions themselves. But as we know, this is not always possible.)

In most conversations, the business people talk for a bit and then the IT people jump in with a discussion of IT mechanisms such as user interfaces or databases that could help. The system is inadequately described. No wonder it doesn’t turn out right.

Green suggests that to solve this problem business people should describe the system in terms of five dimensions: Values, Processes, Events, Content, and Trust. Then with a complete understanding of what the business wants, IT can build it.

The Agile Approach

Agile development can be seen as another way to bridge the requirements gap. One of the fundamental insights of agile is that it pays to be suspicious of requirements. Agile consciously reduced the requirements process to describe the smallest possible system using artifacts such as stories. A system is built and incrementally perfected as users work with the system and provide feedback.

MicroPact’s Structured Data First Approach

Employing their entellitrak platform, MicroPact’s structured data first approach offers a different way around to bridge the requirements gap, one that fits it with the natural inclination of end-users to start by thinking about the data. At the outset, users are asked to describe the data that they are interested in capturing to help do their work. This description takes place using a web-based interface that allows data objects to be defined and then included on user-interface screens. In this way, the ability to model data goes beyond the grid of the spreadsheet, but not so far as to be confusing and unnatural like the declaration of an object in Java.

The ability to draw data on a screen is also crucial, especially for the market that MicroPact first addressed, the federal government, which is often focused on capturing data in forms and then processing it. Many businesses see applications in the same way.

Therefore, an entellitrak application starts out as a bunch of data screens that allow people playing different roles to view the data they need to carry out their part of the task.

Then, with that structure in place, policies and procedures are described. Instead of focusing on an end-to-end process, the process is derived from the bottom up by first defining states for a case being handled, then describing the rules for moving from one state to another, and also describing events. The states, rules, and events are described through visual, BPM-like transformations using UML diagram semantics, or code by using Java, Scala, or Groovy. MicroPact is focused on languages that use the Java Virtual Machine.

Look at the House, Not the Blueprints

As with Agile methodologies, the system comes to life gradually. First you have the data, then rules about the data, then some simple processes, and finally more advanced connections to the outside world through code. Gradually the model of the business becomes more complete, adding data objects that describe people, roles, and other entities such as cases or transactions.

“When you buy a house, you usually don’t start by looking at the blueprints—you look at the rooms and think about how you are going to live there,” said Cerniglia. “Starting with BPM diagrams or most other ways of developing applications is like looking inside the walls at the plumbing or the electrical wiring first. We feel that our data first approach matches the way people think about doing work and provides IT professionals with a better picture of the desired user outcome.”

MicroPact has a strong record in serving federal government agencies, which tend to avoid enterprise applications because they have unique requirements. MicroPact’s entellitrak platform is used by more than 200 federal agencies and organizations including every cabinet level department. The company has operated in a low-key manner, but is now focused on gaining traction in the enterprise.

A simple example of the data first approach can be found at the University of Texas, which uses entellitrack to track and manage the disposition of requests for media credentials to athletic events. Once a media organization has signed up for an account, each request is treated as a case that flows through several states on the way to being approved or rejected.

BI tools, spreadsheets, and BPM are all playing a role in helping people do the work that happens in this whitespace. As I have explained here, it is time for people to start using a data first approach to attack it.

Follow Dan Woods on Twitter:
Follow @danwoodscito

Dan Woods is CTO and editor of CITO Research, a publication that seeks to advance the craft of technology leadership. For more stories like this one visit www.CITOResearch.com. Dan has done research in the past for Appian, SAP, and MicroPact and co-wrote Twikis for Dummies with Peter Thoeny.


Great ! Thanks for your subscription !

You will soon receive the first Content Loop Newsletter