In the fifteen years since seventeen software developers met at a Utah resort and hammered out the Manifesto for Agile Software Development, Agile methodologies for building software have become the predominant approach worldwide. Yet, while there is broad agreement that Agile improves the chances of successful software initiatives as compared to the deeply flawed waterfall approach, Agile still faces many challenges, even after so many years.
Fifteen years is a long time in IT, after all. Technology refreshes occur typically once every three years. Given the increasingly rapid pace of change – not only in the software world, but across today’s increasingly software-driven business environment – perhaps we should move on. Is it finally time to kill Agile?
Problems with Agile Continue to Pile Up
Even though the lightweight, iterative roots of Agile go back well into the 1950s, the 2001 creation of the Agile Manifesto (as it’s come to be called) has become a watershed moment for the approach, defining its four core values and a dozen principles for driving the creation of better software.
In and of itself, however, Agile has never been a methodology. Rather, methodologies like Scrum and Extreme Programming rose to the fore, espousing the principles of Agile. Today, Scrum in particular has achieved a remarkable level of adoption.
Nevertheless, the problems and limitations of Scrum and the broader Agile movement have proven surprisingly persistent, in spite of the throngs of very smart people who have worked in various capacities with Agile over the years.
The numerous complaints about Agile include its lack of focus on software architecture, its emphasis on one-off software projects as opposed to building reusable code, and the reinforcement of the notion that the software development team is a self-contained group, as opposed to participants in a broader collaborative effort.
Perhaps the greatest concern over Agile, however, is the ambiguity of Agile’s principles themselves. Agile calls for self-organizing teams, but there remains no clear understanding of how best to self-organize. Agile also calls for the stakeholder or customer to be an active part of the team – but stakeholders have always resisted this participation, and when they do join the Agile team, they struggle with their role.
This ambiguity, in fact, has led to overly dogmatic interpretations of Agile’s values and principles – essentially locking in particular methodologies, contrary to Agile’s exhortation to respond to ongoing change. “For organizations who use Agile well, the watchword is to improve continuously in all aspects, from portfolio management at the top down to development techniques at the bottom,” says Ron Jeffries, one of the original signatories of the Agile Manifesto and a creator of XP.
Continuous improvement is a noble aspiration to be sure, but the devil is in the details. This lack of a strict approach for ongoing improvement leads to issues, according to Mukund Balasubramanian, CTO at digital consultancy Photon. “This has resulted in lack of predictability, having a scalable model, overwork resulting in decreased employee satisfaction and engagement (waiting on a Product Owner to make a decision as an example) and decreased quality of work product.”
Such problems extend well beyond the Agile team itself. “The challenges for Agile practitioners today relate to understanding the role and position of the Agile team in relation to other stakeholders outside the team, like third party service providers, downstream API providers, package vendors, etc.,” explains Justin Arbuckle, Vice President of Worldwide Transformation and Chief Enterprise Architect at Chef Software. “In other words, working with others who do not hold the same beliefs about team effectiveness and product focus.”
If Not Agile, Then What?
As many enterprises still struggle with the transition from waterfall to Agile, any talk of Agile’s shortcomings may suggest that a move back to waterfall is in the offing. Fortunately, few enterprises are foolish enough to make such a move. Clearly, we must move forward, not backwards. What, then, is the proper evolution of Agile?
In many circles, the answer is Lean. “Agile has outlived its usefulness and have moved more than 40% of our development into what we call ‘Lean Development’,” reports Balasubramanian. “It is as much of a radical shift as the transition from waterfall to Agile was.”
Lean development is an outgrowth of the Lean Manufacturing movement that Toyota championed in the 1950s. Applied to software development, Lean focuses on eliminating any activity that doesn’t add value right away, and emphasizes how the team operates as a whole.
Characterizing Lean as what comes after Agile, however, doesn’t accurately represent the relationship between the two movements. In fact, Lean principles helped drive many aspects of Agile. “There was a connection between lean manufacturing and agile software from the beginning in that many of the developers of the various agile methods were influenced by the ideas of lean manufacturing,” according to Martin Fowler, software development thought leader and Chief Scientist at ThoughtWorks.
At development shops like Photon’s, however, there is a clear, intentional move from Agile to Lean. “We call the next big move ‘Lean Development,’ taking a leaf out of the Lean Manufacturing movement where small, co-located fully empowered teams are responsible end to end for the work product,” Balasubramanian explains. “But the process is an automation first approach towards build, test, deploy and requires significant buy in from senior management in terms of being even more change friendly than Agile and accepting failure.”
DevOps: Beyond Agile and Lean
Chef’s Arbuckle, however, cautions against making the Agile vs. Lean discussion into a religious debate. “The religious debates that so many of us love to hate to have – such as whether this or that practice is originally a ‘Lean’ or ‘Agile’ practice, or whether that idea is really part of ‘DevOps’,” Arbuckle says.
Throwing a mention of DevOps into the mix is hardly a way to tone down the rhetoric – but Arbuckle makes an important point. Today, software development best practice requires breaking down the silos of development and quality assurance (something Agile purported to accomplish), as well as IT operations, security, and even the lines of business that drive development in the first place. DevOps is the best word we have for such a cultural shift.
However, nobody is saying that DevOps supplants Agile. If anything, DevOps is ‘Agile on steroids,’ as it ratchets down Agile’s delivery timeline from weeks to goal of ‘continuous delivery.’
That being said, there is much to DevOps that is not part of the Agile movement, and vice-versa. Continuous delivery, in fact, is more of a Lean principle than an Agile one. “Lean Development addresses [Agile] issues by fully empowering a team but limiting the team size, mandatory co-location in a single room and continuous delivery to guarantee quality,” Balasubramanian explains.
It seems, therefore, that the answer to what’s next after Agile is some kind of convergence of Agile, Lean, and DevOps principles. “We will see that their practices become so intertwined in reality as to merge into the search for a single outcome,” Arbuckle says. “High velocity development in teams that continually improve not only the product but themselves too. I refer to the convergence of these outcomes as ALDO: Agile, Lean, DevOps Outcomes.”
Do we have more work to do in order to establish this converged set of practices as the preferred approach for building software? Absolutely. After all, Agile is confusing enough on its own. Mash it together with Lean and DevOps and you have a recipe for chaos.
Choosing not to implement this new set of practices, however, is becoming the increasingly risky choice in today’s rapidly changing, software-driven world. Enterprises in particular are rapidly finding that not making such a move is even riskier than jumping on board the Agile/Lean/DevOps train. One thing’s for sure: we’re in for a bumpy ride.
Intellyx advises companies on their digital transformation initiatives and helps vendors communicate their agility stories. As of the time of writing, Chef Software and Photon are Intellyx customers. None of the other organizations mentioned in this article are Intellyx customers.
This article was written by Jason Bloomberg from Forbes and was legally licensed through the NewsCred publisher network.