Distributed computing is the future of technology, but you’d never know it from how many tech organizations continue to operate. While our systems run across multiple servers and even data centers, our development teams too often sit within the same office. While there are real benefits to a co-located development team, the costs of competing for local talent may well outweigh them.
Ironically, open source has taught us how to effectively work in a distributed fashion, but far too few organizations have heeded its lessons.
The Costs Of Homogeneity
Facebook is the world’s largest open source company, as I wrote recently. If any company groks open source, it’s Facebook.
And yet Facebook has a very 20th century approach to development. When my company was acqui-hired by Facebook a few years back, Facebook had little interest in any engineer that wasn’t located within Silicon Valley. Move to the Valley or move on, was the mantra.
There are good reasons for this mindset. Studies have found that co-located engineering teams can be more productive, with less time needed to coordinate resources, make decisions, etc. Embedded in that decision, however, is a tradeoff, which Jeff Waugh uncovers in reference to why companies elect the co-located or distributed development approach:
@mjasay They want the highest potential bandwidth between people, or they want the greatest potential range of people. Flip for foolish.
— Jeff Waugh (@jdub) June 23, 2014
Not only does distributed development yield a greater range of people, but it also provides access to talent at far lower cost. This is a big deal given the escalating cost of recruiting and hiring engineers. Seen the price of an engineer in Silicon Valley these days?
In an email conversation, former MySQL and ZenDesk executive Zack Urlocker pushed the advantages of distributed development. Not only is it “much easier to hire when you have distributed teams,” he told me, but it turns remote locations into real offices and “not just sales outposts,” which is good for both productivity and morale beyond the engineering team.
But set aside the costs for a moment. Can distributed development actually work?
What Open Source Teaches Us
For those in the open source world, this is almost a silly question. Some of the world’s greatest software, from Linux to Hadoop, gets written by disparate engineers huddled over their laptops in far-flung locations. By its very nature, open-source development favors online collaboration on code, with all communication happening on mailing lists, chat channels and more.
In the open source world, you can collaborate with your colleague sitting next to you, but any relevant communication needs to happen online, or the model falls apart. If anyone is privileged by sitting next to their peers in the office, the model breaks.
GitHub can help. Increasingly, development happens on GitHub, brilliantly explained in non-geek speak by Lauren Orsini. GitHub works so well because it allows developers to work on their own time, forking and merging code asynchronously, either collaboratively or competitively.
Your Mileage May Vary
Of course, the subtext here is that distributed development works better when the software architecture allows it, not to mention company culture. As Charles Wise posits, “Tightly integrated apps are much tougher” to pull off than systems that involve modular architectures. Given the increasing prevalence of open-source code emerging from tech titans like Facebook and Google, perhaps the default should be modular architectures that are a natural fit for distributed development?
Even with tightly integrated systems, companies have shown that it can work, and well. Marten Mickos, a veteran of MySQL, which had a very distributed workforce, and current CEO of open-source cloud company, Eucalyptus, emailed me some thoughts on how Eucalyptus works:
We have software developers in China, Bangladesh, India and all over the US. A third are located in Santa Barbara where we have an office. The rest work from their homes or collaborative workspaces.
Everything we do is online. We use Jira, Confluence, Github, wiki pages, email, IRC and so on.
Our developers have small private (and personal) clouds in order to be more productive.
We do regular online meetings for specific teams, specific features and specific releases. We follow an agile model, shipping major feature releases twice a year and maintenance releases every second month.
A lot of discussions happen on email. We have an alias for fun@ and one for life@ and discuss@ and biz.intel@. On these lists, people can exchange information and debate various topics. And if they are busy, they can of course ignore those emails.
Because of our model and our commitment to openness, we have been able to recruit some of the absolutely best engineers in the field of distributed systems. The fact that they don’t have to move to wherever the HQ are is a huge plus from them.
This has been my experience as well, having worked at highly distributed teams at Canonical and Nodeable and more tightly coupled teams at Alfresco and MongoDB. Both work, though in different ways.
What Works For You
Ultimately, though, whether distributed development can work for your company comes down to—surprise!—the particular nuances of your company, as Twitter’s open source guru, Chris Aniszczyk suggests:
@mjasay if you have a culture that values remote work… the right tools that enable async/recorded/searchable communication… remote wins
— Chris Aniszczyk (@cra) June 23, 2014
@mjasay co-located wins when you have different levels of non-async communication… along with lack of empathy/culture for remote work
— Chris Aniszczyk (@cra) June 23, 2014
All true. But my modest proposal is that many companies haven’t even tried to experimented with distributed development, and they should. As Urlocker insists, co-located development is “good but it’s limiting. Over time there is tremendous advantage to having multiple Dev teams in multiple cities.”
Better, more diverse talent at less cost. That sounds like a winning formula.