Pied Piper and the Distributed Internet
During the long hiatus of Game of Thrones, the hottest TV show among the tech crowd is HBO’s Silicon Valley. This brilliant send-up of the tech industry follows the rollercoaster fortunes of a Valley startup named “Pied Piper” as it repeatedly pivots from one Next Big Thing to the next. The brilliance of the show (apart from its spot-on caricatures of every known denizen of the tech universe) lies in the fact that these Next Big Things always turn out to be not only plausible, but downright compelling. Pied Piper’s clueless but visionary CEO is determined that his company is not going to merely become a billion dollar “unicorn”, but that it would make the world better along the way.
This season’s Next Big Thing was particularly interesting: Pied Piper’s plan was to create the “New Internet”. Instead of sending all our data to one of a few service providers or “cloud services” where it can be controlled, organized, throttled and often inspected by our Network Overlords, the New Internet would be peer-to-peer. Pied Piper’s plan was to use the vast spare capacity of the world’s cell phones and other devices to build an ocean of data, controlled by no one, as a common resource for all humankind. If you and I are sitting in the same room and I want to send you, say, an email message, no longer would that message have to travel to some distant city just to get from my machine to yours—our devices would simply communicate directly, with no giant all-powerful corporation as part of the story.
Fantasy? Well, not really. The irony is that this “new” model is exactly how the Internet was intended to work all along. Folklore tells us that the ARPAnet (the prototype for the Internet) came into existence because the US government was worried about its ability to communicate in the wake of a nuclear war, when all centralized infrastructure could be assumed to have been destroyed. If “Ma Bell” and her centralized trunk telephone lines were gone, how would we ever pull the nation back together? They sent the problem to the best technical minds of the era. The answer that came back was a radically distributed ocean of computers—a network with no “center” and with no one in charge. The basic idea of the ARPAnet was that countless tiny packets of data would find their own way to their destinations, “routing around failures”, and taking advantage of any available path, no matter how obscure. Pied Piper’s Next Big Thing turns out to be a Very Old Thing indeed.
Not only is it a Very Old Thing, it is also a Very Good Thing. The architecture of the ARPAnet was so well-conceived and so properly engineered that almost 40 years later, these basic ideas are still there, doing their job without anyone giving them a thought. The Internet’s basic architecture is one of the great engineering achievements of all time. The thing is this, though: We no longer are taking full advantage of what the Net can do. Most of this self-healing democratic peer-to-peer awesomeness goes unused as we force all of our data through a small number of immense, centralized worldwide data channels, and store it in a relatively tiny number of “data centers”, each built on an epic scale. These facilities are “distributed” in a fashion (they have to be or they wouldn’t work at all), but not in the way that the creators of the Internet intended. Rather than emerging from the chaos of trillions of independent, democratically-determined decisions, the patterns of today’s Internet are the result of highly-orchestrated, centralized decisions made by a few techno-bureaucrats, ultimately guided by the interests of a small number of corporations.
Why did things evolve the way they did? The answers are not technical, they are economic. All technologies are dependent on funding, and long-term funding is always dependent upon profitability. It is not that distributed computing cannot be profitable, it is just that the path to profitability was a bit less obvious in that direction than in the direction of centralization. We already knew how to operate and profit from large, centralized enterprises, so it was only natural to pattern modern computing in the image of General Motors and WalMart. This approach has gotten us a long way: It has put a supercomputer on every lap and a Star Trek communicator in every pocket. But it will not get us much further. The so-called Internet of Things (IoT) is a whole new ball game. First of all, the sheer numbers of devices is vastly greater. For every cell phone, there will soon be a thousand lesser devices controlling every aspect of the built world. More importantly, the negative consequences of centralization become much more clear in such a world. Do we REALLY want every light switch in our living rooms (or lifesaving devices in our emergency rooms) to be dependent on some distant data center? Do we REALLY want to be forced to push every proprietary detail of the operations of our factories to some third-party “cloud” service whose data integrity and security practices are out of our control? The degree of centralization implied by such a world just doesn’t make sense, and it will not happen. (For example, home security/automation provider Nest had its entire ecosystem go off-line due to some technical glitch, totally interrupting all non-physical interaction with all of its smart locks, cameras, etc).
Today, a hundred real-life Pied Pipers are working on a new generation of edge-computing devices, making use of peer-to-peer mesh networking, local computing, and distributed storage. They are swimming upstream at the moment—the development environments, management tools and user expectations all need to be reworked to pull this off. But the center will not hold. The long term value proposition is clear and the trend is inevitable. We will follow these pipers into a new world that will work very differently from the one we are used to—much more local, much more robust, much more secure, and ultimately much more democratic. It is not that “cloud computing” will go away. For many purposes it makes perfect sense. Buying computing resources as needed from Amazon adds agility, scalability and robustness to software startups. Cloud-based visualization and monitoring systems that “roll up” data across the enterprise is by definition a centralized function. But a great deal of functionality that is now being done in a centralized manner does not belong there. The “New Internet” really is coming, and it is going to be very, very distributed.
Peter Lucas is board chair of Interstacks, Inc, and co-author of the book Trillions: Thriving in the Emerging Information Ecology.