Distributed systems trickle down into Enterprise IT

In my new role at Cisco I’ve had the opportunity to observe and study something happening that is (in my opinion) truly significant and mind blowing.  The IT data center landscape as we know it is on the precipice of a major upheaval.  I’m not talking about virtualization, and cloud, and all that other stuff that’s been obvious in enterprise IT for the last five years..

I’m talking about the trickle down effect of distributed systems into the enterprise IT data center.

Distributed computing is not something new.  The ideas and implementation of these concepts date back to the 1960’s.  It’s how distributed systems have evolved only recently (in the internet era) that is interesting and profound.  Starting with the publication of Google’s GFS paper in 2003, followed by Google’s Map Reduce in 2004.

In the early 2000’s large internet properties such as Google and Yahoo! were faced with problems nobody had ever dealt with before.  They had to scale their application to a population of millions, store and analyze tremendous amounts of data, all while providing a consistently responsive and quality user experience.  Google’s publication of GFS and Map Reduce began the conversation of how to solve these very unique problems using an intelligent software infrastructure managing a warehouse sized distributed system of standard low cost x86 rack servers with local disk.  No big expensive SAN.  Instead, there is a software layer between the server hardware and application that pools all of the compute, network, and storage into one abstracted logical resource.

During this time the enterprise IT data center was not faced with any of Google’s problems, nor was there any anticipation it ever would.  Google published their papers and nobody outside of academia and other growing internet properties took notice.  Understandably so, because scaling an enterprise IT app to millions of users and petabytes of data would be considered quite unusual (even still today).  Rather, enterprise IT was focused on problems of complex management and inefficiency.  Life went on and enterprise IT continued down a path of server virtualization and private cloud, implementing blade server technologies and catapulting the rise of new stars such as VMware.

Meanwhile, the problems solved by Google in scaling internet applications brought rise to new internet properties such as Facebook, Amazon Web Services, and many others.  Each one taking Google’s original ideas and improving upon or customizing them for their own applications needs and publishing additional papers.  Amazon published Dynamo in 2007, describing a distributed system which provides a massive amount of object storage powering their S3 offering.  In 2008, Facebook open sources Cassandra, a highly scalable transaction oriented distributed storage system powering parts of their social media application (chats and messages).  Yahoo! engineers develop and open source Hadoop, a distributed system which provides analytics over large data sets.  These are just a few examples of many.

It would appear the two worlds of Enterprise IT and Web didn’t have much in common.  One was trying to solve the scale and cost problems associated with massive amounts of users and data for their one or two apps.  The other trying to solve the problems of infrastructure inefficiency and complex management for their numerous but relatively smaller scale apps.  And, naturally, each taking two different approaches to their problems.  Enterprises sought after infrastructure consolidation and virtualization solutions provided to them by capable vendors, while the web application providers had no choice but solve their problems on their own, using self developed infrastructure software with an army of in-house software engineers.  Two parallel worlds of data center IT.  Neither world having any influence or effect on the other.

This is the point at which most people understand the industry, as I did, until I started paying closer attention.

As properties such as Yahoo!, Google, Facebook, Amazon became great successes, their architects and software engineers realized that they had moved mountains.  Accomplishing the unthinkable.  The tremendous problems of efficiently running large scale applications on low cost infrastructure had been solved.  Publishing papers about your work was the perfect way to claim well deserved credit and establish name recognition in the industry.  At the very same time, enterprise IT begins to encounter some of the very same problems solved by the large web provider, such as scalable data warehousing and analytics (so called “Big Data”).  Additionally, the software driven distributed systems that solve problems of infrastructure efficiency and management at very large scale could also be applied to infrastructure at a smaller enterprise IT scale (why not?).  And finally, the cost savings of an application infrastructure designed to operate on low cost commodity hardware can be realized at any scale, large web or enterprise IT.

Problem: The average enterprise IT shop doesn’t have the same army of in-house software engineers to stand these systems up to be production ready with any kind of speed or operational efficiency.

Solution: A new business opportunity has presented itself. New problems are arising in enterprise IT that have already been solved by the large web properties.  And (potentially) a new way for enterprise IT to solve the same old problems with lower infrastructure costs.  Why not take these very smart distributed software engineers and put them into start-up companies with the mission of delivering distributed systems  for commercial consumption?

Here are just a few examples of start-ups targeting the “Big Data” space:

Cloudera & Hortonworks – made up of former Facebook and Yahoo! engineers each provide optimal packaging, training, support, and consulting services for Hadoop.

Datastax – provides packaging, consulting services, and training for Hadoop and Cassandra.

MapR – provides optimal packaging and support for Hadoop.

You can begin to see how distributed systems technology originally developed by Google, Yahoo!, Facebook, etc. will begin to trickle down into pockets of the enterprise IT data center with the help of these start-up companies.  In fact, this is already happening.  Enterprises are staring to deploy “Clusters” of rack mount servers and network gear for sole purpose of “Big Data” analytics and data warehousing.  These big data pods might snap-in to the rest of the general purpose infrastructure with a clean Layer 3 hand-off.  Once these big data clusters are in place, the enterprise has a chance to gain familiarity and expertise in the general framework of a distributed computing architecture, opening the door for other existing application environments to begin leveraging this technology.

Today, Big Data is a relatively new problem for enterprise IT and therefore tends to be deployed as a new application environment.  The existing server virtualization infrastructure (VMware, NetApp, EMC, etc.) supporting all of the other traditional applications remains untouched and unchanged.  For now.

Even more interesting is that we are beginning to see distributed systems and open source software based technologies enter the traditional general purpose server virtualization and private cloud environment.  Consider that a server virtualization deployment often requires a large centrialized storage system, often provided by a storage vendor like EMC or NetApp. Can the very same technology that drives “Big Data” also be used to provide the storage infrastructure for server virtualization? Why not?  That’s what the folks at Nutanix set out to accomplish, and they appear to have been successful.  Their solution which claims to eliminate the need for a SAN or NAS for server virtualization has been made generally available.

With distributed computing one thing remains certain: You need servers, you need a network, and you need software to manage the infrastructure.  Most importantly though, to be relevant in this space as a customer or a vendor, you need to understand the application.

I’m spending a lot of time studying the application framework and operational model of things like Hadoop, Cassandra, and OpenStack, and how that translates into current and future software and hardware infrastructure considerations, as well as possible turn-key solutions moving forward.  If these technologies gain a real foothold and you don’t understand how these applications work, you run the risk of loosing relevance and credibility whether you’re an engineer in the enterprise IT data center or a vendor trying to sell switches and servers.  You can run these apps as virtual machines on your desktop, or in a lab with a handful of rack mount servers and a switch.  All of the documentation to learn is out there and readily available.

Here are some places to start: Cloudera training, certification, and videosApache Hadoop DocumentationOpenStack documentation

Stay tuned here for more information and discussion on this topic.



Disclaimer: The author is an employee of Cisco Systems, Inc. However, the views and opinions expressed by the author do not necessarily represent those of Cisco Systems, Inc. The author is not an official media spokesperson for Cisco Systems, Inc.


  1. Petr Lapukhov says


    Hey, I didn’t see you mentioning Cosmos, Dryad and SCOPE! 😉 Oh well, maybe it’s just not so well known outside of Microsoft.

    Next, to the point of your blog post. The computation model you are talking about is known as SIMD (single-instruction, multiple-date) and I believe have been firstly extensively used on NUMA supercomputers. However, it only fits well the applications supporting *data parallelism*, and this is only a subset of all possible applications. A lot of data mining apps fit this model, of course, but building an expensive cluster just for that purpose is something that only a company, which specializes on that single function can afford.

    Secondly, even if an existing application potentially supports data parallelism, re-writing it to fit the distributed model may require tremendous efforts. Might worth it in some cases, but for the most part all people care about is just to make better use of their hardware by means of statistical multiplexing.

    Thirdly, you made a great example with distributed emulated SAN. Distributed file systems have been in existence for quite a while, and as usual they have their limitation. For SIMD commodity clusters, one prominent feature is the fact that applications are assumed to be robust in presence of multiple failures (hdds, machines, etc), which in turn requires special application design (normally handled by job scheduler in M/R, Cosmos, etc). Personally, I see failures in every SCOPE job I run! Exposing block-level access to the duplicate data chunks spread across multiple machines connected by potentially congested network may not provide the level of expectation that a traditional application has. So it’s hard to say whether this model will be more predictable over a traditional SAN.

    Aside from that, I would say that I absolutely love massively parallel computations on commodity clusters. I’ve been more or less involved with that since late 90s (using PVM and MPI mostly then, though) and now kind of getting back into this again (I hope). So I totally share your enthusiasm and hope to see more applications coming up in the future :)

    • says

      Hey Petr! Long time no see.

      I was not aware of Dryad and Cosmos (until now), which is telling because that shows these are not coming up in any discussions or study on this topic. Dryad and Cosmos appear to be based on Windows Server which explains the lack of presence because the systems at the big web properties now trickling down into enterprise IT are all based on Linux.

      As for the applications, you are going to have the old existing applications and some new applications. Big Data is a relatively new problem for enterprise IT and therefore this will be a new application either coded from scratch for Map Reduce or more likely the app will be provided by a vendor playing in the Big Data space.
      For the old existing applications (running on a virtual machine), you wont need to touch that application at all for a distributed system to transparently provide the storage volumes to those virtual machines, or virtual desktops. Perfect example of this is Nutanix.

      Nobody is going to re-code all the old apps for a distributed computing model. But that doesn’t mean the app itself won’t be leveraging or riding on top of an infrastructure powered by a distributed system.


      • Petr Lapukhov says

        Brad, I believe you missed my argument about Nutanix 😉 My point is that emulating block I/O over a commodity cluster could be very challenging task. The commodity cluster is designed as an unreliable system, where probability of failure scales linearly with the cluster size. If you look (I believe you already did) into GFS/Cosmos/HDFS architecture, you will notice that they don’t provide true POSIX file-system interface (e.g. vnode like) but rather a special API that is to be handled appropriately by an application. Having unpredictable latency when accessing your “distributed” HDD could be a serious problem, that is not easy to solve: we just went through the battle of running SCSI over a specially designed network :)

        • says

          Correct, off the shelf distributed file systems available today such as HDFS are not going to provide block I/O to your virtual machines. They weren’t designed for that. Block I/O from a DFS is a hard problem to solve, I agree, as were all the other problems that have been solved up to this point. At the foundation what you have is a software driven architecture that can move mountains with the only limitation being your imagination and skill. It appears the folks at Nutanix have figured out how to minimize block I/O from traversing the network, keeping things local to the server as much as possible, while still providing the illusion of centralized storage to the VM environment.

  2. says

    Very cool stuff Brad. So many of the technologies between the analytics/Big Data space and enterprise/cloud are colliding. OpenFlow was started 3 years ago to help solve some of the scalability issues of large Hadoop clusters and now a hot topic for future enterprise networks. Analytics requires lots of compute and cloud solutions have the resources. Very exciting times – thanks for sharing.

    • says

      Hey Stuart! Thanks for stopping by my friend.

      I’ll disagree with you that OpenFlow was designed to solved scalability issues, but if you are also saying that OpenFlow may have a valuable role to play in these systems, that I definitely agree with.


  3. Joe Smith says


    There is a foundational point that is being made that I find a bit confusing – albeit, I have not read the white papers presented by Google, et al, yet. If my understanding is correct, these massive internet players do NOT virtualize their application servers. They are not concerned with server consolidation. Instead, it seems that these distributed applications are custom-written to be executed on large physical server clusters, not VMs. Why is that? Are the two mutually exclusive?

    Also, can you elaborate a bit more on the block I/O component? What is the key point to understand with regard to block I/O and these distributed systems?

    Thanks you

    • says

      The massive data and user population that Google was faced with resulted in servers with very high cpu and storage utilization. Server consolidation was a non-factor. The factors instead were related to scaling the data and user population while keeping costs low and performance high. The primary assumption with distributed systems is that one machine is not enough to run the application. Therefore virtualizing that machine with a hypervisor is pointless. Rather, you add machines to your cluster as needed to meet the scale and performance requirements.

      While the Enterprise IT data center may not be faced with millions of users, the problem of exponential data growth and the cost of that is very, very real. These are problems that have already been solved by Google and others, so why re-invent the wheel? The influx of distributed systems into Enterprise IT will be especially disruptive to the storage industry as we know it today, IMHO.

      Case in point, Nutanix. If you are going to use a distributed system to provide storage for virtual machines, you can’t just through the VM’s virtual disks on an HDFS cluster and be done with it. The hypervisor doesn’t know how to access HDFS, and HDFS doesn’t have the locking capabilities required and all of the other file system semantics expected in this environment. So the folks at Nutanix built software that allows this to work by showing the hypervisor a standard iSCSI environment on top of what is actually a scale-out distributed storage cluster comprised of the very same machines hosting the VMs. That’s an example of another fundamental design goal of distributed systems, that the data and the application should be on the same system whenever possible, to get the best possible performance.


Leave a Reply

Your email address will not be published. Required fields are marked *