Inverse Virtualization for Internet Scale Applications

The word virtualization is one of those words that can mean a lot of things, sorta like cloud.

When people think of server virtualization today they often think of it in terms of taking a physical server and having it host many virtual servers each with their own operating system and application (One to Many).  Many applications running on one physical machine.  This model is enabled by virtualization software (e.g. VMware, KVM, Hyper-V), and perpetuates the classic application development model of One Application to One Server into a much more efficient infrastructure comprised of fewer physical servers.  One Server to Many Applications.  As a nice side effect, the virtualization software also enabled the classic application model to gain high availability and agility often missing before.  I’ll refer to this as Classic Virtualization.

Therefore, server virtualization is a slam dunk, right?  Who in their right mind would not do that? Well, to the shock and surprise of many, Facebook recently said they where not using “server virtualization”, and instead planned on buying more servers, not less <GASP!>.  This made headlines at Computer World today as Facebook nixes virtualization.

The shock and surprise, I believe, comes from the fact that Facebook (being an internet scale application provider) has an entirely different application development model from the typical IT organization that most people don’t understand.  Facebook, like Yahoo, Google, Apple iTunes, Twitter, etc. is providing just one or only a few applications that must scale to a global demand.  This is very different from the typical IT data center supporting hundreds of applications for a much smaller corporate community.  Rather than the classic IT application model of One Application to One Server (for each of the hundred apps), the large internet providers like Facebook have a development model of One Application to Many Servers.

The fact is, Facebook has implemented server virtualization at the application level.  Many Servers to One Application.  This is another form of server virtualization common in the massively scalable internet data center.  Each cheap little low-power server is just an insignificant worker node in the grand scheme of things.  I’ll refer to this as Inverse Virtualization.

Another interesting observation here is the paradigm shift with storage from the typical IT data center.  In the classic IT virtualization model, operating system and application data from each of the hunderds of virtualized servers is encapsulated and stored on a big central pool of storage with hardware from EMC, NetApp, Hitachi, etc.  With the internet scale data center this is completely different.  Remember each insignificant server is a worker node providing a small slice of client processing for one big application.  The same can be true for storage.  Each insignificant server also has a cheap desktop class hard drive that provides a small slice of storage to one massive logical pool of distributed storage.  See for example OpenStack Swift, or Hadoop Distributed File System (HDFS), Google File System (GFS).

Why cheap low power servers for “Inverse Virtualization”?  As Facebook points out, a bigger server doesn’t necessarily provide a faster experience for the consumer. A bigger server could process more concurrent consumers on one physical machine, however should that server fail, or the rack it’s located in, that’s a much larger swath of affected users. Keeping service levels consistent despite all types of failures (occurring on a daily or hourly basis in such large data centers) is very important to the application provider. Moreover, a bigger server doesn’t provide a better bang for the added equipment and power buck. This is not something unique to Facebook. Other internet application providers are coming to the same conclusion, such as Microsoft.

Microsoft presented the following data at the Linely Data Center conference earlier this year, which shows that lower power Intel processors provide better performance when considering system cost and power requirements:

The natural conclusion from this is that with Inverse Virtualization it makes sense to build out large scale-out farms of insignificant low power servers to achieve better overall performance with lower power requirements and lower cost.  Furthermore, less risk is placed on each server to better absorb failures at the server or rack level, while keeping the overall user experience as consistent as possible.

Facebook didn’t really “nix” server virtualization.  I beg to differ.  Facebook, like many of the other internet scale application and cloud service providers have been implementing virtualization from the get go, Inverse Virtualization.  A virtualization model that arguably results in a more efficient scale out architecture of low power servers – compared to traditional IT server virtualization – if you consider the bigger picture of Performance vs. Cost vs. Power.



  1. says

    So Facebook finally learned a lesson that Google learned long ago.

    I would submit that the only reason virtualization is at the level it’s at right now is that applications will not scale. For years and years, developers coded server applications to run on an operating system that keeps them from behaving in the way Facebook, Google, and other widely distributed, very resilient applications behave. Therefore, we must virtualize just to maximize resources in this old model.

    Once we get rid of that OS layer, and developers begin coding applications that are more scalable, and (sorry for this) cloud aware, we will see more and more companies taking this route.

    Nice write-up Brad.

  2. says

    Excellent explanation, but I have a short question :)

    How is Inverse Virtualization different from Load Balancing (just as example)?
    Let’s take this simple scenario. Web server application, one LB and multiple servers in the back of LB to assure a load distribution. In both LB and Inverse Virtualization you have to use multiple machines in backend. This machines can be concentrate in a datacenter or split around multiple ones.
    I’m sure that something is slipping to me in this explanation, but I don’t know what.
    I’m not an expert in virtualization, so sorry if my question sounds silly.


    • says

      An internet facing application (e.g. Google search, Facebook) is going to have multiple components/tiers collaborating to form the functional application, with traffic flowing from one component to the other. One of those components is the Web front end (the user interface). In the Web front end, absolutely, you will have Load Balancers that spray web requests across thousands of web servers. In this component you can view the Load Balancers as key to enabling the Inverse Virtualization model for the Web front end component.

      Other components of the application might provide more back end processing work to fulfill requests that were not found in cache from the previous Web layer, such as a really obscure search query you might challenge Google with. This type of workload also uses the Inverse Virtualization model built into the application code — such as using MapReduce computations across thousands of nodes to build your search result. Load Balancers in the traditional sense are not really involved here, the MapReduce framework inherently load balances across the worker nodes.


  3. joe smith says

    Brad, nice write-up, but I’m not too sure about what is so revolutionary or innovative in this so-called “Inverse Virtualization” model. Organizations have been spreading out the components of applications (web, database and application) over commoditized clustered servers for years and tying them together with either Ethernet or Infiniband.

    What am I missing?

    • says

      Sure, having a server dedicated to one application among a farm of other servers is nothing new. What is different here is the volume of traffic and unique applications of these massive internet data centers that lend itself to distributing small chunks of processing across a wide array of low power servers. Rather than scaling up the CPU cores, it makes more sense to scale out the cores. This is a foreign concept to the typical IT data center. Hence, “Inverse Virtualization”, it may not look like virtualization the way everyone else does it, but it is, just implemented differently.

      • Carlos Pereira says

        The storage access model being used by these MSDC-based distributed applications is another big point of difference. The concept of “Big Data” (when data becomes large enough that cannot be processed via conventional methods) “Analytics” applies here. SMAQ stack, for instance (cross referring the MapReduce example used by Brad).

        • Drew Pletcher says

          Carlos, yes and no on the storage model. You have to look at the application implementation and associated data relationship. There are a number of implementations for the storage constructs in play, while there is the analytics piece using hadoop/MapReduce, “big data”, scribe, etc. There is also a primary scale out model that use multi-tiered federate storage based upon the Hadoop Distributed File System (HDFS). Given that HDFS is used and understanding that you can overlay intelligent storage use algorithms on top of HDFS for movement of the data to various tiers – hot, warm, cold, archival, etc.

          The mindset is one that allows these federated/distributed file systems to be used for multiple storage clusters to provide a foundation for multiple implementations and applications.

  4. says

    I’ve been saying for a while that there is very little for an Enterprise company to learn from Yahoo, Google, Facebook etc because their systems are so radically different from Enterprise IT. To a lesser extent, this applies to different companies e.g manufacturing has somewhat different needs to an online company.

    I really don’t take time to read about the big companies technology because it has little relevance to the ‘real world’.

    Good writing.


  5. Jamie Sinclair says


    So Facebook, a good candidate for UCS… fill the chassis with lots of lower power blades and benefit from the scale-out architecture of the fabric interconnects.

    nice article..


    • Chris J Cooper says

      Not really. They don’t need the physical densities, and additional costs realized in a complex, expensive blade chassis. They just load up cheap racks with boatloads of cheap, skinless servers, with almost no redundant or often even replaceable components. Google used to literally screw old server boards onto plywood, and wouldn’t bother replacing dead machines until enough had died in an environment to bother sending someone out with a stack of replacements (just go in with a cart and swap out all the dead ones in a server room in one trip). The fabric interconnects are also very expensive when you break it down per port, and most of them still have scalability limits, and are ultimately not necessary anyways. A few copper gigabit ports are good enough, and still cost considerably less, and use far less power than a fiber connection. Infiniband is actually good for these environments as well, when they need the extra bandwidth…

  6. Yamin says

    Hi Brad,

    Nice write up here, may I know how to achieved the many to one kind of virtualization ?
    What component is required to do this ?


Leave a Reply

Your email address will not be published. Required fields are marked *