Inverse Virtualization for Internet Scale Applications

The word virtualization is one of those words that can mean a lot of things, sorta like cloud.

When people think of server virtualization today they often think of it in terms of taking a physical server and having it host many virtual servers each with their own operating system and application (One to Many).  Many applications running on one physical machine.  This model is enabled by virtualization software (e.g. VMware, KVM, Hyper-V), and perpetuates the classic application development model of One Application to One Server into a much more efficient infrastructure comprised of fewer physical servers.  One Server to Many Applications.  As a nice side effect, the virtualization software also enabled the classic application model to gain high availability and agility often missing before.  I’ll refer to this as Classic Virtualization.

Therefore, server virtualization is a slam dunk, right?  Who in their right mind would not do that? Well, to the shock and surprise of many, Facebook recently said they where not using “server virtualization”, and instead planned on buying more servers, not less <GASP!>.  This made headlines at Computer World today as Facebook nixes virtualization.

The shock and surprise, I believe, comes from the fact that Facebook (being an internet scale application provider) has an entirely different application development model from the typical IT organization that most people don’t understand.  Facebook, like Yahoo, Google, Apple iTunes, Twitter, etc. is providing just one or only a few applications that must scale to a global demand.  This is very different from the typical IT data center supporting hundreds of applications for a much smaller corporate community.  Rather than the classic IT application model of One Application to One Server (for each of the hundred apps), the large internet providers like Facebook have a development model of One Application to Many Servers.

The fact is, Facebook has implemented server virtualization at the application level.  Many Servers to One Application.  This is another form of server virtualization common in the massively scalable internet data center.  Each cheap little low-power server is just an insignificant worker node in the grand scheme of things.  I’ll refer to this as Inverse Virtualization.

Another interesting observation here is the paradigm shift with storage from the typical IT data center.  In the classic IT virtualization model, operating system and application data from each of the hunderds of virtualized servers is encapsulated and stored on a big central pool of storage with hardware from EMC, NetApp, Hitachi, etc.  With the internet scale data center this is completely different.  Remember each insignificant server is a worker node providing a small slice of client processing for one big application.  The same can be true for storage.  Each insignificant server also has a cheap desktop class hard drive that provides a small slice of storage to one massive logical pool of distributed storage.  See for example OpenStack Swift, or Hadoop Distributed File System (HDFS), Google File System (GFS).

Why cheap low power servers for “Inverse Virtualization”?  As Facebook points out, a bigger server doesn’t necessarily provide a faster experience for the consumer. A bigger server could process more concurrent consumers on one physical machine, however should that server fail, or the rack it’s located in, that’s a much larger swath of affected users. Keeping service levels consistent despite all types of failures (occurring on a daily or hourly basis in such large data centers) is very important to the application provider. Moreover, a bigger server doesn’t provide a better bang for the added equipment and power buck. This is not something unique to Facebook. Other internet application providers are coming to the same conclusion, such as Microsoft.

Microsoft presented the following data at the Linely Data Center conference earlier this year, which shows that lower power Intel processors provide better performance when considering system cost and power requirements:

The natural conclusion from this is that with Inverse Virtualization it makes sense to build out large scale-out farms of insignificant low power servers to achieve better overall performance with lower power requirements and lower cost.  Furthermore, less risk is placed on each server to better absorb failures at the server or rack level, while keeping the overall user experience as consistent as possible.

Facebook didn’t really “nix” server virtualization.  I beg to differ.  Facebook, like many of the other internet scale application and cloud service providers have been implementing virtualization from the get go, Inverse Virtualization.  A virtualization model that arguably results in a more efficient scale out architecture of low power servers – compared to traditional IT server virtualization – if you consider the bigger picture of Performance vs. Cost vs. Power.


Cisco UCS Networking videos (in HD), Updated & Improved!

One of my most popular posts ever is perhaps Cisco UCS Networking Best Practices (in HD) posted last June (2010).  So what do you do with a good thing?  You figure out how to make it even better, right? Of course!

On that note I am thrilled to present a new and improved 12 part video series covering Cisco UCS Networking!  This series obsoletes the prior set with new, updated, and re-recorded content inspired from new developments in UCS Manager since the last series.  Much of this content I created for and presented at Cisco Live Europe 2011 (London) for the session BRKCOM-2003 (UCS Networking 201 – Deep Dive) on February 4, 2011.  Thanks to those that attended!  It was a fun session and a great audience.

This content and video series is not really a “Deep Dive” in the true technical sense.  Rather, this content is intended to be more of an Intermediate technical level geared towards the Data Center Architect, Network Designer, or IT Manager, to aid in understanding the overall architecture, best practices, and system level capabilities Cisco UCS brings to the data center.


Part 1 – The Physical Architecture of UCS

In this video we take a look at the physical network architecture of Cisco UCS and incorporate the new capability of connecting both blade and rack mount servers to UCS Manager.

Part 2 – Infrastructure Virtualization & Logical Architecture

Here we look at how Cisco UCS virtualizes every significant component of the physical architecture; switches, cables, adapters, and servers. Then we look at how that virtualization creates a more simplified logical architecture transposed from the physical architecture.

Part 3 – Switching Modes of the Fabric Interconnect

In this video the unique behavior and advantages of End Host mode are discussed and compared and contrasted to Switch Mode, and a traditional Layer 2 switch.

Part 4 – Upstream Connectivity for SAN

Here we take a look at the different ways to integrate Cisco UCS to the data center SAN with FC End Host mode, and the connecting storage directly to UCS with the new FC Switch Mode.

Part 5 – Appliance Ports and NAS direct attach

In this video we discuss the new Appliance Port and how it can be implemented for connecting NAS or iSCSI directly to the UCS Fabric Interconnect.

Part 6a – Fabric Failover

The unique Fabric Failover capability is explained and its “Slam Dunk” uses are shown such as with Hyper-V, and bare metal OS installations.

Part 6b – Fabric Failover (cont)

We continue with discussing the potential of using Fabric Failover with VMware software switches and VM FEX.  The best practice design for integrating Nexus 1000V with Cisco UCS is also briefly discussed.

Part 7a – End Host mode Pinning

Here we take a look at the dynamic and static pinning behavior of End Host mode, and how load balancing works.

Part 7b – Upstream LAN connectivity

In this video we look at the different ways to uplink UCS to the upstream network, how failure scenarios are handled, and comparing individual uplinks vs. port channel upliks vs. vPC uplinks.

Part 8 – Inter-Fabric Traffic and Recommended Topologies

This video examines different examples of inter fabric traffic and the recommended topologies for linking UCS to the upstream LAN network that provide the best handling of all traffic flows.

Part 9 – Connecting UCS to Disjointed L2 Domains

Here we discuss the problems you can run into when connecting UCS to separate Layer 2 networks, and ways to make it work.

Part 10 – Gen2 Adapters

This is brief video covering the new Gen2 adapters from Emulex, Qlogic, Broadcom, and Intel. The Cisco VIC (Palo) adapter is also discussed and with it’s unique VM-FEX integration with VMware vSphere.

Part 11 – Cisco VIC QoS

In this video we take a deeper look at the advanced QoS capabilities of the Cisco VIC, and how that can be leveraged in server virtualization deployments as one example.

Part 12 – SPAN and IPv6

In closing the comprehensive SPAN capabilities of UCS are briefly discussed. Also, I pay some lip service to IPv6 (grin).

Disclaimer:  The views and opinions expressed are those of the author, and not necessarily the views and opinions of the author’s employer.  The author is not an official media spokesperson for Cisco Systems, Inc.  For design guidance that best suites your needs, please consult your local Cisco representative.