What is a Distributed Firewall?

In the post “What is Network Virtualization?” I described a model where the application’s complete L2-L7 virtual network is decoupled from hardware and moved into a software abstraction layer for the express purpose of automation and business agility. In this post I’ll focus on network security, and describe an imminent firewall form factor enabled by Network Virtualization — the Distributed Firewall.


If InfoSec ruled the world … well, OK, maybe not the world … if InfoSec ruled the data center network design, and if money was no object, we would probably have something like this. Every server in the data center directly connected to its own port on one massive firewall. Every packet sent from every server would be inspected against a stateful security policy before going anywhere. And every packet received by every server would pass one final policy check before hitting the server’s NIC receive buffer. The firewall wouldn’t care about the IP address of the servers, for the simple reason that it’s directly connected to every server. E.g. “The server on this port can talk to the server on that port, on TCP port X”. And if that wasn’t good enough, the firewall knows everything about the servers connected to it, and can create rules around a rich set of semantics. All of this with no performance penalty. That would be awesome, right?

Let’s pretend money was not the issue. How would you design this massive omnipresent data center firewall? I can think of three ways off hand.

    1. You design a monstrous power sucking stateful firewall chassis with thousands of line-rate ports. At this point it’s time to route a ghastly mess of cables from every server to this centralized mega firewall core chassis – but that’s somebody else’s problem. Oh, and don’t forget you’ll need two of those bad boys for “redundancy”. Your monster firewall is pretty freaking awesome at security, but only so-so at basic L2 and L3 networking. But so what — the network team can learn to like it or find a new job. And if you run out of ports … no worries; just wait another few years for a bigger chassis and do the rip/replace routine.
    2. You design a line rate stateful firewall ToR switch. Rip out the network team’s favorite ToR and put this one in its place. Tell them to stop throwing a fit and just deal with it. You’ll have hundreds of these ToR firewalls to manage and configure consistently. No problem … just let the network team re-apply for their jobs as firewall engineers.

Go ahead and pinch yourself now. This is nothing but a fantasy nightmare.

The interests of security often poorly translate into networking. Comprehensive security ~= Compromised networking.

What about design #3? More on that in a minute. (Hint: title of the post)

In the real world, rest assured we do have firewalls to provide some security. But this security is not ubiquitous, nor is it assured. Instead, we have firewalls (physical or virtual) hanging off the network somewhere catching steered packets – and we can only hope the network was configured correctly to steer the right traffic to the right policy.

In this post we’ll briefly review the physical and virtual firewall, followed by a discussion on the Distributed Firewall.

Physical Firewall

This is the firewall we’ve grown up with; the trusty old physical firewall appliance. It has some very specialized hardware that does network security really well – within certain performance specifications of packets/sessions per second and raw throughput. 10Gbps would be considered high end (and costly). There aren’t enough ports on the firewall to connect every server directly to it, so instead we hang it off the Core switch and tell the network team to figure out how to steer the traffic through it.

For east-west traffic between servers it’s a choke point. It can only go so fast – but don’t tell the network team that or they might start blaming the firewall for every little performance problem. Meanwhile, we’ll just throw more firewalls at the problem, hang them off more ports on the Core network switch, and tell the network team to figure out the traffic steering part all over again.

The physical firewall is hanging off the network somewhere catching packets – it’s not directly connected to the servers. Consequently, security policy is only as good as the information available in the packets; such as IP addresses and TCP/UDP port numbers. So we build our firewall security rules around that basic context. E.g. “This IP address can talk to that IP address on TCP port X”, and so on.

Deploying a new App? No problem! Just submit a change ticket to the InfoSec team describing the App you want to bring online, along with its list of server IP addresses and TCP/UDP port numbers. A few weeks later (after it’s been decided which physical firewall your App will be physically anchored to) the security rules for your App will be added to the existing 5000 lines of rules collecting dust. A few weeks after that, the network team will (hopefully) engineer the traffic steering to the right interfaces on the right firewall. And if the App is decommissioned later on? Don’t ask. The rules are never cleaned up because that’s too much work, and that would require another change ticket anyway.

Deploying a virtualized multi-tenant agile cloud with automated provisioning? OK, that might be a problem. The physical firewall doesn’t do a very good job at automation and multi-tenancy.

Maybe virtualized Apps need a virtual firewall?

Virtual Firewall

The virtual firewall is just like the trusty old physical firewall but only now it’s delivered as a virtual machine instead of a physical appliance. This might appeal to you if it was determined that automation and multi-tenancy are important. The rationale being that each App instance (or tenant) can deploy their own virtual firewalls in their own enclaves, and if the firewall is a virtual machine it can be deployed automatically like any other virtual machine. And performance of a virtual appliance is just as good as a physical.

The virtual firewall is, again, this choke point hanging off the network somewhere catching packets (this time on a vswitch somewhere). We still need to steer traffic through it, and it’s still an object that needs to be managed and cared for instance by instance. And the rule structure still revolves around the basic context of IP addresses – no real improvement there.

Deploying an App in a virtualized multi-tenant cloud? No problem! Just tell the App owner that security is now their problem, and point them to the URL where they can download the firewall virtual machine and user manuals. When one virtual firewall becomes too much of a traffic bottleneck, the App owner can just throw more virtual firewalls at the problem and figure out how to design, manage, and automate around that. If the App is decommissioned later on the virtual firewalls and their rules just go away.

Did you say the auditor is here to validate compliance? OK, that might be a problem. There are hundreds of Apps in this cloud, which means potentially hundreds of virtual firewalls floating around. Let’s hope the App owners deployed them correctly – if at all. And it might take some time to find that one virtual firewall we need to look at and grab its configuration – cross your fingers!

Maybe virtualized apps in a distributed infrastructure need a distributed firewall?

Distributed Firewall

As we’ve just discussed, the physical and virtual firewalls are really the same old firewall model, just in different form factors. The distributed firewall, on the other hand, is when the firewall is no longer a form factor at all, rather the “Firewall” is now embedded as-a-service in the programmable hypervisor kernel networking stack. All participating hypervisors collectively become one “Firewall”. Every virtual server is connected to a hypervisor. By consequence, in this model, every virtual server is directly connected to one omnipresent “Firewall”. A firewall that knows everything about those virtual servers. Sound familiar? ALL YOUR PACKET…

Every packet sent from a virtual machine can be inspected by the stateful firewall service present in the hypervisor kernel, before the packet even touches the network. Only after packets have been permitted through the stateful “Firewall”, they hit a network built with any standard networking equipment and architecture of choice. No need to steer traffic. No need to compromise the network design. And every packet delivered to a virtual machine can be inspected one last time by the same firewall service, at the destination hypervisor, before the packet finally hits the virtual NIC.

The distributed firewall security policy is centrally defined (via API) through a notion of logical “containers”. Workloads are then placed in containers programmatically, and the enforcement is programmed into the hypervisor kernels. The security rules are based on container membership – not based on IP addresses (although you can still do that if you want). For example, things in container “Web” can talk to container “App” on TCP port X, and so on. Of course this means the IP address of your virtual machines can change, without changing the security policy. And container membership can be static or dynamic, programmed to trigger on just about any arbitrary metadata about the workload; such as user identity, the presence of a virus, various operating system characteristics, etc. Thus, security policy is decoupled from IP addressing.

The distributed firewall is multi-tenant and capable by virtue of its presence in the hypervisor kernel. It’s attached directly to the workloads and follows them wherever they go. Just place the workload in its security container and you’re done. Every packet sent or received flows through a stateful “Firewall”, and the security policy follows the virtual machine when it migrates to another hypervisor. With a distributed firewall, traffic steering is completely removed from the process of implementing security policy – for the simple reason that it’s directly connected to every virtual machine. In other words, security is omnipresent, at the very first and last hop.

With security decoupled from both traffic steering and IP addressing, it becomes possible to consider more simplified, flatter application architectures. For example, Web and App “tiers” could be on the same network segment, separated by their container membership.

Deploying an App in a virtualized multi-tenant cloud? No problem! The App owner just describes the desired security from one group of virtual machines to another. No need to fumble around with virtual firewalls, and the infrastructure owner can enforce certain enterprise policies right at the first hop. And if the App is removed later on its associated security rules are removed along with it.

Bottlenecks? Choke points? Not anymore. The distributed “Firewall” performance scales out with the hypervisors. Meaning, the theoretical throughput of the distributed firewall is some calculation of (number of Hypervisor machines) * (Specs of each machine).

The auditor just showed up? No problem! Go to the central policy console and view everything in real time, in one place.

Despite all of its awesomeness, the distributed firewall is not the firewall to end all other firewalls. The slam dunk deployment will be for securing east-west traffic within and between tiers of a virtualized application, Web > App, etc. and virtual server (or desktop) access control. At the north most edge of an App architecture you might still want a traditional physical or virtual firewall providing an additional set of perimeter edge services, such as NAT, VPN, etc.

Distributed Firewall: Key Takeaways

  • Directly connected to virtual machines
  • No traffic steering
  • Omnipresent security at the first and last hop
  • Security implementation decoupled from network architecture
  • Not a choke point
  • Scale-out distributed kernel resident data plane
  • No device to manage – Firewall as-a-service
  • Centralized policy automated with APIs
  • Security rules decoupled from IP addressing
  • Workloads added to “containers”, with static or dynamic membership
  • Security enforced based on container membership
  • Ideal for east-west application traffic, and access control
  • Stateful security that migrates with the workload
  • May be combined with traditional physical or virtual firewalls in an overall solution



  1. Oded ROtter says

    I like this post (like your other posts-as usual)- i thought about these issues for a while.
    In my mind i can see a Scale-out distributed kernel resident data plane on hypervisor kernel & ToR switches with shared control plane.
    Only that can be complicated.
    Assume you have Mucrosoft Hyper-v + vmware + KVM + Legacy servers…nightmare …
    But , if you “offload” the dataplane to the TOR switch (Assume a 8 * 10GE ports per BladeServer rack) maybe it will be easier and cheaper to implement ?

  2. says

    Hey Brad. A distributed hypervisor level firewall sounds great but as ever I’m looking for more detail. Is this a shipping or due to be shipped product? Does it have a name? Can a guest be in more than one container? Can you combine the container and IP approaches (source is container, destination is IP or visa versa)? Are there plans for other host classification methods? What’s the logging like? Are protocol specific features supported (active FTP, TCP connection control etc.)?

    More generally, the functionality, abstracted host classification and central control is very attractive and something I’d hope to see in SDN products sometime soon as well. If this is a product close to market, well done VMW.

  3. says

    Fascinating – but a bit beyond my tech grade… Could a BOINC volunteer grid do this? Would it be worth it?

    We have 20,000 machines about to be freed from their current task, looking to put them to good use. Currently selling their GPU compute for 3p/hour, but no idea if distributed firewall could earn that – or if it is even possible on a high-churn public grid.

    Your input would be most appreciated!

  4. says

    Great Post Brad, as usual.

    One fourth possibility is that security techniques evolve as well. Once you have a distributed L3 Gateway, all sorts of things become possible/interesting. Why not just use unique VXLAN IDs for each ‘container’? Nearly the same as what you describe. Sure there are important differences (filtered v. stateful inspection), but when you can isolate things with finer granularity, the security perimeter shrinks accordingly.

    All you need is:

    1. VXLAN everywhere
    2. A way to manage 16M VXLANs.


  5. Ajay C says

    This sounds exactly identical to the Juniper VGW solution that uses the VMWARE Fast Path API that has been around for a while.
    This is a great solution but reality is that most datacenters have a mix of virtualized and physical servers. What would be great is a unified control plane that can implement policies at the edge – irrespective of whether the the protected entity was physical/virtual. Thoughts?

      • says

        Not exactly. With Cisco VSG, only the first packet of a flow is inspected. Also, I could be wrong, but it looks like the Cisco VSG data sheet recommends deploying multiple VSG instances for a multi-tenant environment. A kernel resident distributed firewall is deployed once, supporting the multiple tenants using the hypervisors.

        • stefan avgoustakis says

          Hi Brad,

          Correct – initial packet processing occurs in the Cisco VSG for policy evaluation and enforcement. Subsequent policy enforcement for packets is offloaded directly to vPath (similar to how a physical firewall uses a fast path for traffic optimization) which is a component of the Nexus 1000v distributed virtual switch (but you know that – better then me :)).

          A single VSG instance supports multiple tenants, multiple VSG’s might be required for scaling.

          If I understand your post correctly you’re advocating that the policy creation and distribution is a centralized solution but the enforcement and inspection happens in the hypervisor’s – depending on the number of VM’s on that hypervisor and the amount of traffic HW resources required by the hypervisor might become a challenge (stateful inspection is a resource intensive process) hence some environments might want to have a dedicated hypervisor hosting service nodes such as virtual firewalls which makes traffic steering a necessity.

          Although its an interesting evolution seeing the protection services moving from the network to the hypervisor and even further down into the CPU’s.

          Maybe the real distributed firewall is the one that is managed and controlled by an open standard protocol and has enforcement point within each layer of the stack.

          Thank for your great articles on this blog –


  6. Vinay Bannai says


    Well written article.
    This has been one of the issues that are being tackled by the Openstack Neutron (formerly Quantum) through security groups and FWaaS.
    We have run into all or most of the issues that you highlight. We like to use the terminology of perimeter firewall which are typically handled by big iron firewalls and once inside, the different tiers or apps communicate using security zones or groups.

    Thanks for the writeup.

  7. Carlo Gagliardi says

    Excellent post Brad, I have been discussing these points with my colleagues for some time now, the main spur being the failure of physical or context based firewall solutions to robustly support our intent of workload mobility between datacenters. We have been struggling with concepts on how various services, such as firewalling and wan optimization, can share state across datacenters to allow not only DR fast turnup scenarios but full DC to DC live migrations. We seem to be making progress with the LISP/OTV/VM-Mobility combo from Cisco, VMware’s latency requirement improvement for vmotion, VXLAN to allow L3 barriers to be crossed etc, but I feel the firewall issue has been left open too long. I have briefly looked at a few different products for firewall security, and we seem to have all the features we need, but not all in one product. My understanding is that Cisco’s new ASA clustering feature allows physical/context based firewalls to be split amongst DCs with low latency between. This is a nice step but my understanding is that if a workload were to migrate, the system would trombone that traffic back and forth through the original firewall that recorded the state. Not exactly awesome for long lived flows if no handoff of that state occurs. My brief look at the VMware Networking and Security product indicated to me that it wasn’t as stateful as other products but more of a filtering solution. Perhaps you can correct my misperception if it is one, or clue us in on VMware’s new kernel based stateful distributed firewall that allows state to be replicated along with the vmotion????? I do need more information on the Juniper product, I understood that it is hypervisor based but wasn’t clear on statefulness or robustness of inspections. I have even joked that maybe we can go back to a host based firewalling solution where the state is held in the VM uOS :) This way no matter where the OS, so goes the policy and state. Nobody laughed.

    Thanks again for posting what we have all been thinking about or what we should really be thinking about, I would appreciate more discussion on the topic.


    • Agreed says

      ” I have even joked that maybe we can go back to a host based firewalling solution where the state is held in the VM uOS :) This way no matter where the OS, so goes the policy and state. Nobody laughed.”

      Why would they. This is truly the end state where a workload and its security are one unit. If someone would develop a strong central management platform, segmentation and policy management would be much more finite.

      • Carlo Gagliardi says

        I thought that security folks would argue that we need to separate the security intelligence from the object being protected (or compromised). All too often we see malware, viruses, and hackers disable the AV, Malware and intrusion protection on a system and hijack it all. If the security is in the hypervisor like the NVP and Juniper product, the OS is unable to manipulate the external security processes and may not even be aware of them either. This tiny bit of separation is what may keep the solution intact under attack.

        • James Hess says

          Not necessarily “need to separate the object being protected”. There may be a benefit to doing so, to the extent that the security intelligence on the object is intended to control outbound traffic.
          If you have security on every object being protected, then if an object is compromised, the other objects’ security intelligence is still independent of the compromised objects.

          *Many* networks do no egress filtering and did no extra firewalling on hosts before; they often do *one or the other*…. firewall at the perimeter, or firewall on each machine. Therefore, many organizations might not benefit from isolation between systems performing security roles, since, there is no real reason the inbound protection has to be separate.

          In case there is no egress filtering in the traditional network; the only real disadvantage of moving security into the object being protected and not having it separate — Now the sysadmin of the system, or random software vendors may accidentally disable the security or introduce misconfiguration or unapproved configuration to make their lives easier.

          The sysadmin may disable the firewall, or they may call up support, and the first thing support tells the sysadmin is they have to disable the firewall on the host…. which the vendor does, and as soon as they turn it off: the host gets covertly attacked and compromised.

          Or possibly they find the Firewall had nothing to do with the issue but forget to turn it on, or the vendor tells them they can’t turn it back on, or it will no longer be supported.

          By having separately managed infrastructure that the security team is responsible for maintaining under a change control process outside the systems sysadmins are allowed to touch, the risk of such incidents can be mitigated.

          But I suppose a good enough multi-host guest Firewall management system could be a decent replacement.

          Especially if your distributed guest Firewall management system does not allow configuration changes to be made locally, if it monitors the health of the firewalls on the guests, and by design, the infrastructure will shutdown connectivity to a guest if its Firewall is missing.

  8. Gurjar says

    Why do we even need the term NETWORK? Why do we even need to know these detials?

    I would like everything to be taken care by software so NETWORK is ubiqotous so there is no need for Virtual network admins….just click here and there and be and done with it…thats it….the word NETWORK and NETWORK ADMINS will be history…VMWARE needs to improve a lot to make that happen.

    Am a VM guy and would not like to work with these pesky network admins…….

    • Carlo Gagliardi says

      Until we have 10Gb/s full duplex wireless service with signals that travel 1000 times the speed of light to cut latency down to microseconds, you are stuck with us networkers! You have to connect to something, software doesn’t run the universe, yet! Perhaps the quantum movement will set you free?

      • Ronald says

        This day would come….software will rule the world..

        We will have evrything Software defined…

        SDE — Software Defined Exchange — no need for MS Exchange admins
        Call Managers — no need for Vocie personell
        and blah..blah…AND….

        Software Defiend Programming.—Programs being built by Software..so we need just a few handful of programmers…

        ALL but one field will die…that field will be SAN and VM admins….

  9. says

    Long time reader, first time writer. I appreciate many of your posts Brad, so take my light criticism with a grain of salt please.

    I think I can agree and disagree on many things. If you are positioning a distributed cloud based enterprise, the vAppliance is “good enough” in most cases. However, cloud providers have caught on to these appliances and are starting to charge for CPU more than disk in some cases.

    That said, there are many hardware appliances that far outstrip their virtual brethren simply because of hardware acceleration. In the case of load balancing, SSL stripping and re-encryption CANNOT be done at line-rate without specialized hardware. The same goes for HTTPS inspection of UTM (flow or proxy based in-line Antivirus), IPS, and firewall traffic. Packet-forwarding is a quaint topic. Claiming high through-put for routing pretty much means nothing as those functions are increasingly commodity and being stuffed in devices that are capable of much more. If you think stateful firewalls still make your network safe, you need to lift the rock you have been living under.

    Sorry to be so harsh, but some of these articles are very myopic and don’t really address the issues of a modern network. That is why the state funded phrackers (my word, it is a play on water-fracking for natural gas, because this is analogous to how modern ‘hackers’ mine data from networks) pwn you.

    I think a mixture of SSL stripping hardware accelerated load balancers or physical firewalls may help decryption in the short term, but you should probably pay for or build out a layered approach with hardware accelerated appliances TOR and drop the SSL traffic off to the blade computer infrastructure with multi-tenants and then have them control their own diced out vAppliance firewall. If your hardware firewall supports the idea of VDOM\Instance\Slice\Context then you can give them control of their Virtual Domain on the hardware (if they want to pay a little more for control) and they can stand up another software version of the same and handle if with the same Operations Manager but with far more granular policy.

    I get kinda tired of people pimping specific products in these forums when their products are a neutered and inefficient form of security management. But there it is…

    Until more\better hardware acceleration comes to virtual platforms, the idea of true high-performing distributed security will need a mixture of physical and virtual hardware. Does your vendor support these scenarios?

    • Donny says

      Sorry, but I have to jump on this SSL decryption point as it breaks the entire trust model. The participants in the transaction (sender/receiver) are the only ones who should be privy to the contents of the encrypted communication. This man in the middle routine is just another threat.

      The network is hostile. Be it internal or external, once it leaves the application no transport is to be trusted. SSL was to assist in this be providing a unique and certified session between two participants in a communication. With firewall termination, SSL middle men, and other network invasions another solution/protocol will need to be established to provide true privacy.

      Also, with defense in depth, the network is only one facet…

  10. Aron says

    I strongly beleive there is no need to ‘virtualize’ a Network. SDN is not Virtualizaiton, its network programming.
    Thats it. There’s nothing to network as all components sit inside the server fabric.

  11. GM says

    How would rationalize communication between NSX security groups and external networks behind a Cisco ASA or Checkpoint device?


    • says

      Hi George,
      You can create NSX security groups that represent those external networks, for example groups with names like “External-1” and “External-2”. Once you’ve done that you put IP address ranges as the member of those groups. You can then compose NSX firewall rules using the groups. For example “Web-servers” to “External-1” DENY, and so on. The rules will be enforced at the NSX layer.

      The other approach is use an integration with the external Firewall vendor and NSX (such as Palo Alto Networks). This is where NSX will share its group information with the 3rd party firewall, allowing rules to be composed on the external Firewall using the NSX group names. This is already possible with Palo Alto, and I believe a similar integration with Checkpoint is coming soon.



Leave a Reply

Your email address will not be published. Required fields are marked *