Incomplete thought: Just in time QoS

Imagine a data center where network QoS is adaptive and self adjusting to the current application conditions. Just in time QoS. Something much different from the static, stale, and generic configurations we are accustomed to today.

Configuring QoS is a pain, most people avoid it to begin with. Yet some switches are very good at it, resulting in unused and wasted potential.

If you do configure QoS, since the configuration is static, you may tend to implement a very generic policy to address all possible applications. Yet not all applications may use the network at the same time, or some apps may have several different types of traffic profiles, and the generic policy doesn’t provide the granularity that would provide the best results.

What if the network was programmatically configured for the best application relevant QoS policy, at the best time?

What if the painful burden of switch-by-switch QoS configuration was replaced by a single instance of a (dynamic) policy that kept each switch in lock step, automatically?

Such a thing might be possible in a Software Defined Network (SDN).

What do you think? Am I crazy? Why?



  1. Stefan says

    Not crazy at all :) Actually this reminded me of a project I had in grad school, over 10 yrs ago, ref “active networks”. At the time we attempted something similar to ANTS (couldn’t find other, more significant links, but this should give you an idea of what I am talking about): … Your idea of JIT QoS could be implemented in “intelligent” nodes along the path, capable of running code and adjusting conditions of traffic based on some specific criteria, I’d venture to guess.

    • says

      Yes, I have heard of that. I don’t have quite as many years in the business as you do Ivan, but I do have a few, and to this day have yet to see MPLS enabled ToR switches in the data center. Moreover, do you think software virtual switches will implement MPLS? I think not. MPLS based QoS is not viable in the DC, in my humble opinion.

      • says

        @Stefan & @Brad: of course we haven’t seen MPLS-enabled ToR switches. The reason is very simple: nobody (big enough) is asking for that feature. Why not? Because most people think that technologies solving problems in domain A cannot possible work and solve very similar problems in domain B. It’s like thinking you can’t possibly drive a right-hand-drive car in US. Why would the engineers choose to remain domain-limited? Beats me …

        So what happens is this: instead of trying to see whether an existing (field-proven and well-understood) technology can fit the bill, we’re reinventing elliptic and hexagon wheels, because it’s more interesting and intellectually challenging.

        @Brad: As for “virtual switches implementing X” – you’ve seen how far into the morass the L2-only virtual switches have brought us. It’s time to take a step back and ask ourselves “What is the optimal solution” not “what’s easiest/cheapest for VMware/Citrix/Sun/Microsoft to code”. After all, you have your own virtual switch – go out and innovate.

        Just my grumpy โ‚ฌ0.02 ๐Ÿ˜‰

        • says

          I’m biting my tongue a bit here. There are things happening with the Nexus 1000V. That said, if it was an MPLS based solution almost nobody would be able to use it. The access layer switch device supporting MPLS is extremely rare.


          • Petr Lapukhov says

            What’s more interesting, there is no need for MPLS TE in the DC networking altogether. All the DC needs from the network is full bisection bandwidth and bounded latency. Both could be achieved by using appropriate simple and symmetric topology, where path utilization is achieved by simple ECMP spraying (there are caveats, of course, but that’s a separate discussion). TE normally has its place in asymmetric, partially meshed long haul networks, where optimizing utilization for expensive links is top priority. For the DCs, networking cost is on the oder of magnitude lower compared to server cost. Hence the drive for “flat interconnection cloud”, as that’s all that servers really need from the underlying “fabric”.

  2. Dave says

    While I like the idea in theory, a look at the practical execution warrants a bit more attention. In order to effectively deliver an end to end QoS solution, we have to understand the required performance metrics associated with a given application/traffic flow and have a corresponding queueing/policing/etc. policy to accommodate it. In order for that policy to be effective, it has to be in place ahead of the traffic flow that will use it. In the case of latency and loss sensitive traffic, FCoE, voice, video, iSCSI, FIP, trading platforms, etc, I would still lean toward statically defined QoS mechanisms because we don’t really have the luxury of queueing while we determine the appropriate strategy. We’ll call this the proactive QoS handler, depending on traditional traffic marking and trust boundary mechanisms.

    On the other hand, what about the non-realtime traffic classes? I think a foray into dynamic QoS policy would really have to start there, because the reconfiguration of in-path forwarding hardware in such a case is likely going to have to be reactive in nature. In a switching environment, it’s not likely that we can buffer or discard frames while we identify a flow and decide what to do, so the initial traffic goes through as part of a default strategy. Once identified, having an adaptive element that could adjust the policy on the fly could be helpful as an adaptive rate limiter.

    Am I on the same page, or are you thinking more about a predictive model that uses traffic trending data to adjust QoS policy on a time basis?

    • says

      Hi Dave,
      If the QoS policy was applied reactively, I agree that would be a sticky issue. In other words, if the switch has to learn about the flow it may already be too late. Fair point.

      However, a reactive system is not what I’m thinking here. With SDN, it’s entirely possible that a controller linked programmatically to a job scheduler knows ahead of time what flows will hit the network and can prepare the switches for the best QoS at that point in the time, then clean up afterwards.

      You make an interesting point about real time vs. non real time traffic. Some real time traffic may be able to be predicted by the controller, others perhaps not.

      Good contributions to the discussion.


      • says

        Disclaimer: I’m a server/apps guy, not a network guy, so go easy on me. ๐Ÿ˜‰ What about going even further – and adaptive QoS policy that could “learn” what QoS settings are optimal based on historical traffic flows that it’s been watching (with maybe a bit of input from the application itself)? Think of the example of a backup job running each night – could an intelligent application with the right hooks into the network tell the switch that hey, I’ve got 5 TB to backup to another system over there, and 6 hours to do it in. You, Mr. Smart Switch, have an idea of what else flows through your doors typically at this time of night. Set a QoS policy just high enough to make sure I finish on time, but otherwise I don’t need to be prioritized. However, if halfway through my 6 hours, if you see I’m not going to make it, start prioritizing me! Oh, and by the way, don’t try to WAN-optimize me since this is fairly unique backup data that isn’t going to be asked for again by anybody else.

        OK so there’d have to be some pre-defined rules to say which applications were authorized to make such requests, and some default QoS policies to resolve conflicting requests. But I think there’s potential here.

  3. Drew says

    So a few thoughts and musings from the sideline.. This is something Jef Squyres and I kicked around a number o,f times but we also took it a step further as the problem with many switches is how buffers are implemented in a default state versus with QoS enabled. In many implementations you have more buffer in best effort queue that high priority queue, but once you enable QoS you realign buffers to high priority queue which in turn levels off best effort queues which my end up impacting a greater number of application flows than benefitting the few number punted to the high priority queue. So you will toss VoQs at me but then again, you have to look at how you allocate to the VoQs and mapping of QoS/CoS/DSCP to the these shallower queues,… The more queues the shallower you go as buffer memory isn’t cheap.

    So let’s take a different path on this. So let’s say you provide a mechanism for the application to identify to the switch size, type of message and characteristics at time of application launch and you dynamically configure the buffers for large or small messages, incast based or single flow, sustained or bursty. Now you can have a far more effective JIT mechanism to address the application demands. Even if you have a ToR or Agg switch that is VoQ based you could provide a waterfall mechanism to allow multiple VoQs to support a given application flow.

    Just the ramblings of a want to be mad scientist at 11:45pm..


    • says

      This speaks to the idea that SDN provides the opportunity to evolve switch engineering to create higher value, rather than the view that SDN simply commoditizes switching as many tend advocate or believe.
      Excellent comments. Thank you.

  4. says

    In the end QoS means dropping or slowing something down. So, someone still needs to profile the traffic to decide what traffic is more important than others, so I guess you would need some sort of setup tuning where you introduced the traffic to the network, or like Sean McKeown mentions above, an adaptive learning mechanism that recognizes patterns.

    • Drew says

      It is all about your latency budget and how much time you can afford in the “slow down”, buffer versus NAK/drop/recovery….

    • says

      That, or.. you have SDN employed where the SDN controller is linked with the applications via some kind of API, and therefore knows when and what flows will be on the network, and can therefore preemptively prepare the network with the best configuration and QoS, at the right time.

  5. joe smith says

    Add more bandwidth. There – your QoS problem is solved. No philosophizing from Ivory towers is necessary. :-)

    • says

      That sounds nice on paper, but I’m afraid it’s not that easy in real life, or least in the web 2.0 data center.
      Any time you have (2) or more servers trying to talk to (1) server at the same time, you will have contention. A wire rate switch with zero oversubscription interconnects doesn’t change that.
      Consider that in a medium or large web 2.0 data center, such as Twitter or Facebook, you have lots of east-west traffic, where hundreds of servers may be simultaneously sending data to one server.

  6. Troy Levin says

    Hi Joe,

    More bandwidth (assuming you are NOT kidding), will not solve every situation. Obviously adding bandwidth isn’t always cost feasible. However, consider for example, in the digital media production environment where it is common to have flows characterized by very large bursty files (GB in size & MTU 1500) transmitted simultaneous at high speeds. More bandwidth will not overcome exhausted buffers on a switch resulting in packet loss on a microscopic timescale even though the total bandwidth of the link has the capacity to handle the aggregated flows. Such a scenario is very well documented.

    With SDN’s, perhaps allocating buffer resources as apart of its dynamic ability to apply QoS could solve this issue.

  7. Joe says

    I can see this happening with current technology. Snort sensor and a switch interface mechanism across a control channel works well at dynamic ACLs and QoS control.
    Excellent blog, tnx for sharing.

Leave a Reply

Your email address will not be published. Required fields are marked *