While MPLS technology is fairly standard in terms of the RFCs and overall base functionality, there are many differences in the features and capabilities that are implemented and offered from one vendor to the next.  We will examine some of these differences so that Enterprises will have the right knowledge base when evaluating solutions for meeting both their current and future business requirements.

MPLS Access Technologies

While traditional serial access technologies are considered the global standard for local access connectivity. Many MPLS service providers have added additional access technologies to their portfolio in order to either a) provide cost competitiveness, whether deployed natively or as a solution over existing copper (i.e. T1 or DS3) facilities, or b) add more value to their solution offering.

Ethernet

The first, as mentioned, is that the cost of Ethernet equipment and Ethernet interfaces are less expensive for both the service provider, as well as the client; and in addition to being a lower cost solution in terms of hardware, it also provides clients with the ability to set incremental ranges in terms of the bandwidth that they would be contractually committed. For example, if an Enterprise has an office in what appears to be a growth market, you may want a solution that provides you with the opportunity to grow the bandwidth as the number of people, as well as the application resources used by those people, increases. If the current requirement only justifies 2Mb worth of connectivity, but the business needs flexibility, then deploying a 10Mb Ethernet Access solution with a 2Mb MPLS port commitment would be a nice option; allowing the Enterprise to upgrade their bandwidth (usually with 48 hours) without having to go through a re-deployment (usually 6 to 8 weeks) in order to get a larger circuit.

However, like with any technology, Ethernet as an access solution does have some drawbacks that every Enterprise must be aware of. Ethernet, while being a very mature LAN technology, isn’t yet widely deployed to business complexes throughout the United States and Europe, as it is in many Asian and Indian markets. This means that the Enterprise will often see additional costs due to new build outs, as well as lengthier (4 to 6 month) installation time frames. However, in most cases, both of these issues are out weighed by the cost effectiveness and contractual bandwidth flexibility that’s offered by Ethernet.  In addition to the extra cost and time associated with getting Ethernet to the actual building, there are also the costs of getting Ethernet extensions (whether copper or fiber) from the building’s main point of entry (MPOE to the offices server or Telco room. Quite often, whether the access solution was Native Ethernet or Ethernet over Copper, Enterprise will either need to pay for fiber extensions and/or conduit (ranging from $1500 to $5000) or pay for multiple copper pair runs (ranging from $1500 to $5250) to accommodate Native 10/100BaseTX Ethernet extension or Ethernet over Copper (i.e. 10Mb over 7 T1s) extensions.

In relation to cost, the Enterprise must compare the total cost of deployment with the needs of the business. For example, if the organization is going to be operating at a particular location for several years, and if that office needs bandwidth flexibility to accommodate both expansion and contraction, then committing to longer contractual terms can often reduce or eliminate the up front cost associated with an Ethernet deployment. Otherwise, an organization may need to look at alternative access methods and/or cover these addition costs.

The second major value point offered by Ethernet is that it can flexibly scale. If an Enterprise has a 10MbE access circuit for a given office and they are only committing to 5Mb for MPLS network access, then that leaves the organization with 5Mb of bandwidth to spare on that access connection. Many vendors offer additional services and features that allow an Enterprise to utilize that bandwidth in several advantageous/cost effective ways. For example, if the vendor supports VLAN tagging in addition to QoS enforcement at the edge of their network, the Enterprise could easily add a VLAN for Internet access and terminate it to a firewall. If VLAN tagging is not supported on the access link, the Enterprise could inquire about setting up special QoS classes that can be dedicated to either external, SIP based voice services (i.e. IP based Toll-Free or Long Distance) or for sending internet traffic to a cloud based firewall. These additional services, when integrated over existing Ethernet access links, are often much more cost effective vs. deploying these service on their own.

Internet VPN via Broadband & Wireless

Internet VPN can be a very cost effective means for accessing an MPLS network. Many Enterprises will deploy this type of access solution when connecting Small Office/Home Office (SOHO) sites (i.e. executive suites like Regus, home offices, as well as teleworker mobile devices and laptops) to an MPLS network. Typically, this solution is driven primarily by both cost and availability. Usually it’s not cost effective to deploy a dedicated connection to a SOHO location due to the number of people being supported or it’s location, and often it’s not available to organizations that utilize executive office suites to policies outside of their control.  However, in addition to meeting the SOHO connectivity needs of the Enterprise, Internet VPN access can also provide an organization with a very cost effective MPLS WAN backup solution that doesn’t completely rely on the public internet. For example, if the organization has a large corporate office or data center that’s housing many of the organization’s applications, it might make more sense to have the VPN tunnels terminating back into the service providers cloud instead of between sites directly. This would ensure the following: 1) if a host site failed, all remote sites would stay on their primary MPLS connection and the host site would use it’s VPN connection to the MPLS network, 2) the VPN connection to the MPLS network will often provide critical Enterprise applications with much better transport performance during an outage by providing lower latency than the public internet, as well as by enabling QoS enforcement once traffic is on the MPLS network.

As a an access technology, the type of Internet connection being deployed is the key to this solution. If the location in question doesn’t have symmetrical bandwidth (i.e. like an ADSL connections with 256Kbps upload/1.5Mbps download, wireless 3G/4G data cards), then it’s likely that this solution will only be able to provide basic, private connectivity for specific data applications (i.e. email, file sharing, etc…). However, if the location has a dedicated, symmetrical connection (i.e. SDSL 1Mb up/down, Share/Dedicated Internet, etc…) or a guaranteed asymmetrical high bandwidth connection (i.e. Cable Internet (2Mb up/12Mb down) with specific packet delivery SLAs, then Enterprises may be able to subscribe to deliver multiple services (Voice, Video, and Data) over these Internet VPN access connections.

As far as cost, an Enterprise must absolutely research and compare one provider’s offering with another and negotiate using that research. If the service providers oversubscribe their VPN access points with both existing and future customers, and don’t offer any bandwidth guarantees, then their prices should be inline with such offerings. However, if the service provider offers dedicated bandwidth, monitoring, and the ability to setup policies on traffic that’s entering and exiting their network, likely there is much more value, and the price will be inline with those additional capabilities.

QoS Capabilities

Once an Enterprise has determined the amount of raw bandwidth that’s required by the organization, as well as the amount of flexibility they’ll need to have for either expanding or contracting that bandwidth, the next major feature or capability an organization must research are the QoS offerings available from different providers. Because the service of MPLS can be looked at as being an intelligent router in the cloud that’s being managed by the service provider, it is very similar to Software as a Service where all the capabilities and features of an application live off premises.  Thus, while provider SLAs are still a valuable component in determining which provider has the right QoS offering, there are additional details that an enterprise must know.

MPLS QoS Enforcement

The first of these is how and where a service provider enforces QoS. Some providers use a non-hierarchical QoS enforcement policy. This means that some or all traffic classes (i.e. usually voice and video) get allocated a specific amount of bandwidth (i.e. as a percentage of the whole or a as a specific value) for their particular forwarding class. If one bit of data hits that forwarding class, then the entire amount of bandwidth dedicated to that class is carved out of the pipe to support that one bit. For example, if an Enterprise has a 10MbE access circuit with a 5Mb MPLS port and subscribes to 2Mb forwarding class that provides a specific packet delivery guarantee for any traffic that traverses that class; if that Enterprise should send one voice call, the QoS policy doesn’t just allocate bandwidth for that one voice call, it utilizes the entire 2Mb of that forwarding class. Obviously, this isn’t an efficient use of bandwidth for something like a voice call; however, for “chatty” applications that dynamically scale their bandwidth usage while in session (i.e. Telepresence, Desktop Video Conferencing, etc…), reserving the forwarding class’s entire amount of allocated bandwidth may be required. The other enforcement policy is exactly the opposite (i.e. a hierarchical) policy. In this type of policy, forwarding class bandwidth reservations are allocated on a FIFO basis. For example, if traffic assigned to a higher class arrives at the same time as traffic in a lower forwarding class, the high class traffic not only gets access to the bandwidth first, but it’s only allocated enough bandwidth from its forwarding class (up to that classes max bandwidth reservation) that it needs to complete it’s transaction within the defined SLA parameters of that particular forwarding class. This means that instead of a single 83Kbps VoIP call taking up the full 2Mb of the forwarding class on the circuit that’s been allocated, it will only take what it needs, allowing lower classes to utilize the remaining bandwidth. Thus, an efficient solution that’s making the best use of the available bandwidth.

Number of MPLS QoS Classes

While managing how much bandwidth a given class with take from the overall circuit is important for application performance, it is also important to have the right number of classes. Often, Enterprises will account for application traffic (i.e. citrix, email, ftp, voice, video, etc…), but tend to forget about signaling traffic (i.e. BGP and OSPF updates, SIP Signaling, ICA signaling, etc…); this type of traffic is just as critical as end-user facing application traffic. Therefore, ideally an organization should assign the highest forwarding class to your connection state traffic (i.e. routing traffic and signaling that’s responsible for keeping the link active).  Failing to isolate this traffic into the highest forwarding class can lend to protocol and session flaps during times of congestion, and thus affect the entire circuit performance, regardless of the application. Next, an organization needs to account for both the real-time traffic streams, as well as the signaling session associated to them. Since signaling is usually only an incremental piece of any voice or video call, ideally that traffic should be classified to use the same forwarding class that actual voice and video RTP traffic will use. Thus, when calculating the amount of forwarding class bandwidth that’s required, be sure to include the signaling traffic. Once the most critical traffic types are classified, an organization must determine how many classes will be needed to meet the available SLAs associated with the Enterprise’s application portfolio. Some Enterprises may simply need one data class (i.e. usually because all of their applications are wrapped within citrix or terminal services), others may need more forwarding classes for many different applications (i.e. such as citrix or terminal services, Internet access via centralized sites, non-critical unified communication services such as messaging, etc…).

End-to-End QoS Enforcement & Monitoring

Once an Enterprise has determined the “how” (i.e. the type of QoS needed and the number of classes needed), the next major QoS related requirement that must be reviewed with the service provider is where QoS is actually enforced and if the service provider gives the organization visibility at these enforcement points.

This item is one of the most the most critical QoS components as it relates to successfully meeting the application delivery SLAs of the organization. Nearly all service providers will enforce QoS at the edge of their networks as traffic enters their backbone; this ensures that the appropriate policies are being applied. However, if an Enterprise is oversubscribing its resource locations (i.e. corporate offices, data centers, regional offices, etc…) by not having bandwidth that is equal to the sum of the organization’s field offices, then it is equally important that a service provider enforce QoS at the egress point of their network. For example, if an Enterprise has 2 resource centers and 8 remote offices, and those 8 remotes have 10Mb each and the host site has only 50Mb, then the organization will experience oversubscription should all of the remotes sites start communicating at the same time.  This oversubscription would not be an issue if the service provider enforces QoS as traffic exits their network (i.e. allowing higher priority traffic to exit first, while buffering or dropping lower priority traffic as needed). Most Enterprise IT Architects, software engineers, and network engineers realize that TCP based application can sustain an acceptable amount of retransmission while still meeting user expectations; this then allows the organization to save money by oversubscribing available bandwidth. However, if the service provider doesn’t provide QoS enforcement for traffic exiting the network, the Enterprise will need to buy enough bandwidth to meet the SLA requirements dictated by the organization.

While it is not always a key requirement for helping an Enterprise IT Architect to meet their organizational SLA objective, having visibility to what is being classified, forwarded, buffered, and dropped by the service provider can dramatically improve an organizations application troubleshooting capabilities when issues arise. Thus, if an application is classified to use a given forwarding class, and then exceeds the bandwidth allocated to that forwarding class, traffic may be re-classified, buffered, or dropped; all of which can impact user experience. Typically, a user will open a ticket for application related issues with the help desk; the help desk then reviews the applications attributes as well as the users device, the server, the network; escalating the issue as necessary. If an organization doesn’t have visibility to a critical piece of the network because it resides outside of the premises, not only is precious time wasted on troubleshooting to get to the root cause of a possible chronic issue, but the root cause itself may turn out to be inaccurate; causing unnecessary expenses.

MPLS Network Support

After researching both the MPLS access technologies and QoS capabilities, an Enterprise must have a clear understanding of how the solution is supported by the service provider. In an environment where the transport features and capabilities are 100% owned by the service provider, an organization must have proactive notification and trouble ticketing for elements outside of their control. The interface on the service provider’s network should be visible to their NOC, consistent packet drops on that interface should be visible to their NOC, routing session changes should be visible to their NOC, network backbone issues should be visible to their NOC, and so on. Moreover, since these issues should be visible by the service provider’s NOC, proactive alerts and ticketing should be included in the service offering.

Ironically, many service providers do not monitor the connections between the customer premises and their own network, and if they do, quite often they put the responsibility of notification for resolution onto their clients. If this is the case, it is absolutely imperative to understand where the demarcation point exist between the organization’s network support team and the service provider’s NOC support group. Moreover, if proactive alerts and ticketing are not part of the service offering, then it is imperative that organizations negotiate any necessary discounts that will be needed to supplement the lack of monitoring being provided by the service provider.

On the other hand, if the service provider offers proactive alerts and ticketing, the Enterprise must compare both the type of alerts being ticketed by each service provider, as well as the NOC processes that each service provider uses to work individual tickets to resolution. If two providers have similar service offerings and features, but one of them offers an MTTR (mean time to repair) of 4 hours and the other is 2 hours; then the Enterprise should use the provider with an MTTR of 2 hours, correct?  No, the Enterprise should request a report from that service provider that shows what their network wide/customer wide, average MTTR is for the different categories of outages. An SLA is only a written document with credits associated to it. Most of the credits don’t come close to the dollar figure associated with productivity loss and/or revenue loss.

Please note that this document assumes that you have some form of MPLS, VPLS, and/or Point-to-Point Wide Area Network transport in place for connecting your locations.

The first step in isolating video & audio quality is a physical inspection of the cabling plant that connects the conferencing system together (i.e. Codec, Flat Screens, and Audio Systems), on occasion, bad crimps and/or faulty connectors will seemingly impair a video quality session as it relates to the haptic elements associated with both Audio and Video quality.

The second step is to review the network connectivity that links the Video Unit in the LAN/WAN infrastructure; this includes checking the in-room Ethernet cabling, room jacks, IDF patch points, as well as the device’s physical interface settings. As was mentioned above, you want to ensure that there’s no faulty wiring or connector points. Moreover, the network interfaces of different switches & codec manufacturers react differently when they’re set to auto/auto and rely on MIDI or MIDIX for determining speed and duplex settings. As such, you should avoid any form of auto-negotiation on the network interfaces between the Codec and the switch it connects to. Assigning a 100/Full setting to both respective interfaces will ensure a constant high-quality configuration regardless of whether the Codec reboots or spanning tree changes/resets on the switch.

Logical Connectivity Assessments

Once you’ve completed the physical network configuration inspection, the next step is to configure your logical paths within the LAN network so that video is isolated from other types of LAN traffic. Although it is not widely documented, mostly due to the large back-plane and heavy forwarding capacity of high-end LAN switches, it is highly suggested that you separate your Video, Voice, and Data Traffic into individual broadcast domains using VLANs. The reason for this is that quite often, Video and Video RTP Streams use the same UDP ports that other devices may use, such as replication engines, SharePoint, accelerators, load-balancers, etc…. When you isolate traffic into a broadcast domain using a VLAN, where the VLAN is trunked between multiple switches, you are assured that traffic will be isolated to that specific service/traffic from a forwarding perspective between the different devices that are residing on this specific virtual network. Moreover, multicast, spanning tree, as well as rapid spanning tree works faster when individual interfaces are isolated to a specific broadcast domain/virtual network; essentially giving you the extended capability to modify the setting of these elements on the interfaces that are associated to this specific virtual network without affecting your other networks within the LAN core.

Also, there is an additional advantage to isolating traffic to a VLAN. If your virtual networks traverse a switch core that is responsible for switching/forwarding an exceptional amount of UDP traffic (i.e. 10Gbps or more) within the spanning tree/forwarding core, you can further isolate your virtual network by associating priority and/or quality to the VLANs directly (802.1q/p). This provides you with several advantages.

• You can ensure that all traffic traversing your segmented Video network has a higher priority over other types of traffic (this will ensure low jitter and constant packet delivery)

• You can create forwarding classes on the VLANs within your switches to set priority on the allocation of the available buffer space to ensure that Video traffic is prioritized within the backplane on forwarding (this is absolutely critical if you’re using a video bridge to run multi-site/multi-room connections and/or if you’re streaming live content from video servers).

• You can also create VLAN inference to ensure that all traffic gaining access to the VLAN is assigned a Class of Service value on ingress and egress (this is a great feature to use if you have mobile video stations that do not have a locked configuration) – more on this point shortly.

In additional to these physical and logical configuration best practices, there are several network based configurations that should be associate to your video network. We will discuss the top three in the next section.

To continue from the previous section, once you’ve confirmed the physical connectivity of your Video System, as well as configured & confirmed the logical isolation of your video traffic, the next step is to ensure that you have quality associated with the individual traffic streams (Video, Audio, and Signaling). As we mentioned previously, you can use VLAN inference to associate Class of Service to the IP Headers of the individual streams that are traversing your Virtual Video Network. The reason you want to have Class of Service associate to individual streams is twofold:

1. You never want to have to worry about what will happen to your video traffic as it relates to using both switched and/or routed interface within your LAN core(s).
2. You want to ensure that video traffic is associate to the proper forwarding queues when that traffic is routed through the interfaces that are connecting both the LAN & WAN links.

As such, it is best practice to assign class of service directly on the individual units. For example, on most Polycom devices you can set Quality of Service by going to the Administration Menu of your Polycom via Setup\LAN Settings\IP Settings\QoS Settings section. This may vary from model to model, but Class of Service should be listed under the IP settings menu.

By default , most network documentation suggests using the following ToS/DCSP setting for each of the three elements of a video call:

However, these suggested settings do not always meld well with how your Wide Area Network service provider recognizes traffic for QoS enforcement of specific traffic types. As such, it may be less complex (and work just as well) to simply mark all three streams with the same Class of Service forwarding value that your WAN provider expects to see for you to achieve specific service level for a given type of traffic. This will ensure configuration continuity throughout the network over that specific provider.

There are also situations (typically due to network inheritance, multiple providers, and/or management simplicity) where you may choose not to set Class of Service directly on the Video Units. In these situations, we typically recommend the following configurations (again, these suggestions are based on the assumption that you’ve already isolated your Video Traffic into a segmented Virtual Network within the LAN):

• Infer the Class of Service assigned to the IP headers from the VLANs themselves (this will ensure that all traffic is marked while traversing the LAN core, even if it’s traversing intermediary/campus routing points). When you do this, you must make sure that any interface associate to this virtual network has a QoS trust policy associated to it. If you do not have a campus network, but you’re still routing from core to edge, then refer to the next option.

• Associate Quality of Service (not class of service) on the Video VLANs so that traffic arrives at core & edge routing points “un-marked”, but with forwarding priority. On those intermediate routing interfaces, create a Policy/Class Map, and then associate that Policy to these interfaces for outbound traffic using the “service-policy output” command. Be sure to select a single CoS value (i.e. ToS 4 or AF41) for all traffic…this will allow you to create simple remark policies on your edge WAN routers if necessary to accommodate multiple service providers.

• If the VLAN does not have an intermediate routing point for getting traffic to your edge network devices (i.e. the VLAN/Subnet gateway resides on the edge router(s)), then you assign the policy map on that particular LAN/Gateway interface. The different between doing it at intermediate point versus at the actual edge is the way the Policy/Class Map is assigned. In the previous example, we would assign it on “output” as traffic left the core destined to your routed edge. In this example, since the edge device is the actual LAN gateway of the virtual Video network, you will use the “service-policy input” command so that traffic is marked as it enters the edge router from the LAN. In this example, if only one service provider is connected to that edge router…you would mark traffic to match that service provider’s specific QoS policies; if two providers exist…then you would again assign a single CoS value for that traffic, and then place remark policies (assigned as “service-policy output” commands) on the interfaces connecting to the two service providers.

There are a variety of ways to assign class of service on Layer 3 Switches and Edge Routers. We’ve provided an example of how to assign class of service to the interfaces connected to a couple of service providers from a single router:

Sample Class of Service Configuration

!

class-map match-all video

match access-group name video

!

policy-map Provider A-Video

class video

set dscp af31

!

policy-map Provider B -Video

class video

set ip precedence 4

!

ip access-list extended video

permit ip 10.1.1.0 0.0.0.255 any

!

interface Serial0/0

description Connection to Verizon

ip address 172.16.0.1 255.255.255.252

service-policy output Provider A-Video

!

Interface Serial1/0

description Connection to Masergy

ip address 172.16.1.1 255.255.255.252

service-policy output Provider B-Video

!

Again, this configuration would change slightly if this device was an intermediate interface or the gateway for the LAN network. For example, you would use the following mapping if it was a single service provider connecting to a single gateway router:

!

interface GigabitEthernet0/0

encapsulation 802.1 VLAN102

no ip address

speed 1000

!

interface VLAN102

ip address 10.1.1.1 255.255.255.0

service-policy input Provider A-Video

!

This configuration marks traffic as it leave the VLAN and enters the edge router. With traffic being already marked, the router simply has to forward traffic out to the service provider (pending no additional policies are required on the interface for shaping to the service providers policing queue(s)).

To summarize, once you’ve confirmed physical connectivity, configured and confirmed logical network isolation and separation, and have the appropriate Class of Service markings at your QoS forwarding points…your video transport network should be rock solid!!! However, this does nothing to prevent human error in the actual use of the video systems. As such, here are some key pointers and best practices you should assess and implement to ensure Quality of User Experience (QoE) every time you have a video conference.

• Lock the users out!!! Once you have physically setup the system, whenever possible, be sure that the only equipment and wiring exposed is Camera, Screens, and Ethernet Patch Cable, respectively. The less stuff being touched will ensure performance over the long run.
• Lock the users out!!! Make sure you have administrative passwords associated with each video unit so that no one but the assigned team can access the configuration elements.
• Lock the user out!!! Ensure that each individual video system and/or management gateway has a list of the callable end points. Moreover, you should define and hard set specific Audio and Video codec types for each destination point (i.e 384K H.264 to Dallas, 768K H.323 to London, etc…). This will prevent users from selecting either too large or too small of codec, and thus getting bad performance and/or quality.
• Lock the users out!!! If possible, lock down the volume and mute control settings on the remote…making it accessible via the microphones only (this will prevent user error), and awkward situation of a user trying to figure out why he can hear them, but not the other way around.
• Lock the users out!!! If possible, lock down the TV’s remote and video input options, making volume & power the only accessible items. This will prevent situations where users getting audio but no Video.