How to Measure Voice Quality Yourself

The Expensive, Accurate Approach: End-to-End Voice Quality Testers

As mentioned in the discussion of PESQ, existing tools can measure the quality of the voice network by directly pumping in prerecording voice samples andcomparing the output. These tools are either expensive or home-grown, and are used to test large networks as a part of a planning or predeployment phase.

This sort of testing is more of a tuning exercise, and-much like how piano tuning is a rare and complicated enough exercise that it is not performed frequently-direct end-to-end testing is not diagnostic. Telephone equipment testing companies do make the sort of equipment to perform this end-to-end inspection, and these tools can be rented. Unfortunately, it is very difficult to know where to invest in this sort of heavily proactive effort.

More likely, the voice quality is measured by having administrators walk around the network with some number of phones in question, ensuring themselves that whatever problems they may face will likely be manageable. The problem with both forms of proactive testing is that they normally occur on only lightly loaded networks, and thus are not able to measure the effect of network load on voice quality. Network load is generally the largest impact on voice quality, in fact, partly because voice mobility network managers do a good job of testing their networks before they launch them for basic problems, which they quickly correct, and partly because voice mobility networks are more likely to be robust enough out of the box for basic voice connectivity.

Network Specific: Packet Capture Tests

Most of the major packet capture tools, for wireline and for wireless, make modules that are able to indirectly infer the MOS values using E-model calculations. Sometimes, these work by tracing the voice setup protocols, such as SIP, and determining what RTP flows map to phone calls and the properties of the phone calls. Other times, these tools will just look directly at the RTP streams, and not try to find out what phone numbers the streams map to In both cases, the tools then use the sequence number and timestamp fields in the RTP stream to determine values such as loss, delay, and jitter. Using assumed values for the jitter buffer, with the option of having the user overwrite them, the tools then model the expected effect and produce a score.

The major issue with these tools is that they show quality only up to the point where they are inserted. An easy example of the problem is to look at wireless networks. On a Wi-Fi network, a packet capture tool may be able to directly determine what packets it sees and come up with a score. By looking at the Wi-Fi protocol, the tool may do a good job of inferring whether the mobile phone received the packet from the access point, and at what time, and may produce a reasonably close call quality number. On the other hand, the upstream flow is likely to look quite good from the point of view of the test tool, because there is only one network in between the client and the tool. The entirety of the network upstream from the client goes missing, and the upstream MOS value can be entirely misleading.

Some network infrastructure devices are able to do these inferences within themselves, as they pass the data through. This may be a reasonable thing to do, again depending on the point of insertion and how well they are able to capture information as late into the network as possible. It is important, when using all of these tools, for you to consult with the vendor or maker of the tools to find out where the tools are measuring. For a wireless controller with voice metric capabilities, for example, make sure that the downstream metrics are measured on the access point, based on what happened over the air, and not just passing through the controller. For wireless overlay monitoring, make sure that there is an option to do a similar capture using a wired mirror port on one of the switches, for cases in which voice quality might begin to suffer and the network needs direct attention. Overall, do not rely on just one tool, and believe what the users say-no matter what the tool tells you.

The Device Itself

The most accurate and reasonable way to measure voice quality is from the endpoints themselves. Both some handsets and PBXs offer the ability for the device to produce the one-way MOS value or R-value for the receive side at the device itself. These numbers are based entirely on E-model calculations, assuming best-case or known-default scenarios for the rest of the system, but are likely to be the most accurate. Of course, it is difficult to ask a user to determine what the voice quality is of a call while on it, especially given that voice quality is not something a user wants to measure. However, for diagnosing locations that are having troubles, this tool is valuable for the administrator herself, who is able to avoid having to guess as to whether the call sounds reasonable, and may be able to detect variations in the MOS value or R-value.

In the end, keep in mind that the absolute values produced by any of the methods deserve being taken with a grain of salt. As time goes on, the administrator of a voice mobility network should be able to learn what the real quality means for any given value the tool suggests, even when the tool is placing results a half a MOS point too high or too low. However, the variation of the scores, especially when the network has changed, can be a valuable tool for point the way towards the solution.

Jitter | What Makes Voice over IP Quality Suffer

Jitter is the variation in delays that the receiver experiences. Jitter is a nuisance that the user does not hear directly, because the phones employ a jitter buffer to correct for any delays. Jitter can be defined in a number of ways. One way is to use the standard deviation or maximum deviation around the mean delay per packet. Another way is to use the known arrival intervals (such as 20ms), and subtract consecutive delays of packets that were not lost from the known arrival time, then take the standard deviation or the maximum deviation. Either way, the jitter, measured in times or percentages against the mean, tells how variable the network is.

Jitter is introduced by variable queuing delays within network equipment. Phones and PBXs are well known for having very regular transmission intervals. However, the intervening network may have variable traffic. As the queue depths change and the network loads fluctuate, and as contention-based media such as Wi-Fi links clog with density, packets are forced to wait. Wireless networks are the biggest culprit for introducing delay into an enterprise private network. This is because wireless packets can be lost and retransmitted, and the time it takes to retransmit a packet can usually be measured in units of a millisecond.

A jitter buffer's job is to sit on the receiver and prevent the jitter from causing an underrun of the voice decoder. An underrun is an awkward period of silence that happens when the phone has finished playing the previous packet and needs another packet to play, but one has not yet arrived. These underruns count as a form of error or loss, even if every packet does make it to the receiver, and loss concealment will work to disguise them. The problem with jitter becomes that an underrun must be followed by an increase in delay of the same amount, assuming no packets are lost. This can be seen by realizing that the delayed packet will hold up the line for packets behind it.

Here, the value of the jitter buffer can be seen. The jitter buffer lets the receiver build up a slight delay in the output. If this delay is greater than the amount of actual jitter on the network, the jitter buffer will be able to smooth things out without underruning.

In this sense, the jitter buffer converts jitter directly into delay. If the jitter becomes too large, the jitter buffer may have limited room, and start dropping earlier samples in the buffer to let the call catch up to be closer to real time. In this way, the jitter buffer can convert the jitter directly into loss.

Because jitter is always converted into delay first, then loss, it does not have a direct impact on the E-model by itself, but instead can be folded in to the other measures. However, the complication arises because the user or administrator does not usually know the exact parameters of the jitter buffer. How many samples, how much delay, will the jitter buffer take before it starts to drop audio? Does the jitter buffer start off with a fixed delay? Does it build up the delay as jitter forces it to? Or does it try to proactively build in some delay, which can grow or shrink as the underruns occur? These all have an impact on the E-model call quality.

As a result, a rule of thumb here is to match the jitter tolerance to the delay tolerance. The network, at least, should not introduce more than 50ms of jitter.

Fair Queuing and Weighted Fair Queuing | QoS

Using the new queuing schemes, each flow now has its own queue. With the fair queuing policy, the packets are transmitted round-robin in order to guarantee each flow an equal share of the capacity (possibly penalizing flows that have large packets at times of network congestion). Weighted fair queuing—an algorithm that is widely used in today’s advanced QoS-capable routers—assigns each different type of flow its (by no means necessarily identical) share of bandwidth. Figure 1 illustrates the concept: In Figure 1a, with the first-come, first-served queue, airplanes, cars, and elephants move in the same order in which they have arrived (a scheme that would cause plane crashes and annoy the drivers of the cars following elephants!). In Figure 1b, with fair queuing, the queues are formed per each flow (defined here as a formation of planes or cars or a caravan of elephants), but they are preempted so that bigger things have to wait until an equivalent number of smaller things passes (still, a maddening experience for elephants!). In Figure 1c, with weighted fair queuing, the planes are given the right of way, so they move through the queue almost without slowing down and always keeping formation; the planes are followed by cars, and the cars by the caravan of elephants. This property of keeping the packet “formation” eliminates delay variance (called jitter).

Figure 1: Queuing and scheduling in routers. (a) First-come, first-served queuing. (b) Fair queuing. (c) Weighted fair queuing.

In 1992, A. Parekh and R. Gallager of MIT demonstrated that a flow that experiences a service rate slightly higher than the flow’s data rate has a bounded delay. In other words, by requesting that a flow not exceed a certain rate, the network can guarantee that the delay experienced by the flow does not exceed a certain value. (A good example of a similar result is green streets in cities, where stoplights are adjusted so that a car traveling at a certain speed—for example, 25 mph—is guaranteed a green light at about 9 out of 10 intersections.)

The scientists then augmented the weighted fair queuing with the specification of guaranteed delay for each flow. This work resulted in a new architecture for what its creators called integrated services packet networks [compare with the expansion of the integrated services digital network (ISDN)] in Clark et al. (1992). Two types of services—guaranteed (which supports real-time traffic with determined latency and jitter bounds) and controlled-load (which deals with relaxed traffic)—were defined. At that point, the groundwork was laid for the standardization work in the Internet Engineering Task Force (IETF). The protocol that defines integrated services, called the Resource Reservation Setup Protocol (RSVP)—which is not a routing protocol. In a nutshell, RSVP, which was designed with multicasting (that is, sending a message to multiple receivers) in mind, makes bandwidth reservations—from destination to source—in the routers along the spanning tree covering multicast group members. The routers store the necessary state information, which is then maintained by sending specific RSVP messages in both directions.

The integrated services approach has been comprehensive, but apparently far too ambitious to implement widely. One recurring sentiment is that the overhead associated with reservations is far too large; the other is that it is overkill as far as the short-lived flows (of which most of the present Internet traffic consists) are concerned. (The counterargument to the latter is, of course, that the model was not created with the short-lived flows in mind; but then, something needs to be done about the short flows, too.) The third concern (Weiss, 1998) regarding the integrated services approach is that it would make charging those who request a higher QoS difficult. In any event, while the applicability of the RSVP to wide area networks and the Internet is questioned, it is being implemented for smaller enterprise networks. In essence, the integrated services approach has been a top-down one—guaranteeing absolute QoS in the network on a per-flow basis.

A bottom-up alternative technology, where QoS building blocks (which routers can recognize and act on) are defined, is called differentiated services (Kumar et al., 1998; Weiss, 1998). This technology has been actively addressed by the IETF and has resulted in a standard. The concept behind the technology is definition of various classes of services. The service provider establishes with each customer a service level agreement (SLA). Among other things, an SLA specifies how much traffic a user may send within any given class of service. A particular class of service of a packet is encoded in its IP header. The traffic is then policed at the border of the service provider’s network. Once the traffic enters the network, specialized routers provide it with differentiated treatment, but—unlike the case with the integrated services approach—the treatment is based not on a per-flow basis, but solely on the indicated class of service. The overall network is set up so as to meet all SLAs.