Newsletters

Customer Support:   (972) 395-3225

Home

Articles, News, Announcements - click Main News Page
Previous Story       Next Story
    
Case Study: Customers Benefit From A Tier 2 Competitive Carrier Who Excels At C-Level Technical Expertise and Account Management

by David Deutsch, CTO, Televergence Solutions Inc. - September 30, 2015

CUSTOMERS BENEFIT FROM A TIER 2 COMPETITIVE CARRIER WHO EXCELS AT C-LEVEL TECHNICAL EXPERTISE & ACCOUNT MANAGEMENT

A Case Study Related to Call Center Traffic

David Deutsch, CTO

Televergence Solutions Inc.

(ddeutsch@tsicorp.net)

November 21, 2014

Introduction

My name is David Deutsch and, at the time of this writing have been serving as TSI’s CTO for the past 4 of its 28 year history. During this time, our technical team has developed extensive expertise in managing call center traffic. Additionally, we have developed internal diagnostic utilities specifically tailored for high density predictive and outbound environments. I’d like to share the details of a TSI customer’s issues that were inadequately addressed by traditional upstream carriers and how we were able to diagnose and resolve them quickly and efficiently.

Summary

A 500+ seat call center customer expanded into a new, well-planned location. Although the new facility’s initial production turn up went extremely well; as additional capacity and lines were activated, unforeseen and last minute issues related to their termination carriers began to be reported by operators and floor managers. Call quality began deteriorating, calls to certain regions began to fail, drop ratios and sales-to-call ratios suffered, and finally, many incidents of one way audio were reported. Critical to the financial health of any call center is the ability to maximize call metrics and the conversion of quality data leads. As the customer’s financial performance was being impacted, the customer began a dialog to address the issues with each of their five carriers.

Client Profile

In business since 1982, this call center customer specializes in debt collection, medical notification and seasonal political polling services. Approximately 90% of their business is predictive based dialing using a combination of traditional Dialogic TDM and next generation VoIP based systems. Despite a difficult economic environment, the demand for debt collection and medical notification call center services continued to grow. Expansion was necessary if the customer wanted to capture growth opportunities. The expansion into a nearby facility included a multi-year lease, a floor plan with desks laid out and computers purchased, phones deployed and a new 300 megabit fiber optic installed by a local LEC / cable company. The new location plan included total deployment using VoIP and SIP technology on a high density fiber internet connection, anticipating the saves that these technologies often promise. All systems deployed were heavily tested prior to production.

The Carriers

This customer selected five underlying VoIP carriers based on price, capacity and reputation for service delivery to call center applications. Two are considered Tier 1 due to their size and the other three are considered Tier 2, or regional in nature.

Tier 1 Carriers

When presented with the performance issues the customer was experiencing, both of their Tier 1 carriers assumed that the problems were on the customer’s end and provided little, if any, technical support. Tickets by these carriers are typically processed by helpdesk personnel who lack training in complex VoIP technologies and are under qualified to distinguish between a traditional user with business lines and a call center. Even at a higher level within their support team, it is expected by these carriers that the customer will provide advanced technical details, as well as a perspective on what the problem(s) may be. Frequently, the response by the carrier includes the classic attitude of "dialer = lower quality”. Within a week, both Tier 1 account representatives began to dismiss the issue as a nuisance customer with a poor technical understanding.

Tier 2 Carriers

The first Tier 2 carrier we will discuss is a regional CLEC who had been a supplier to the call center for several years. The call center’s business represented a significant line item. They worked with the customer for nearly two weeks. However, the carrier’s experience was primarily limited to TDM. Without a strong background in VoIP technologies on a large scale, the CLEC’s switch vendor was quick to point out theoretical limits on calls per second (CPS) and other elements of the call center’s VoIP hardware. They suggested that the customer install multiple DS3 loops to carry traffic into a platform in which they had more experience.

The second Tier 2 carrier we will discuss is part of a new group of carriers whose services are exclusively VoIP technology within the call center or "dialer" marketplace. As part of a lowest price marketing strategy and in an effort to keep their costs at a minimum, these carriers typically rent a hosted switch or deploy a freeware-based collage of open source technologies. While I certainly take no issue with open source technologies such as Asterisk, Freeswitch, OpenSIPS and its variants, etc., and being a veteran Linux user myself since it's early inception in 1991, I believe that a carrier must have the technical expertise and related resources to provide its customers with an appropriate level of support. This carrier had a “no skin in the game" infrastructure, and viewed the termination of the customer’s calls as “best attempt”; so ultimately was unable to offer any meaningful technical support. Consequently, the customer’s sales-to-call ratio was very low and wait times by operators were unstable.

The third Tier 2 provider we will discuss is of course ourselves, TSI. Our perception of a call center is not one as a "dialer", but rather as a highly valued enterprise customer who has complex and multi-vendor technologies, resulting in heavy peak traffic loads. We maintain redundant fiber connectivity, a Cisco based network, a Lucent TDM/SONET network, colocation space and a Dell blade server infrastructure. We track voice-related issues at both a signalling and packet media transport level. Routine customer issues are managed by a technical team in our U.S. based, 24/7, network operations center. This single control of infrastructure enabled us to take single ownership of identifying and resolving this customer’s issues.

The Issues

Now that we’ve had a chance to review the carriers, their responses to the customer, and their ability to address the issues, we'll focus on one specific and difficult-to-address issue that was identified and resolved by TSI.

Although the customer was experiencing several issues, each with its own complexities, I wanted to focus on the cause of one way and poor quality audio on a majority of their calls during peak production.

To avoid the bandwidth costs associated with handling full call audio, most carriers only touch the signaling and allow audio (RTP packets) to directly connect between the customer and their upstream endpoint. In contrast, TSI proxies all VoIP media; this approach allows us to perform deep inspection of customer audio and identify issues not often “seen” by a carrier’s networking staff.

Simulated calls between a test machine on the customer’s network and TSI’s lab environment revealed that packets were being throttled from the customer’s connection at a mere 10 megabits per second. We determined that this throttling occurred specifically between their carrier and specific upstream IP connections, indicating that a carrier was either paying a high rate for transport, or did not have stable peering with another.

As it turns out, although the customer had purchased an enterprise grade fiber connection from a regional internet company, which should have guaranteed that their packets had priority on the network, the customer did not realize that traffic shaping and throttling often found in today's tech news regarding video content, such as Netflix, might apply to their connection and type of traffic.

Although Internet Service Providers were initially ordered by the FCC to stop traffic shaping protocols such as VoIP in 2007, Comcast won an appeal of this order (see Comcast Corp. v. FCC, 600 F. 3d 642 - Court of Appeals, Dist. of Columbia Circuit 2010). Consequently, many ISPs still traffic shape today. It may seem like an obvious and logical solution to obtain service from a different ISP who won’t alter or block traffic from the customer’s facilities. However, if you've been involved in large deployments, you're most likely aware that each building has on-network carriers and that other carriers, even 50 feet away, will not service a new address; if they will, it is only with extremely cost prohibitive build outs. This customer did not have the resources to select an alternative provider, nor was traffic shaping a breach of the carrier’s own contract and acceptable use policy.

The Resolution

TSI's technical response team tried several solutions to resolve the issue, including attempts to divert traffic to different destination IPs since the traffic limit was circuit based, not destination based; and encapsulating traffic into an IPSEC VPN tunnel (the carrier shaping technology still limited packets per second). The final solution included two methods.

Method one included the management of our edge network BGP advertisements. BGP is a routing protocol used in the internet's core to decide on the best path to send packets to a specific destination. TSI utilized its BGP relationship with our upstream carriers to block our network from being reached on specific providers that the customer’s ISP had issues with. Therefore, even though the customer’s ISP would normally reach our network via Level3 or Centurylink, we forced their ISP to seek specific alternative routes that did not have this form of throttling enabled. If anyone reading this BLOG posting wants more detail on how we achieved this, please contact me directly regarding "BGP Community Strings".

Method two included looking at how throttling was being achieved, specifically if it was a packet per second or an overall bandwidth limit placed on a "type" of traffic profile for the circuit. Although the customer’s VoIP based predictive system didn't have access to g729 or g723 codec licenses, which are often used to compress the bandwidth necessary for calls, because TSI maintains its own transcoding hardware, we were able to choose codecs that were public domain and apply these transcoding profiles to every single call, regardless of their final destination networks.

Today, the customer is successfully using their existing connection to complete their calls without hitting the traffic shaping limits of their ISP. Naturally, they are also investigating a new fiber provider as well.

Conclusion

The critical importance of a call center’s upstream carrier’s ability to deliver calls to the PSTN is paralleled only by the center’s ability to establish a comfort level with the carrier’s C-Level Team, 24/7 technical team, expertise within the call center segment, and willingness to dedicate the resources necessary to identify and resolve your unique and urgent issues. At TSI, we are your partner, not just your carrier.

If your call center is already VoIP interconnected and has relationships with large national carriers, we suggest you take the time to investigate some of your options and see what a smaller carrier has to offer above and beyond basic origination and termination services.



 
Return to main news page