Getting their level best

If you continually had problems with the quality of the high-speed Internet access provided by your local cable or phone company, wouldn't it be great if you had pre-negotiated a deal with the company that would let you withhold some of your household's monthly fee, as a way to compensate you for your troubles?

In reality, deals like that aren't available to small-scale users like you and me. But they are commonly found in the business world, and they're called service level agreements or SLAs.

SLAs are forged between the customer and the company providing the communications service, whether that service is an e-mail system, Web hosting, or something more arcane-sounding like an IP VPN (Internet-protocol virtual private network). They can apply to a multitude of service parameters, from bandwidth availability to how long it will take to bring the system back up if it goes down.

In a nutshell, SLAs spell out how the service will function and what is supposed to happen if it doesn't meet those levels. But some SLAs aren't guaranteed, and it's not always easy for the client companies to know if service levels are being reached.

That's where Brix Networks comes in. The Chelmsford, Mass., firm develops and markets systems that help companies like Level 3, Genuity and Net2Phone monitor the various communications processes that they provide to third-party clients, typically large firms of 500 or more employees in a host of industries.

Most firms of that size have some kind of SLA with their communication service providers, but until Brix conducted marketing research last year, it was unclear just how important SLAs are to the buyers of communication services and what the consequences are for the service providers who break them. Before the research, it was gut feelings and anecdotal evidence.

To shed some light on the issue, Brix turned to Natick, Mass.-based Sage Research, which conducted a Web-based survey on SLAs with members of its research panel of 3,000 IT professionals.

"You can have a lot of different kinds of SLAs," says Kathryn Korostoff, president of Sage Research. "The most common SLA is up-time, 'If you use our network, we promise it will be up 99.9 percent of the time.' You can say your response time for particular types of traffic will be x milliseconds, or your response times for particular types of applications will be x seconds, etc.

"A lot of carriers, such as Sprint, WorldComm, Genuity, AT&T, etc., have invested money to deploy IP-based virtual private network services, for example. Yet for most carriers, actual sales haven't lived up to the original forecasts. One hypothesis I have always had is that people are reluctant to move to one of these services because they don't see them as being as reliable as other types of services. So how do you get people past that?"

Answering that question was one of the main goals of the research. Along the way, Sage and Brix sought to verify the importance of SLAs and also to understand key issues and perceptions surrounding SLAs. Did respondents believe that their providers met or adhered to their current SLAs? What should the consequences be of non-compliance? How important are they as a service provider selection criteria? In other words, if you are choosing between two otherwise-equivalent service providers, if one has a better SLA does that make a difference? And is that more important than past brand experience, than price?

"Anybody can say they have an SLA," Korostoff says. "But what are the attributes that make customers feel that it is more than an empty promise? Is it about being able to monitor the SLA's performance in real-time? Is it about the kinds of reports you get about how the provider is meeting the SLA thresholds? Is it about the level of detail of the SLAs themselves?"

Other key questions: Do firms feel that communications services like IP-based VPNs are reliable? Would SLAs help them feel better about the services? What kind of guarantees would they need and what would the attributes of those guarantees need to be to really change their perceptions of those services?

Pay a premium

The survey found that not only were SLAs important, but the enterprises were willing to pay a significant premium for verified quality and guaranteed service - and those willing to pay came from companies in a range of industries, not just those like financial services which had shown willingness to pay in the past.

"That was the most startling finding," says Jamie Warter, Brix Networks' vice president of marketing and business development. "And the fact that it was not correlated to any industry - the findings were universal. It basically went across the board. It showed that there is a latent need for quality and people are willing to pay for it."

For example:

  • Over half (56 percent) of the organizations indicated that e-mail services with guaranteed SLAs were worth over 30 percent more than e-mail services without guaranteed SLAs.
  • Half (51 percent) of the organizations said that Web hosting services with guaranteed SLAs were worth over 30 percent more than Web hosting services without guaranteed SLAs.
  • Half (49 percent) indicated that IP VPN services with guaranteed SLAs are worth over 30 percent more than IP VPN services without guaranteed SLAs.

(Even if organizations were overstating the amount they would pay for guaranteed SLAs, the fact that people are willing to pay some kind of premium provides an opportunity to sell differentiated services - like flying first-class instead of coach.)

And if there is a dispute over SLA compliance (43 percent said they had experienced SLA disputes), the companies aren't afraid to take action against the service provider: 38 percent said they withheld payment for services and 31 said they had changed service providers as a result of SLA disputes.

"Those findings gave us really great data," Korostoff says. "After all, we can measure attitudes all we want but what you really want to know is, what behaviors do they result in? That's where the rubber hits the road."

Sales handbook

In addition to providing solid data, the survey also helped Brix create a sales handbook for product managers at the telecom firms. "One of the things we found was that the typical offering manager or product manager at a service provider doesn't have a lot of time and has trouble putting the business case for SLAs together. So this research has been one of the foundations for what we call the offering manager handbook, where we offer a business plan, a rollout plan, and a marketing plan to the carrier on how to compete on verified quality using service level agreements. And a lot of the core data for that came from the study," Warter says.

A white paper based on the research findings has been a popular item to download from the Brix Networks Web site and it has also received a lot of attention in the industry, Warter says. "The fact that we used an independent agency has helped improve the credibility of the report and now a lot more people believe the numbers and support the results."

Short and sweet

The survey earned a good response rate - a testament to the value of keeping things short and sweet, Korostoff says. "We're all plagued by dwindling response rates and everyone in the industry is struggling to keep response rates up. One of the things we ask our clients to work with us on is fine-tuning their objectives and avoiding scope-creep. Brix was a good client to work with in that regard because they really took our advice to heart and as a result we had a nice short survey and got great response rates."

The Web seems like a natural vehicle for a survey of this type, but it wasn't the first choice for Brix. "This is the first time we had done Web research and I was used to more traditional market research," Warter says. "Going in, the thought of using the Web was a negative; I was thinking of a phone survey. But now I wouldn't do another phone survey ever again. After Sage showed us the profiles of the people on their panel and we realized how quickly the survey could be conducted using the Web, we knew it was obviously the right way to go."

"We do a lot of Web-based research at this stage because we are surveying IT professionals," Korostoff says, "and you can't get these people on the phone; they're never at their desks. But with Web-based surveys, they can take the survey when they want to and that works really well. Overall it has been a great methodology for us. We do well over half of our quant work on the Web."

In addition to keeping things brief, she says, it's important to speak the respondent's language. "Many companies are so immersed in their own corporate jargon. So we spend a lot of time with our clients saying 'OK, we know what you want to find out, but we have to word it very differently, because you are using language that already has baggage, or is language that is not commonly accepted outside of your company.' This is particularly true in technology because we are so thick with jargon. If you send out a survey that people can't understand, or is obviously filled with marketing-speak, you'll just frustrate and anger the respondents."

Korostoff says the depth of information Sage has on its Web panel helps make sure the right surveys go to the right people. "We have over 50 variables about them in terms of the possible technologies they might have responsibility for. That way we are only sending relevant surveys to people. We aren't sending a survey about laptops to the person who runs the printers. We are very careful not to over-survey them. We treat them like gold!"

In the Brix study, company size was an important factor. "We only wanted to talk to companies that had at least 500 employees. It's not that SLAs couldn't possibly be of interest to companies that are smaller but we know for certain that companies of that size or greater would definitely use the services that SLAs would be relevant to," Korostoff says.

Not for every situation

While the Web panel approach worked well in this instance, it's not appropriate for every situation, Korostoff says. "I think a lot of us old-time research purists went kicking and screaming into panels, but the truth of the matter is, especially in technology research, I have to have qualified respondents. And it's so hard to get good respondents these days and my best bet for meeting clients' objectives, budgets, and timelines is usually a panel. I wouldn't recommend a panel for every type of project. But there are a lot of times where panels are perfectly fine. You just have to keep a good eye on your statistics, make sure you are testing for reliability, and do your homework."