In search of consistent returns
Editor's note: Michael Shuker is vice president, business analytics lead, at PNC Bank, Pittsburgh. Courtney Stopansky is market research lead at PNC Bank.
Based in Pittsburgh, PNC Bank, one of the largest regional banks in the United States, maintains a customer focus as one of its core values. In order to deliver on this value, PNC has a strong commitment to customer satisfaction. A critical component of this commitment is the use of follow-up surveys to assess customer experiences across a range of PNC Bank products, services and interactions. This voice-of-the-customer tracking has become fully ingrained across PNC Bank’s various business units. The surveys are administered by the insights and market dynamics (IMD) group within PNC Bank’s retail marketing division. Results are reported via continuously-updating dashboards or on an occasional basis and play a key role in PNC Bank’s ongoing improvement efforts.
For several years now, as a part of this customer service experience evaluation, regular surveys have been administered to customers who have interacted with a PNC Bank branch banker or representatives at PNC Bank’s customer care center to assess satisfaction with the respective experience. When customers have an interaction with a PNC Bank employee in a branch or in the PNC Bank customer care center, they are eligible to receive a follow-up e-mail invitation to participate in a brief survey to assess their overall satisfaction.
The surveys are typically brief, three-to-four-minute questionnaires covering the overall experience and a number of scaled attributes to identify drivers of customer satisfaction of working with branch or call-center employees. The scaled attributes explore customer reactions to the experience and perceptions of how well their interaction was handled by the PNC Bank employee. In addition, there is a customer advocacy measure with a follow-up, open-ended question to understand the impact of the branch or care-center experience on the customer’s likelihood to recommend PNC Bank.
Most surveys administered across PNC Bank and, in particular, these follow-up customer experience surveys, utilize seven-point scales to assess performance across a range of attributes. This branch and care-center survey predated efforts at standardizing survey design within PNC Bank; therefore, it instead incorporated five-point scales for attributes. Thus, the scaled evaluations used in the branch and care-center transactional survey are being transitioned from the current five-point scales to seven-point scales, allowing these crucial surveys to be in alignment with others conducted across different lines of business within PNC Bank and allowing for better understanding of how these branch and care-center experiences fit within the broader PNC Bank business.
Some concern
However, since the results of these surveys can have direct and indirect impacts on branch and call-center employees’ internal evaluations, there was some (highly justified) concern from stakeholders within the line of business about the transition. In particular, these stakeholders expressed apprehensions as to whether the past-state and future-state scales would prove to be comparable and if there would be any impact on the customer recommendation scores associated with employees. It was up to the IMD team to provide evidence that switching scales would not cause any significant disruption in what the results of these surveys ultimately mean for the employees being evaluated.
To test any potential impact of switching scales, IMD chose to conduct a split-sample experiment using our internal Customer Insights Community Panel, administered by our external research partner, Morpace Inc. This research panel is a group of approximately 12,000 current PNC Bank customers who have agreed to participate in various research activities with PNC Bank on an occasional basis. IMD works with this panel to conduct all types of qualitative and quantitative research, such as assessing marketing communications, getting feedback on Web site user experience, small business banking customer needs and testing new names for products or services. It is a valuable resource that has impact across the entire business. The panel gives our customers an avenue to provide valuable feedback and ensures that PNC Bank’s products, services and messaging are aligned with genuine customer needs and expectations.
Developed two versions
To understand the potential impact of changing the branch and care-center survey attributes from five- to seven-point scales, IMD developed two versions of the customer experience surveys to be administered within this customer panel. The original version, using the five-point scales for overall satisfaction and attribute evaluations, was fielded along with a revised version using seven-point scales instead. Invitations went out to all members of the panel, which resulted in over 2,600 completed surveys.
The bipolar agreement scales, as used in the survey, are presented here. Note that neither included a mid-point label.
Panelists were directed to either the branch version of the survey or the care-center version of the survey, depending on their most recent, self-reported experience. Half of the branch or care-center survey respondents went through the survey with five-point scale attributes, while the other half went through a version with seven-point scale attributes; similarly, half of the care-center respondents received the five-point version and the other half received the seven-point version. Final sample sizes for each cell are provided in Table 1.
Of particular interest in the results was the alignment between top-box and top-two box scores from the sample using the five-point scale and the corresponding top-box and top-two box scores from the sample receiving the seven-point scale version.
Our hypothesis was that results would show that the top-box scores from the five-point scales would be spread out over the top-two boxes on the seven-point scale. IMD expected that some sort of heuristic could then be developed from the results to understand the relationship between the two scales to allow converting from one scale to the other, so as to eliminate or minimize any disruption in employee evaluations. Alas, the results were not as we expected.
Quite a surprise
The results were analyzed using SPSS and proved to be quite a surprise in that there was near-perfect alignment between top-box scores for the five-point scaled attributes and the top-box scores from the seven-point scaled attributes (Figure 1).
The charts in Figure 1 provide results across four of the attributes used in the branch survey. In no case is there more than a three percentage-point difference between five-point scale results and seven-point scale results on a top-box basis. There is even a high degree of correspondence between middle-box scores as well. In fact, the primary differences appear to be that ratings of 4 on the five-point scale tend to get split up between 5 and 6 on the seven-point scales.
These results are further supported by the results of the care-center surveys, where a similar set of attributes demonstrated a similarly high degree of alignment between the two versions, as seen in Figure 2.
These findings did not match the IMD team’s initial hypothesis that the seven-point scale would show lower top-box scores and would reflect respondents splitting their highest ratings between scores of 6 and 7. Importantly, it demonstrated to our internal business partners that there could be a seamless transition from the five-point scale to the seven-point scale with minimal impact on employee evaluations. However, there was still a question about any impact of switching scales on employees’ customer recommendation scores, a critical measure for employee evaluations.
To test the effect of scale on customer recommendation scores, IMD ran crosstabs in SPSS to determine if there were any differences in the percent of respondents who would be classified in the top-two box (a 9 or 10 on the customer recommendation scale) among those providing top-box scores on each of the two scales. In essence, IMD was seeking to answer the question of whether responding to a five-point scale survey would result in any discernible differences in customer recommendation from responding to a survey with a seven-point scale. The expectation was that there should be no significant difference but IMD wanted to be able to make a solid case to their internal business partners, since customer recommendation scores are very important to their efforts.
The results of this additional test demonstrated that respondents recommend in the top two box at the same rate, regardless of whether they were part of a five-point scale survey or a seven-point scale survey (Figure 3, Figure 4).
This finding added additional support for the fact that the transition from the old surveys to the new would have minimal impact on employee evaluations. It was an additional layer of support for the transition.
Tracked over time
While the results of this test were quite conclusive, one area for further research and discussion concerns the fact that the scores on these evaluations tend to be fairly high (top-box scores in the range of 60 percent to 80 percent are common). There is still a question about whether this degree of alignment between top-box scores would hold up with more critical evaluations (i.e., more negative customer experiences). It is something that will be tracked over time as these surveys continue to be administered in the future.
The results of this experiment have been presented to our internal business partners with the recommendation that they convert from their current five-point scale survey to a seven-point scale variant to get these critical employee evaluation tools in line with the rest of the business. IMD was able to make the strong case that there would be little or no disruption in employee evaluations. Importantly, making this shift will allow for the results from the branch and care-center evaluations to be compared to results from other business units. This will provide for understanding evaluations not just among employees within a specific department of function but also across the various departments and functions. This will allow PNC Bank to continue to assess its internal processes to ensure that it provides the service to customers that ensures their greatest satisfaction.
This test also further highlights the utility and importance of our customer research panel. Having a large group of customers essentially on standby to run this test with was vital in our ability to help business partners make agile decisions. Additionally, it allowed us to continue to demonstrate that PNC cares about our customers’ financial well-being by ensuring that PNC is measuring their experiences as consistently as possible.