Scanning the seas

In their continuing quest to do research better and faster, many researchers are discovering that scannable questionnaires have a lot to offer. On a scannable form, instead of checking boxes or writing in answers, respondents reply by filling in dots. The completed forms are then fed into a machine that "reads" the dots.

Some large-scale users have bought their own scanning systems while others who survey sporadically send the forms to an outside firm for processing. One company, Princess Cruises, does a little of both, using an in-house system for its ongoing satisfaction survey and farming out smaller projects to Pine Company, a Santa Monica, Calif., data processing firm.

Love Boats

Princess Cruises' famous fleet of Love Boats sails to destinations around the world. The company is the third largest in the industry based on market share and carries approximately 450,000 passengers annually.

The company has had an in-house scanning system for four years. Jaime Goldfarb, senior market researcher, Princess Cruises, says that the ability to have immediate access to the data from its on-board surveys was one of the reasons Princess made the investment in a scanning system. "Our use of scanning systems is twofold. First, to monitor on-board performance on a regular basis, we distribute a satisfaction questionnaire on all voyages. Second, in response to our operations and marketing departments, we are continually designing and implementing a number of small, custom questionnaires. In the past, we had the data keypunched. However, by farming out the questionnaires, we found that we had a lag time in processing that would last anywhere from a week to ten days. In order to reduce our processing time, we looked for a method in which we could process the forms in-house. We decided that the only feasible approach was a scannable system."

Another reason for going to scannable forms was increased accuracy, Goldfarb says. "We have found scanning to be a more accurate method of data input than keypunching. When initially considering a move to a scannable system, we ran tests on a set of questionnaires we had already fielded. When we compared the data files resultant from the two methods to an internal review of forms, we found that although the scanned system occasionally missed marks because they had not been filled in properly, the scanned data file was still more accurate."

An additional plus is the public's comfort level with scannable forms, he says. "Because they have become so ubiquitous in society, from test questionnaires in school and college, in some ways people are more comfortable filling in bubbles than they are in having to complete a non-scannable questionnaire."

185,000 forms per year

The on-board survey is distributed one per cabin prior to the last day of the cruise. Passengers are asked to place the surveys in a locked box when they are completed. The 45 question survey is very detailed, looking at opinions of pre-cruise documents and logistics such as airport transfers to the ships, airline flights, and shore excursions. It also looks at presentation of food, quality of ingredients, variety of menu, and service areas such as stateroom service, housekeeping, and front desk performance. The surveys are distributed on every voyage of each of the nine ships in the Princess fleet. Approximately 185,000 forms are processed per year, and the response rate is between 65 and 95 percent.

"The questionnaires from every voyage are processed, and a performance report is distributed to all operations departments," Goldfarb says. "We also produce a monthly summary report that examines on-board performance by cruise destination and vessel, as well as identifying any specific problem areas that need to be addressed. If we see an area that consistently shows a deficiency in performance, we will take whatever steps are necessary to fix the problem.

"Whenever changes are made in our questionnaire, we first approach our operations people for any input they may have in order to make the forms as meaningful as possible for them. We try to make the questionnaires as actionable as possible for these operations people, so that not only can we identify problems we can also provide feedback on how these problems might be rectified."

For example, the ongoing survey has led to changes in the menu items and the on-board buffets. "At certain times during the year we've found problems with our on-board buffets in terms of presentation and variety of items that were offered. Based on that data we've changed the program of the buffets, the menus and the way the food is presented. We also found that certain food items, based on open-ended comments, tend to be less popular so we've taken those items off menus and added things that people indicated they'd prefer."

Custom projects

While Princess uses its in-house scanning capabilities for the ongoing satisfaction study, for certain surveys it has turned to Pine Company. "As I stated previously, we continually have a number of smaller custom projects that arise in response to specific operations and marketing concerns. Because of our satisfaction with scannable systems, we have wanted to continue to employ this method. However, due to time and capability constraints, we have not found it viable to design these questionnaires in-house. Pine Company has been a great resource for us in conducting these types of studies," Goldfarb says.

One of those smaller studies looked at satisfaction with booking of shore excursions, the trips and tours passengers can take when their vessel reaches port. Passengers book shore excursions two ways, either prior to the cruise through their travel agent or Princess or through the shore excursion office on each Princess vessel. "From our standard on-board surveys we've gotten some feedback that (the booking process) is one of our weaker areas. In response we designed a survey that looked in detail at the various attributes affecting shore excursions to try to remedy any problems we might be having."

Coding open-ends

One way Princess is seeking to make the questionnaires more valuable is by coding the open-ended comments passengers make on the satisfaction questionnaire. "With Pine Company's help we have recently begun to code the open-ended comments we receive on our on-board satisfaction questionnaires. We have found that this coding adds to the reliability of our scanned data, as well as providing insight into the actual cause of problems."

In general, Goldfarb says there aren't really any special things to consider when writing a questionnaire that will be scanned, although certain kinds of questions may take up a lot of room on the form. "If you're going to do any kind of age or geographic segmentation, scannable systems do tend to take up a lot of space on your questionnaire because you have to give a full array of numbers for people to fill in. Whereas on a keypunch form you can just ask them for their ZIP code and it only takes up a line. That is something that we've contended with although there are some new scannable systems that can now read hand written material and we're looking at these systems in order to avert these limitations."

Ben Pine, president of Pine Company, says that the next advances in scanning will be in improving the systems' intelligent character recognition, to allow for reading open-ended questions.

Pine says that his company usually sticks to handling the printing and processing of the forms and doesn't develop the questionnaires but will offer suggestions if clients need help. "Generally we want our clients to develop the questions. We'll type them in and print them out and make them look good. But most of our customers are sophisticated research people and they know how to develop a questionnaire.

"We don't get into the analysis of the data but we help them to see things and make suggestions on how they might look at their data so they get something that's actionable and not just stacks and stacks of paper."

Tips for prospective buyers

For researchers considering buying an in-house scanning system, Goldfarb recommends asking manufacturers to do a test processing run of survey forms on their machines. "There are a number of different scanning systems on the market, ranging widely in cost. Although most are very similar in their operation, there are differences. You should opt for the scanner that you feel most comfortable with and, obviously, the one that best fits within your price range."

Goldfarb also suggests that researchers buy a system that appears to have more features than may be needed initially. Chances are you'll grow into it. "The features of scanners vary across makes and models. Although we don't necessarily recommend that you purchase the most expensive machine--we certainly didn't--we do recommend that you consider buying a machine that has more features than you might see an immediate use for, because your appreciation of these features will grow as you become more familiar with the advantages of scanning technologies."

And, if possible, try it before you buy it. "When we went through our last round of purchasing this new scanner we had the companies lend us their machine for two week periods. Otherwise, it's very difficult to differentiate, because when the companies come in and do their demos all the machines look great. Having them there for a longer period of time gives you a better feel for how the machine performs.

"The initial setup of a scanning system can be somewhat frustrating as it involves a number of complexities," Goldfarb continues. "For example, scanners themselves basically all produce a similar type of data file. Generally this is an ASCII file that can be read by whatever type of software you use to process data. We have had an application written for us that produces our performance reports at the touch of a button. However, it took us a bit of time to have the application tailored so that it gave us exactly what we were looking for. Further, in addition to the scanner itself, each scanning company has its own software package that is used to define the way that the questionnaires are read. Insuring that this form definition is correct is sometimes a difficult process."