Editor's note: Tim Macer, managing director of U.K.-based consulting firm meaning ltd., writes as an independent software analyst and advisor.

As research companies struggle to reconcile the demands of clients to work faster and cheaper with their own desire to do the job well, maybe we should be looking more to technology to cut a few corners – safely, of course. Testing surveys, for many researchers, is a pain. It demands time when you don’t have it and there is always a nagging doubt that, despite your best endeavors, something serious could slip through. Testing is a cost, but not testing adequately is potentially even more expensive.

The tools most survey platforms provide for testing surveys vary from rudimentary to non-existent, which is why Knowledge Navigators, a research services and technology company based in Europe, decided to fill a gap with a descriptively-named tool called SurveyTester. It is a Web-based tool that allows survey programmers, researchers and even clients to collaborate on the task of testing an online survey, finding and reporting errors, then, once fixed, re-testing it. Importantly for a tool like this, it is platform-independent – though it does need to be trained by the technicians at Knowledge Navigators to work with a survey product for the first time. Currently SurveyTester supports Confirmit, IBM Data Collection or Dimensions (as it is still known by many), Kinesis, Lime Survey, Firefly and Rogator. I am told it usually takes only a day or two for the developers to add another package.

SurveyTester provides a test environment within your preferred Web browser and then expects human testers to put the survey through its paces, just as you would normally if testing from a test link. There is nothing to change within the survey platform, so you are testing the survey very much in the same conditions that participants will receive it. At a technical level, what SurveyTester does is capture and interpret the HTML code issued by the survey platform and in particular, the header information that contains useful information for survey testing such as the question name. This is why SurveyTester has to be trained to work with different packages, as this metadata varies between products.

What testers see is a special SurveyTester bar at the bottom of the screen, which provides various tools as well as the ability to record comments to the programmer to make changes. There is also a button to auto-fill the question on screen and move on to the next, which in itself can save a lot of time when testing. The tool will not auto-conduct surveys for you. You still need to view each screen but this is not unreasonable as the problem it is setting out to resolve is that of making manual testing easier. This can complement the auto-testing facility that many survey tools already contain, where the survey platform will generate 100 or 1,000 random interviews which you can crosstab to look for discrepancies. Instead, SurveyTester focuses on allowing humans to look carefully at each question to ensure it meets the spec and contains no other glaring errors.

If a tester notices an error, it is a single click on the special toolbar to call up the error-reporting dialogue. You pick the class of error from a drop-down list and simply type in what the problem is that you’ve noticed. Very usefully, the system is able to record the point in the survey where you are and the question name that the programmer needs to locate the correction. It also gets the tester to record the type of error from a drop-down list. This drop-down list is user-definable by the SurveyTester administrator and each survey can potentially be given a different one. Normally, a company is likely to have several template taxonomies for different types of survey or different clients.

Every time a problem is recorded, SurveyTester also automatically snapshots the screen the tester was viewing. In fact, it does more than this: It records every screen viewed, whether or not it was commented on, both as an image and as the HTML. It then runs the HTML again through a variety of other devices that Knowledge Navigators has attached to its servers, which includes a variety of different smartphones and tablets. This means that when the feedback is being reviewed by the survey programmer, he or she can see not only how each question was rendered at the time the tester saw it but also what it looks like on any common smartphone or handheld device.

This screen-capture capability is an invaluable tool in responding to the challenge of the rising tide of participants taking online surveys on their smartphones. Knowledge Navigators has ruled out using simulators, partly because this is something users can already do in their testing and partly because they are not always reliable. However, I’m not fully convinced how their physical solution will scale as more customers come on board and more devices come out, and it can lead to a backlog in rendering time and the screenshots being available because they are processed in the background.

Anything that makes the testing of mobile surveys easier is a good thing and this goes a long way in allowing each screen to be viewed as it would appear on different devices. It does not, however, reproduce executing the survey on the phone – the devices are only used to render the captured HTML for that screen. It does not help with testing surveys on mobile apps – and using this approach, it could not – as these are not delivered as HTML. You can, of course, use SurveyTester on a tablet and a smartphone; however, the testing tools have not been optimized for smartphones so it is not really usable in that mode at present. It is something Knowledge Navigators will look at for the future.

So far, I’ve focused on the role of the tester and what he or she sees but SurveyTester supports other roles, including administrators, project managers and internal testers. As with the error definitions, you can also define your own roles.

The project manager role is aimed at the programmer or scripter and provides him or her with access to the all of the error reports, which now become actions to resolve. In doing so, scripters have access to all the screen-shots and also the ability to reproduce the tests that the tester originally performed. SurveyTester does not record the keystrokes but the actual data entered, so a previous test run can be re-run instantly to any point in the survey.

 

Project managers can provide feedback to the tester, seek clarification or provide confirmation that a problem has been resolved by adding in replies to comments. It means that SurveyTester can be used to centralize all the communication between programmers and testers, regardless of where they are located.

The SurveyTester user experience

TNS Infratest in Germany makes extensive use of SurveyTester when fielding any new online survey. Rudi Hirmke, global sourcing professional at TNS Infratest, says that over 100 individuals have used the tool since its introduction last year. “Everyone involved in testing a survey uses it – researchers, scripters for their tests and clients on one or two occasions too,” he says. It is also used to coordinate testing between Germany and TNS’s technical team in India.

Hirmke found it hard to put a finger on all the time TNS Infratest has saved. “If I had to guess, I’d say it was between a third and a half of the time [spent testing surveys].” In his view, “One of the real advantages of the software is that you can reproduce your test, so if you have found a bug and you want to go straight there you just click or reuse the test run and you have the same answers. Before, you had to start from the beginning and you were never sure if you had picked the right answers. If you are testing a 30-minute interview and the bug is near the end, then that can be 30 minutes each time you test it just to get to that point.”

He has also found that managing the process of correcting errors becomes more efficient too. “It is really helpful that you have one place where all of the changes are documented. Earlier we were using different e-mails and files so you were never sure everyone was on the same page.”

SurveyTester can also provide a closed-loop feedback mechanism to reduce particular classes of systematic errors. “One of the big opportunities you have is in defining the categories of errors in Survey Tester,” Hirmke says. “If people consider them carefully, [it means] you can have a look where you most often have problems with meeting the requirements. For example, if you find that 80 percent of the corrections are layout changes, you can discover where you might be able improve the process or improve the skills in that area.“

He concludes: “If you want to have everyone communicating all their changes in one place, there is no better way. This is the best tool for it, end of story!” 

Sidebar

Pros

  • Consolidates in one place all messages from all your survey reviewers
  • Can save hours of time when repeating tests
  • See how your survey will look on mobile devices too
  • Can improve quality over time by identifying commonly recurring errors across project

 

Cons

  • Only works with browser-based surveys – does not support survey apps
  • Support currently limited to six different survey platforms
  • Not optimized for testing on mobile devices

Pricing

  • Pay-as-you-go pricing, in euro, from €150.00 (approx. $165.00) per project. Annual fixed rental pricing also available depending on volumes.