Hard data on software

Editor’s note: Tim Macer, managing director of U.K.-based consulting firm meaning ltd., writes as an independent software analyst and advisor.

Almost nine out of 10 research companies in North America today use Web survey software, with the rest of the world only a few percentage points behind this now. Around 12 percent of research companies have written their own data collection software, but a staggering 74 percent have developed their own panel management software rather than use an off-the-shelf solution. And across the industry, both by size of company and around the globe, practices vary widely with regard to mixed-mode research, approaches to electronic data delivery and traditional crosstab reporting, and the more mundane activities of editing and cleaning data.

These and more are among the findings of the 2005 Confirmit Annual MR Software Survey which we publish here in Quirk’sfor the first time. Norway-based research software maker Future Information Research Management, or FIRM, who commission and sponsor this research, have provided their kind permission to present these results here. The commentary and interpretation, however, is editorially independent of the sponsor.

The survey is based on an invited sample of senior decision makers at research companies around the world. From an initial sample of 800, 212 participated, with 64 in North America, 117 in Europe and 31 in Asia-Pacific, which is a reflection of the differential response rates in each region. Despite several polite reminders, it seems researchers are becoming as resistant to survey invitations as the rest of the world.

Companies of all sizes were well represented in the achieved sample, with 53 with revenues of under $1 million (small companies in our analysis) and 44 in our large category with annual revenues exceeding $24 million. Neither was the sample dominated by IT specialists, as we were keen to represent the views of both research and operational decision makers. Thirty-nine percent of the sample was at board or senior vice president level, and 48 percent considered themselves to have primarily a research responsibility, against 18 percent IT or DP and 34 percent business or operational. The survey was carried out independently and anonymously by my company, meaning ltd., to professional research standards, adhering fully to the Market Research Society code of conduct.

If you would like to join our confidential panel of technology-aware researchers for our next round of industry research in 2006, please send me an e-mail (tim.macer@meaning.uk.com ).

Types of software utilized by research companies

Three categories of specialist research software have achieved virtual ubiquity in research companies worldwide: software for online surveys, for analysis, and for presentation or publishing of results. While two in every three research companies worldwide may have the capability to do their CATI research, only one in four can provide a CAPI service. However, nine research providers out of every 10 today have the means to field Web surveys.

North America’s research firms still have the edge over Europe on Web survey capabilities, at 89 percent versus 81 percent, but companies in Asia-Pacific are neck-and-neck with North America, at 90 percent. Asia-Pacific also leads in the prevalence of panel management software, where 65 percent of companies have acquired or built these tools, as opposed to 44 percent in Europe and just 36 percent in North America.

CAPI is well established as an important channel to respondents in both Europe, where 32 percent of research companies have invested in appropriate technology for the task, and Asia-Pacific, where 29 percent are similarly equipped. This may come as a surprise to researchers in North America, where geography and custom conspire to keep CAPI as the method that failed to take off in the 1990s. A mere 9 percent of research companies in North America today seem to offer any CAPI capabilities.

SMS instant text messaging on cellular phones, surely the most restrictive of all new interviewing modes, is starting to put in an appearance in a few research companies worldwide (4 percent in North America and 5 percent in Europe), though it seems to be more than twice as popular in Asia-Pacific, where 12 percent of the companies surveyed have the software to engage respondents’ thumbs in text surveys.

Developing their own

Given the widespread availability of mature, well-designed and well-maintained software for every research application imaginable, it is perhaps surprising to see the extent to which research companies have developed their own solutions in-house. Many companies use more than one software package for the same task, so we asked first about the origin of all the software they use in each of the four categories, then, where more than one solution was in use, we asked them to nominate which was the “main” package used for each task.

From Figure 2 we can see that the own-grown solution does not always become the software of choice, which implies that software is often developed in-house to supplement shortcomings in the mainstream bought-in packages. This is particularly the case with analysis software, where own-grown tools are relatively common, but seem to be used alongside other tools most often.

Web survey (WAPI) software is the most popular candidate for own-grown solutions, with 19 percent of companies developing their own software, and 15 percent adopting their own-grown solution as their main package. This equates to more than three in four who had the option of an own-grown tool. Given the resources and expertise that are required to develop and maintain software, and both the time and risk that this represents, it is even more puzzling that the survey has identified those most likely to be collecting surveys or analyzing results using self-built tools to be smaller firms - something that was replicated across all four categories of software.

Mixed-mode capabilities

Opinions remain divided on the importance of mixed-mode research, and the need for software to provide support for combinations of CATI, WAPI and possibly CAPI within the same package. The question asked was, “If you were choosing new software, or reviewing your current solution, how much importance would you place on the tool’s ability to mix and combine different data collection modes?” with a five-point scale of answers from unimportant (1) to essential (5).

Overall, 53 percent of the companies surveyed rated such support at the top end of the scale (essential or very important) and 29 percent settled for the midpoint (moderately important) while only 14 percent considered it to be either fairly unimportant or unimportant. However, a significant split emerges at the top end of the scale, with 37 percent of large firms considering mixed-mode functionality to be essential, against 19 percent of mid-sized and a slender 6 percent of small companies.

This is reflected, in more moderate degrees, in the average scores derived from the five-point rating scale, shown in Figure 3, with unimportant scored as 1 and essential as 5.

Mixed-mode support

What is meant by mixed-mode software varies widely from one user to the next, so we also asked respondents to rate where, on a spectrum of capabilities, they considered their needs to fall in the ability to mix modes. This ranged from “common authoring” - simply using the same authoring tool to prepare surveys each designed and administered in one mode alone (e.g., Web-only or CATI-only) through “mixed modes in parallel,” where one survey instrument will be administered in two or more modes, but only one for any given interview, up to the most technically challenging category of mixed-mode surveys where interviews may switch to and fro between modes as demanded by the script or by the interviewer or respondent.

As we saw with the importance rating for mixed-mode support, the level of sophistication required divides along company size lines, with small and mid-sized companies having broadly similar needs. A significantly different set of needs is revealed among the large firms, 43 percent of whom see complex mode-switching surveys as the goal, against 23 percent of mid-sized and 21 percent of smaller firms.

Despite this, the underlying pattern is one of divided opinions about whether mixed-mode should be used to allow simultaneous interviewing across different channels in one survey, or simply to streamline production of surveys each destined for a different but single-mode treatment. This probably reflects the widely voiced though not always well-founded concerns in the industry about the perils of drawing comparisons from research data that derive from fundamentally different interviewing methods.

Panel software

One neglected area of research software development continues to be the provision of tools to manage online or offline panels. In Europe and also in Asia-Pacific, panels are assuming an even greater importance in enabling online research than they have in North America, and there is currently an unprecedented level of activity worldwide in building panels of every complexion and specialty in order to deliver the promise of high-speed, high-quality research-on-demand.

The support offered for panel management in the plethora of Web survey tools available on the open market is far from keeping up with demand, and this is clearly shown in this survey in the extent to which research companies are taking matters in hand for themselves. While around one in 10 companies may be developing their own software for the classic research tasks of CATI, CAPI or data analysis, as we have already seen, and 15 percent - still a minority - have felt it necessary to build their own Web survey tools, that situation is turned on its head with panel management software. Even in North America, the split is 55/45 for own-grown panel software as opposed to bought-in. Elsewhere, in the areas where the demand is greatest, the gulf widens considerably. Seventy-two percent of companies in Asia-Pacific and 81 percent in Europe are going through the effort of building and maintaining their own tools rather than use what is provided.

Report delivery formats

We devoted several questions in the survey to the area of results presentation and delivery to end clients. This is a major growth area being led both by client demand and technological innovation. Though our survey did not attempt to capture figures on the frequency of use of each method, or the volumes involved, our introductory question in this section was: “In which formats are you required to deliver your reports and research results?”

Not surprisingly, Word and PowerPoint dominate as delivery methods (82 percent PowerPoint and 70 percent Word) as these are tools which both research companies and clients are likely to have on their PCs and use every day.

Around half the sample are still delivering printed tables (57 percent worldwide, 53 percent in North America), but nearly the same amount (62 percent worldwide, 56 percent in North America), are using Acrobat PDF to deliver reports and tables: usually the same format but without the paper. Encouragingly, from a technological point of view, the figure for online static reports and tables hovers just a couple of percentage points behind printed tabs (55 percent worldwide, 52 percent in North America).

Providing clients with interactive analysis, either on the desktop or via a Web portal, is still very much a minority undertaking, with only 22 percent worldwide delivering in this format, and as few as 11 percent in North America.

Changes in research delivery modes to clients

Our subsequent questions focused on electronic delivery using specialist research software, rather than standard office automation products. We asked participants the extent to which they considered demand for each delivery method would increase (or decrease) over the coming 12 months. Demand appears to be strongest for online delivery of static reports, with 76 percent overall seeing some increase in demand, and 22 percent seeing a strong demand. Research portals are also considered fertile for growth by research companies, with 68 percent predicting some increase in demand, and almost one in five (19 percent) predicting a major increase in use.

Demand appears slightly softer for interactive analysis for clients, at 63 percent overall for any increase, but with 18 percent forecasting a major increase in demand. Across all three areas, only a small handful in the sample consider that demand for electronic delivery will diminish, the rest predicting that it will remain the same.

Whither the bulk crosstab report?

Researchers are convinced that many clients never look at them, but clients are often suspicious that their agency is short-changing them when they suggest phasing out the deck of tables. Judging by the response to our question on the future of the bulk crosstab report, it looks as if it will take a while for the industry to be weaned off the blocks of everything-by-everything crosstabs. The actual question asked was, “When considering analysis and reporting tools for the future, how important is it that these should be able to produce volumes of crosstabular reports?”

Overall, 55 percent considered this an essential requirement. Importance relates closely to the size of the research company, with smaller companies most likely to consider crosstabs essential (58 percent), against 47 percent of large companies.

It is still only a tiny, brave minority who see no future for bulk crosstabs, and these are more likely to be found in large companies (7 percent) as opposed to small (4 percent) or mid-sized companies (2 percent). However, the same trend applies to those viewing it as relatively unimportant, which stands at a sizeable 26 percent in large companies, followed by 19 percent in small companies and only 8 percent in mid-sized companies. It seems the opinion-formers here are to be found among the biggest and the smallest research companies.

Unusually, there was very little difference between global regions in expectation here, with Europe and America closely mirroring the overall picture.

Cleaning online data

Our next two questions focused on the practice of editing and cleaning data. Software suppliers are often fulsome in praising their software’s benefits in delivering increased accuracy, both by avoiding a separate data entry step and through enforcing logical conditions and validation on screen. The unwary can interpret this to mean that editing and cleaning of data from WAPI has become superfluous.

In our first question, we asked companies who do online research (almost all companies in the sample) the extent to which they edited and cleaned data. It appears that the world is divided roughly 80/20 in favor of cleaning or taking what comes, though only one in 10 admit to never cleaning their data. Opinions seem to differ by company size, with mid-size and large companies engaging in very similar practices: around one-half cleaning all projects, and one-quarter performing selective cleaning.

Small companies seem to take a more all-or-nothing approach, with 64 percent - the highest proportion of any group - routinely cleaning all projects, and 19 percent, also a much higher proportion than other groups, leaving their data untouched, or 28 percent if you include those who rarely clean data too.

There were some interesting global variations. North Americans are more likely to clean all their projects (69 percent), with only 14 percent rarely or never cleaning projects; the equivalent figures for Europe are 54 percent and 27 percent respectively.

Editing capabilities

It is possible that some of the differences in editing practice already observed stem from the variations in the functionality offered by the different software packages used. Some provide a broad range of editing tools while others expect users to tip out their data into an uncontrolled environment such as an Excel spreadsheet or an Access database in order to make changes. We therefore asked companies what editing and cleaning capabilities they looked for in their research software overall. Both batch cleaning - the ability to set cleaning rules and then apply them automatically to data - and the ability to preserve the raw data when making changes are virtually universal requirements (76 percent and 72 percent respectively), especially in North America (84 percent for both batch cleaning and preserving raw data), which we have already seen is where companies are particularly hot on cleaning their data.

Even though so few software packages actually provide the capability to roll back changes and wipe out erroneous edit changes on demand, or provide an automatic audit trail of the actual changes applied, demand is still strong for these features: 57 percent of companies globally and 67 percent in North America want roll back and 51 percent (61 percent in North America) would like to have an automatic audit trail. It is an area of unmet demand in much of today’s research software.

Accessibility requirements

Any research company engaged in public or social policy research today is likely to be aware of growing demands from their clients to ensure that online surveys are accessible to respondents with disabilities, and possibly the published results too. We therefore asked research companies how important it was for them that the software they used complied with disability legislation (such as Section 508 in the United States), or allowed surveys to be constructed to AA or AAA rating under the W3C accessibility standards.

Worldwide, a slim majority (54 percent) considered it either essential or moderately important. Perhaps surprisingly, the reverse is observed in North America, where 58 percent considered accessibility in online surveys, analysis or published results to be unimportant and a slender 8 percent viewed it as essential. Company size had very little influence on consideration, but respondents who described their job functions as either IT or DP were less equivocal. This is one group that would be acutely aware of the difficulties in making a survey Section 508-friendly when the tool is not designed to do this and 61 percent of this group thought it was moderately important, with a further 12 percent considering it essential. And although several of the major software suppliers are now working to ensure that their products are disability-friendly, support across the range of products on the market is patchy and too many would still fall at the first hurdle if put to the test.

Information sources

Our final question was, “When you wish to find out about new research software, what sources of information do you use?” Unfortunately for those responsible for marketing among the software providers, the top two are channels on which they can only hope to exert an indirect influence: word of mouth comes in on top at 76 percent, and software reviews second at 62 percent. Yet the fourth, cited by 53 percent, is the downloadable software demo, which should make this a must for the many software providers that still do not let you take a peek at their software or even a screenshot before tangling with the sales team. Yet the personal touch clearly does have its place: a sizeable 45 percent of decision makers find meeting with exhibitors at conferences and trade shows a good way to keep up with the latest software.

Any software company considering axing the ad campaign needs to think again on these figures: the 29 percent using ads in publications should be viewed as those who actively seek them out, as opposed to the much larger figure who are subject to their influence in a more passive way, which is not something that this survey could measure accurately.