Considerations in quantitative data collection in China

The rules are changing

Editor’s note: David Ying Hon Ho is research director (APAC) at Market Strategies International, a Livonia, Mich., research firm.

Commercial market research is a relatively recent industry in China: The first Chinese market research provider, Guangzhou Marketing Research, just celebrated its 20th anniversary in 2008. As a young and growing industry, operation practices have undergone significant changes. In this article, we lay out considerations in data collection of quantitative studies specific to the Chinese market, with a focus on the technology market.

Suit the local context

All data collection methods, such as door-knock surveys, central-location tests, mail surveys, phone interviews and Web surveys, exist in China today. What differentiates China from other countries is the popularity of certain methods and how those methods are adapted to suit the local context. The three most-common methods are described below.

Face-to-face interviews

Face-to-face interviewing has been the default mode of data collection from the beginning of commercial market research in China. However, because of the growing popularity of phone and Internet use among businesses and consumers, more studies are now implemented by phone and by Web. Face-to-face is, however, still widely used in China and has its merits in at least two situations.

Under the Confucian influence, showing respect is necessary to successfully interview an executive in a large company, a medical doctor or a quasi-government officer in a state-owned enterprise. The face-to-face interview is the only method that offers the respect these respondents expect. They tend to demand human contact, and interviewers must employ proper identification, business cards, documents, invitation letters and, most importantly, politeness and respect. This is in big contrast to Western customs. In the U.S., for example, senior business executives prefer to be interviewed by phone for the purpose of saving time and simpler logistics.

In another situation, face-to-face interviewing is the norm for data collection and quality control practices for local field providers in second- and third-tier cities. Different geographical regions of China exhibit wide discrepancies in their economic developmental stages. Prices could vary significantly across city tiers. Compared to metropolitan Beijing, Shanghai or Guangzhou, low labor and rental costs in lower-tier cities mean that Web and phone surveys offer little cost advantage over face-to-face. Also, field providers in small cities have their practice more firmly built upon traditional methodologies.

Phone interviews

The prevalent mode of data collection is phone interviews, especially for business-to-business research in China. Most Chinese research companies are now equipped with a CATI center. The general pros and cons of phone interviews as a method apply to China the same as anywhere else. Here, we would like to bring up a special consideration in China, which is China’s heterogeneous linguistic scene.

The widely-accepted view is that Chinese is a single language with various dialects. However, some linguists view Chinese as a language family with numerous mutually-unintelligible languages in it. We do not plan to explore this academic issue here but, for practical purposes, it makes better sense to consider Chinese as a family of dozens of languages: Mandarin (Guanhua), Cantonese (Yue), Shanghainese (Wu), Taiwanese (Min, Hokkien), etc.

Phone interviews in China are often conducted in Mandarin, the standard national language. One practical problem is when a Mandarin native-speaking interviewer pronounces a brand name and the non-native Mandarin-speaking respondent does not always understand. In oral communication, when unfamiliar names or words are heard, differences in accents must be considered. In the mid-2000s, we had a CATI project across China. In the interview, the interviewer and the respondent discussed a number of software brands and product names. Unfortunately, in the technology market, these names are often in English and not translated into Chinese. Local pronunciation of English names can vary considerably. As a result, a significant portion of the interview time was spent on the interviewer and respondent trying to understand what the other party was saying. We had also seen interviewers spending more than a minute trying to understand a single response due to similar language barrier. In some cases, an answer might have been mistaken after all. In response to this language issue, well-trained interviewers and experienced field providers are particularly important in China, as opposed to markets like the U.S. that are linguistically more homogeneous.

The adoption of mobile phones in China has been growing at a double-digit rate. In early 2009, according to the Ministry of Industry and Information Technology, the national penetration was about 52 percent. It has exceeded 100 percent in the biggest cities but remains below 30 percent in rural areas. The research industry has been thinking about conducting research via the mobile phone platform in first-tier cities of China. However, cold calls or text messaging that targets the mobile platform is still largely unacceptable to respondents. While there have been sporadic attempts and experiments to do quick and simple studies, no appropriate methodologies have yet been created by researchers.

Web surveys

Many Chinese researchers have serious doubts about implementing Web surveys in China because of skepticism on sample representativeness and the impossibility of respondent authentication. We will discuss the sample issue in the next section of this article.

Regarding authentication, one cannot be certain that the recruited target respondents are filling out the Web survey themselves. Proxy respondents, usually subordinates of busy executives and IT managers, are known to be asked by their bosses to fill out surveys. In response, some research companies discourage the use of Web surveys. Others research companies might go back to some form of phone or face-to-face interviewing to establish the identity and legitimacy of the respondent. The task starts by recruiting respondents by phone and finishes by inviting them to a self-administered Web survey.

In the last few years, the trend in China, compelled in part by international research standards established in developed markets, is to do research online. As the Chinese research market matures, we expect that more studies will be administered online. And as more questionnaires become complex or require visual stimuli, Web surveys will become more frequent and advantageous. In the meantime, stringent quality control and data cleaning procedures are the remedy to current quality concerns.

Three broad areas

Sample is generally sourced from three broad areas: client lists, public sources and field provider proprietary databases.

Client lists

In studies that target the research client’s existing customers, the sample is sourced from a client-provided list extracted from a customer database. It is a frame-based sample source from which a probability sample can be drawn. The precondition is that the list must be updated and accurate, which unfortunately is often not the case in China. When employing customer lists in China, we find that it is best to assume a higher rate of “non-working” customer information compared to mature markets. Why Chinese lists are less accurate is not well-studied by the industry yet. We advise researchers to buffer during sampling in this market.

Public sources

Because of certain past criminal incidents, the Chinese government does not allow the publishing of a comprehensive listing of phone numbers. Residential phone directories basically do not exist. While business phone directories are available in most cities, these directories represent mainly, or only, paid listings and exclude phone subscribers who are not advertisers. In other words, only Yellow Pages-style directories exist. Random-digit dialing is the remaining sampling strategy more widely used in CATI centers in China. Despite being a legitimate method, it should be noted that the preparation of the sampling frame and the configuration of dialer software are not carefully considered in some call centers. This is another pitfall researchers have to pay attention to.

Commercial databases are published and sold by business information and intelligence providers such as Dun and Bradstreet. For business-to-business studies in developed countries, these databases are important resources. In China, given the socialist legacy of complicated organization of state-owned, quasi-state-owned and other businesses, many of these databases have a far-from-ideal coverage of the target industry. Directories published by companies affiliated with official bodies, such as the National Bureau of Statistics, or industry associations are usually biased, as they are likely to contain only “related” companies. And the rapid change in China means these databases are usually outdated by the time they are in wide circulation.

In China, panels are widely used in fast-moving consumer goods research but less so in other product sectors. This is partly because consumer goods research makes up the bulk of all research activities. Technology research panels and sub-panels do exist, but many of these panels are nascent and growing. Panel providers generally do not disclose the exact procedures or history of panelist recruitment, and the benchmarks of quality panel maintenance (and thus the “representativeness” of the panel) remain murky.

A significant concern in Internet and qualitative panel samples is identity fraud. It is commonplace for respondents to participate in research that they are not qualified for and thus earn incentives as their full-time job; sometimes contract researchers participate in a kickback scheme that benefits both the recruiter and the “participant.” Identity fraud exists in all markets, but given the current Chinese economic and social context, organizations such as the China Marketing Research Association have identified fraud as a more serious problem in China than in many other markets.

Field provider proprietary databases

As with other markets, each field provider in China aggregates contacts of people to form a proprietary respondent database that can be used for various research modalities. How this database is built is often unclear to the research buyer, and field providers are often unwilling to provide specifics. Network sampling seems to be the usual mechanism: A field provider accumulates former respondents from projects it has conducted historically or recruits potential respondents via other channels. Sometimes a search is conducted on an ad hoc basis based on each project’s specifications. Because respondents are highly interconnected (A refers B, who refers C and D, etc.), no probability sample can be obtained and the selection of respondents in the pool essentially comprises a convenience sample.

Mix and match

In practice, under time, pressure and cost considerations, research providers in China tend to draw a sample by combining various sample sources mentioned above. This reduces risk by ensuring that if any one sample source has significant flaws, it is only part of the picture - not the sole source on which an entire project is reliant. While this approach has pragmatic appeal, it raises serious questions. Can we combine sample sources in a statistically-acceptable way? If not, will choosing and relying on one particular sample source generate a heavily-biased sample? Can we measure how biased this resultant sample is?

Seeing the coexistence

As the China market research industry enters its third decade, we are seeing the coexistence of several common modes of data collection, with strong growth in online methods. On the surface, China supports the same sampling and data collection methods used in developed markets, yet the day-to-day “rules” for operationalizing these methods can be completely different. China will continue to face the challenges discussed in this article, and research practitioners will need to be diligent in updating their understanding of the market and customizing their studies accordingly.