The impact of gender in e-mailed survey invitations

No need for a woman’s touch?

Editor’s note: Philip Derham is a director of Derham Marketing Research Pty., Surrey Hills, Victoria, Australia.

In the February 2007 Quirk’s article “Does the survey sender’s gender matter?” Stefan Althoff reported that when e-mail invitations to participate in some German industries’ online surveys appeared to come from a woman, the response rates were better than when men had sent the invitations. He also noted that this difference had not been replicated in general-community online survey trials.

These initial German findings suggested two gender-related hypotheses which we subsequently tested in Australia. The first and obvious hypothesis was whether response rates from e-mailed invitations to participate in online surveys were higher when the e-mails are apparently from a woman.

The second hypothesis was that women answer online survey questions less openly than men, because of a lesser willingness to expose themselves in a more assertive Australian male culture.

If either hypothesis were valid in Australia, changes in marketing research practice would be needed for better data collection and strengthened online survey response rates.¹ Hence, it seemed appropriate to test the hypotheses to determine the impact, if any, of gender.

The summary findings are that in the Australian context, the gender of the e-mail invitation-sender does not influence the response rates achieved in online surveys, and in online surveys, women answer questions as openly as do men. Additional study is recommended, as there is room to test the hypotheses further among specific audiences and topics.

Hypothesis 1: Australians respond better to an e-mail invitation from a woman.

On first view, the hypothesis that Australians would respond better to a woman’s e-mail appears reasonable. Australian men can appear more threatening than women - we know that four in five Australian murders and four in five Australian assaults are undertaken by men, making men actually and apparently the more dangerous sex.²

Conversely, gender in e-mail invitations may be an irrelevance as the e-mail appears on a computer screen as gender-neutral text, and many spam-type e-mails purporting to be from one person are commonly thought to have been sent by someone else.

To test the hypothesis, we used clearly gender-different first names in e-mail invitations to participate in four test surveys conducted between October 2007 and June 2008. The first three surveys were of women invitees only and the fourth test survey was with a larger female and male universe.

In accordance with the Australian Market and Social Research Society’s Code of Conduct, the e-mail addresses used were those of real people and the gender-identifying names used are, in the Australian culture, mainstream names.³

Women-only surveys

The first three online surveys were of respective samples of 3,500 women, 3,600 women and 1,000 women aged 25 to 54 years, who were invited to participate in a shopping center café survey. Each woman received one personally-addressed e-mail invitation, had eight days to respond and was offered the opportunity to enter a competition to win one of 13 individual $A100 cash prizes.

It was expected that the only known element was a subject-line note that the e-mail was being sent to people who had completed a First Direct Solutions survey some time before receiving this survey invitation. (It was considered unlikely that the respondents would have heard of the research company or of the researchers before the invitation e-mail.)

Each sample was divided into state groups and each state’s sample was divided on an A-K and L-Z basis so that in each state, half the sample received an e-mail from the named female researcher and the other half received an e-mail invitation from the named male researcher. To avoid any possible alphabetical order or state bias, the alphabetical order of sample e-mailed by the female researcher in the first survey was e-mailed by the male researcher in the second survey.

The first two surveys used different survey links so the identity and gender of the e-mail sender was clear. The third survey used one link and used the respondents’ e-mail addresses to determine whether the female or the male researcher had invited the respondent to participate.

The gender of the e-mail senders was confirmed when the recipients opened or clicked on the invitation, as they were asked to advise the female or the male researcher, by name, that they had received the invitation.

The results were interesting. The first survey (undertaken in October 2007) had a one-in-four response rate, and the second and third surveys were undertaken shortly before or after the Christmas-New Year holiday season and had one-in-five response rates. The differences between the response rates from e-mails sent by either researcher were statistically insignificant, as Table 1 shows. Each survey’s results were within standard error (2 percent and 3 percent for the first two surveys and 4 percent and 7 percent for the third survey, at the 95-percent confidence level).

Noticeably, the main influence on response rates appeared to be timing. Surveys dispatched within a month of the Christmas holiday periods (which, in Australia, began on December 25, 2007) had noticeably lower response rates than the survey invitations sent some months before. This is definitely an Australian cultural issue, as December is generally dedicated to finishing the year’s business, school and social activities, to shopping for Christmas and preparing for holidays after Christmas. Then, until mid-to-late January, more are on vacation or in caretaker mode than at other times of the year.

Almost all respondents provided their e-mail addresses, as these were needed to enter the competitions to win one of the prizes. In survey one, 93 percent gave an e-mail address; in survey two, 96 percent; and in survey three, 89 percent. In the fourth (male and female universe) survey, 94 percent of women and 94 percent of male respondents provided their e-mail addresses to the male researcher and 91 percent of men and 94 percent of women gave their e-mail addresses to the female researcher.

These high e-mail address response levels indicate the respondents were unconcerned about providing their e-mail addresses to a male or to a female researcher, indicating little concern for the gender of the e-mail invitation to participate. Once the decision to participate in the survey had been made, the sender’s gender appeared to be irrelevant.

A female and male sample

A fourth survey test was undertaken in June 2008 with people who lived near shopping centers. The net universe of 21,900 prospects was drawn from the First Direct Solutions panel previously used and a third of the sample were men (31 percent).

Each invitee was sent a one-time e-mail invitation and in this survey, the panel source was not indicated in the subject line. Instead, the subject line stated the e-mail was about a “Food Court Survey - with cash prizes draw!” The e-mails were again personally-addressed to each prospective respondent. (By title, first name, family name, home suburb, Australian state and post code and addressed to them as Dear Title and Family Name.) The e-mail letters were signed by either the female or the male researcher, with the same e-mail and receipt details as before.

There was one unintended difference between the two samples in this fourth survey. For dispatch ease, both samples had been grouped by the recipients’ home state and were sent out in mail-merged small batches of less than 2,000 e-mails per time. During the tedium of sending, as each batch was sent separately, the invitation line was inadvertently changed.

Three quarters of the invitations from the female researcher were sent with the planned “Food Court Survey - with cash prizes draw!” subject line but a quarter were sent with the residents’ home state shown at the start and all the male researcher’s invitations were sent with the home state (e.g., NSW or New South Wales) shown at the start - e.g., “NSW Food Court Survey - with cash prizes draw!”

This difference, rather than the sender’s gender, seemed to affect the response rates between the two senders, as Table 2 shows.

The three-percentage-point difference in response is greater than the standard error would indicate (a one-percentage-point difference between the two would be expected). This difference appears to have been driven more by the state localization of the survey subject line as both the male and the female senders received almost identical numbers of unsubscribe requests and other e-mails from the people contacted, as the next section notes.

Larger universe

The shopping center customers’ survey test had a larger universe of invitees and also had a larger number of responses than the earlier three surveys. The larger number was in surveys completed but also in the number of unsubscribe requests received and in e-mail correspondence from invitees.

The researchers responded to all unsubscribe requests, acknowledging that the request would be honored, and responded to all other e-mails with appropriate answers. In doing so, it seemed that more e-mails came from recipients of the female researcher’s invitations and were often chattily addressed.

When analyzed, there was minimal difference between the researchers in unsubscribe requests received or in e-mail correspondence, as Table 3 shows.

This lack of gender difference in the shopping center survey unsubscribe or e-mail correspondence rates suggested we review these measures from the earlier surveys. As Table 4 shows, there was essentially no difference in unsubscribe or in the e-mail correspondence rates.

Not an influence

The responses to invitations from a male or from a female e-mail address indicate no online survey response rate differences with the samples tested. This then indicates the e-mail invitation sender’s gender is not an influence in the decisions respondents make whether to complete online surveys - at least for these topics and with these samples.

The survey results suggest that e-mail invitations appear genderless as electronic text on electronic backgrounds on computer screens. Factors other than the sender’s gender influence decisions to participate in a survey. In particular, the impact of inviting respondents to participate in a survey about their local (state) area seems to generate better responses and future testing will look to that as a way to strengthen response rates.

The further analyses of unsubscribe rates and of correspondence with the researchers also found no difference by the inviting e-mail sender’s gender.

Hypothesis 2: Women respond differently to men.

In testing whether women answered differently to men, we looked for a question that would be commonly asked of both and would be independent of gender-related behavior. Bradburn’s findings that respondents in self-completion surveys underreport answers to questions that make them feel uncomfortable⁴ suggested such questions could be logical tests for gender differences.

As our previous online surveys had not asked which, if any, questions respondents were uncomfortable answering (the absence of this does suggest a possible future project!), the “What is your total annual household income?” question was considered likely to be an uncomfortable question. This was because clients often expressed concern that it was intrusive, and Australian face-to-face interviews often use pre-coded answer sheets that give a letter or number for each income range, so respondents need not state an actual dollar amount or range. The assumption was strengthened when we found that 15 percent of recent telephone respondents and 10 percent of recent mail survey respondents refused or did not answer when asked for their household income (based on findings from three recent Derham Marketing Research community and three customer audience telephone surveys, and four recent postal self-completion surveys).

Responded differently

The online surveys analyzed for household income response differences all had optional household income questions. Answer places were shown but respondents could proceed to the next screen without answering (though respondents had not been told that). The results were analyzed by respondents’ gender to see whether men and women responded differently to the presumed uncomfortable question about household incomes.

The first online survey results reviewed were from a sample of 6,420 Australian financial institution customers and the one-time e-mail invitations were sent from an unknown and gender-neutral e-mail address (the sender was derhamp@derhamresearch.com.au). The e-mail subject line said the e-mail was an invitation to participate in the financial institution’s customer survey and participants could win a prize.

When the e-mails were opened or viewed in a reading panel, they were seen to be on the financial institution’s letterhead and signed by the financial institution’s male chief executive officer. When the readers moved their cursors to another e-mail or sought to close the e-mail, they were asked to acknowledge receipt of the e-mail to a male researcher.

The non-response to the household income question differed minimally between men and women. Only 3.7 percent of men and 4.1 percent of women chose not to answer the household income question - the difference of 0.4 percent was well within standard error (1.8 percent at the 95 percent confidence level). This study suggested gender had little influence on household income answers.

The second online survey was of 4,350 customers from another Australian financial institution but differed because a three-contacts approach was used⁵ to generate response. The same e-mail dispatch and e-mail receipt procedures were used and again, the e-mail components were mostly from men.

The non-response to the household income question in this second financial institution’s sample differed markedly from that of the first study. In the second study, 32 percent of men but 40 percent of women chose not to answer the household income question - an 8 percent difference and a significant difference (the standard error was of up to 5 percent).

This second study suggests women and men do answer questions that may cause discomfort differently or the women surveyed may not have known their household incomes. The survey had not asked why the household income question was not answered.

The findings of the two small-sample studies were contradictory and thus inconclusive. The third study reviewed was of a large-scale survey. In that study, 20,000 Australians were invited to participate in an online survey.⁶ The e-mailed survey invitation was sent from a gender-neutral and unknown-to-the-recipient e-mail address. The subject line said the e-mail was an online survey invitation, named the panel source (First Direct Solutions), and said there were potential prizes of cash, iPods and cinema tickets.

When the e-mail was opened or seen in a reading panel, the sender was identified as a male researcher and when the cursor was moved or the opened e-mail closed, the recipient was asked to acknowledge receipt to the male researcher.

In this larger survey, 0.2 percent of men and 0.4 percent of women chose not to answer the household income question. The 0.2 percent difference in response from women was well within the 1.7 percent standard error expected. This finding in itself begins to question conventional wisdom that the household income question is an uncomfortable question, when the assumption is made in relation to these online surveys, where the household income question is asked as the one question on its own screen, as was done on these surveys (Figure 1).

The fourth study reviewed was the June 2007 shopping center survey, with its male and female universe. In that survey, 4 percent of men and 5 percent of women did not answer the household income question when asked by the male researcher; and 4 percent of men and 6 percent of women did not answer when asked by the female researcher.

The two larger-universe survey findings suggest no gender difference on response to a possibly uncomfortable question. So the mostly similar (with one discordant) results from these four studies suggest gender is probably not a factor in not answering questions about household income.

Not to be valid

The first hypothesis - that Australians respond better to an e-mail invitation from a woman to participate in an online survey - was found to be not valid for the respondent groups tested. Rather, the survey results indicate other factors influence decisions to participate in a survey. In particular, the impact of inviting respondents to participate in a survey about their local (state) area seems to generate better responses and future testing will look to that as a way to strengthen response rates.

The further analyses of unsubscribe rates and of correspondence with the researchers also found no difference by the sender’s gender.

The second hypothesis - that there are gender differences surrounding the answering of uncomfortable questions - was not substantiated. Rather, in general, men and women appear to answer uncomfortable questions in online surveys at the same level. In addition, the gender of the e-mail invitation sender was not a factor affecting the level of answer given.

Hence, the overall conclusion is that gender has no impact on online survey response rates or on the decision to answer uncomfortable questions among Australians. Other factors - time of year, localization, topic, and not-tested factors such as survey design - are more likely to influence the response rates and question answers.

Tested further

The conclusions drawn are based on Australian finance industry customers, women aged 25 to 54 years, shopping center customers (male or female) and on a general-population sample that was researched about financial affairs. Hence, it may be appropriate to test these findings in other industry sectors, about other topics and with senders from minority groups, to be sure of the conclusions reached.

A more promising area for immediate response improvement seems to be named geographic proximity. That will be tested further.

The findings reported in this article expressly relate to Australia and to the online survey method and we will look at the relevance of these findings to other research methods in the future and would welcome other researchers’ findings on those and for other cultures.

Notes

¹ The Australian Market and Social Research Society Professional Code of Conduct requires all contact be from a legitimate and real person. Having a male or a female sender of an online survey invitation would be, for most Australian marketing research companies, an easy matter to manage, as the professional and managerial elements of this sector are almost equally divided between males and females, according to government statistics.

² Criminal Courts, 2006-7. Australian Bureau of Statistics, January 28, 2008. Ref. 4513.0.

³ To comply with the Australian Market and Social Research Society Code of Conduct, respondent contacts must be from identifiable, contactable and real people, so dummy names could not be used. The sender e-mail addresses used in all tests were sue.derham@derhamresearch.com.au (the female researcher) and philip.derham@derhamresearch.com.au(the male researcher), as in the Australian culture, Philip is a man’s name and Sue a woman’s name. As the researchers’ names are mainstream, the findings relate to mainstream gender names, and may or may not also relate to the findings that could follow from the use of names associated with specific ethnic or religious groups. Further research on this may be advisable.

⁴ “Question Threat and Response Bias.” Norman M. Bradburn, Seymour Sudman, Ed Blair and Carol Stocking; Public Opinion Quarterly, 1978.

⁵ The three contacts were a pre-survey notice that the survey would follow next week, the survey invitation and a “quick, the survey finishes shortly” reminder a day before the survey closed.

⁶ The sample had been drawn in proportion to the population, by Australian state and by age groups within each state. Each sample had a 60 percent female component. The end sample matched the state, age and gender distributions of the original universe invited to participate.