The value of small sample sizes
Editor’s note: Nicola Vyas is the senior research director at Healthcare Research Worldwide. The HRW team can be reached at info@hrwhealthcare.com. This is an edited version of an article that originally appeared under the title “Small is Beautiful: Optimising Quantitative Approach in Healthcare.”
If you have previously worked in consumer research, in the first few weeks of working in the pharmaceutical industry, you may have raised an eyebrow when presented with a quantitative sample size of 30 respondents. While you may have initially scoffed at such a paltry base size, after only a few months you will find yourself actively defending these sample sizes and bandying about terms such as “robust”, “statistically significant” and “conclusive evidence.”
So where does that leave us? Should we be worried about researching with such small base sizes or is this just a fact of life in pharmaceutical market research?
I firmly believe that we should always aim for the largest achievable sample size. Recent projects we have conducted have included sample sizes of over 1,000 respondents and with this number of respondents we can all feel confident that the results we obtain deserve the moniker “robust.” A larger sample size provides us with a much richer data set in which we can explore subgroups, test hypotheses and run more complex statistical analysis.
Where we struggle is when researching small population sizes from which it is neither practical nor appropriate to draw a large sample of respondents. This is when we are forced to take a more pragmatic approach. Let’s face it, budgetary and time constraints nearly always impact the feasibility of a quantitative research study. But does that mean we are just making do? No, I don’t believe it does. However, there are several pitfalls which we need to avoid.
Data has limitations: Over-analyzing and over-interpreting data
One of the key considerations when working with a small sample is simply to remind yourself of its limitations. Complex statistical analysis may not be appropriate, and researchers need to remember that subgroup analysis can be misleading. This doesn’t mean that you can’t interrogate the data, but it does mean that you need to approach the analysis with a critical mind. Analyzing subgroups of less than 30 respondents is clearly suboptimal, so if you find yourself doing so, think again.
Indeed, it is all too easy to fall into the trap of analyzing a small sample in the same way as you would approach a larger sample. It is either negligent or disingenuous to gloss over the fact that we are dealing with low numbers so we need to ensure that our debrief document gives a fair and honest representation of the data.
Over-reliance of statistical significance testing
Statistical significance tests provide us with an essential tool for analyzing market research data. Without these tests, we would need to rely solely on our own judgement – a definite risk. However, there is a tendency to rely on significance testing above all else, and this is particularly problematic when dealing with a small base size.
Yes, significance testing provides us with a great deal of reassurance that our findings are genuine. The issue comes when we rely solely on statistical tests and fail to take note of the overall pattern of response. Even with a small sample size, statistical tests will undoubtedly pick up some differences, but how do we know that these differences are real?
In my opinion, we know this only by looking at the data set as a whole. Checking the consistency of response and themes within the data is a good start.
The importance of a small sample size
Another factor we need to consider is the overall population you are researching. Within small populations, you would expect a proportionately small sample size. For example, a very large sample size of 10,000 respondents amongst the total U.K. population accounts for only about 0.16% of the population. Compare this to a sample size of 100 GPs in the U.K., which accounts for almost 2.5% of the population. This starts to put the numbers into perspective.
In addition, even with a very small base of respondents, the degree of consensus is important in defining how confident we can be in the results. If 18 out of 20 people agree on a particular point, this is a powerful result regardless of the small number of people questioned.
In pharmaceutical market research, small sample sizes are unavoidable but worthy of consideration nonetheless. The views of even a few people add to our knowledge and understanding and help us make sound business decisions. What we often forget is that small sample sizes need to be handled differently. Respect the statistics but don’t overplay them, be honest in the way data is presented and don’t try to hide the fact that it has limitations. Clients respect our opinions and expertise, so a discussion around the sample size is entirely appropriate.