Forewarned is forearmed

Editor's note: Based in Chicago, John Garza is vice president at Ipsos.

Failure mode and effects analysis (FMEA) is an ominous-sounding name for a tool that is actually quite simple and profoundly useful in identifying risk areas in any business process. In short, with FMEA we identify all of the possible ways a process can break, the likelihood of them occurring, the potential impact of these breakages and our ability to detect them or even prevent them from happening in the first place.

In this article, we are demonstrating FMEA in a market research context – specifically, for use within tracking study survey processes. That said, the FMEA approach described here can benefit almost any business process, in market research and beyond.

Many daily checks

We had been managing a long-running survey-based B2C customer experience tracking study for one of our clients. From an overall technical program perspective, all was running smoothly, as we had had in place many daily checks in our process (both automated and otherwise). These checks looked for any issues around e-mail invitation delivery and survey programming, for example.

We eventually noticed that there had been a marked shift in the proportion of mobile completes in the quarter. For some time, we knew that the proportion of mobile survey takers was steadily on the rise (keeping in line with the global increase of mobile users). However, we realized that in recent months the rate of mobile completes on our survey program had accelerated beyond expectations. This was important as mobile respondents on this program tend to give lower customer experience scores than do desktop respondents. Furthermore, our client had set customer experience growth targets for the year within their organization, in part based on an expectation of mobile survey respondents. And now those targets had to be readdressed.

Clearly, it would have been better for us to have caught this for our client immediately versus one to two months after this marked shift in proportion of mobile surveys. In doing so, we would have helped our client by giving them a bit more time to manage internal communications and planning, while we contributed to this process with analytical deep-dives and best practices. In the end, our client was very gracious and understood that this real-world shift in mobile usage was unavoidable and to a large degree unpredictable. But at the same time, we realized that as trusted partners and advisors we needed to augment our survey monitoring and that FMEA was the right tool for the job.

Used across many industries 

The FMEA procedure was developed in the 1950s by the U.S. military but was later used by NASA in the Apollo, Viking and Voyager space programs, just to name a few. The civil aviation industry followed suit, along with automotive giants such as Toyota, DaimlerChrysler, General Motors and the Ford Motor Company. Today, FMEA is widely used across many industries.1 It is also in the standard toolkit for various quality-related disciplines, such as TQM and Six Sigma.

One of the fundamental questions FMEA asks is, “What could possibly go wrong?” In answering this question we are able to account for almost any conceivable problem before it rears its ugly head. We can then improve our processes and prevent these issues from happening. Or at the very least, we can establish steps to detect these issues as soon as they happen.

Additionally, FMEA can be expanded to measure not just when a process clearly breaks but when something in the process might possibly be broken. In this example using a survey process, we are allowing FMEA to alert us to changes in our survey’s output. These changes in output may indicate that something is wrong.

As we mentioned earlier, FMEA is a very straightforward process. As shown in Figure 1, here are the steps:

Measurement: First, list all of the things you want to inspect as possible areas of concern. Using FMEA nomenclature, these are called “failure modes.” However, as we said earlier, we can also list those things that point to possible failure modes.

Potential cause(s)/mechanism(s) of failure: Describe the possible cause of the failure. For survey-related failure modes, often the causes are tied to programming issues, e-mail invitation scheduling issues or sample file issues. There may be more depending on the program.

Severity: Using a 1-10 scale, where 10 is the highest level of severity, rate the potential impact of an issue occurring

Probability of occurrence: Using a 1-10 scale, where 10 is the highest probability of occurrence, quantify the chance that an issue in that particular area may occur.

Detection risk: Using a 1-10 scale, where 10 is the lowest probability of detection, quantify the chance that you will be able to catch an issue in that particular area any time it occurs.

RPN: Calculate the risk priority number (RPN). RPN is a simple multiplication of the severity by the probability of occurrence by the detection risk. The RPN will help you see where your process is the most vulnerable. Use the RPN to prioritize where and how marginal resources are allocated to improve your process.

You’ll notice that many of these steps ask for quantification of an element in your process. In some cases, this quantification will need to be your best guess. This is alright – as long as this best guess is an educated one, made by those familiar with the process, subjectivity should not be an issue. It is better to have a structured monitoring approach with some subjectivity than no monitoring approach at all.

Prioritized for review 

You may already be able to see the benefits of applying FMEA to a process. Now that you have calculated RPN across all pertinent areas in your tracking study process, any area with a higher RPN should be prioritized for review. We should then ask ourselves the following questions: What process improvements can we make in order to prevent this issue from happening? What can we do to detect this issue immediately if and when it does happen?

Obviously, prevention is an ideal solution to any potential problem. Where prevention is impossible or unrealistic, detection is the next best thing. In a survey process, quickly identifying unexpected fluctuations in survey scores, response rates, demographic mix, etc. will allow your team to investigate and either take remedial action where necessary or explain it away as normal variation. The situation we want to avoid is one where we identify an issue farther downstream when it is too late!

Changed immensely

Survey-based tracking studies have been around a long time but have also changed immensely over that same time period. Where once paper and telephone surveys were common, online surveys took over as the preferred data collection methodology. Online surveys further include complexities such as varying device types and varying invitation and survey delivery methods. Your survey program may even be multimode, using a combination of data collection methodologies.

Changes in the technologies used for data collection will require you to revisit your FMEA design frequently, in order to ensure it is as all-encompassing as possible. More holistically, as the design of your survey changes over time, you will want to ensure that you have reviewed your survey process FMEA and have addressed necessary adjustments there as well. Revisit your FMEA design at least each quarter and see if any adjustments need to be made to it. 

Lastly, some other thoughts on FMEA that we have found to be helpful:

• Start now! FMEA can be implemented on any process, however long it has been running. It is never too late to start improving your quality processes.
• FMEA is best when used on new processes, both during and after the design phase. Using a garbage-in, garbage-out mentality, introducing FMEA early on will only ensure that you are executing a process with the highest possible quality output. 
• Reward participation and improvement! A quality process such as FMEA may be perceived as “extra work” by some, especially if you introduce FMEA onto a long-running survey process. Get buy-in by talking up FMEA’s benefits to your staff (less re-work, less stress, etc.) but also by rewarding improvements in process quality. Track your quality by using metrics such as the number of days between failures/issues or the average failures/issues per quarter and reward that improvement.

Practicality and simplicity 

FMEA has been a very valuable tool for quality professionals for decades. While it has been very popular in more technical industries, our use of FMEA does not need to be limited to those industries. Its practicality and simplicity allow us to easily appropriate FMEA into our researcher toolkit. In doing so, we will be able to improve our survey processes, shielding ourselves from quality risk and other unwanted surprises. The time we save on avoiding re-work can thus be redirected to other value-adding tasks, which is the best possible situation for both researchers and our clients.

Reference

1 “Implementation of failure mode and effect analysis: A literature review,” International Journal of Management, IT and Engineering. http://www.ijmra.us.