Watch and learn

Editor's note: Tom Donnelly is vice president, head of medtech, at research firm Branding Science. Jason Mandelbaum is senior research executive at Branding Science. Hiba Rahman-Vyas is research executive at Branding Science. 

This past year has been a challenging one for conducting research in person. While many types of projects have been successfully converted to virtual approaches with relative ease, observational research and hands-on work with medical devices and technology have proven to be more difficult. In this article we will: discuss traditional observational research approaches; mention the standard web-assisted telephone interview approach; detail advantages and disadvantages of three virtual observational approaches; and review two case studies: home health infusions and preparing/administering IV chemo. In our conclusion, we reflect on the need for observational research, point out the strengths of conducting research in phases and provide tips for success.

Emphasis on observation 

When designing a new medical device or modifying an existing one, observational research is used to observe workflows, workarounds and unmet needs. In market research, we may use the term ethnographic research, while in human factors we may call it contextual inquiry (CI). Regardless of what you call the approach, the idea is to watch health care practitioners or patients using the medical device. Using an observer-as-participant approach,1 there is an emphasis on observation, interjecting questions as needed for better understanding, as compared to active moderation during the use. 

Trained medical moderators enter a medical environment – such as a hospital, an ambulatory surgery center or a pharmacy – to follow and observe medical personnel performing the procedures of interest. When the device is self-administered, patients are observed to better understand how they use their medical device. Moderators go to the patients’ homes to watch them administer their own treatments. In both settings, researchers video-record, take pictures, use handwritten notes to capture critical steps in device use, determine workflows and examine the environment of use. Analyses are conducted using various types of data, such as careful observation of the videos showing how the end user holds and manipulates the device, self-reported data and the moderator’s observational notes.

Web-assisted telephone interviews (WATIs) are the backbone of virtual qualitative market research and almost all readers are already using this kind of approach. However, here is a short review to set the stage for discussing virtual approaches. As the name suggests, respondents are interviewed by the moderator while viewing a stimulus on a computer (or tablet). In some cases, respondents and/or moderators are using their webcam, which allows for viewing facial expressions. Depending on the platform, additional ways of communicating and collecting data can be incorporated. For example, if there are multiple respondents, they can individually provide ratings or open-ended responses such that the moderator can see their responses but their fellow respondents cannot. You could also have people mark-up stimuli individually, as well as collaborate on a white board. Some platforms provide a virtual backroom with a chat function where the moderator can chat live with their clients without respondents seeing the discussion. 

Virtual observational approaches: pros and cons

Virtual approaches have been successfully used for various types of research projects but less so for observation and device testing research. The advantages and disadvantages of three approaches are described: WATI device testing, virtual ethnography and virtual observation. The right corner of each column in Table 1 denotes which approach was used for the two case studies (C1 or C2).

WATI device testing

This approach uses the standard WATI platform to observe use of devices sent to the respondent. You may need to send the respondent a webcam in addition to the device to be tested. After the device testing is complete, the respondent ships it back to the researcher so that it can be sanitized for use by another respondent. In our case study, we found it cost- and time-efficient to use a courier service in two metropolitan areas. The courier was shipped the prototype, disposables and a webcam. Before each scheduled interview, a box was dropped off for the respondent. After each interview, the box was picked up. The prototype and webcam were sanitized and the disposables were refreshed before delivering them to the next respondent.

There are some advantages to the WATI approach. The moderator can observe the device use while interacting with the respondent. It allows clients to watch and interact with the moderator. The moderator can provide live instructions and probe where desired. The approach is often iterative, so that learnings from earlier interviews can benefit the approach in subsequent interviews. 

There are also some disadvantages to this approach. There is a lower video quality versus what can be captured with traditional video equipment. By the nature of this approach, it is a scheduled set time for the interview. You can run into technical issues with Wi-Fi and webcams. Another downside is there is a set point of view for where the webcam is located. 

Virtual ethnography

Another approach can be called virtual ethnography, where respondents record themselves, typically with a smartphone, while they discuss their condition and show the use of their device. The videos are then uploaded to a platform. An optional add-on is the use of WATIs to get context with follow-up questions. 

There are some advantages of this approach. You have the ability for much longer observations and multiple observations over time. The research can be done in parallel with many respondents at once. Another important advantage is that the observation is more natural since it is done in respondents’ own environments, on their own schedules. 

There are some disadvantages. For example, there is no interaction between the moderator and the respondents. This leads to the need for pilots to check the method and make sure everything in the process is working smoothly before fully launching field. Also, respondents are responsible for uploading their videos, which can pose a challenge and require the researcher to follow up. And finally, the point of view is often that of a selfie, which may or may not be problematic. The use of a smartphone stand can alleviate this issue.

Virtual observation

The third approach can be called virtual observation. In this case, the researcher ships a recording device, such as a GoPro camera, to respondents. You can provide the option of using a head strap or a stand. As with the prior approach, respondents are recording themselves but the visual perspective could more closely simulate in-person observation or the perspective of the user. After respondents finish the research, they ship back the recording device, the video is downloaded, the device gets sanitized and shipped to another respondent.

There are some advantages. As with the last approach, you can have a longer observation time and the research can occur in parallel with multiple respondents. An important advantage is the high-quality video. In addition, you can have multiple points of view because of the use of the stand or head strap. 

One of the disadvantages is similar to the last approach: you cannot interact with the respondent. This means you need to conduct pilots to check the method. Another disadvantage is that you are limited by the number of recording devices that you have available, in our case GoPro cameras.

Case studies

The two case studies provide context for the use of these three approaches, as well as shows some of the challenges faced in conducting virtual observation.

Case study 1: home health infusion unmet needs

In the first case study (Figure 1), the goal was to understand the workflow for patients and home health nurses with large-volume infusions and identify the unmet needs that exist for both types of end users. The study was originally planned as four-hour in-home observations, including both shadowing nurses as they treat patients and observing patients as they self-administer. COVID-19 required pivoting to a virtual approach in observing infusions without entering people’s homes. A virtual two-phase approach was used. 

The first phase used WATIs with patients and nurses. The WATIs yielded good information on the detailed steps users took in their procedure. They provided foundational learnings, helped identify the overall workflow and identified some of the user needs. These findings would have seemed adequate had a second phase of observational research not been conducted. In the second phase, GoPro cameras were shipped to patients and nurses to video-record infusions. The analysis of the videos showed how there are some unspoken user needs and workarounds that were not mentioned during the WATIs. People were unaware of all the steps that they take in a procedure. After performing the procedure repeatedly over time, it had become automatic and procedural memory took over. Similarly, workarounds that were adopted to overcome the limitations of the device or their environment were not easily recalled because they had become automatic. 

Comparing the information obtained across the two research phases is revealing. There is a lot of information gathered in the second phase of virtual observations that was not mentioned during the interviews. For example, in the interviews, patients noted the benefits of the portability of their infusion devices allowing them to move around, even go out to dinner while infusing; however, the virtual observation showed that patients were generally uncomfortable while connected and simply sat on the couch during their infusion, as opposed to doing other tasks mentioned during the interview. During the interviews, there was no mention of issues around personal protective equipment (PPE) but in observations there was some difficulty managing some of the devices while wearing gloves. Interviews did not reveal difficulty with tourniquets but challenges were noted on videos with single-handed tying of tourniquets. There was no mention of port-related difficulties during the interviews but in the videos patients with chest ports did have some difficulties with their self-administration, such as straining their necks. Nurses did not report any priming issues during the interviews but in the videos it was obvious that nurses were looking for a place to prime tubes, such as in discarded packaging or in a plastic bag. Finally, during virtual observations, many weaker patients had significant difficulties opening the packaging materials and disconnecting components, resorting to using their teeth or asking for help. 

Case study 2: preparing/administering IV chemo unmet needs

The goal of the second case study (Figure 2) was to understand the unmet needs for preparing and administering an IV chemotherapy, as well as test a prototype to fulfill an unmet need. Originally, most of the research was to be conducted in person but COVID-19 caused a shift to virtual research. The three research phases are detailed below. 

In the first phase, pharmacists and pharmacy technicians recorded themselves (“virtual ethnography”) in their preparing the IV chemo with different setups. The observation allowed researchers to see where in the pharmacy specific materials were stored and if there were any organizational habits in storing. Specific brands of the materials used, different sizes and protective gear worn could be seen in videos. The location of the preparation was shown. Some were in rooms that had hoods to prepare the medication underneath, while others did not have hoods and used extra PPE to handle the materials. Researchers could observe if other HCPs were involved in the process, such as pharmacists signing off for pharmacy technicians.

In the second phase, WATIs were conducted with pharmacists, pharmacy technicians and nurses. Some of the information that would have been gathered during observation was obtained through interviews, such as detailed steps the respondent takes while preparing/administering the treatment. Respondents also discussed unmet needs and reacted to different profiles of potential device prototypes to help with the preparation/administration of the IV chemotherapy. 

In the third phase, WATI device testing was conducted using prototypes with pharmacists, pharmacy technicians and nurses. A courier service was used to ship the device prototype to respondents. Respondents simulated use of the prototype during the infusion and preparation process. Some HCPs had questions during the procedure, such as how to attach a part of the device, so live moderating was needed for this phase. Instructions were provided on how to use the device and the moderator purposely did not provide a demonstration. It became apparent what aspects of the directions for use needed improvement. The observation of the device use revealed user errors for these naïve users. Some respondents made unconscious mistakes due to their ingrained approach over many infusions in their career. 

Challenges and solutions

Specific challenges were identified during the two case studies and some solutions can be suggested (Figure 3). With the first case study, the researchers had a limited number of GoPro cameras and the process takes substantial time. In both studies, field was extended to include shipping, as well as pilots to determine if the approach and directions were successful. It is important to coordinate schedules of moderators, respondents and shipping. Another challenge is that there are multiple levels of consent: home health companies; nurses; and patients. Safety protocols were another challenge during COVID. The research team developed a protocol, including instructions for the respondents and the courier service about sanitizing materials and use of gloves. There were some unanticipated added expenses. Given the research was planned for in-person, planned travel expenses were used to cover some of the new expenses. It is suggested to include a budget contingency (~5-10%) that gets triggered when unexpected costs are incurred. 

There were some technical issues that warrant mention. Some of the respondents may be unfamiliar with the recording device used, in this case, the GoPro camera. Initially, written instructions were provided but the pilots showed the need for video instructions to show how to use the camera. Pilot interviews also revealed issues with the camera angle used. Instructions were updated to ensure the appropriate camera angle. Some nurses were uncomfortable wearing the camera on their heads, so including a tripod is another way to facilitate videotaping. Other technical issues faced were with webcams and Wi-Fi. It is important to have buffer time built into interviews (+15-20 mins.) to account for technical issues being resolved. Some respondents had issues with the webcams on their computer. It is suggested that all respondents are shipped webcams. Having a technician on hand to help with issues is also advisable. Given prototypes can break and additional PPE may be needed, it is best to have extra prototypes, PPE and disposables available. Finally, video files tend to be large and difficult to share. Sometimes file-sharing sites can be used, while in other cases an external hard drive with password protection is needed. 

Conclusions

When should you use these three approaches (Table 2) and how can the challenges faced be overcome? WATI device testing is most similar to traditional in-person interviews and is already the replacement for face-to-face interviews. It is the best approach when a live moderator is needed, such as when directions are complex or if follow-up questions are likely. It fits well with standard scheduling of research. It should not be used if high-quality video is needed or when a stationary point of view from a webcam is not acceptable. 

The virtual ethnographic approach is perfect when you want more depth or intimacy. It allows for long and/or multiple observations. It provides the ability to conduct field in parallel. If scheduling is difficult, this is also a great approach. Instructions should be straightforward because there is no live moderation. If a selfie point of view is not desired, this approach is less viable.

The virtual observation approach should be used if you want high-quality video. You can conduct field in parallel. This is great for long procedures and works well when scheduling is difficult. Instructions need to be straightforward given there is no live moderation. You can have different points of view from the video, which can be important for certain projects. 

The case studies provide a reminder that observational research is needed. Respondents are not able to recall their actual step-by-step procedures. Habits and workarounds just become part of procedural memory and are not always consciously available. Additionally, the case studies show that the iteration of phases provide a much better outcome than one phase can. The initial phase allows the research team to develop frameworks that lead to specific things to observe or probe on at a later phase. The case studies show that in-person observation can be successfully moved to a virtual approach during this global pandemic; however, even when in-person work is feasible, these virtual approaches can prove useful. 

Reference

1 Privitera, M.B. (2015). Contextual Inquiry for Medical Device Design. Elsevier: Waltham, Mass.