Trouble getting a signal
Editor's note: Michael M. Wehrman is senior manager, growth insights at Comcast NBCUniversal. He can be reached at michael_wehrman@comcast.com.
Most of us can easily picture what we think typical household internet use looks like. Smart televisions streaming the latest hit shows, mobile devices displaying viral content on demand, consoles delivering lag-free multiplayer gaming sessions, smart devices playing music and podcasts, perhaps even long-distance commands to remotely change the thermostat or activate the vacuum. Such an example is one end of the spectrum – what of the other? What do we know about households that use very little data at all?
That audience, which we are calling “nontraditional internet” households, was the center of a research challenge. Difficulties in defining the audience (not just by the audience but the researchers) led us to pause mid-pilot; difficulties in interpreting quantitative results led us to pivot quickly to qualitative research in order to make sense of what could have been either a massive error in data collection or simply us missing vital context (fortunately, it was the latter). Through the team’s curiosity and nimble efforts, we were able to make sense of this audience and realized just how much variety there is in how households entertain themselves today.
But before all that, we discovered the first hiccup: With a survey in-field, how could we be sure we’ve correctly identified nontraditional internet households in the first place?
Admitting your audience is hard to reach
While it’s typically good (or great!) practice to write simple questions, when you have a difficult-to-identify audience, it’s possible to miss the mark by being too simple. Our first major challenge was that our questions were too simple to start with:
- Who is your home internet provider?
- Who is your mobile/cell service provider?
- Which do you use more often?
Simplicity in the question masked complexity in the responses – we quickly realized that mobile devices connected to Wi-Fi (since we were asking specifically about activity in the home) might be incorrectly counted. To make matters more difficult, respondents also tend to struggle with accurately recalling how much time they spend connected to local Wi-Fi when in the home. It’s all too common to scroll social media while watching streaming television; does that count as home internet, cell service – or perhaps both? With the right array of updated questions, added mid-pilot and aimed at validating that respondents were confident in the information they were providing us, we in turn were confident that we correctly identified the precise audience we were looking for: low-data-usage households.
We had little time for reprieve, however, as we immediately discovered that for an audience that is defined by how little they use the internet at home, on average they had about six devices per household.
When quant gives you more questions
Digging deeper into the data, it wasn’t just the devices that upended the existing hypotheses about low-data-usage households. Our hypotheses included some expected patterns around household sizes (they would skew smaller), geography (more likely to be rural) and income (overindex in lower income brackets). The quantitative research showed results spanned all of these criteria and more. Nearly every single hypothesis we started with was nullified – admittedly, accepting that possibility is a cornerstone of why consumer research is needed in the first place.
Still, the number of devices being so high stood out not so much as a busted hypothesis but a logical improbability: With households using little data, relying on cell phones and/or low-speed internet services, how could they be so connected? After examining the data in the ways one typically might (was it reverse-coded or alternately-coded by the data collection tool?) to ensure the results are accurate, we decided to stand up qualitative research as quick as we reasonably could. Stakeholders were eagerly anticipating results and we couldn’t possibly share insights that didn’t quite make sense absent any context. We needed answers and we needed them fast.
Our team designed an ethnographic study that was comprised of diary entries for a week at different points of the day – household usage varies during the day (as people are more prone to be at home vs. out) and across the week, as weekday and weekend habits can vary as well. We also followed up with in-depth interviews where we wanted to see participants’ houses – where their internet connection sat in the house, where their connected devices were and to ask additional question on usage that emerged from the diaries.
But first we had to recruit and even though we had solved for the screening questions (given the need to do so in the quantitative portion), we found several fresh new challenges in front of us. When we were conducting this study (late 2022), households all across the world (and certainly the United States as well) were slowly emerging from habits and practices learned during the 2020 COVID-19 pandemic. One of those was an extreme reluctance to invite a group of (two or three) strangers into their house for an in-depth interview (at any level of participant compensation). Many potentially useful candidates instantly opted out of participation on this premise alone. Yet on the other hand, remember that this audience has lower-speed (or cellular-only) internet connections; their internet connections often weren’t good enough to sustain a video call. With speed as a necessity, ultimately we mixed our approach – we went to households that would have us but also conducted video interviews with a number of others, hoping for the best (connection). At least we would have the diary entries as a single, steady point to collect data.
We had primed ourselves for dropped video calls, out-of-sync audio, poor frame rates and lots of inaudible answers. In truth, there was a bit of that but mostly in video quality; thankfully, the audio came through. Getting a tour of a participants’ home isn’t quite the same spatially/sensorially when viewed through the lens of a camera phone but we finally, at long last, had context!
They’re doing what? (Or: Make it a Blockbuster night)
This entire project threw me into a bout of nostalgia for the 1990s that I grew up in. After completing the quant portion, I felt a bit like the character of Willam in the Kevin Smith movie “Mallrats.” He spent the film staring at a “Magic Eye” drawing, unable to see the sailboat hiding in plain sight. Our team had the data, yet no matter how much time I spent staring at it, I couldn’t see the image beneath it all.
First things first, how did we explain the high-devices/low-data relationship? For the latter, it was simply a matter of finding entertainment outside of streaming. This audience was lagging behind typical American households in terms of adopting streaming platforms (and given limited bandwidth, this even extended to ad-based video on demand and free ad-supported TV services). For those households that did watch streaming services, the limited bandwidth meant that they could watch one program on one device at a time (and thus the household would have to agree on what everyone watched). Even households without streaming spent plenty of time watching television, however, relying on over-the-air broadcast television, home video collections and even physical disc rentals (via Redbox and similar services at retail stores they frequented). In short, what we discovered was not so much a difference in amount of time spent watching television but rather these consumers recognized the data limitations of their households and developed strategies to work around them. Renting videos through physical media? Certainly seemed as if the 1990s nostalgia proved to be a prescient framework for thinking through this research after all.
But Redbox rentals don’t provide context on the number of devices (and certainly the DVD players weren’t being counted among connected devices). When we asked about connected devices such as phones, laptops and so on, what we found was both quite surprising yet also quite simple: sometimes a connected device is not a connected device any longer. The most common application of this was the discovery that when it was time for this audience to upgrade their cell phones, rather than trading in their old phones, many were keeping them instead. This was prominent in households with children. Old phones would have their SIM cards removed or deactivated so they could not be used as cell phones; parents would load games, videos, educational apps and more onto the phones and then give the device to their children to use as a constant source of offline entertainment. This practice saved money, kept children busy (and entertained) and did not put a strain on the limited data capacity of the household. Everyone wins!
Everyone save for the few children whose gaming habits demanded connectivity. We did hear from a few parents that their children needed to get online to play games (most commonly Roblox). While these households rarely had connections that could sustain a lengthy Roblox session, industrious children often found ways to get their playtime in, typically going to a friend’s house or the local public library to be able to play online.
One crucial bit of context here is to point out that the disconnected devices didn’t solely serve the purpose of circumventing bandwidth limitations but also to provide some early socialization to devices while also keeping children off the internet. While parents understood that they would be connected elsewhere (at school or in one of the aforementioned locations where they might play Roblox), the desire to limit the amount of time their children were connected as well as have some control over where and how they connected was cited as both a justification and benefit of being a low-data household. Cost savings might have been one driver of using a low-bandwidth or cellular-only service but many also mentioned safety/control for the connectivity habits of their children as an additional benefit.
To sum up what we discovered about the entertainment habits of limited-connectivity households: offline gaming, over the air television and physical media (owned and rented) were consistent habits. A perfectly fine 1990s weekend was also a perfectly fine 2022 weekend. With that, we had resolution – vital context to resolve the seeming contradiction between limited connectivity and lots of connected devices. In “Mallrats,” Willam did eventually see the sailboat after all – and so did we.
The value of taking time
For us, the critical takeaways start with the value of taking time during a pilot phase, particularly if you think the audience is hard to reach or hard to define. As insiders at a telecommunications company, if it’s hard for us to settle on criteria, expect that to be doubly true for research participants. Having the ability to pause a day into data collection, gather our thoughts and try something new was essential – and beneficial. Calling a timeout is not the same thing as throwing in the towel and, in our case, likely saved the project.
Not being hesitant to fast-follow quant with qual was another major revelation. Conventionally, researchers use qual to generate insights that can inform gaps in knowledge essential to field quant research. In our case, the quant research revealed a relationship (limited connectivity and lots of connected devices) that, after accounting for all data-based possibilities, simply did not afford any intuitable explanations. Fast-following with qual was critical given our stakeholders’ needs. The story and the business implications matter far more than the percentages and quantitative data. Discovering that context was precisely what we needed to unlock our understanding of this audience.