Data Use: Beyond Excel: Why you should integrate software development tools and principles into your research

Editor's note: Adi Sarid is CEO of Sarid Research Institute. He can be reached at adi@sarid-ins.co.il.

A good friend once shared his perspective on spreadsheets: Excel is the second-best tool for anything. The “best tool” depends on the specific goal you are trying to accomplish.

In the far-away past, we extensively used Excel (and SPSS) in our daily market research activities and the statement that Excel is a great tool seemed reasonable. I no longer agree with that. The allegory is that we were using basic analysis tools, feeling great about it, without knowing we were shooting ourselves in the foot. These basic tools have significant pitfalls and this article is about how we now avoid them by integrating software development tools and principles into our day-to-day market research routine.

The following is not a complete list but are some powerful concepts and tools that guide us and can revolutionize any business in market research if utilized correctly. Some are easy to implement and some are advanced, but even implementing a small portion can make a big difference. Also, note that this article’s perspective is highly biased towards quantitative research, the field I’m primarily associated with and active in. Some of my principles might also apply to qualitative research but that is not the focus here. 

Coding: Building reproducibility into market research

I’m a data scientist and most of my team is also. Our analysis is conducted primarily with R (and some Python) by coding. R is a statistical-oriented programming language and Python is an all-purpose programming language (with data science extensions). After we collect or receive data (e.g., from a surveying tool or other sources), we code a pipeline that loads the data into R and conduct all the analysis in R. 

Coding this way is usually referred to as “writing a script.” It creates a straightforward recipe that includes the steps to generate the results. Similar to how software works when it runs its code, we are building software that runs our statistical analysis.

The significant benefit of this approach is reproducibility. Anyone allowed access can run the same script and get the same results. Testing is easier; one can review the code and the results and find errors, if any, in the computation. The available packages in R can produce presentations (e.g., PowerPoint), reports (HTML, PDF) and even interactive dashboards. 

Another advantage is that you can start building the analysis before the data is wholly gathered because refreshing the results is just re-running the already-written script. This method helps shorten the time to complete the project.

Older software supports scripting or coding but to a lesser extent. For example, SPSS has a scripting language but “encourages” the use of a graphical user interface for analysis. Excel (although supporting VBA for programming) is widely used as a spreadsheet with formulas and user-interface actions. Documentation is poor, it’s hard to test and verify results and Excel users tend to corrupt data files (i.e., with merged cells, dates, encoding meaning with “cell-coloring” and other harmful practices). Also, more than once, I ran into users who could not explain the complex Excel file they had built just a few months prior (let alone someone trying to understand a file created by someone else).

To summarize this point, replace old tools (such as Excel and SPSS) with R (or Python). The effort is significant but the benefits are enormous. An excellent book to start with is “R for Data Science,” available online for free (https://r4ds.had.co.nz/).

Coding

Implementation effort: significant 

Benefit: enormous

Version control: Safeguarding accuracy and promoting collaboration in market research

Following the previous point, when you use code to create your analysis, you can also start using version control practices (i.e., with services like GitHub). Version control is a familiar term to most developers and it means that every change in a script (in analysis) is “committed” to a code repository. The repository tracks all changes and lets us go back in time to undo any changes or compare versions. We can compare two versions of the analysis (termed “branches”) and combine them (merge) – making it easier to collaborate without interfering. In addition, all projects are managed in repositories within an organization so it is easy to retain, search and share all the organization's knowledge.

If you ever had to save files like: “file_ver1,” “file_ver1_backup,” “file_ver1_backup_of_backup,” “file_ver2,” “file_ver2_after_cleanup,” etc., then you should stop and move to using version control. I can’t imagine a world without it.

If you are already familiar with coding, the implementation effort is small and the benefit is enormous. There are many tutorials online (just search for “learning git” on Google).

Version control

Implementation effort: moderate
(assuming you are coding) 

Benefit: significant

Automating deliverables: Scaling your analysis

As a benefit of coding, you can easily automate deliverables. Specifically, a use case we often see when working with customers whose activities are spread across multiple regions or branches is the need to generate segmented reports, e.g., presentations for numerous stakeholders across different areas or departments. Such automation becomes relatively straightforward when the deliverables are script-generated. All you have to do is loop over the various regions and run the same script to generate your delivery. Manual work might be needed if the delivery requires human intervention for insights (though this might also be automated with AI integrations). 

Similarly, a report that is required on an ongoing basis (e.g., weekly or monthly) can be easily automated with minimal effort.

Automating deliverables

Implementation effort: limited
(assuming you are coding)

Benefit: significant

Cloud infrastructure in market research: Security and robustness

When I say “cloud,” do you think of … rain? Most people think of precipitation. Those who write documents or create presentations also think of Google Workspace (formerly G Suite) or Microsoft 365. In the software development industry, however, cloud is a term that describes a wide range of services that companies like Amazon (AWS), Microsoft (Azure) and Google (GCP) provide. These services include servers or other integrations such as AI solutions, custom automation, etc. Most of our analysis is done on cloud infrastructure (and not on local computers). The benefits you gain from using cloud infrastructure include improved security and robustness (i.e., from a business continuity perspective).

Cloud infrastructure enables automation without user interventions (such as daily reports or online dashboards). At the more advanced level, cloud infrastructure allows us to serve customers with customized prediction models developed based on their data (generally called machine learning models) or push processed data files related to ongoing surveys.

Using cloud infrastructure opens up a lot but requires significant expertise. Suppose you’re serious about moving to the cloud. In that case, you might consider hiring or consulting with companies that provide DevOps (developer operations) or MLOps (machine learning operations) as a service.

Cloud infrastructure in market research

Implementation effort: enormous (requires a high level of expertise)

Benefit: significant

Documentation: Transparency and organizational memory

A fundamental pillar in software development is documentation. It allows for transparency and improves the organizational memory. When you work like a software developer in your market research analysis, you document the processes you do and the results you get, either via comments in your script or via “notebooks,” which are a combination of text, code and outcomes of that code (e.g., charts and tables). These notebooks have many forms depending on your programming language (e.g., RMarkdown for R, Jupyter for Python or Quarto for a more language-diverse approach).

Using notebooks also opens up many other options, like quickly building a website, a blog or a statistic dashboard that documents your work. I even used notebooks to create interactive tutorials for my undergraduate students learning statistics.

Documentation

Implementation effort: moderate 

Benefit: significant

Packages: Efficiency, consistency and speed in market research

A saying goes: “When you find yourself repeating a piece of code more than three times, it's time to write a function.” As mentioned earlier, our analysis is conducted by running scripts. When you use scripts, you find yourself repeating many operations, like refactoring variables the same way or creating the same type of graph repeatedly (but on different variables or even different projects).

When you utilize code, the practice you should stick to is to break your code into functions, i.e., smaller units that do a specific task. Then, you can combine these functions into a library (“package”). This improves the work in several ways:

• You become more efficient – you run the function instead of rewriting your script.
• You become consistent and more accurate – the function’s correctness is verified once and reusing it puts you in a safe place.
• You can share your function internally (proprietary packages) or even with the world (open-source packages), which lets other people use your functions and enjoy these benefits.
• From an organizational perspective, it shortens the learning curve for new employees and also guides the organization to converge on a unified standard for analysis and research.

Packages need to be built and maintained, which requires a slightly higher level of expertise from a development perspective. However, the benefits are across the board.

Packages

Implementation effort: significant

Benefit: enormous

Team management strategies: Synchronizing efforts for success

Compared to the previous concepts, which were primarily technical, this is more of a “soft” concept relating to managing market research teams and projects. 

A few years ago, I started using daily meetings of about 15-30 minutes each morning for the entire team. It might be trivial but I didn’t think about it until a friend suggested it as an excellent way to stay on top of things (the same friend from the beginning of this article). 

Daily meetings are a common practice in software development teams and they are handy for syncing the team, sharing challenges and ensuring that everyone knows what they are doing. It also lets me know when someone is about to finish up and needs the next task – resulting in improved planning and fewer surprises.

Many elements of software development management can be adopted into market research project management. In addition, many tools support project management and draw from valuable elements such as kanban, agile, scrum, sprints and more. This article is focused on something other than project management but browse the different project management methods and tools used during software development and consider what would work best for you. There's a bit of trial and error involved. 

As a general start, I can recommend “Death by Meeting” by Patrick Lencioni; it’s a fun read and sets the tone for an excellent organizational culture in terms of setting meetings.

Team management strategies

Implementation effort: moderate

Benefit: Moderate-significant (depending on how efficiently you are managing today)

Immense potential

In conclusion, the fusion of market research with software development principles is a paradigm shift that offers immense potential for revolutionizing the industry. 

By embracing coding as a fundamental aspect of analysis, market researchers can achieve unparalleled reproducibility and efficiency, leading to more reliable and insightful results. Adopting version control practices ensures accuracy and fosters seamless collaboration, while automation streamlines processes and enhances scalability. Leveraging cloud infrastructure enhances security and robustness and enables advanced functionalities such as predictive modeling and automated data processing. Moreover, prioritizing documentation cultivates transparency and preserves organizational knowledge, facilitating continuous improvement and innovation. Developing and utilizing proprietary packages further amplifies efficiency and consistency while effective team management strategies ensure synchronization and maximize productivity. 

By embracing these principles and tools, market researchers can break free from the limitations of traditional approaches, paving the way for unparalleled insights and success in an ever-evolving landscape.

In short, think more like software developers. To support such a transformation, you might want to consider hiring a few senior data scientists (if you don’t already have them in your organization).