Creative learning

Editor's note: Jeff Vitale is a director of research and consulting at Market Strategies International, a Livonia, Mich., research firm.

If you were asked which U.S. college or university has had the greatest global social and economic impact through the achievements of its faculty and students, how would you answer? Harvard? Stanford? MIT? Or would you be tempted to cite one of the larger public universities boasting 500,000+ alumni such as the University of Michigan, The Ohio State University or Penn State?

In a recent study of the adult U.S. population conducted by Market Strategies International, all of the above institutions received multiple votes but Harvard was the clear leader, being mentioned on an unaided basis by one in five respondents; the others followed markedly behind. Perhaps this is not surprising, considering that venerable Harvard is the oldest educational institution in the U.S., founded in 1636 by a vote of the Great and General Court of the Massachusetts Bay Colony more than a century before the American colonies declared their independence from England.

It’s commonly believed that Harvard alumni have had a noteworthy social and economic impact but having that belief and defining the impact are different matters entirely. Getting a robust answer to this question would seemingly require a significant investment of time and resources, leading researchers and marketers to ask, “Why bother? What’s the value proposition?” The answer is simple: Impact data is a measure of the return on investment for a variety of stakeholders ranging from the university itself, faculty, students, business leaders and even government agencies. The kinds of questions impact data can answer in the academic space include:

  • Is it worth the investment to create centers of learning or degree programs for particular specialties or industries?
  • How must the curriculum change to meet the needs of an evolving global society?
  • Is funding a new building or modernizing medical equipment money well spent?
  • Are federal grants likely to yield results?
  • Does the university have a synergetic relationship with the local community?
  • What is the expected lifetime value of a university education that an alumnus might expect?
  • Does alumni diversity translate into impact diversity?
  • Are there emerging impact trends that could result in social, economic or political evolution?

So, in January 2015, the Harvard Business School and Market Strategies teamed up to develop and execute a study that would quantify the collective social and economic impact of the living alumni of all 12 degree-granting Harvard schools as well as the Radcliffe Institute for Advanced Study in a tangible, rigorous and repeatable way.
While the results are not the focus of this article, I wo

uld be glad to send you a copy of the executive summary or you canalso find more information at http://bit.ly/1lMvDfn. Instead, this article focuses on a few of the novel approaches we used to meet the objectives, including an innovative partnership with LinkedIn.

Getting lost in a forest

As you might imagine, there are almost limitless ways to measure economic and social impact, and there is the danger of getting lost in a forest when seeking the perfect tree. Working closely with Harvard faculty and leadership, we ultimately decided that the best approach was to measure a specific type of impact, which could be expanded upon in future studies. After careful thought, we settled on a measure of economic impact that assessed the ongoing entrepreneurial efforts of living Harvard alumni.

Specifically, we defined economic impact as the aggregated annual revenues (or assets under management, in the case of financial services organizations) and employees of currently-operating profit and non-profit ventures founded in full or part by one or more living Harvard alumni. We adopted this definition because it measures a contribution that would not have existed if not for the founder’s efforts. So, while a Harvard alumnus might be responsible for the success of a particular company, it would only be included in the study if that alumnus was also a founder of that company. Similarly, while you might rightly argue that a Harvard alumnus who becomes president of the United State will undeniably have global impact, it is not within the scope of this first study because presidential impact exists independently from the alumnus to a greater or lesser degree, and, as such, cannot be readily disentangled from the impact of the office itself.

In addition to non-profit entrepreneurial activities, we included alumni board service and alumni volunteer hours to augment measures of social impact. In case you’re wondering, the roughly 375,000 living Harvard alumni serve on nearly 300,000 boards and collectively volunteer more than 1.6 million person-hours a month, which is equivalent to 6,575 full-time jobs. Where Harvard alumni are concerned, it is not all about economics.

Focused on methodological rigor

With the scope of the study defined, we focused on methodological rigor to ensure accurate insight. Getting a respectable response rate and minimizing the potential for response bias were two of our chief concerns, especially considering that many alumni are very busy people. Minimizing response bias is particularly important given the study questions. Any incentives of monetary value could differentially appeal to people of varied economic strata or professional success. Similarly, incentives involving donations could differentially appeal to people with different levels of social awareness or sense of responsibility. As such, we decided not to use incentives because the risk of non-response bias would be far too high.

Further, we determined that placing any urgent plea in the invitation letters focusing on the need to measure impact would also potentially encourage or discourage participation, depending on each individual respondent’s self-assessed success level – also introducing unneeded response bias.

With two of a market researcher’s best tools unavailable (incentives and urgent language), we did what we could to increase awareness among alumni and make the survey experience as easy as possible to maximize survey completes.

LinkedIn became invaluable

Increasing awareness was one of the first ways LinkedIn became invaluable. Sending a survey to all living Harvard alumni requires accurate, up-to-date contact information. LinkedIn is the undisputed social network of professionals, so advertisements urging folks who attended Harvard to update their alumni information as well as advertisements announcing an upcoming study were efficient ways to increase awareness that have not long existed. In fact, in 2005 LinkedIn reported 5 million members; in 2006, membership had risen to 12 million. LinkedIn became truly global – by its own admission – as recently as 2008. By 2010, it had 90 million members and at the close of 2015, it boasted more than 414 million members. Suffice it to say that this is quite a potential audience.

The next step of optimizing the survey experience for online respondents also involved LinkedIn. As you may have experienced when job hunting, there is often an option to fill in some or all of the job application by logging into your LinkedIn account and automatically populating career and educational information. For individuals with long careers, this represents a great deal of information that might be very time-consuming and tedious to enter by hand. Recognizing the importance of the Harvard Impact Study as well as the benefit to LinkedIn members who wanted to participate, LinkedIn gave us special permission to develop a custom API that allowed respondents to log into their LinkedIn accounts and automatically populate the relevant portions of the Harvard Impact Study. Knowing that fields often populate awkwardly, Market Strategies ensured that the API processed the information and presented it back to each respondent efficiently and correctly. Further, respondents were able to use simple check-boxes to include, exclude or edit each line of data. Respondents were able to add new careers and educational accomplishments as well.

This auto-population process was highly successful and many respondents reported a dozen or more posi-tions, which would not have been practical using manual entry. Once we collected basic career data, we asked respondents foundership questions for each company or venture they mentioned. Because so much of the required information was inputted with the click of the mouse, the survey experience was simple for many respondents and optimized for others. We believe the LinkedIn integration – and the improved completeness of the data it afforded – resulted in more accurate projections.

Address the question of outliers

Once we collected, cleaned and validated the data using an array of secondary and tertiary research tools, we needed to address the question of outliers. As researchers, we all know about outliers and the dilemma of whether they should be kept or dropped in an analysis. Often, we are tempted to simply drop them because they don’t play well with many of our statistical assumptions. Yet they can be legitimate observations and are sometimes quite interesting and important, as is the case here.

Given that measuring the total economic impact of Harvard Alumni was one of our stated goals, it would be inappropriate to drop a company like Staples or Microsoft from the analysis as an outlier simply because it has great impact. Yet, considering the sheer size of a company like Microsoft, we had to ask whether it is reasonable to include it in any population projections made from sample data. In simple terms, if a founder of the largest company in the world fills out a survey and the study response rate is 25 percent, can we assume there are another three such companies among non-respondents? The answer is probably not.

In such cases, researchers often adopt inclusion/exclusion rules based on historical experience, standard deviations from the mean, visual examination of scatterplots or even gut-feels. Often, a conservative approach is thought best and, when in doubt, outliers are excluded. Here, the goal was to get a robust and accurate estimate of impact without bias in any direction. As such, outlier companies, or “luminary ventures” as we branded them to avoid any negative connotation, needed to be identified and treated differently than the other ventures in the analysis.

To this end, our data scientists employed a simple process inspired by variance shift outlier modeling to determine appropriate cutoff points. In a nutshell, we ordered companies by revenue and then calculated the variance over and over again, adding companies of increasing size. The first meaningful spike in variance was then used as the cutoff point and all companies with values above that level were designated as luminary ventures. These companies were so large that while we included them in the total estimates, we did not use them to make non-responder projections. We identified a number of sample records using this technique for annual revenues, assets under management and employee number and these cases were not used to make non-responder projections, improving the validity and robustness of the final projections.

Of course, non-response bias is always a possibility when making projections from survey data. In order to address this concern, we used a variety of techniques to detect any non-response bias, including:

  • comparing impact metrics calculated from fully-completed versus partially-completed surveys;
  • comparing data from surveys completed without reminders, a single reminder, two reminders and three reminders or more; and
  • using historical alumni data to predict founding behavior via a discriminant function analysis and then scoring both alumni respondents and alumni non-respondents to detect any bias.

Extracted randoms samples

LinkedIn proved to be a useful method for assessing non-response bias as well. We extracted random samples of founder and non-founder respondents – as well as non-respondents – from the sample list and then conducted manual research using LinkedIn as a secondary data source. This helped us determine if LinkedIn members who responded were more or less likely to be founders than those who did not respond. Once again, the answer was no.

So what are the estimated 2015 global annual revenues of companies founded by Harvard alumni? Glad you asked: approximately $3.9 trillion which, incidentally, rivals the 2014 GDP of Germany.