Editor’s note: Randy T. Catron is senior research consultant, The Principal Financial Group, a Des Moines, Iowa, financial services company.

Intranet sites are becoming a common tool for communication within many companies. Unlike corporate Internet sites where the audience can be anyone in the world with Internet access, the intranet audience is typically limited to employees and business partners associated with an organization.

The limitations on who may access an intranet create opportunities to communicate a variety of information that wouldn’t necessarily be provided to the public, such as certain operational goals, performance results, or employee policies. It can also provide forums for employees to discuss and debate issues that are relevant to a company’s progress.

Although intranet sites have a more narrowly defined audience, the challenges of communicating through the intranet are very similar to those faced when attempting to communicate through the Internet. Issues such as navigation ease, site content, and graphic presentation are common to both intranets and the Internet.

The intranet at The Principal

The Principal Financial Group is a diversified financial services company with more than $72 billion in assets. There are more than 18,000 employees worldwide who rely primarily on an electronic mail system and paper reports to communicate and share information.

During the past two years more than three dozen intranet sites have been developed at The Principal by various departments, business units, and groups representing job functions. Typically, their purpose is to provide wider and faster dissemination of information to various target audiences, as well as to reduce overhead costs by using less paper and postage. Nearly all employees at the corporate headquarters in Des Moines, Iowa, have access to the intranet sites, as do many employees at external locations.

Developers of some of the intranet sites have asked the Management Services Research Team to analyze their respective sites with the purpose of refining the sites’ information content, improving their usability and graphic appeal. Because intranets are a relatively recent communication medium and one that the team had not worked with, a new methodology was created to answer the research questions.

A qualitative approach

It is not uncommon to see questionnaires attached to Internet or intranet sites where the users are asked to rate the site using scales and text boxes. This methodology seems to make sense in this environment, since the questionnaire is delivered electronically in the same environment as the site.

However, this methodology does not capture the subtle reactions of the user to various elements on the site. Because the population we were researching was employees to whom we had direct access, we believed a qualitative, observational approach to the research would provide greater insight and richer information.

Our first customer requesting intranet research inquired about a focus group-type setting for the research. They envisioned eight to 10 of their potential users exploring the site in a computer lab, with a moderator in the room capturing their comments and ideas. The research team proposed a modification of that idea, suggesting a one-on-one user-researcher format rather than the group format.

A one-on-one format provided the following advantages:

  • an opportunity to observe the user in his or her work setting, which would be a more comfortable and natural environment, and the one he/she would be in when accessing the site;
  • a better setting for detailed questioning;
  • a better opportunity to observe the user’s cursor movements, mouse-click actions, and other physical movements that may explain how he/she interacts with the site.

Preparation phase

The research process began by meeting with the intranet site project leader and the designers (the research customers). At the meeting, the customers provided information such as their purpose for creating the site, any specific details related to the site they want explored by the research, and information about their target audience (the users). Most often the sites are intended for a segment of the employee population, such as employees who work with information technology (IT), although there are some intranet sites created for all employees, such as the human resources site at The Principal.

The customers either defined the potential user population so a sample could be randomly selected, or they specifically identified potential users who they wanted to participate in the research. We believe our customers were not attempting to bias the research with their specific selections; every person they selected provided both positive and negative feedback and none had worked on the specific intranet site project.

After eight to 12 research participants were identified, they were contacted and asked to schedule one hour of time for the research. Generally one hour is sufficient for data gathering, and in many cases, the participants completed the process in less than an hour.

After conducting two intranet projects, it seems there are two primary classifications that apply to the participants: those who are experienced or inexperienced with Web-type applications, and, those who are interested or mildly interested in the information contained on the intranet site. Ideally it is best to include a mix of participants from each of these classifications.

As is common with much qualitative research, many participants brought up similar issues, ideas and concerns. However, it was interesting that nearly every participant identified or commented on at least one aspect of the site that was not noticed by or did not concern the other participants.

The data gathering process

A three-step process is used to collect data from the participants: a pre-browse interview; the browsing activity; and a post-browse interview. Ideally the three steps will occur at the participant’s workstation with the researcher observing from a side chair.

The pre-browse interview is analogous to a focus group warm-up. It provides an opportunity for the participant to become comfortable with the researcher and to initiate communication. It occurs prior to the participant bringing the site up on their computer screen. Usually the following information is gathered during the pre-browse interview:

  • basic demographic questions such as job title, years employed;
  • current sources of information related to the focus of the site;
  • types of information would they expect to find on the site;
  • previous experience with intranet or Internet sites.

For the browsing activity, the participants are asked or instructed to load the site on their computer. At this time the researcher is looking diagonally at the screen, mouse, and keyboard with the participant’s back to the researcher. From this vantage point it is possible to observe the participant’s actions and reactions to the intranet site.

Often the browsing activity starts by asking the participant to verbally describe his or her reactions to the overall design and content of the site’s home page. Next the participant is asked to view every major element of the site and think out loud, describing what he or she is thinking while exploring the site. The researcher listens to the participant, watches the cursor and mouse movements, and looks for other cues (such as a pause in action) that indicates a reaction to some aspect of the site. If the participant does not verbalize a particular action, the researcher probes for understanding. (An example might be a participant’s quick click on some words that are not hypertext. This occasionally happens when the words are a color other than black or the color used for hypertext.)

The researcher may record the sequence followed by the participant when exploring the site. Another helpful observation is to ask the participant to start at the site’s home page and then locate a very specific piece of information (such as the date of a particular training class). It is valuable to note how participants use different methods to locate information. (Some participants will use the site’s search feature; others will go to one or more locations on the site where he or she thinks the information should be located and then drill down.)

During the browsing activity the participant is not required to view every element of the site, just the front page of every major section. Some will choose to drill down several layers into one section, but only look at the front page in another. This may be a reflection of natural behavior had the participant explored the site in a non-research setting.

Often a participant’s comments and reactions to an intranet site can be assigned to one of three categories:

  • Content: the type of information presented on the site, how it is organized, and how it is presented.
  • Navigation: how easy or natural it is for the participant to move within and around the site.
  • Graphics/design: whether the color or design of the site enhances or inhibits the participant’s ability to process the information or navigate the site.

The following are examples of participants’ comments about intranet sites we have researched:

Content: Several participants commented that a "job listings" section was in the incorrect location. It was placed within a section titled, "People," but most participants said it should be in the section titled "Career Management."

Navigation: Several participants found links that were dead-ends (incorrect reference addresses). Also, navigation buttons were not available in the locations where the participants expected to find them.

Graphics: Several participants commented that a light gray patterned wallpaper made the black text difficult to read. Also, a site required too much scrolling down to access the content that was often off the bottom of the screen.

The final step in the process is the post-browse interview. The purpose is to ask summary-type questions and to provide an opportunity for the participant to share final suggestions and ideas. Often this step requires very little time unless the participant wants to re-review certain sections of the site.

Analysis and reporting

After collecting the data, the notes on each participant are reviewed for differences and common themes. The primary objective of the analysis is to provide information and guidance to the site’s creators that will enable them to communicate more clearly and provide content that meets the needs of their target audience.

Typically the report follows a format that includes background, methodology, and a review of the participants’ reactions to each major section of the site. The comments can often be categorized as content-related, navigation-related, or graphic/design-related. The report usually is completed by the researchers’ recommendations about the major items that should be modified on the site.

Provide guidance

It would be helpful if the creators of an intranet site conducted research on their potential users prior to developing the site. This would provide them with guidance on the specific types of information that should be included and excluded from the site, as well as information on how it should be organized. However, many times this doesn’t happen.

This type of research, like the intranet, is new for our research team and we are continuously refining the methodology. Overall, our customers have been pleased with the outcome of the intranet research and have perceived it to be valuable.