Design Survey Questions to Increase the Value in Analysis

Quantitative custom surveys are a very powerful tool for companies seeking information about their customers. If data from these surveys is valid and reliable, companies can collect valuable information on what products and attributes customers prefer, or how satisfied they are with goods and services.

However, companies often overlook an important question: what type of data should I collect? This seems like a straightforward question—but can be more complex than it appears. The type of data you collect has important implications for how you can accurately analyze data and how useful the information will ultimately be to the company that collects it.

Imagine a restaurant chain, Peppy’s Perfect Pepperoni Pizza (4Ps); 4Ps decided to undertake a customer satisfaction survey. In the process of survey development, they came to understand there were different types of data they could collect that would yield different levels of value for their analysis. 

Categorical Data

Categorical data (also called nominal data) has two or more categories, but there is really no intrinsic ordering to the categories. For example, 4Ps wanted to know which of 5 toppings customers liked best on their Peppy’s Meat Lovers Special. Management did not assign any particular ordering (highest to lowest, best to worst, etc.) to the toppings. They simply wanted to know that their customers preferred.

• Chicken
• Pepperoni
• Italian Sausage
• Ham
• Bacon
• Ground Beef

Categorical data is not particularly good for more complex types of statistical analysis beyond frequency distributions. For example, it does not make sense to compute means for pizza toppings. A mean of a categorical variable is not logical because there is no fundamental ordering of the categories—chicken isn’t better than bacon (well, some would disagree with that).

Gathering categorical is often necessary in market research because it gives companies some insight into what product attributes customers prefer—and there is simply no other way to ask the question. However, the data gathered still gives management important information. If a frequency distribution for this question reveals that 50% of customers prefer pepperoni out of the 5 categories, this tells management a lot about the dominant topping. Categorical data can also help companies better understand the demographics or personas of their customers (male or female, married or single, etc.). Consequently, categorical data remains a popular data type for its simplicity and ease of understanding. 

Ordinal Data

Ordinal data is similar to categorical data, except that there is a clear ordering of categories. For example, 4Ps’ management also wants to know how satisfied customers are with Peppy’s Meat Lovers Special. So they ask their customers how satisfied they are with the Special.

• Very Satisfied
• Satisfied
• Unsatisfied
• Very unsatisfied

There is a clear ordering between very satisfied and very unsatisfied. One connotes happy customers and one connotes unhappy customers, with levels in-between. Like categorical questions, these types of ordinal Likert-scale questions are also a staple of customer surveys. The important purpose of better understanding how customers feel about specific products and services.Another common type of ordinal variable is a numeric ordinal scale. 4Ps’ management would also like to know the yearly total household income levels of their respondents in order to develop customer segments and personas. They use the following scale:

• $25,000 or less
• $25,001 -$50,000
• $50,001 -$100,000
• $100,001-$150,000

Again, this type of question is very common, and can give companies some valuable information. This type of question is often used when asking sensitive numeric information such as income that respondents might not want to directly answer.

However, with both the types of data, the scales are not necessarily equally spaced. For the Likert-scale question, we cannot really know if the scale is equally spaced in the mind of respondents. In other words, we don’t really know where respondents anchor their responses. It is probably safe to say that the interval between unsatisfied and satisfied is probably larger than the one between very satisfied and satisfied. For the numeric ordinal scale, the intervals are clearly uneven. While we can say that there is a clear ordering of the data and we can compute frequency distributes, like categorical data, it less meaningful to do more complex statistical analysis.

In general, it is preferable to use an ordinal scale for Likert-type questions rather than a numeric ordinal scale (for that interval or continuous data discussed below is preferable). While the underlying issue of response anchoring is a challenge (although there are survey writing techniques to help anchor the questions and make the intervals as even as possible), such scaled questions can let companies know if customers are happy or unhappy, satisfied or unsatisfied, etc. Response anchoring is very important for issues customer satisfaction, branding and persona development.  

Interval and Continuous Data

Interval data is like ordinal data in that it is clearly ordered. The primary difference is that the intervals are also evenly spaced. So, if 4Ps’ executives wanted interval data on household incomes, they might offer the following answer choices:

• $25,000 or less
• $25,001 -$50,000
• $50,001 -$75,000
• $75,001-$100,000
• $100,001-$125,000
• $125,001-$150,000

This scale is much better than the previous presented as an ordinal scale. The equally spaced intervals mean we have more precise information about yearly household income. However, in this example, the main issue is that it is still presented in ranges, which still limits some types of statistical analysis. 

Continuous Data

Therefore, an important subset of interval data is continuous data, which is data that is generally asked directly and can be measured in small sums (and for all intents and purpose is considered continuous). For example, if 4P management wants to really know in great detail about the average household income of its customers, it may just directly ask what yearly household income is to the nearest dollar. What is your yearly household income?(Please answer to the nearest dollar). Annual Income Asking for continuous data is very powerful because some information can be lost even when asking for interval data (as in the range example above). The statistical analysis of continuous data is powerful as we can easily calculate means and much more advanced techniques such as regression analysis and factor analysis, among others. As a general rule, if you have the opportunity to collect continuous data from your respondents, it is a good idea to do so.  Unless you have serious concerns about the sensitivity of interval data –which is valid for some variables –continuous data will give you the most powerful opportunity to derive the greatest value from your data analysis.

Do the Statistics Matter that Much?

It is clear that some questions simply require companies to capture categorical or ordinal data because there are few (if any) alternatives; however, interval and continuous data may be superior in some situations. While many companies think it best that a responding customer should “click a box” when taking a survey, there is certainly nothing wrong with asking respondents to provide a more specific numerical answer to capture continuous data. In addition, keeping in mind that evenly spaced numeric intervals will provide more opportunity for advanced statistics, will help companies write surveys that provide more meaningful and precise results in the long run.

One potential objection that some companies have to the need for continuous data may be, “but do the statistics matter that much?” When considering the additional value that advanced analytical techniques can bring to a research study, the statistics can matter a great deal.

While a basic analysis involving frequency distributions certainly provides a wealth of useful information initially, more advanced statistical analysis can look beyond the initial analysis to help reveal surprising or counter-intuitive results about demographics, personas and products. In short, it can give companies more insightful and thoughtful conclusions when undertaking custom surveys, which is good for them and their customers.

When it comes to survey writing, the type of questions you ask, and the types of data they bring to your analysis matter a great deal.