Saturday, August 4, 2012

Scale Preferences in Surveys

Is there an ideal scale solution?

I’m not sure they answer is completely definitive, but as an analyst and researcher I’ve primarily a fan of the zero to ten scale (with each end given an extreme response description).

First, why are scales important?  I’ve touched on the importance in previous posts like Evaluating Media and Advertising Allocation, Brand Differentiation Research Strategy, and The Improvement Gap Analysis & Method.  To summarize, scales help differentiate the degree to which people feel toward certain questions.  Yes or no responses are not always an option in consumer perceptions and feelings.

This brings to bare a couple of additional questions: What’s the best numeric scale to use in analyses and what scale is easiest for the respondent to interpret?  If it’s a paper of phone survey, sophisticated and easily interpreted scale questions can be a real challenge.  It can’t have too many radial points (e.g., you don’t want to ask on a scale of 0 to 100, where 0 = ABC and 100 = XYZ) or too many descriptors defining each point (e.g., completely, somewhat, rarely, never, neutral, etc.).  If we don’t include a number of options, our ability to analyze differentiation in responses becomes more limiting (e.g., on a scale of one to three doesn’t give us a whole lot of information to differentiate between responses). See Exhibit A for varying examples of scale questions.

Thankfully interactive online surveys exist, but surprising very few seem to have taken the leap from the traditional phone and written survey methods.  Online surveys have some real untapped potential for finding the sweet spot between maximizing participation and analytic reliability/differentiation.

Given the challenges presented in traditional collection methods, I’m going to focus on the ideal for interactive online scales.

Even vs. Odd Number of Scale Collection Points:
  • 1 to 10 (10 options) or 0 to 10 (11 options)
  • 0 to 7 (8 options) or 1 to 7 (7 options)
  • 0 to 5 (6 options) or 1 to 5 (5 options)
I’ve usually heard the value of even numbered scales is that it requires respondents to choose or lean toward one of the extremes on the scale presented.  I understand the desire to acquire definitive feelings, but the truth is some people have no feeling, one way or another, toward certain things (an inconvenient reality to some decision makers).  I believe eliminating the neutral option interjects bias into the results and ultimately the analysis.  I feel it’s “extremely important” to use odd numbered scales; ones that have the option to select a true mid-point of neutrality.  But sometimes you have to deal with the scale that may be given to you to analyze.  We’re not always in a position to choose.

Number of Scale Points:

3, 5, 7, 9, 11, 13….?

Well, we know one scale point isn’t an option and I’m ruling out even numbered scales.  Here’s what I do know: “Research indicates a positive relationship between the number of scale points and reliability.*”

*Marketing Research, Methodological Foundations, Tenth Edition, © 2010

Having a large number of scale points is important for analyses.  If you’re using a radial point scale collection method, having a scale exceeding zero to ten or one to eleven (eleven points) can look overwhelming.  With a radial scale point display, I would recommend not exceeding eleven.  It’s important to note: if you are going to display the numbers on the scale, your maximum scale should probably be zero to ten.   Fred Reichheld has written books on the importance of this scale and its ease of translation to potential respondents.  Zero is typically defined as bad, and ten is usually associated with the highest of marks.  One to eleven wouldn’t work, because, eleven is not commonly associated with perfection or rankings.  Sometimes one is considered the best as well without a clear definition (very similar to the defining an ace as high or low in card games).  This lack of a clear association may create confusion.

Here’s where an interactive scale can help us overcome participation and visual fatigue.  The use of sliding scale displays enables us to remove the need to display numbers (see options 5 and 6 in Exhibit A for visual examples of sliding scales).  With the numbers coded into the background, you need not worry about confusion or overwhelming radial point displays.  In fact, the scale coded into the background is ultimately up to the analyst developing the interactive survey.  The sliding scale can even get us a scale greater than eleven points to help maximize reliability.  Technically we would be limitless, but zero to a billion sounds like a bit much.  If you don’t like the idea of 101 points spanning zero to one hundred, simply create 101 points spanning zero to ten.  It’s simply moving the decimal point to the tenths (e.g., 0, 0.1, 0.2, 0.3 …all the way up to... 9.8, 9.9, 10).  I have yet to see this offered as a scale option, but I’d love to have this capability.  This brings us to our last quandary of where to start our scale.

Starting a Scale with a One or a Zero:

If you’re displaying numbers, it’s actually pretty arbitrary in terms of what you use as long as the numbers are clearly defined.  It’s ultimately the number of scale points that dictates the strength of your analysis.  A one to ten scale is essentially the same as a zero to nine scale, or as crazy as it may sound, a two to eleven scale.  I would hope it’s self evident that if you’re using a low end descriptor of “not at all, none, never”, or anything that’s a definitive null, than zero is the best number to start with on the scale.  I really don’t have a case for using one and I’m not completely sure why scales do start with a one for display or analysis purposes.  Until I hear a solid case or rationale for using one to start the scale, I’m going to stick to zero when given the option.  Zero to ten data collection also has the easiest conversion to percentage analyses.  I’ll touch on the analyses of varying scales, demonstrate this conversion to percentage analyses, and limitations of starting with a one on a scale in a future post.  I’ll also include some best practices in analyzing scale results.  I’ve seen some people use average scores, others median, and some use percentage score specific to ranges (e.g., percent of respondents scoring eight or higher on a scale of zero to ten).  I’ll point out issues with some of these different analysis options.  The short answer to this future post would simply say: “use the average scores when completing analyses,” but I’ll demonstrate the reasons.

Obviously the choice and preference in scales used is ultimately yours, but it is worth considering the implications of your choice.  Exhibit B represents my preferences.

I’m hoping to see some of interactive scales I reference here available through some of the mechanisms I use in the near future.