Wednesday, August 8, 2012

Scale Survey Question Analyses

How different methods can impact your analyses

First, why median scores are a bad idea:
Medians are good for analyses that incorporate extreme outliers in data.  Household income is probably the best example of when to use a median in analysis.  One billionaire can make an average income analysis skyrocket.  Since we’re analyzing a scale, there’s a distinct and established range.  The percent differences can be significant from one data set using averages in analysis versus another using medians.  See Exhibit A for a random example of 100 respondents analyzed using medians versus averages.

The average analysis was 19% higher than the median analysis in 2011 and 11% lower in 2012.
Conversely, when you analyze the change from between 2011 and 2012, we see a 12% increase in average scoring, while the median analysis shows a 50% increase.  If this isn’t enough to put the nail in coffin of median analyses on scale questions, I don’t know what is.
Second, why rounding averages is another bad idea:
In Exhibit B’s example of two different average scores where Group A = 4 and Group B = 5, Group B’s average score is 25% higher than Group A’s.  What if Group A’s average was actual 4.49 and Group B’s was 4.51?  The difference would be minimal.  What if Group A’s average was actually 3.50 and Group B’s was 5.49?  Group B’s average score would be 57% higher.  The range in difference is anywhere from 0% to 57%.  This would indicate to me that’s it’s a bad idea to round results in analyses.  See Exhibit B for examples of the impact in rounding averages.

Third, why percentage groupings can actually misinterpret results:
I’ve seen reports and analyses that group numbers together from a scale question.  See Exhibit C for an example of grouping sevens or higher on a scale of zero to ten.  All of the examples use in Exhibit C have 25% scoring a seven or higher.
It didn’t occur to me until recently how inaccurate these groupings can be in analysis.  When you start creating a number of different scenarios, some random and some extreme, the variations are a wake-up call.  In Exhibit C, Ex. Max’ and Ex. Min share the same 25% scoring a seven or higher, but Ex. Max’s average score is 19% higher than Ex. Min’s. 

At its most extreme in scoring, the maximum can be 300% higher than the minimum.  See Exhibit D for analysis comparisons in averages (non-rounded) versus the grouping method for 25%, 50% and 75% scoring seven or higher on a scale of zero to ten.

Fourth, converting the scale analysis into percentage representation:
Scale analysis can be converted into a percentage analysis (as already seen in Exhibits C and D).  If you’re using a zero to ten scale, simply moving the decimal point converts your average scores into percentage representation.  At its lowest, the entire sample giving zeros equates to 0% and at its highest, the entire sample giving tens equates to a 100%.  If you’re using a scale other than zero to ten, you’ll need to use a less obvious conversion formula.  See my original scale post on why zero to ten is a superior scale.  The conversion formula for varying scales other than zero to ten can be found in Exhibit E.

For an example of scale impact and conversion on a scale of one to ten, see Exhibit F.

Not all analyses are equal.  In fact, some can be downright deceptive.  My advice: when you can, use zero to ten scales, conduct average (non-rounded) analyses, and convert to percentage analyses when needed.