Friday, May 11, 2012

Research Weighting

Part 2: How to apply to analyses
Now that we understand the value of weighting and how to calculate them (click here for Part 1 of post), let’s look at how to incorporate them into analyses.
To simplify, I’m going to use a smaller hypothetical to demonstrate how to apply and analyze research weights.  I also wanted to ensure the exhibits don’t get ridiculously large.  Mid-Atlantic Sales has 124 employees, of which 52 completed an employee survey.  Each employee is based in a territory devised by states.  Each of these territories is very different, so using weights will help ensure that no territory will be over or under represented in the overall results/analysis.  Exhibit 7 includes the breakdowns and weights we will need for our analysis. 

As you can see without weighting, North Carolina and Washington D.C. would have been over represented, while Delaware, Maryland and Virginia would have been under represented.
First we need to add a column to the survey results that includes the appropriate weight for each respondent (See Exhibit 8). 

For our example, any respondent working in Virginia would receive the weight of 1.258065.  By adding a sum at the bottom of the weight column we are able to verify the weights are accurate (in this case 52).
Results Tabulation/Analysis:
To calculate percentage based questions from the respondent data without weights we simply sum the responses specific to the answer and divide by the total respondents.  To calculate a weighted percentage, we sum the weight of the responses specific to the answer and divide by the weight sum (52 in our hypothetical).  See Exhibit 9 for formulas and examples of calculations and results. 

As you can see, the percentage of unweighted respondents answering yes to “Question 1” is 13% higher than the weighted results (52% unweighted vs. 46% weighted).  This demonstrates how different results can be if we don’t include appropriate weights.
To calculate averages in scale questions (on a scale of 1 to 10, or 1 to 5, etc.) we first need to include a column for a “Scale Weight.”  It’s simply multiplying the scale response number by the assigned weight for the respondent (See Exhibit 10 for example). 

We also want to add a sum of the “Scale Weights” at the bottom of the column.  The unweighted average is simply the sum of responses (scale numbers) divided by the total respondents.  The weighted average is the sum of the Scale Weights divided by the Weight Sum (52 in our hypothetical). Exhibit 11 provides the calculations for the scaled “Question 2” in percentages and averages. 

Again you can see how unweighted results can vary from weighted results.
While my hypothetical is pretty simply, the method still applies to more complex examples and surveys.
I have to give this two part post geek approval.