Friday, May 4, 2012

Research Weighting


A two part post on how to calculate and apply to analyses

A lot of secondary research companies provide data/information that has been weighted to reflect the general population surveyed based on gender, age, ethnicity, geography and income (sometimes more).  It sounds extremely complicated, but I’ve learned it’s not necessarily as complicated as some would lead us to believe.  If you’ve conducted a primary study and need to provide basic weighting to ensure there isn’t over or under representation from specific segments, this two part post may help with providing a basic solution.  In Part 1, I discuss the importance of weighting and how to create weights.  In Part 2, I will demonstrate how to apply the weights and integrate them into your analyses.

Part 1
Why is weighting important or necessary
he hypothetical employee satisfaction study in Exhibit 1 shows us how actual distribution and respondent distribution can impact analyses.

ACME Corporation has more than 1,200 employees and received nearly 60% participation (750 respondents) in their employee survey.  The company is divided into eight different departments and the results are intended to help prioritize direction at the company and department level.  Since the dynamics spanning each department can be very different (pay, hours, responsibilities, work environment, etc.), it’s important to understand that looking at respondent based results can give you the wrong impression of the reality of the overall business. 

The Sales Department represents less than a fifth of employees (17%), yet they represented more than a quarter of feedback on the survey (27%).  If the Sales Department feels strongly toward a specific question, it can give the appearance that the company feels this way as well.  This department is over represented in the respondent based results/analysis.

On the other hand, a third (33%) of the company’s employees works in the Production Department, yet less than a fifth (16%) of the survey respondents works within this department.  If their responses feel strongly toward a specific question, it may not be as evident in the overall results.  This department is under represented in the respondent based results/analysis. 

Apply or removing weight can help alleviate the issues of under or over representation.  See Exhibit 2 for an example of the difference in results to questions when they are analyzed with and without weighting.

Without the weighting, decision makers may be lulled into believing things are better or worse than they really are (better in Exhibit 2’s case).  The weighting helps us paint a more accurate representation of reality.

How to create/calculate basic weights

Using the same hypothetical employee survey from “ACME Corporation,” I will show you a simple means of creating individual weights to apply to each respondent’s answers.  First, we need to determine to complexity of our weighting.  In our example, we simply want to ensure each department has equal representation in the overall results, which means each respondent will have 1 of 8 possible weights apply to their responses.
There’s actually two different ways you can use to calculate weights.  See Exhibit 3 for formulas.

Each of these calculated weights will need to be appended to the results, corresponding to the appropriate segment.

How do we know these weights are accurate?  First, when each weight is applied to the 750 respondents, the sum of the weights equals 750.  Second, the sum of each department’s weights divided by the sum (750) equals the same percentage of the actual employee count.  See Exhibit 4 for a demonstration.

For more complex weighting, the same formulas hold true; the biggest difference being the number of weighted segments you end up using.  In my second hypothetical, we conducted a consumer survey in the three counties of Ocean, Suburbia and City Center.  We want to ensure that no county, gender, or age segment is over or under represented (assuming these metrics were collected in the survey).  This adds two additional layers of complexity to our ability to calculate and apply weights. At its most granular we will need every respondent to receive 1 of 18 possible weights.  Three counties broken down by two genders give us six segments to weight.  Then, these six segments are then broken down by three age segments giving us a total of 18 in our example.  See Exhibit 5 for an example of the breakdown in actual data and survey data by the 18 segments. 

Using Weighting Formulas A or B should give use the weights we desire.  Using the same exercise of evaluating weights in Exhibit 4 will help validate your weighting.  See Exhibit 6 for an evaluation of our more complicated weighting example.

In Part 2 of Research Weighting, I will demonstrate how to apply the weights to results and analyses.