Friday, December 14, 2012

Reverse Logic Questionnaire Method

    
Reducing bias/priming and strengthening reliability through analysis
I’ve seen a lot of surveys that following a stream of questions in an order similar to this (I’m using new vehicles simply to give context):
1.) When was the last time you bought a new vehicle,
2.) When do plan on buying a new vehicle,
3.) What sources do you use to help make decisions to buy a new vehicle,
4.) When you are looking to buy a new vehicle, how important is it to you that the dealership offers/provides the following,
5.) Based on your perceptions/experiences how would you rate the following dealers in the areas of quality, service, etc.,
6.) Which dealership would you considering buying from first, second or third.
I never gave it much thought until the last few years that the order presented above should be reversed.  In its current order we are priming potential respondents.  Not “obvious priming” like:  “Dealership X has a bad reputation for service, how likely are you to consider buying from Dealership X.”  This priming is more akin to leading consumers through a decision making process they never normally gave much thought as they experienced it.  While the current order is in a similar order to how people make decisions, but this doesn’t make for a good question order.
I want to peal the layers of the onion back to the core, not cut straight to the core and try to reconstruct the onion.  We instinctively have an idea where we would likely buy from, but don’t necessarily have an instinctive idea of why; basically because we’ve already gone through an evaluation process.  This is why I like the idea of reversing the traditional method of the question stream outlined above.  If we get respondents thinking about what they believe is important to them and then follow this with questions about how businesses rate on some of these same areas, it may lead them down a different path of consideration.
Some use the initial questions as qualifiers for the survey (or to provide skip patterns).  Of those who haven’t bought or plan to buy don’t answer any remaining questions.  Skip patterns have a place and value, but if you want full market perception and analysis, why would we want to discriminate.  This is where the argument between the value of questionnaire development and analysis meets.
The great thing about the reverse order is how it helps qualify the survey you conducted.  You can see how responses of those that have bought or are in the market to buy differ from the rest.  If there are not any differences, they may be something wrong with the sample.  The differences need to make sense too, for example the average scores for importance questions should be higher among respondents in the market to buy the specific good/service you’re measuring.
The elimination of skip patterns also helps to provide an analysis of cold, warm and hot prospects, and identify overall market perceptions.  I’ve found the reverse logic creates more reliable and diverse analyses.