Using “Research on Research” to Improve Market Research Survey Design
Market researchers rely on a variety of ways of obtaining information from respondents who complete online surveys. Some survey methodologies are very similar to each other, attempting to collect the information in different ways. Recently we evaluated two methods of ranking attributes: the drag and drop method compared to numeric entry. We investigated two aspects of each of these: Did these methods provide us with different results? Is one method more appealing to respondents compared with the other? We sought to understand whether one of these methods was better than the other as way to increase respondent satisfaction (and thus future response rates) without a change in the end-result of the data.
To conduct this research we used KJT Group’s LightSource Poll. The LightSource Poll is a nationally representative, online public-opinion poll that is conducted quarterly. As part of the LightSource Poll, we conduct “research on research” to test quantitative research methods. Using these results, we can provide evidence-based recommendations to our researchers and our clients in an effort to design better surveys in the future.
As part of the survey, we randomly assigned half of all respondents (n=1,000) to the drag and drop and half to the numeric entry method. Using the drag and drop method, respondents drag attributes to physically rearrange them from best to worst. By comparison, numeric entry utilizes a text box next to each attribute and respondents type in the rank of each attribute. Respondents were asked to rank 11 issues based on which is most important to the United States. The list of issues was identical, regardless of ranking method, and the issues were randomized for every respondent. After completing the exercise, we included follow-up questions to gauge how much respondents enjoyed the exercise and how difficult it was to complete the exercise. We used the Wilcoxon rank-sum test to determine if the rankings for the two methods were significantly different and student t-test to assess ratings regarding the techniques and time to complete
The results of the ranking exercises did not reveal statistically significant differences in mean rankings, difficulty of completing the exercise and time to complete. However, respondents reported the drag and drop method to be significantly more enjoyable than the numeric entry method (p value <.05.)
Based on these findings we plan to utilize the drag and drop method for attributes rankings in our online surveys whenever possible. Given equivalence across all other attributes, we will leverage the exercise that is more enjoyable for respondents.
At KJT Group, we believe using evidence to guide our research is critical. We will continue to conduct “research on research” using the LightSource Poll as a way to provide greater evidence as to why certain methods may be preferred to others. We look forward to sharing future posts comparing various research methodologies.