Are Red Herring Questions Effective In Improving Survey Validity?


We’ve all been there: you’re asked to fill out a survey to claim a small prize, be it a pair of movie tickets or a free sandwich, but the last thing you want to do is a survey. So you just barely skim the questions or “Christmas tree” the responses, submit the survey and claim your prize. Easy win for you, but a statistical nightmare for researchers.

Such situations were the subject of research by Facebook and Survey Monkey (Liu & Wronski, 2018), appearing in the International Journal of Market Research, which focused on the practice of including red herring questions, aka trap questions, in surveys; questions which assess how focussed participants are while completing the questionnaire.

Studies have shown that while responses to online surveys can be more honest, participants can also be otherwise less engaged than they would be in interviewer-administered surveys. Red herring questions in surveys, which may ask participants to read a passage and select the order in which certain items appear, or a question which directs respondents to select a specific answer, can gauge how closely a surveyee is paying attention to the questions, and help researchers weigh results accordingly. 

It has been shown that respondents fail a red herring question can add noise to survey results, and in the worst cases, make the conclusions drawn from the survey completely invalid, according to  Berinsky, Margolis, and Sances (2014). The researchers recommend using multiple red herring questions, and lieu of simply disregarding those respondents who failed, analyzing them separately from those who answered correctly. 

Participants can choose to answer falsely for many reasons; perhaps the survey is too long or tedious, or perhaps there is a prize at the end that they are too eager to reach. While making surveys short and sweet and adding a screening phase to increase the sense of commitment in participants are both good strategies to improve validity, it also helps to throw in a red herring question or two to keep participants on their toes. This may lead to some respondents feeling their trust has been violated, it is a minor infraction and researchers must have a way to demonstrate that participants are appropriately tuned in.