Surveys can help researchers with many aspects of understanding a given market and answering important questions such as:
- How have trends changed?
- What does a brand’s target audience look like?
- What’s the best way to segment a market?
- Which segments are the most important to target?
- What are potential areas for market expansion?
Because of the weight that surveys hold on informing business recommendations, it is so important that the survey responses are accurate. As this type of research is used to represent a specific population, a large number of respondents are needed to complete the survey. Most of the time respondents are incentivized to participate in the study. While in a perfect world every respondent is truthful, sometimes people are more focused on receiving an incentive than answering the survey truthfully. A few bad seeds also have created survey bots, which is a way for one person to collect multiple incentives quite quickly.
So how can you prevent bad or fake respondents from getting into your survey? There are several methods to help make your survey as accurate as possible.
Tips for Catching People Mindlessly Speeding Through Surveys
How Long it Takes Respondents to Answer a Question
The simplest way to tell if a person is speeding through the questions and not taking the time to read them is by looking at how long it takes them to complete the survey. After the researchers have determined a normal time to go through the questions, they can weed out the respondents who took an absurdly short time to answer. For example, respondents who take less than a second to answer each question are red flags.
Straight Lines and Patterns
Often, respondents rushing through the survey will fill out the same answer to every question. For example, if it is a grid-style question where respondents are asked to indicate, through a scale, how likely they are to do something or agree with a statement, they may choose neutral for all responses. The phrase Straight Lines gets its name because, for a grid, the shaded-in responses will form to create an image of a straight line column going down the page. On the other hand, instead of a straight line, respondents filling out the survey in a patterned format may also indicate that the questions aren’t being fully read. These responses may look like Christmas trees or triangles.
Selecting All Possible Responses
Sometimes survey questions will ask respondents to select all the answer choices that apply. For example, a question might ask what the respondent has purchased in the past month. Possible choices could include fifteen choices such as a house, car, watch, and groceries. If the respondent chooses all of the possible choices, which appears highly unrealistic, this may indicate that they are not spending enough time going through the survey.
Tips for Catching Respondents that Answer Surveys Illogically or Incorrectly
ReCaptcha In Surveys
This is the feature that is added to prevent bots from entering the survey in the first place. It is an interface that determines if the user is human based on cursor movements and clicks. Think of when you’re asked to confirm “I’m not a robot” and then to choose all the pictures that include a specific object, such as a stop sign or a certain color car.
Opposite Answers
Respondents that answer survey questions that directly contradict something they have previously recorded are important to look out for. For example, if a respondent records that they are vegan at the start of the survey and then subsequently answers a question indicating that they eat meat often, that is a red flag.
Red Herrings
Researchers often throw in unrelated questions or answers to surveys to ensure that the respondent is paying attention. For example, surveys might include a trick question, where the statement will say, “Pick Neutral to show that you are paying attention”. It also might be a simple arithmetic question such as, “what is three plus four”. A third example is including ridiculous examples for answer choices. A question could ask about what the respondent has done in the past month and give an improbable choice such as “winning $1 million from the lottery”. All of these questions or answer choices are used to weed out participants who aren’t paying good enough attention or are zoning out.
Tips for Open-Ended Questions in Surveys
Nonsense Open-Ended Questions
When surveys ask open-ended questions like, “Why are you optimistic about the future” and the answer is something that doesn’t make sense or is just a series of randomly generated letters or words, it is most likely a bot. Even if it isn’t a computer-generated response, respondents who answer nonsensically should be deleted regardless because they aren’t taking the survey seriously.
Duplicate Responses
When responses to open-ended questions are generic and seem repetitive among different respondents, they are probably bots. Especially when the answers don’t exactly fit the question, such as the answer “I am optimistic” to the question above, “Why are you optimistic about the future”. Additionally, instead of hand-sorting through the responses, many companies will use IP address blockers to prevent a single device from submitting the survey more than once.
Conclusion
It is important to review and “clean” the data after receiving all of the survey responses to ensure that the conclusions are as accurate as possible. Also, it is worth mentioning that if the survey response meets one of the criteria mentioned above, it isn’t necessarily grounds for an instant deletion of the response. It is important to go back and check through the whole survey to determine holistically if the respondent is suspicious.
Interested in reading more on market research? Check out our other blogs here.
Are you interested in our market research capabilities?
