Day: January 4, 2021

The Current Status

Screening questions are regarded by the research industry as the boring part of a questionnaire, a collection of hurdles to be jumped or failed by the respondent. Most researchers will typically copy and paste them from previous questionnaires once targeting criteria have been established… then give them little thought.

And why should they? Each screener is simply a stepping stone to the more important questions in the survey, a binary ‘yes’ or ‘no’ onto the next question or into the pile of failed respondents. In fact, screeners are so often looked at as poor relations that they are usually ignored when counting the number of questions in the survey. 

Moreover, to go beyond the formulaic approach is to invite the dual dangers of bias and jeopardizing consistency with past research into the same target group. 

My thoughts on the matter are informed by reading thousands of expert consumer and B2B questionnaire assessments. The experts in question were the respondents themselves who take online surveys. And because I come from the panel side of research, I see all too often what happens when screening goes wrong – delays with re-drafting, frustration, data quality issues, and ‘blame games’ between fieldwork providers and researchers. 

So, to avoid those kinds of problems, I’m offering up seven tips to improve your screening questions in the future… 

1. The Golden Rule

If you are going to tinker with screeners, avoid the opportunity for bias, and deploy a rigorously neutral tone.

Start by using standard approved categories and variables for job title, function, vertical, industry sector, company size, and so on. But also use variation to ‘hide’ the titles, sectors, functions, etc., screening in if they are quite specific and would otherwise stand out.  

When using non-hierarchical categories such as the industry sector, consider randomizing the options to avoid bias.  

2. Priming & Chunking

Sectioning sets of screening questions with headings can help to prime respondents. This works well with knowledge-assessment and behavioural screening questions, and narrative logic can be used to smooth natural transitions. Using headings or introductory priming tends to increase the thinking time taken on a question. 

Example: you have already established the subject’s role and decision-making capacity concerning xxx. “Now, we would like to you to consider your role with your company’s use of xxx” would provide a precursor to questions such as: “How often do you use yyy?” or “Which consultancy services have you considered when decided on zzz”. This ‘priming’ will produce more thoughtful answers than launching straight into the question. If used sparingly, this technique will produce better data, particularly when you want careful thought applied to answers.

3. Global Corporation Size 

Ensure you are capturing all representatives from a target group of companies which may naturally consider themselves separately. Ask first whether the respondent has offices outside their country to ensure that the global headcount is considered, then route accordingly.

4. Don’t Force Answers

There is nothing more frustrating for respondents than encountering a question they cannot honestly answer. Feedback from respondents bears this out. Early on this happens in the survey, the more your data quality will suffer. If you’ve ever taken a multiple-choice personality test and wished ‘None of the above’ was an option, you should sympathize. However, ‘None of the above’ or ‘If Other, please specify’ should only be used if the most likely options have been included. Also beware of ‘false alternatives,’ – forcing the respondent to select one option when two or more apply equally.

5. Avoid Huge Grid Questions…

… they only increase dropout and reduce your available sample pool.

6. Are You Sure You Don’t Want Them? 

Think about the class of excluded respondents. Some may have views that might inform the research. Think about technology products, for example. It’s not just CIOs and CTOs that need to be invited to participate in a study, what about IT Managers and Analysts, the ones who actually would use the product?

7. And Finally…

The screen test: always put yourself in the respondent’s shoes. Do the screeners flow? Are the options exhaustive and clear?

Screening questions are never the sexiest part of a B2B survey, but for good research, they must be well-thought-out and implemented properly. Good luck! 

Any comments welcome: David.winter@www.irbureau.com