12.03 Bias in questioning

Collecting data from people in a sample or population, often involves asking them questions and recording their responses. One of the most common survey methods is the questionnaire, where participants answer a set of questions on a printed or online form. Other survey methods, like personal interviews, can be used to collect data in a similar way. 

The quality of the data collected depends a lot on the quality of the questions being asked. Some important considerations when designing questionnaires are:

• Using simple language that is easy to understand
• Respecting people's right to privacy
• Considering the ethics of the issue being analysed
• Asking questions that are clear and concise, with no ambiguity
• Asking questions that are fair and unbiased
• Only asking questions that are relevant
• Allowing for a range of different responses
• Keeping the overall length of the survey appropriate
• Ensuring the survey is easy to complete

The two main types of questions used in questionnaires are either 'closed' or 'open'.

• Closed questions include those where the participant answers by selecting either YES or NO or they choose their answer from a list of options (multiple choice or indicating a value on a scale). This is by far the most common question type because it is relatively easy to answer and can be recorded automatically by a computer. 
   
• Open questions are those that require a more detailed written response. These questions may allow for a more accurate response and can often pick up details that are missed in closed question types. The main drawback with open questions is that they are difficult to analyse using computerised methods. They may be more appropriate though for interview-style surveys.

Our goal in surveying is to obtain accurate information about from our sample that we can make useful inferences about the population as a whole. As we saw previously, inaccurate data can arise from sampling errors and/or measurement errors. 

For surveys a common form of measurement error is response bias. This is when participants respond inaccurately or falsely to questions. This can occur for a wide variety of reasons:

• The participant has not fully understood the question, this is more likely to occur if questions are complicated or ambiguous.
• Subjects may like to present themselves in a favourable light and under-report information about drinking habits or unpopular beliefs, particularly if survey results are not confidential.
• Phrasing of the question may lead a subject to give a particular response, this could include use of emotive language or body language of the interviewer that implies a certain answer is preferable.

Non-response to part or all of a survey can also lead to biased results. For example, posting a survey on the location of a nuclear waste site will likely only see those with a strong opinion or directly impacted make the effort to answer. This would bias the results as we are excluding those with moderate views. Subjects may choose not respond if:

• They see the survey as inconvenient, such as extended answers are required, the survey is considered too long or requires effort to return.
• They do not see the topic of the survey as relevant to themselves or may not have strong opinions on the topic.
• They feel sceptical about the confidentiality of the survey.  

In summary, a well designed survey is key to obtaining quality data. Further detail on some aspects of survey design can be found in our investigation.

 

Practice questions

QUESTION 1

QUESTION 2

QUESTION 3