topic badge

5.03 Bias in questioning

Lesson

Collecting data from people in a sample or population, often involves asking them questions and recording their responses. One of the most common survey methods is the questionnaire, where participants answer a set of questions on a printed or online form. Other survey methods, like personal interviews, can be used to collect data in a similar way. 

The quality of the data collected depends a lot on the quality of the questions being asked. Some important considerations when designing questionnaires are:

  • Using simple language that is easy to understand
  • Respecting people's right to privacy
  • Considering the ethics of the issue being analysed
  • Asking questions that are clear and concise, with no ambiguity
  • Asking questions that are fair and unbiased
  • Only asking questions that are relevant
  • Allowing for a range of different responses
  • Keeping the overall length of the survey appropriate
  • Ensuring the survey is easy to complete

The two main types of questions used in questionnaires are either 'closed' or 'open'.

  • Closed questions include those where the participant answers by selecting either YES or NO or they choose their answer from a list of options (multiple choice or indicating a value on a scale). This is by far the most common question type because it is relatively easy to answer and can be recorded automatically by a computer. 
     
  • Open questions are those that require a more detailed written response. These questions may allow for a more accurate response and can often pick up details that are missed in closed question types. The main drawback with open questions is that they are difficult to analyse using computerised methods. They may be more appropriate though for interview-style surveys.

Our goal in surveying is to obtain accurate information about from our sample that we can make useful inferences about the population as a whole. As we saw previously, inaccurate data can arise from sampling errors and/or measurement errors

For surveys a common form of measurement error is response bias. This is when participants respond inaccurately or falsely to questions. This can occur for a wide variety of reasons:

  • The participant has not fully understood the question, this is more likely to occur if questions are complicated or ambiguous.
  • Subjects may like to present themselves in a favourable light and under-report information about drinking habits or unpopular beliefs, particularly if survey results are not confidential.
  • Phrasing of the question may lead a subject to give a particular response, this could include use of emotive language or body language of the interviewer that implies a certain answer is preferable.

Non-response to part or all of a survey can also lead to biased results. For example, posting a survey on the location of a nuclear waste site will likely only see those with a strong opinion or directly impacted make the effort to answer. This would bias the results as we are excluding those with moderate views. Subjects may choose not respond if:

  • They see the survey as inconvenient, such as extended answers are required, the survey is considered too long or requires effort to return.
  • They do not see the topic of the survey as relevant to themselves or may not have strong opinions on the topic.
  • They feel sceptical about the confidentiality of the survey.  

In summary, a well designed survey is key to obtaining quality data. Further detail on some aspects of survey design can be found in our investigation.

 

Practice questions

QUESTION 1

Which of these questions are fair?

  1. Do you feel that the TV news is an inaccurate portrayal of life’s problems?

    A

    Don't you think this newspaper is biased?

    B

    Do you prefer the look and feel of thick lush carpeting in your living room?

    C

    Do you take these extra strength multi-vitamins to supplement your diet?

    D

    None of them

    E

QUESTION 2

QUESTION 3

Adults attending a local cinema were asked the following question:

“How many times did you see a movie at this cinema last year?”

  1. Is this an example of Sampling Error or Measurement Error?

    Sampling error

    A

    Measurement error

    B
  2. What type of Measurement Error is this?

    The scale provided is inadequate.

    A

    This is an assumption based question.

    B

    Poor and/or leading question wording.

    C
  3. What is the main reason why this question is poor?

    Poor and/or leading question wording

    A

    The people being surveyed need more information.

    B

    Relies too heavily on respondent memory.

    C

Outcomes

3.4.7

investigate questionnaire design principles; for example, simple language, unambiguous questions, consideration of number of choices, issues of privacy and ethics, freedom from bias

3.4.8

describe the faults in the collection of data process

3.4.9

describe sources of error in surveys; for example, sampling error and measurement error

3.4.10

describe possible misrepresentation of the results of a survey due to the unreliability of generalising the survey findings to the entire population, for example, because of limited sample size or chance variation between samples

What is Mathspace

About Mathspace