Pros and Cons of Samples (Investigation)

Investigation

You can review ideas about statistical sampling by referring to some or all of the following chapters: 

Obtaining representative samples

How do you collect your data?

Sampling techniques

In this chapter, you are invited to devise and to carry out your own statistical investigation, possibly making use of the following examples and the accompanying discussion for guidance. Substitute your own scenarios and make your own decisions about sampling methods.

Sampling is done to gather information about a population of experimental subjects when it is not possible to obtain data about every member of the population or when it is too expensive to do so.

In another chapter, the example of conducting a taste test on a batch of cookies was discussed. Clearly, it would be impossible to test every cookie because the testing process is destructive and there would be none left if all were tested. In other situations where the cost of measuring the experimental subjects in a population is high, it is recognised that a well-designed sample survey can provide data that is almost as good as that which would be obtained by measuring every subject.

The phrase 'almost as good as' in the previous sentence calls to mind the question of how large a sample should be in order to be confident that inferences drawn from it will be sufficiently reliable. There are ways of doing this, to be explored in subsequent work.

Example 1

Often, the question asked in a survey or in an experiment has either a 'yes' or a 'no' answer for each experimental subject. Thus, the end result is a count or a proportion of the number of positive responses in the experiment.

Quality control in a production process can be of this nature: items either do or do not meet the required specifications.

Sampling of products coming off an assembly line will often be done by  systematic sampling in which, say, every 20th item is tested. This method of sampling will usually give a representative sample of the production process provided the actual failures occur randomly.

But, consider a situation in which the production process is for some reason producing some faulty items randomly and also some faulty items systematically so that, say, every 15th item would fail if it were to be tested. The 1 in 20 testing procedure might pick up the correct proportion of randomly occurring faults but at best it would detect the systematically faulty products at the rate of only one in sixty, 60 being the lowest common multiple of 15 and 20, and it would miss the systematically faulty items completely in four experiments out of five, 5 being the highest common factor of 15 and 20.

As a consequence, the business selling the product would  make an inadequate allowance for warranty in its pricing to cover the cost of one claim in every fifteen items. Furthermore, the business would not be alerted to the systematic failure effect in its production process.

For entirely different reasons, another researcher might want to know how many people had ever been to Portugal. To the question 'Have you ever been to Portugal?' each subject would be expected to answer either 'yes' or 'no'.  

The investigator might choose to question random people encountered in a shopping mall. This is called a convenience sample. The sample might provide good information about the population restricted to those people who visited the mall when the survey was being carried out and who were willing to speak to the investigator, but it is unlikely that it would be representative of the wider population.

To provide reliable information about a population, a sample needs to accurately represent the population from which it is drawn. This tends to occur when every subject in the population has the same probability of being selected for the sample. 

Example 2

In many investigations, a measurement of some kind is made on the experimental subjects. We might want to know how tall some grass grows after a certain treatment or how bright the stars are on average or into what weight range most trombone players fall. In each case, a sampling process will be needed.

In the case of the grass, the experiment might involve treating sample plots of land on which grass is to be grown with two different levels of superphosphate fertiliser. Ideally, the plots would be selected randomly from some larger area of land considered to be the 'population'. Each plot could then be divided into two portions with a different treatment level being applied to each portion. The random selection of plots would be done with a view to eliminating unknown bias effects from the experiment. Sampling by this method produces a simple random sample.

However, within the larger area of land, there are some hilltop regions and some low-lying areas and it seems clear that these known differences could influence the outcomes of the experiment. The experimenter determines that 25% of the land can be classified as 'hilltop' and so the sample of plots should include 1 in 4 randomly selected hilltop plots in order to ensure that the sample is truly representative of the population.

This kind of sampling is called stratified random sampling. It is used to minimise known sources of bias, for example in allowing for differences between male and female trombone players. When strata in a data set are recognised, it is possible to report the results for each stratum separately and to check whether the supposed stratification effects are real.