Confidence intervals and p-values

Introduction

When we get a result from an epidemiological study, the result can be due to three things:

  • Chance (also known as random error)
  • Bias or error
  • Confounding

The impact of chance on the results of a study is usually expressed as p-values and confidence intervals. The impact of chance on results is mostly determined by the sample size of the study.

P-values

The p-value (or probability value) represents the probability that the association shown in a research study could have occurred by chance alone, if there was no actual relationship between the exposure and outcome.

The size of the p-value helps you understand the possible impact of chance on the result.  A statistically significant p-value is traditionally set at less than or equal to 0.05 – this number means that if you repeated the same study 20 times, the probability that the observed result could occur due to chance alone is 5 out of 100 (or 1 in 20). Similarly, a p value of 0.001 means that the probability of the result being due to chance is 1 in 1000. The smaller the p-value, the less likely the results are due to chance.

P-values can be very useful because their method of calculation includes a lot of information, including the sample size, the sampling variation, and difference from expectation, all in one convenient number. Many modern statistical software packages such as SPSS or SAS will calculate the p-value automatically for many different types of statistical test. However, the p value can often be misused. The p value is heavily influenced by sample size – the bigger the population the more likely you are to find a significant result of some kind, even if the statistically significant result is clinically meaningless.

Confidence intervals

Confidence intervals provide an indication of the precision of the result obtained from the study. The confidence interval expresses the concept that the result achieved from the study is probably not 100% accurate, and that the real answer lies somewhere within a given range – the confidence interval is the range within which the true answer is likely to lie.

Traditionally confidence intervals of 95% are used. A 95% confidence interval means that if we repeated the same study many times, then we would include the true result in our interval 95% of the time.

Confidence intervals are influenced by sample size; a small sample will typically give a wide confidence interval whilst a large sample will give a narrower confidence interval. A narrow confidence interval indicates that our result is quite precise.

Question

An outbreak of Salmonella Dural occurred in a large town. To identify risk factors for infection, the public health unit conducted a case-control study with 13 cases and 25 controls frequency-matched for age and sex. Telephone interviews were conducted using a standardized questionnaire. Raw dragonfruit consumption was the only exposure significantly associated with illness (OR, 26.7; 95% CI, 5.4-101.5; P = .003).

  1. What is the confidence interval?
  2. What is the p-value?
  3. Write a sentence interpreting these two results
  4. What do you think of the size of the confidence interval? What could you do to change the size of the confidence interval?
​​​

Answer

Current as at: Friday 22 December 2017
Contact page owner: Health Protection NSW