The Level of Significance: Setting Criteria for Hypothesis Tests

Every time a researcher runs a hypothesis test, they face a key decision: how much risk of being wrong are they willing to accept? This threshold – known as the level of significance – is one of the most fundamental concepts in statistics. It determines whether findings from a study are considered statistically meaningful or simply the result of random chance. Whether you’re testing water quality in a river, measuring the effect of a new fertilizer, or studying the impact of air pollution on respiratory health, the level of significance shapes the conclusions you draw from your data.

What is the level of significance? 🔗

The level of significance, denoted by the Greek letter α (alpha), is the probability of rejecting a true null hypothesis – in other words, the risk you’re willing to take of concluding that an effect exists when it actually doesn’t. This kind of mistake is formally called a Type I error, or a false positive.

Here’s how it works in practice. Suppose an environmental scientist wants to determine whether a new water treatment method reduces bacterial contamination. The null hypothesis (H₀) states that the treatment has no effect. The alternative hypothesis (H₁) states that the treatment does reduce contamination. Before collecting any data, the researcher sets a level of significance – say, α = 0.05. This means they are willing to accept a 5% chance of falsely concluding the treatment works when it doesn’t.

After running the experiment and performing the appropriate statistical test, the researcher obtains a p-value. If this p-value falls below α, the null hypothesis is rejected, and the result is deemed statistically significant. If the p-value is equal to or greater than α, the researcher fails to reject the null hypothesis.

Common choices for α 🔗

Researchers don’t pick α arbitrarily – they choose it based on how much error they can tolerate given the context of their study. The three most widely used values are:

α = 0.05 (5%): This is the standard threshold across most scientific disciplines, including environmental science. It represents a balance between being too lenient and too strict. When a researcher uses this value, they accept that there is a 5% risk of concluding a difference exists when there is no actual difference.

α = 0.01 (1%): This stricter threshold is used when the consequences of a Type I error are severe. For example, in pharmaceutical trials or public health safety studies, a false positive could lead to harmful treatments being approved. The 0.01 level demands stronger evidence before the null hypothesis is rejected.

α = 0.10 (10%): A more lenient threshold sometimes used in exploratory research or when sample sizes are limited. It carries a higher risk of false positives but allows researchers to detect potential effects that might be missed with stricter criteria.

It’s worth noting that the choice of 0.05 as the default isn’t based on a mathematical certainty. As explained in the medical literature, early statisticians found that a one-in-twenty chance of being wrong was acceptable for many applications, and the convention simply stuck. Researchers should always justify their choice of α based on their specific research context, not just convention.

Critical regions and decision rules 🔗

Once the level of significance is set, it defines what’s called the critical region (also known as the rejection region). This is the area on the probability distribution curve where, if the test statistic falls, you reject the null hypothesis. The size and location of this critical region depend on two things: the chosen α value and whether the test is one-tailed or two-tailed.

Two-tailed tests 🔗

A two-tailed test is used when the researcher wants to detect a difference in either direction – whether a parameter is greater than or less than a hypothesized value. In this case, the significance level is split equally between the two tails of the distribution.

For example, if α = 0.05 in a two-tailed test, 2.5% (0.025) is placed in the upper tail and 2.5% in the lower tail. A researcher testing whether the average dissolved oxygen level in a lake is different from the national standard of 6 mg/L would use a two-tailed test. They care about whether the level is significantly higher or lower than the standard. The null hypothesis is rejected only if the test statistic falls into either extreme end of the distribution.

Two-tailed tests are the default in most scientific research because they allow detection of effects in both directions, which is generally more conservative and responsible.

One-tailed tests 🔗

A one-tailed test is used when the researcher has a specific directional prediction. All of the significance level is concentrated in one tail of the distribution. This makes it easier to detect an effect in the predicted direction because the critical value is less extreme.

For instance, an environmental agency testing whether mercury levels in a river exceed the safe limit of 0.001 mg/L would use a right-tailed test. They aren’t interested in whether the level is below the limit – only whether it’s dangerously above it. With α = 0.05, the entire 5% sits in the upper tail, and the critical z-value is 1.645 instead of 1.96 (which would be required for each tail in a two-tailed test at the same α).

Conversely, a left-tailed test would be appropriate if a researcher wants to determine whether a conservation programme has decreased the rate of deforestation compared to previous years.

The decision rule in action 🔗

The decision rule is straightforward: compare the p-value to α. If the p-value is less than α, reject H₀. If the p-value is equal to or greater than α, fail to reject H₀. Alternatively, you can compare the test statistic directly against the critical value obtained from statistical tables. If the test statistic exceeds the critical value in the direction specified by the alternative hypothesis, H₀ is rejected.

Consider a practical scenario: a soil scientist tests whether a bioremediation technique reduces lead concentration in contaminated soil. After treatment, the mean lead level in the sample is 380 ppm, while the pre-treatment standard was 420 ppm. Using a one-tailed t-test at α = 0.05, the scientist obtains a p-value of 0.03. Since 0.03 is less than 0.05, the null hypothesis (no reduction) is rejected, and the scientist concludes the bioremediation technique significantly reduced lead levels.

Practical applications across fields 🔗

The level of significance isn’t just an abstract statistical concept – it has real-world consequences in how decisions are made. The choice of α directly affects how cautious or liberal a study’s conclusions are, and different fields adopt different standards based on the stakes involved.

Environmental monitoring and public health 🔗

Environmental regulatory agencies frequently use α = 0.05 when evaluating whether pollutant concentrations in water, air, or soil exceed permissible limits. However, when public health is directly at risk – such as testing drinking water safety – agencies may adopt a stricter α = 0.01. The logic is clear: falsely concluding that contaminated water is safe (a Type II error facilitated by a lenient threshold) could endanger lives. A stricter α reduces the chance of false positives, adding an extra layer of caution.

For example, the United States Environmental Protection Agency relies on rigorous statistical testing to determine whether industrial discharge violates the Clean Water Act standards. Choosing the right α ensures that enforcement actions are based on solid evidence while still protecting communities.

Climate science 🔗

Climate researchers typically use α = 0.05 for routine analyses of temperature trends or precipitation patterns. But when studying phenomena with major policy implications – such as whether a tipping point has been reached in Arctic ice melt – scientists may require α = 0.01. The stricter level reflects the gravity of the conclusion: declaring that a climate tipping point has occurred when it hasn’t could lead to misallocation of billions of dollars in resources. Conversely, missing a real tipping point could delay urgent action.

Wildlife and conservation biology 🔗

Researchers studying endangered species sometimes use a more relaxed threshold like α = 0.10, particularly when dealing with small sample sizes that are common in wildlife studies. When studying a critically endangered amphibian population of only 30 individuals, for instance, using α = 0.05 might mean the study lacks the statistical power to detect a genuine decline. A slightly higher α allows the researcher to flag potential threats even if the evidence isn’t overwhelming, which is prudent when the cost of missing a real decline could mean species extinction.

Medical and pharmaceutical research 🔗

Clinical trials typically use α = 0.05, but some high-stakes trials use α = 0.01 or even more stringent thresholds. In drug efficacy studies, a false positive could mean approving a medication that doesn’t actually work – exposing patients to side effects without any benefit. Some fields go much further: particle physics famously requires a 5-sigma threshold (roughly equivalent to α = 0.0000003) before claiming a discovery, as was the case with the confirmation of the Higgs boson.

Agricultural and ecological research 🔗

Agricultural scientists testing new crop varieties or fertilizers generally use α = 0.05. But the practical significance matters as much as statistical significance here. A new fertilizer might produce a statistically significant increase in yield at α = 0.05, but if that increase is only 0.5% above the existing product, it may not be worth the additional cost. This highlights an important distinction between statistical significance and practical significance – a result can be statistically significant without being meaningfully useful in the real world.

Statistical significance vs. practical significance 🔗

One of the most common pitfalls in research is treating statistical significance as the final word. A p-value below 0.05 does not automatically mean a finding is important or actionable. As healthcare research literature emphasizes, professionals should always consider clinical or practical importance alongside statistical results.

For environmental scientists, this is especially relevant. A new filtration method might reduce a pollutant by a statistically significant amount, but if the reduction is negligibly small compared to regulatory thresholds, it may not warrant investment. Similarly, a study might fail to reach statistical significance at α = 0.05 – yielding a p-value of, say, 0.07 – but the observed trend could still be ecologically meaningful and worth investigating further with a larger sample.

The scientific community is increasingly encouraging researchers to report effect sizes and confidence intervals alongside p-values. These provide a richer picture of both the magnitude of an effect and the uncertainty around it, rather than reducing findings to a simple “significant” or “not significant” binary.

Common mistakes to avoid 🔗

Confusing the p-value with the probability that H₀ is true: A p-value of 0.03 does not mean there’s a 3% chance the null hypothesis is true. It means that if H₀ were true, there would be a 3% probability of observing results as extreme as (or more extreme than) what was found.

Choosing α after seeing the data: The significance level must be set before data analysis begins. Adjusting α after the fact to make results appear significant is a form of data manipulation that undermines research integrity.

Ignoring Type II errors: While α controls the Type I error rate, researchers should also consider β (beta), the probability of a Type II error – failing to reject a false null hypothesis. The power of a test (1 – β) represents the ability to detect a real effect when one exists. A very strict α reduces Type I errors but can increase Type II errors, potentially causing researchers to miss genuine effects.

Over-reliance on α = 0.05: There’s nothing magical about the 0.05 threshold. Some statisticians have even proposed shifting the default to 0.005 for new claims to improve reproducibility. The appropriate α always depends on the specific research context, the consequences of errors, and the available sample size.

Choosing the right level of significance 🔗

Selecting an appropriate α requires weighing several factors. First, consider the consequences of a Type I error. If falsely rejecting H₀ could lead to costly or dangerous outcomes (approving an unsafe chemical, declaring an ecosystem recovered when it hasn’t), a stricter α is warranted. Second, evaluate the consequences of a Type II error. If missing a real effect could be equally harmful (failing to detect a declining population), a more lenient α might be justified. Third, account for sample size. With larger samples, even trivial effects can become statistically significant at α = 0.05. With smaller samples, the opposite is true – meaningful effects may go undetected without relaxing the threshold.

Finally, consider field-specific norms. While these norms exist for good reason, they shouldn’t be followed blindly. The best practice is to state and justify your chosen α in your research methodology, explaining why it’s appropriate for the particular study.

What do you think? In environmental research, where the consequences of both false alarms and missed detections can be severe, how should scientists decide between a stricter or more lenient significance level? Can you think of a situation where using α = 0.10 would be more responsible than using α = 0.01?

References

1. https://www.ncbi.nlm.nih.gov/books/NBK459346/
2. https://www.ncbi.nlm.nih.gov/books/NBK557421/
3. https://blog.minitab.com/en/blog/adventures-in-statistics-2/understanding-hypothesis-tests-significance-levels-alpha-and-p-values-in-statistics
4. https://www.geeksforgeeks.org/maths/level-of-significance-definition-steps-and-examples/
5. https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/faq-what-are-the-differences-between-one-tailed-and-two-tailed-tests/
6. https://www.sciencedirect.com/topics/mathematics/tailed-test
7. https://en.wikipedia.org/wiki/Statistical_significance