Hypothesis Tests

Links to introductory web resources on hypothesis tests

http://www.quantdec.com/Articles/pcbpipe/pcbpipe.pdf  This report, which I wrote for an industrial client, illustrates how many ideas from this course--estimation, testing, the meaning of probability, rational decision theory--can be applied in a practical environmental setting.  The sensitive details have been removed, of course.  What remains is "a self-contained analysis of a tolerance limit test ...  An appendix develops the theory, beginning with definitions, and assesses the performance of the test in two ways."

http://www.statistics.com/content/teaching/Biology/HYPOTEST.htm  Notes from a U. of Waterloo course on "Statistics and experimental design."  Thoughtful discussion of hypothesis tests.

http://www.cnr.colostate.edu/~anderson/thompson1.html  A lengthy bibliography on articles and books about statistical hypothesis testing.

http://www.sjsu.edu/faculty/gerstman/EpiInfo/basics.htm#The%20Basis%20of%20Inference  "A brief review of some principles" includes some choice remarks about hypothesis testing.  (From San Jose State University, California.)

http://ericae.net/edo/ED366654.htm  "The Concept of Statistical Significance Testing: ... This Digest will help you better understand the concept of significance testing."

http://www.cnr.colostate.edu/~anderson/nester.html  "A few quotes regarding hypothesis testing" (most of them arguing against it).

Discussion

The mythology of hypothesis testing

An amazing amount of nonsense has been written about statistical tests of hypotheses.  Some of the myths include

The logic of hypothesis tests

Statistics is empirical: it uses observations to help us draw conclusions.  The goals of statistics are to improve how we reason with data and to help us understand our reasoning.

Empirical deductions do not have the same finality as logical proofs in mathematics.  Using data alone we can never incontrovertibly establish that something is true.  There is always the possibility of exception.  Uncertainty attends all empirical reasoning.

Therefore statistical tests cannot, and do not, aim to establish that something is the case.  They can only evaluate how strongly the data support or contradict a conclusion.

We hypothesize a class of probability models for an experimental outcome.  These are the possible "states of nature."  Estimation (discussed previously) consists of identifying the model most consistent with the outcome.  By contrast, hypothesis testing consists of identifying some special model or models within the class (the "null hypothesis") and assessing the degree to which the experimental outcome constitutes evidence against the special model, relative to all other models we are considering.

If the test concludes that the outcome is strong evidence against the null hypothesis, then at least one, and maybe more than one, of the following are true:

If the test concludes that the outcome is not strong evidence against the null hypothesis, then

There are several schools of thought concerning hypothesis tests and many different views about what they are for and what they should tell us.

One view is that a test assigns a number, the "confidence," to the data and that the confidence is the probability that a particular state of nature (the "hypothesis") is true.  That is not the view taken in this course, in part because environmental statistics are used by many different stakeholders in any real application: property owners or managers, local, state, and national regulatory agencies, people who live nearby, and people whose livelihoods depend on the outcome.  It is not possible to provide an objective meaning to the probability of a hypothesis in this setting.

To maintain objectivity, we assert that an hypothesis is either an effective model of the experiment's behavior or it is not.  We then try to select procedures that have a good chance of interpreting the data correctly in this regard.  One function of the statistician is to select good decision procedures.  One merit of this approach is that the applicability and desirability of a statistical procedure can usually be evaluated before all the data are available.  This makes it much easier to achieve agreement among the stakeholders about which procedure to use.

(For instance, it is commonplace for investigators to get into trouble by choosing the procedure after reviewing all the data.  As we have seen, different procedures can give different results.  A naive or biased or dishonest statistician can sometimes "steer" the statistical results in a desired direction by a clever post hoc choice of test.  Everybody knows this, which is the basis for the adage that statistics is the "third lie" (after white lies and black lies).  This possibility leaves the honest investigator in an indefensible position.

A better approach is to formalize the choice of statistical procedure.  That is the role of the investigation work plan.)

From our point of view, then, the "confidence" is the chance--computed before the data are gathered--that our procedure will arrive at the correct decision provided a particular probability model (the null hypothesis) is sufficiently accurate.  Clearly the confidence does not tell us everything we would like to know.  It might not even be particularly relevant: often the null hypothesis is not an accurate model of the experiment at all.

These considerations tell us that

The latter is measured by the "power" of the test.  U.S. EPA guidance (see DQO or RCRA GW guidance, for instance) frames the test selection process in terms of power.

An example

The need to distinguish a special class of models, or null hypothesis, is not universal, as the following example shows.  This example uses the language and concepts described in the Decision Theory article.

Parts of the southeastern United States consist of limestone formations that contain caves and solution channels which may rapidly conduct groundwater for great distances beneath the surface.  In the 1930's, the Tennessee Valley Authority created a large system of dams in this area, inundating many natural springs and seeps where this groundwater was ultimately conducted to the surface.

In the 1990's, an investigation of groundwater contamination included a dye test to track groundwater flows.  Groundwater was believed to flow beneath a nearby range of hills to emerge beneath the surface of a large nearby TVA lake.  Historical maps showed (and here I simplify the actual problem) that two large springs had existed there but were inundated.

A non-toxic dye was introduced into sinkholes and wells onsite.  Any dye subsequently emerging at the lake bottom from a former spring would appear as a diffuse colored spot near the surface.  One object of the test was to look for these colored spots to determine whether the groundwater was flowing into the lake, and if so, to establish which spring it was being carried to.

The dye was indeed detected in the lake within a few days of its introduction into the groundwater.  Let us, therefore, consider the secondary objective, that of identifying the spring from which the dye emerged.  To do this, physical theory suggests the dye spot will be located randomly above the spring with a Normal distribution centered at the true spring location.  The depth of water over each spring was about the same, so the variance of the distribution should be the same, regardless of the spring.

Let's set up a system of local coordinates and length measurements so that  the springs are at coordinates (-1, 0) [spring A] and (1, 0) [spring B].  The probability model is that the dye spot is governed by one of two (bivariate) Normal distributions, N( (-1, 0), sigma ) or N( (1, 0), sigma ).  A dye spot observation consists of its coordinates (x, y).

We contemplate two decisions, although there are really three:

1. The groundwater emerges at spring A.
2. The groundwater emerges at spring B.
3. (The groundwater emerges at some unexpected location.)

Without additional information (such as some idea of what sigma is), we will not know when it's appropriate to make the third decision.  Therefore, until we get such information, we shall limit our decisions to one of the first two.

The hypothesis test in this case therefore consists of some rule that tells us, for any possible dye spot location (x, y), which spring (A or B) produced it.

The test divides all coordinates (x, y) into two sets: the set A associated with decision A and the set B with decision B.  Conversely, any subset of the set of all coordinates corresponds to the decision that associates that set with spring A and its complement with spring B.

To decide among possible tests, we need a loss function.  Classical hypothesis testing uses simple loss function which is 0 when the decision is correct and 1 when it is not.

The risk function is a pair of values: one for when spring A is the correct choice and another for when spring B is the correct choice.  Let's compute it.

When spring A is the correct choice, the probability distribution of the dye spot is modeled by N( (-1, 0), sigma ).  For simplicity, let's just call this distribution F_A (with a similar notation for spring B's distribution).  If this model is at least approximately correct, then the probability of making decision B is close to F_A(B), the total probability associated with the set where decision B will be made.  The probability of making decision A is F_A(A).  The risk is F_A(A) * Loss(make decision A when A is correct) + F_A(B) * Loss(make decision B when A is correct) = F_A(A) * 0 + F_A(B) * 1 =  F_A(B).  This is r(A), the risk for state A.

Similarly, r(B), the risk for state B, is F_B(A) = 1 - F_B(B)  (the probability of making decision A when B is correct).

To reduce the risk, we therefore need simultaneously to reduce F_A(B) and increase F_B(B).  Since the forms of F_A and F_B are beyond our control, the only variable left to consider is set B.

We can make F_B(B) as large as we like by making set B very, very large.  Unfortunately, this makes F_A(B) very large too.  The trick is to focus set B in the regions where spring B is the most likely source and to have set B avoid regions where spring A is the most likely source.

The balanced (minimax and invariant) solution chooses set B to coincide with those points where spring B is more likely than spring A to have been the source of the dye.  Those are exactly the points closer to spring B than to spring A (the ones with positive x coordinate)..  Set A therefore coincides with the points closer to spring A than to spring B (the ones with negative x coordinate).  (Points on the line equidistant from the springs--the y axis--may belong to either set, but it doesn't matter: the probability that the dye appears exactly halfway between the springs is so small we needn't worry about it.)

If the springs are very close compared to the dye dispersivity (sigma), then r(A) and r(B) can be almost as large as 0.50--the test is simply too imprecise to distinguish the dye's source and choosing among the springs is little better than flipping a coin.  If the springs are very far apart compared to the dispersivity, then r(A) and r(B) are both practically zero.  The risk cannot exceed 0.50 for the minimax solution.

There are some very interesting features to this example:

This practical example shows how the statements quoted in the introduction can be false or misleading.

Return to the Environmental Statistics home page

This page is copyright (c) 2001 Quantitative Decisions.  Please cite it as

This page was created 4 April 2001 and last updated 9 May 2001.