Alternatives to Hypothesis Testing

Effect size statistics

The simplest and easiest alternative is to report the size of the effect observed, together with a confidence interval, instead of just the evidence that it is not zero. Most statistical software capable of NHST can also produce confidence intervals.

• Confidence intervals are better than standard errors, in spite of the difficulty of interpretation (see here). This is particularly so when errors are not distributed symmetrically around the point estimate, as often occurs with proportions, probabilities and counts.
   
• If the response variable was transformed for the analysis, convert the point estimate and the confidence limits back to "real" values for reporting.
   
• Confidence intervals can be calculated for nonparametric tests, including randomization tests, though it can be tedious (see Ramsey & Schaffer 2002:93).
   
• The confidence interval will show whether the data are consistent with (a) no effect and/or (b) an effect large enough to be biologically important.
   
• The overlap of 95% confidence intervals for different samples and the position of the point estimates allows one to assess the evidence for a difference (Ramsey & Schaffer 2002:139):

Effect size with confidence intervals is generally the best choice for descriptive studies, which are probably the most common in management-oriented wildlife science.

Information Theoretic methods

The basis of information theoretic approaches is the estimation of the relative distance or discrepancy of a proposed model from the truth. The actual distance is unknown, but Akaike's Information Criterion (AIC) and its derivatives allow the relative distance for a set of candidate models to be estimated.

Model likelihoods are at the core of the method, and it is often referred to as the "likelihood approach" to distinguish it from Bayesian methods.

At its simplest, we could use AIC to compare a model including the effect of interest and a null model without the effect, analogous to NHST, but now assessing the relative support for the two models in the data - excluding any data which weren't observed (David Anderson has an example here).

Much more interesting, though, is the ability to fit and compare multiple models, and to produce estimates averaged over several models to incorporate uncertainty as to the best model, ie. model selection and multi-model inference (Burnham & Anderson 2002).

Information criteria are built-in to several packages used for wildlife data analysis, including PRESENCE and MARK, where it is used to distinguish between different models including covariates. In the DISTANCE package, it is used to select the model for the detection function.

An introduction to the basics of modelling and model selection using IT methods is here. The recognized text is Burnham & Anderson 2002.

Bayesian methods

Bayesian methods tell us what we want to know, such as the probability that our hypothesis is true, or the probability distribution of a parameter. They also allow previous knowledge to be incorporated into the analysis.

For much of the 20th century, Bayesian analysis was vilified as unscientific; and indeed, in the early part of the century, most "previous knowledge" in ecology had little empirical basis. Moreover, Fisher held that Bayesian methods were just plain wrong. Today we have a corpus of evidence-based information, as well as ways of dealing with situations which really are novel.

The use of Bayesian methods was also held back by the calculations required. Apart from the simplest cases, the results cannot be calculated algebraically. The availability of computers and the development of ingenuous algorithms mean that analyses can be done on any desktop. Current methods still have their pitfalls, and are not for the faint-hearted.

An introduction to Bayes and some simple examples are here. A useful introduction to Bayesian statistics is Bolstad 2004. For MCMC methods used to implement Bayes with more complex models see Ntzoufras 2009, though the forthcoming Kéry 2010 may be an easier introduction.

When is NHST appropriate?

Assessing model fit: Model comparison methods (whether IT or Bayesian) select the best model among the candidate set, but do not indicate whether any of them is 'good enough'. A low P-value indicates a model which needs further work. This is not strictly a hypothesis falsification application.

Challenging research hypotheses: Several authors (eg. Stephens et al, 2007; Robinson & Wainer 2002) envisage the continued use of NHST within a programme of rigorous experimentation designed to test working hypotheses sequentially. Robinson & Wainer in particular warn of the necessity of replicating experiments before accepting their results.