Tool
Could my tasters really tell the two batches apart?
From a triangle, tetrad, duo-trio or paired test: how likely the number of correct answers would be if every taster had guessed, worked exactly from the binomial distribution, and how many correct answers the panel needed. It says what a significant result does not mean, and why a result that is not significant does not show the samples are the same.
Discrimination test result
Each assessor counted once, every one of them giving an answer.
Three coded samples, two alike and one different; pick the one that differs.
Chosen before the test, not after seeing the result.
What the count shows
A difference was detected at α = 0.05 (p = 0.028)
Needed at α = 0.05: 13 of 24
How this is worked out
- Chance of a correct answer by guessing: 1 in 3 (triangle test)
- Correct answers expected by guessing alone: 8.0 of 24
- Correct answers observed: 13 of 24 (54%)
- Probability of 13 or more correct if every assessor guessed: 0.028 — exact one-tailed binomial
- Correct answers needed at α = 0.05: 13 of 24
What this assumes (2)
- The probability is only meaningful if the test was run as designed: every assessor answered (a forced choice, no "can’t tell"), nobody knew which sample was which, the serving orders were balanced across assessors, and each assessor is counted once.
- A significant result says a difference was perceived by more assessors than guessing explains. It does not say how large the difference is, which way people prefer it, or whether ordinary drinkers would notice.
What the number is
If nobody on the panel could perceive any difference, every answer would be a guess, and the number of correct answers would scatter around what guessing gives: a third of the panel in a triangle test, half in a duo-trio. The probability shown is how often guessing alone would produce at least as many correct answers as you observed. It is computed exactly from the binomial distribution, not from the normal approximation, which is poor at the panel sizes a cidery actually has.
The significance level is a threshold chosen before the test. Choosing it after seeing the result, or running the test again until it comes out, turns the probability into a number with no meaning.
Which test answers which question
The triangle, tetrad and duo-trio tests ask whether the samples differ at all, and do not tell the assessors what to look for. A paired comparison or a 3-AFC names the attribute — which is sweeter, which is more bitter — and answers only that question. None of them says which sample people prefer; that is a consumer test, on untrained people, in numbers no small panel reaches. How a panel is built, and what it can and cannot establish, is set out on how a trained sensory panel works; choosing the test and the panel size for a given question is on choosing a sensory test.
A small panel that fails to detect a change of supplier, yeast or process has not shown the change is safe. It may simply have been too small to see it. Showing that two samples are similar enough needs a test planned for that question, with the panel size worked out beforehand.