The short answer
If you are comparing percentages between groups, start with a chi square test. If you are comparing averages between two groups, use a t test, and for three or more groups use ANOVA. If you want to predict an outcome from several factors at once, use regression: linear for numeric outcomes, logistic for yes or no outcomes.
Everything else in this guide is about checking that the simple answer still holds for your data.
Three questions that pick your test
1. What type is your outcome? Categorical outcomes (such as yes or no, or agree or disagree) are summarized as percentages. Numeric outcomes (such as income, age or test scores) are summarized as averages.
2. How many groups are you comparing? Two groups and three or more groups need different tests, because comparing many groups pair by pair inflates the chance of a false positive.
3. Are the groups independent? Different people in each group are independent. The same people measured before and after are paired, and paired data needs a paired test.
Decision table
| Your question | Outcome | Standard test | If assumptions fail |
|---|---|---|---|
| Compare percentages between groups | Categorical | Chi square test of independence | Fisher's exact test when expected counts are below 5 |
| Compare the average of two independent groups | Numeric | Independent samples t test (Welch's version) | Mann Whitney U test |
| Compare the average of three or more groups | Numeric | One way ANOVA | Kruskal Wallis test |
| Compare the same people measured twice | Numeric | Paired t test | Wilcoxon signed rank test |
| Measure the link between two numeric variables | Numeric | Pearson correlation | Spearman rank correlation |
| Predict a numeric outcome from several factors | Numeric | Linear regression | Robust or quantile regression |
| Predict a yes or no outcome | Binary | Logistic regression | Exact or penalised logistic regression |
Checking assumptions
Parametric tests such as the t test and ANOVA assume roughly normal data within each group and, for classic ANOVA, similar spread across groups. With larger samples the t test is fairly robust, and Welch’s version does not require equal variances, which is why many statisticians use it by default.
When data are heavily skewed, ordinal or the sample is small, the nonparametric alternatives in the table are safer. They compare ranks rather than raw values.
Survey data needs extra care
Most textbook tests assume a simple random sample. Many real surveys use weights, strata or clusters. Running ordinary tests on that data can make results look more certain than they are. Design based versions, such as the Rao Scott chi square test, account for this, and software such as R, Stata and SPSS Complex Samples supports them.
Likert items deserve a note too. A single five point item is ordinal, so rank based tests fit well. A scale built by adding several items is often treated as numeric, after checking its reliability with Cronbach’s alpha.
Beyond the p value
A p value tells you how surprising your result would be if there were truly no difference. It does not tell you how large or important the difference is. Always report an effect size, such as a difference in means or an odds ratio, with a 95% confidence interval. When you run many comparisons, adjust for multiple testing so chance findings do not slip through.
Need a second opinion on your analysis?
Our statisticians review methods and results for researchers and teams. See survey analytics or request a free assessment.
FAQ
Can I use a t test on Likert scale data?
For a single Likert item, a rank based test such as Mann Whitney U is usually more appropriate. For a scale made by combining several items, a t test is commonly accepted once reliability has been checked.
What sample size do I need for a t test?
It depends on the size of the difference you want to detect and how variable the data are. A power analysis gives the answer; as a rough guide, detecting a medium effect with 80% power needs about 64 people per group.
Should I use a one tailed or two tailed test?
Use two tailed tests unless you decided before seeing the data that only one direction matters. Two tailed tests are the standard in most journals.
What if my data are weighted?
Use design based tests that account for weights, strata and clusters, rather than standard tests that assume a simple random sample.
[Author name], [credential]
[One line author bio with link to Team page and research profile]