Students tend to approach this backwards: they collect data, open SPSS or R, and then search for a test that will produce a small p-value. That sequence is how projects end up with a t-test on ordinal data or an ANOVA on repeated measurements treated as independent.
The correct sequence runs the other way. Your research question and your measurement decisions determine the test before a single observation is recorded. This post gives you the three questions that decide it, a decision table you can work through in two minutes, and what to do when the assumptions do not hold.
Write the analysis plan before you collect data
ThesisAI drafts your methodology and results sections from the literature in your field, so the test you name is the one comparable published studies actually used.
Kezdj el írniThe Three Questions That Decide the Test
- What kind of question are you asking? Are you comparing groups, testing an association between two variables, or predicting an outcome from one or more predictors? These lead to three different families of test.
- What is the level of measurement of your outcome? Continuous, ordinal, nominal, count, or time-to-event. This narrows the family to a handful of candidates.
- How are your observations structured? Independent groups, or the same participants measured more than once? How many groups? Are observations nested inside classes, clinics or families?
Answer those three and the test is usually determined. Everything after that - normality, variance, sample size - decides whether you use the standard version of that test or a robust alternative, not which test you are doing.
Levels of Measurement
Get this wrong and everything downstream is wrong, so it is worth being precise.
| Level | What it means | Examples | Summary statistic |
|---|---|---|---|
| Nominal | Named categories with no order | Country, treatment arm, blood type | Counts, proportions, mode |
| Ordinal | Ordered categories, unequal or unknown intervals | Likert item, education level, tumour stage | Median, interquartile range |
| Interval | Equal intervals, no true zero | Temperature in Celsius, calendar year | Mean, standard deviation |
| Ratio | Equal intervals with a true zero | Reaction time, income, height, count of errors | Mean, standard deviation |
The contested case is the Likert scale. A single item ("strongly disagree" to "strongly agree") is ordinal, and treating it as continuous is defensible only with reservations. A summed or averaged multi-item scale is routinely treated as continuous, and that is accepted practice in psychology and management research. Whichever you do, state the choice in your methodology and justify it rather than leaving it implicit.
The Decision Table
Find the row matching your question and your outcome type.
| Your question | Outcome type | Design | Test |
|---|---|---|---|
| Compare groups | Continuous | 2 independent groups | Independent samples t-test (Welch by default) |
| Continuous | 2 paired measurements | Paired samples t-test | |
| Continuous | 3+ independent groups | One-way ANOVA | |
| Continuous | 3+ repeated measurements | Repeated measures ANOVA or a mixed model | |
| Ordinal or skewed | 2 independent groups | Mann-Whitney U | |
| Ordinal or skewed | 2 paired measurements | Wilcoxon signed-rank | |
| Ordinal or skewed | 3+ independent groups | Kruskal-Wallis H | |
| Ordinal or skewed | 3+ repeated measurements | Friedman | |
| Compare proportions | Nominal | 2+ independent groups | Chi-square test of independence |
| Nominal | Small expected counts | Fisher's exact test | |
| Nominal | Paired, 2 categories | McNemar's test | |
| Test a relationship | Two continuous | Linear, no severe outliers | Pearson correlation |
| Two ordinal, or non-linear monotonic | - | Spearman rho or Kendall tau | |
| Two nominal | - | Chi-square with Cramer's V | |
| Predict an outcome | Continuous | One or more predictors | Linear regression |
| Binary | One or more predictors | Logistic regression | |
| Count | One or more predictors | Poisson or negative binomial regression | |
| Time to event | With censoring | Kaplan-Meier, log-rank, Cox regression | |
| Any | Observations nested in groups | Mixed effects or multilevel model | |
| More complex designs | Continuous | Two independent variables | Two-way ANOVA (main effects plus interaction) |
| Continuous | Groups plus a continuous covariate | ANCOVA | |
| Several continuous outcomes | - | MANOVA, or separate models with a correction |
Parametric vs Non-parametric
Parametric tests (t-test, ANOVA, Pearson, linear regression) assume something about the shape of the underlying distribution. Non-parametric tests (Mann-Whitney, Kruskal-Wallis, Spearman) work on ranks and assume much less.
Two corrections to the usual folklore are worth making.
First, the normality assumption applies to the residuals, not the raw data, and with a reasonably large sample the central limit theorem makes t-tests and ANOVA robust to moderate departures from it. Switching to a non-parametric test because a Shapiro-Wilk test came back significant at n = 400 is a mistake - that test is so sensitive at large n that it flags trivial deviations. Look at a Q-Q plot rather than a significance test.
Second, non-parametric tests are not free. They generally have less power when the parametric assumptions do hold, and they test slightly different hypotheses. The Mann-Whitney U tests whether one group tends to produce larger values, not whether the means differ, which matters when you write the interpretation.
Use a non-parametric test when your outcome is genuinely ordinal, when the sample is small and clearly skewed, or when outliers dominate and you cannot justify removing them. Consider a transformation, a robust standard error, or a bootstrap instead when you want to keep the parametric interpretation.
Assumptions You Have to Check
| Test | Assumptions | If violated |
|---|---|---|
| Independent t-test | Independence, approximate normality of residuals, equal variances | Use Welch's t-test (unequal variances) or Mann-Whitney |
| Paired t-test | Normality of the differences | Wilcoxon signed-rank |
| One-way ANOVA | Independence, normality of residuals, homogeneity of variance | Welch's ANOVA or Kruskal-Wallis |
| Repeated measures ANOVA | Sphericity | Greenhouse-Geisser correction, or a mixed model |
| Chi-square | Expected count of at least 5 in most cells | Fisher's exact test, or merge sparse categories |
| Pearson correlation | Linearity, bivariate normality, no dominant outliers | Spearman rho |
| Linear regression | Linearity, independent errors, constant variance, normal residuals, no severe multicollinearity | Transform, use robust standard errors, or a different model family |
| Logistic regression | Linearity in the logit, no complete separation, adequate events per predictor | Add splines, use penalised (Firth) logistic regression |
Always check independence first, because it is the assumption with no easy fix. Students measured within the same classroom, patients within the same hospital, and repeated observations from the same participant are not independent, and analysing them as if they were produces standard errors that are far too small and p-values that are far too optimistic. Nested data needs a multilevel model, not a t-test.
Three Worked Examples
Example 1
Question: Does a four-week mindfulness programme reduce anxiety compared with a waitlist control?
Outcome: GAD-7 total score, treated as continuous.
Design: Two independent groups, measured once after the intervention.
Test: Independent samples t-test, Welch version. If you also measured baseline anxiety, ANCOVA with baseline as a covariate is the better choice - it has more power and adjusts for chance imbalance at randomisation.
Example 2
Question: Is employment status after graduation associated with degree subject?
Outcome: Employed or not employed, nominal.
Design: Five subject groups, independent observations.
Test: Chi-square test of independence, reported with Cramer's V as the effect size. Check that expected counts are adequate; if one subject group has very few graduates, merge it or use Fisher's exact test.
Example 3
Question: Do study hours, prior GPA and attendance predict final exam score?
Outcome: Exam score, continuous.
Design: Three continuous predictors, one observation per student.
Test: Multiple linear regression. Report the model R-squared and each standardised coefficient with its confidence interval. Check variance inflation factors, because study hours and attendance will be correlated.
How to Report the Result
A test statistic on its own is not a result. Report the descriptive statistics, the test, the exact p-value, and an effect size with a confidence interval.
Weak: "The t-test was significant (p < 0.05), showing that the intervention worked."
No descriptives, no degrees of freedom, no effect size, no interval, and a causal claim the test does not support on its own.
Strong: "Post-intervention anxiety was lower in the mindfulness group (M = 7.2, SD = 3.4, n = 42) than in the waitlist group (M = 9.8, SD = 3.9, n = 41), t(78.4) = 3.24, p = .002, d = 0.71, 95% CI [0.26, 1.15]. Welch's correction was applied because Levene's test indicated unequal variances."
Gives the reader everything needed to judge the size of the effect, its precision, and why that version of the test was used.
Two further points. Report exact p-values rather than thresholds (p = .002, not p < .05), except below .001. And a non-significant result is a result: report it fully with its confidence interval rather than burying it, because an interval that spans zero but excludes large effects tells the reader something a bare "not significant" does not. Our results section guide covers the write-up in detail.
Sample Size and Power
Decide your sample size before collecting data, using a power analysis based on the smallest effect that would be meaningful in your field. Conventionally you target 80% power at an alpha of .05, though 90% is increasingly expected in clinical work.
Post-hoc power analysis - calculating power from the effect you observed, after a non-significant result - is not informative and is criticised in the methodological literature, because it is a deterministic function of the p-value and tells you nothing new. If a reviewer asks why your study was underpowered, the honest answer is a confidence interval showing what effect sizes your data can and cannot rule out.
Common Mistakes
| Mistake | Why it matters |
|---|---|
| Running several tests until one is significant | Inflates the false positive rate; this is p-hacking whether or not it was intended |
| Ignoring the repeated-measures structure | Treats correlated observations as independent; p-values far too small |
| Multiple comparisons without correction | Twenty tests at alpha .05 produce roughly one false positive by chance alone |
| Reporting p without an effect size | Significance says an effect exists, not that it is large enough to matter |
| Reading "not significant" as "no effect" | Absence of evidence, particularly in a small sample, is not evidence of absence |
| Choosing the test after seeing the results | Undermines the inference entirely; pre-specify the analysis plan |
| Using a chi-square test on paired data | McNemar's test is the paired equivalent; chi-square gives the wrong answer |
FAQs About Choosing a Statistical Test
Can I use a t-test on Likert data?
On a single item, it is questionable and Mann-Whitney is safer. On a summed multi-item scale with several response options, treating it as continuous is standard practice in most social science fields. Say which you did and why.
How do I test for normality?
Look at a histogram and a Q-Q plot of the residuals. Shapiro-Wilk and Kolmogorov-Smirnov tests are available but misleading at both extremes - almost nothing is significant at n = 20, and almost everything is at n = 1,000. Visual inspection plus sample size is the better judgment.
What do I do about outliers?
Investigate before deciding. Data entry errors get corrected or removed and you say so. Genuine extreme values are usually kept, with a sensitivity analysis showing the result with and without them. Deleting points because they weaken your finding is not defensible.
Which correction should I use for multiple comparisons?
Bonferroni is simple and conservative, suitable for a small number of planned comparisons. Holm-Bonferroni is uniformly more powerful and has no real downside. For a large number of exploratory tests, Benjamini-Hochberg controls the false discovery rate and is the usual choice in genomics and neuroimaging.
ANOVA told me the groups differ. Which ones?
ANOVA only tells you that at least one group differs. Follow it with post-hoc comparisons: Tukey's HSD for all pairs with equal variances, Games-Howell when variances are unequal, or planned contrasts if you specified specific comparisons in advance.
Should I use SPSS, R, Python or Stata?
Whichever your department supports, since you will need help at some point. R and Python are free, reproducible from a script, and increasingly expected. Whatever you use, report the software and version in your methodology - it is part of making the analysis reproducible.
The reason to settle the test before collecting data is not bureaucratic. A test chosen in advance is a prediction you committed to; a test chosen after seeing the numbers is a description of the numbers. Examiners and reviewers know the difference, and it is the single easiest thing to protect in your write-up.