Neue Veröffentlichung in Psychological Methods

Steinhilber, M., Schnuerch, M., & Schubert, A.-L. (2024). Sequential analysis of variance: Increasing efficiency of hypothesis testing. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000677

Abstract: Researchers commonly use analysis of variance (ANOVA) to statistically test results of factorial designs. Performing an a priori power analysis is crucial to ensure that the ANOVA is sufficiently powered, however, it often poses a challenge and can result in large sample sizes, especially if the expected effect size is small. Due to the high prevalence of small effect sizes in psychology, studies are frequently underpowered as it is often economically unfeasible to gather the necessary sample size for adequate Type-II error control. Here, we present a more efficient alternative to the fixed ANOVA, the so-called sequential ANOVA that we implemented in the R package “sprtt.” The sequential ANOVA is based on the sequential probability ratio test (SPRT) that uses a likelihood ratio as a test statistic and controls for long-term error rates. SPRTs gather evidence for both the null and the alternative hypothesis and conclude this process when a sufficient amount of evidence has been gathered to accept one of the two hypotheses. Through simulations, we show that the sequential ANOVA is more efficient than the fixed ANOVA and reliably controls long-term error rates. Additionally, robustness analyses revealed that the sequential and fixed ANOVAs exhibit analogous properties when their underlying assumptions are violated. Taken together, our results demonstrate that the sequential ANOVA is an efficient alternative to fixed sample designs for hypothesis testing.

Impact Statement: In scientific research, the analysis of variance (ANOVA) is frequently used to assess statistical differences in mean values across multiple groups. Essential to this process is an a-priori sample size calculation, ensuring that the researchers collect enough data to be able to detect an effect size of interest with a high enough chance. However, accurately determining the required sample size can be challenging. Moreover, finding small differences requires a lot of data, making it expensive and sometimes not feasible to collect enough data. We introduce an alternative method implemented in the R package “sprtt,” termed sequential ANOVA, as a more resource-efficient alternative to the traditional fixed ANOVA. The sequential ANOVA, based on the sequential probability ratio test (SPRT), uses a likelihood ratio to compare two competing hypotheses while adjusting for long-term error rates. The sequential ANOVA accumulates evidence iteratively until sufficient evidence is collected to accept one of the two hypotheses. Our simulations confirm that the sequential ANOVA outperforms the traditional fixed ANOVA in efficiency and maintains the long-term error control. In cases where the underlying assumptions are not met, the sequential ANOVA is as robust as the fixed ANOVA. Consequently, our findings support the use of sequential ANOVA in studies that have limitations on sample size, offering a robust and resource-efficient solution for hypothesis testing.