Ways to increase statistical power

Michael Großbach

Hochschule für Musik, Theater und Medien – Hannover

Institut für Musikphysiologie und Musiker-Medizin

2023-12-07

Sample sizes are always too small

“Increase your sample size!”

  • Lack of resources
    • Money
    • Staff
    • Time
  • Lack of subjects / small populations

Determinants of statistical power

  • The statistical power (i.e. probability) to detect an existing effect depends on
    • \(\alpha\)
    • \(\beta\)
    • The ‘true’ effect size
    • Sample size
    • Measurement error

But there’s more

Increase statistical power

by choosing an appropriate experiment design

Simple nested design

Code
subj_n <- 40  # total number of subjects
b0 <- 0       # intercept
b1 <- 0.5     # fixed effect of condition
u0s_sd <- 0.5 # random intercept SD for subjects
sigma_sd <- 2 # error SD
# set up data structure
nested_data <- function(sn, intercept, slope, varinterc, sigma) {
  add_random(subj = sn) %>%
  # add and recode categorical variables
  add_between("subj", cond = c("control", "test")) %>%
  add_recode("cond", "cond.t", control = 0, test = 1) %>%
  # add random effects 
  add_ranef("subj", u0s = u0s_sd) %>%
  add_ranef(sigma = sigma_sd) %>%
  # calculate DV
  mutate(dv = b0 + u0s + b1 * cond.t + sigma) ->
    nd
  return(nd)
}
nest_dat <- nested_data(subj_n, b0, b1, u0s, sigma_sd)

Difference between conditions/groups?


    Welch Two Sample t-test

data:  dv by cond
t = -0.7, df = 37, p-value = 0.5
alternative hypothesis: true difference in means between group control and group test is not equal to 0
95 percent confidence interval:
 -1.743  0.813
sample estimates:
mean in group control    mean in group test 
               -0.054                 0.411 

Power analysis by simulation

Control and Test groups

Simple crossed design

Code
subj_n <- 40  # total number of subjects
b0 <- 0       # intercept
b1 <- 0.5     # fixed effect of condition
u0s_sd <- 0.5   # random intercept SD for subjects
sigma_sd = 2 # error SD
# set up data structure
crossed_data <- function(sn, intercept, slope, varinterc, sigma) {
  add_random(subj = sn) %>%
    # add and recode categorical variables
    add_within("subj", cond = c("control", "test")) %>%    # <- CHANGED!!!
    add_recode("cond", "cond.t", control = 0, test = 1) %>%
    # add random effects 
    add_ranef("subj", u0s = u0s_sd) %>%
    add_ranef(sigma = sigma_sd) %>%
    # calculate DV
    mutate(dv = b0 + u0s + b1 * cond.t + sigma) ->
    cd
  return(cd)
}
cross_dat <- crossed_data(subj_n, b0, b1, u0s, sigma_sd)

Power analysis by simulation

Each Subj their own control

Crossed design with replicates

Code
subj_n <- 40  # total number of subjects
b0 <- 0       # intercept
b1 <- 0.5     # fixed effect of condition
u0s_sd <- 0.5 # random intercept SD for subjects
u1s_sd <- 0.2 # random slope for replicates        # <- NEW !!!
r_01s <- 0.7  # correlation between intercept & slope # <- NEW !!!
sigma_sd = 2 # error SD
# set up data structure
cross_repl_data <- function(sn, intercept, slope, varinterc, varslope, varcor, sigma) {
  add_random(subj = sn) %>%
    # add and recode categorical variables
    add_within("subj", 
               cond = c("control", "test"),
               replicate = 0:9) %>%  # <- NEW !!!
    add_recode("cond", "cond.t", control = 0, test = 1) %>%
    # add random effects 
    add_ranef("subj", 
              u0s = varinterc,
              u1s = varslope,                       # <- NEW !!!
              .cors = varcor) %>%                  # <- NEW !!!
    add_ranef(sigma = sigma) %>%
    # calculate DV
    mutate(dv = b0 + u0s + b1 * cond.t + u1s + sigma) ->
    cd
  return(cd)
}
cr_dat <- cross_repl_data(subj_n, b0, b1, u0s_sd, u1s_sd, r_01s, sigma_sd)

Power analysis by simulation

Each subject their own control; 10 trials per condition

Know your parameters

To be able to simulate realistically…

  • \(\alpha\), \(\beta\)
  • effect size estimate (literature, experience)
  • inter-individual variation (literature, experience)
  • inter-condition variation (literature, experience)

Takeaway message

The required sample size to achieve the desired statistical power can be manipulated by choosing a suitable experimental design!

This requires a lot of prior research!