Trial and error: A hierarchical modeling approach to test-retest reliability

•Conventional ICC formulation does not account for cross-trial variability and thus may attenuate test-retest reliability.•The extent of ICC attenuation depends on the number of trials and cross-trial variability.•A hierarchical model can characterize test-retest reliability as a correlation that is separate from trial-level effects.•The uncertainty of test-retest reliability can be expressed through its distribution or a quantile interval.•Cross-trial variability is much larger than cross-subject variability for typical tasks.•A large trial sample size (e.g., greater than 100) may be required to achieve a reasonable precision of reliability estimation.