The conclusiveness of trial sequential analysis varies with estimation of between-study variance: a case study

Abstract Background Trial sequential methods have been introduced to address issues related to increased likelihood of incorrectly rejecting the null hypothesis in meta-analyses due to repeated significance testing. Between-study variance (τ2) and its estimate ( $$\widehat{\tau }$$ τ ^ 2) play a crucial role in both meta-analysis and trial sequential analysis with the random-effects model. Therefore, we investigated how different $$\widehat{\tau }$$ τ ^ 2 impact the results of and quantities used in trial sequential analysis. Methods This case study was grounded in a Cochrane review that provides data for smaller (< 10 randomized clinical trials, RCTs) and larger (> 20 RCTs) meta-analyses. The review compared various outcomes between video-laryngoscopy and direct laryngoscopy for tracheal intubation, and we used outcomes including hypoxemia and failed intubation, stratified by difficulty, expertise, and obesity. We calculated odds ratios using inverse variance method with six estimators for τ2, including DerSimonian-Laird, restricted maximum-likelihood, Paule-Mandel, maximum-likelihood, Sidik-Jonkman, and Hunter-Schmidt. Then we depicted the relationships between $$\widehat{\tau }$$ τ ^ 2 and quantities in trial sequential analysis including diversity, adjustment factor, required information size (RIS), and α-spending boundaries. Results We found that diversity increases logarithmically with $$\widehat{\tau }$$ τ ^ 2, and that the adjustment factor, RIS, and α-spending boundaries increase linearly with $$\widehat{\tau }$$ τ ^ 2. Also, the conclusions of trial sequential analysis can differ depending on the estimator used for between-study variance. Conclusion This study highlights the importance of $$\widehat{\tau }$$ τ ^ 2 in trial sequential analysis and underscores the need to align the meta-analysis and the trial sequential analysis by choosing estimators to avoid introducing biases and discrepancies in effect size estimates and uncertainty assessments. Graphical Abstract