gms | German Medical Science

Introduction: The simultaneous confirmatory analysis of multiple research questions usually requires the control of a global significance level, due to an increased risk of Type-I errors. A huge variety of multiple comparison procedures (MCP) have been proposed and can be applied to specific testing problems. A general approach was introduced by Bretz et al. 2009 [1], based on a graph representation of the testing problem. This approach is a framework for sequential rejective testing procedures and comprises also widely known methods, such as Bonferroni correction [2] and the Holm-Bonferroni procedure [3].

Methods: To select the best statistical design in the graph-based framework a comprehensive analysis of the procedure’s characteristics under changing parameters is necessary. To obtain fitness landscapes we performed a full search with respect to planning alternatives (expected result of a clinical trial) and target functions (optimization aim). Fitness evaluation is based on simulations of weighted Bonferroni tests. All calculations have been performed in the statistical software R [4] using the gMCP package [5], [6]. Resulting fitness landscapes can be computed for different target functions, i.e. local power values, power to reject at least 1, power to reject all null hypotheses, and the number of expected rejections. We analyzed a multitude of scenarios applying optimization of node weights and edge weights alone, as well as joint optimization. Comparisons for widely used MCPs such as Bonferroni, Holm-Bonferroni, and Fixed-Sequence designs will be presented. Effect sizes were varied in magnitude and distribution among the family of null-hypotheses.

Results: Fitness landscapes of the analyzed scenarios show specific properties, with respect to optimization aim and type. In general, fitness landscapes with respect to node weight optimization (with fixed graph) show clear gradients with increasing slopes when effect sizes increase. For very small effect sizes and also very larger effect sizes, variations in parameters have only a small impact, due to flat fitness landscapes. Our data shows that Holm-Bonferroni is only optimal in certain scenarios, e.g. equally important null-hypotheses with comparable effect sizes. In cases of unequally distributed effect estimates the optimal solutions shift towards parameter combinations favoring the null-hypothesis with the largest effect. In situations where all null-hypotheses should be rejected, procedures that exhibit two or more unconnected subgraphs (e.g. such as Bonferroni correction) are usually sub-optimal, due to the significance level that cannot be redistributed among the graph’s components. Finally, we will present the results from gradient-based optimization techniques and a differential evolution (DE) approach, and compare these with the full-search results.

Discussion: The full-search approach allows to evaluate and compare different optimization algorithms. Furthermore, our approach can also identify solutions of multi-criteria optimization. Overall, the analysis of fitness landscapes will enable statisticians to identify the best graph-based MCP. A comprehensive R-package will be developed to provide easy access to MCP optimization.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.