Predicting future outcomes of patients is essential to clinical practice, with many prediction models published each year. Empirical evidence suggests that published studies often have severe methodological limitations, which undermine their usefulness. This article presents a step-by-step guide to help researchers develop and evaluate a clinical prediction model. The guide covers best practices in defining the aim and users, selecting data sources, addressing missing data, exploring alternative modelling options, and assessing model performance. The steps are illustrated using an example from relapsing-remitting multiple sclerosis. Comprehensive R code is also provided.

Aim: Comparative effectiveness research using real-world data often involves pairwise propensity score matching to adjust for confounding bias. We show that corresponding treatment effect estimates may have limited external validity, and propose two visualization tools to clarify the target estimand.

Materials & methods: We conduct a simulation study to demonstrate, with bivariate ellipses and joy plots, that differences in covariate distributions across treatment groups may affect the external validity of treatment effect estimates. We showcase how these visualization tools can facilitate the interpretation of target estimands in a case study comparing the effectiveness of teriflunomide (TERI), dimethyl fumarate (DMF) and natalizumab (NAT) on manual dexterity in patients with multiple sclerosis.

Results: In the simulation study, estimates of the treatment effect greatly differed depending on the target population. For example, when comparing treatment B with C, the estimated treatment effect (and respective standard error) varied from -0.27 (0.03) to -0.37 (0.04) in the type of patients initially receiving treatment B and C, respectively. Visualization of the matched samples revealed that covariate distributions vary for each comparison and cannot be used to target one common treatment effect for the three treatment comparisons. In the case study, the bivariate distribution of age and disease duration varied across the population of patients receiving TERI, DMF or NAT. Although results suggest that DMF and NAT improve manual dexterity at 1 year compared with TERI, the effectiveness of DMF versus NAT differs depending on which target estimand is used.

Conclusion: Visualization tools may help to clarify the target population in comparative effectiveness studies and resolve ambiguity about the interpretation of estimated treatment effects.

To the Editor: We recently became aware of the study by Chen et al. Notable differences in the 3-month confirmed disability progression (CDP3M) outcome in this analysis have been identified compared with previously published network meta-analysis (NMA). More specifically, the results for CDP3M greatly differ for interferon (IFN) beta-1A 30 mcg every week, IFN beta-1A 44 mcg 3 times a week, IFN beta-1A 22 mcg 3 times a week, natalizumab 300 mg every 4 weeks, and ocrelizumab 600 mg every 24 weeks. The published comparative estimates by Chen et al. may compromise the external validity of the SUCRA ranking results given that it is inconsistent with the totality of the existing body of published evidence. For example, the NMA of Chen et al. includes only one trial assessing the efficacy of natalizumab (where it was compared with placebo). Because there are no trials comparing natalizumab with other active treatments, the pooled effect estimate for natalizumab versus placebo (hazard ratio = 0.85) should remain similar to the treatment effect estimate from the original trial (hazard ratio = 0.58). However, this is not the case in the review of Chen et al. Similar discrepancies appear for ponesimod where the original trial reported a hazard ratio versus teriflunomide 14 mg of 0.83 (0.58; 1.18), whereas the NMA reported a hazard ratio of 1.39 (0.55; 3.57). Therefore, we respectfully request additional transparency from Chen et al. regarding the NMA methods and additional clarity supporting their results.

Real-world data sources offer opportunities to compare the effectiveness of treatments in practical clinical settings. However, relevant outcomes are often recorded selectively and collected at irregular measurement times. It is therefore common to convert the available visits to a standardized schedule with equally spaced visits. Although more advanced imputation methods exist, they are not designed to recover longitudinal outcome trajectories and typically assume that missingness is non-informative. We, therefore, propose an extension of multilevel multiple imputation methods to facilitate the analysis of real-world outcome data that is collected at irregular observation times. We illustrate multilevel multiple imputation in a case study evaluating two disease-modifying therapies for multiple sclerosis in terms of time to confirmed disability progression. This survival outcome is derived from repeated measurements of the Expanded Disability Status Scale, which is collected when patients come to the healthcare center for a clinical visit and for which longitudinal trajectories can be estimated. Subsequently, we perform a simulation study to compare the performance of multilevel multiple imputation to commonly used single imputation methods. Results indicate that multilevel multiple imputation leads to less biased treatment effect estimates and improves the coverage of confidence intervals, even when outcomes are missing not at random.