Objective: To externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19.

Design: Two stage individual participant data meta-analysis.

Setting: Secondary and tertiary care.

Participants: 46914 patients across 18 countries, admitted to a hospital with polymerase chain reaction confirmed covid-19 from November 2019 to April 2021.

Data sources: Multiple (clustered) cohorts in Brazil, Belgium, China, Czech Republic, Egypt, France, Iran, Israel, Italy, Mexico, Netherlands, Portugal, Russia, Saudi Arabia, Spain, Sweden, United Kingdom, and United States previously identified by a living systematic review of covid-19 prediction models published in The BMJ, and through PROSPERO, reference checking, and expert knowledge.

Model selection and eligibility criteria: Prognostic models identified by the living systematic review and through contacting experts. A priori models were excluded that had a high risk of bias in the participant domain of PROBAST (prediction model study risk of bias assessment tool) or for which the applicability was deemed poor.

Methods: Eight prognostic models with diverse predictors were identified and validated. A two stage individual participant data meta-analysis was performed of the estimated model concordance (C) statistic, calibration slope, calibration-in-the-large, and observed to expected ratio (O:E) across the included clusters.

Main outcome measures: 30 day mortality or in-hospital mortality.

Results: Datasets included 27 clusters from 18 different countries and contained data on 46 914patients. The pooled estimates ranged from 0.67 to 0.80 (C statistic), 0.22 to 1.22 (calibration slope), and 0.18 to 2.59 (O:E ratio) and were prone to substantial between study heterogeneity. The 4C Mortality Score by Knight et al (pooled C statistic 0.80, 95% confidence interval 0.75 to 0.84, 95% prediction interval 0.72 to 0.86) and clinical model by Wang et al (0.77, 0.73 to 0.80, 0.63 to 0.87) had the highest discriminative ability. On average, 29% fewer deaths were observed than predicted by the 4C Mortality Score (pooled O:E 0.71, 95% confidence interval 0.45 to 1.11, 95% prediction interval 0.21 to 2.39), 35% fewer than predicted by the Wang clinical model (0.65, 0.52 to 0.82, 0.23 to 1.89), and 4% fewer than predicted by Xie et al's model (0.96, 0.59 to 1.55, 0.21 to 4.28).

Conclusion: The prognostic value of the included models varied greatly between the data sources. Although the Knight 4C Mortality Score and Wang clinical model appeared most promising, recalibration (intercept and slope updates) is needed before implementation in routine care.

While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1-3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance.

Ideally, a meta-analysis will summarize data from several unbiased studies. Here we look into the less than ideal situation in which contributing studies may be compromised by non-differential measurement error in the exposure variable. Specifically, we consider a meta-analysis for the association between a continuous outcome variable and one or more continuous exposure variables, where the associations may be quantified as regression coefficients of a linear regression model. A flexible Bayesian framework is developed which allows one to obtain appropriate point and interval estimates with varying degrees of prior knowledge about the magnitude of the measurement error. We also demonstrate how, if individual-participant data (IPD) are available, the Bayesian meta-analysis model can adjust for multiple participant-level covariates, these being measured with or without measurement error.

Context: While urinary incontinence (UI) commonly occurs after radical prostatectomy (RP), it is unclear what factors increase the risk of UI development.

Objective: To perform a systematic review of patient- and tumour-related prognostic factors for post-RP UI. The primary outcome was UI within 3 mo after RP. Secondary outcomes included UI at 3-12 mo and ≥12 mo after RP.

Evidence acquisition: Databases including Medline, EMBASE, and CENTRAL were searched between January 1990 and May 2020. All studies reporting patient- and tumour-related prognostic factors in univariable or multivariable analyses were included. Surgical factors were excluded. Risk of bias (RoB) and confounding assessments were performed using the Quality In Prognosis Studies (QUIPS) tool. Random-effects meta-analyses were performed for all prognostic factor, where possible.

Evidence synthesis: A total of 119 studies (5 randomised controlled trials, 24 prospective, 88 retrospective, and 2 case-control studies) with 131 379 patients were included. RoB was high for study participation and confounding; moderate to high for statistical analysis, study attrition, and prognostic factor measurement; and low for outcome measurements. Significant prognostic factors for postoperative UI within 3 mo after RP were age (odds ratio [OR] per yearly increase 1.04, 95% confidence interval [CI] 1.03-1.05), membranous urethral length (MUL; OR per 1-mm increase 0.81, 95% CI 0.74-0.88), prostate volume (PV; OR per 1-ml increase 1.005, 95% CI 1.000-1.011), and Charlson comorbidity index (CCI; OR 1.28, 95% CI 1.09-1.50).

Conclusions: Increasing age, shorter MUL, greater PV, and higher CCI are independent prognostic factors for UI within 3 mo after RP, with all except CCI remaining prognostic at 3-12 mo.

Patient summary: We reviewed the literature to identify patient and disease factors associated with urinary incontinence after surgery for prostate cancer. We found increasing age, larger prostate volume, shorter length of a section of the urethra (membranous urethra), and lower fitness were associated with worse urinary incontinence for the first 3 mo after surgery, with all except lower fitness remaining predictive at 3-12 mo.

It is widely recommended that any developed - diagnostic or prognostic - prediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment. We discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome. We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c-statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically-based prior distributions to improve estimation of between-study heterogeneity in small samples. Finally, we illustrate all methods using two examples: meta-analysis of the predictive performance of EuroSCORE II and of the Framingham Risk Score. All examples and meta-analysis models have been implemented in our newly developed R package "metamisc".