Article Text
PDF
Abstract
Background The SARS-CoV-2 pandemic led to a steep increase in hospital and intensive care unit (ICU) admissions for acute respiratory failure worldwide. Early identification of patients at risk of clinical deterioration is crucial in terms of appropriate care delivery and resource allocation. We aimed to evaluate and compare the prognostic performance of Sequential Organ Failure Assessment (SOFA), Quick Sequential Organ Failure Assessment (qSOFA), Confusion, Uraemia, Respiratory Rate, Blood Pressure and Age ≥65 (CURB-65), Respiratory Rate and Oxygenation (ROX) index and Coronavirus Clinical Characterisation Consortium (4C) score to predict death and ICU admission among patients admitted to the hospital for acute COVID-19 infection.
Methods and analysis Consecutive adult patients admitted to the Geneva University Hospitals during two successive COVID-19 flares in spring and autumn 2020 were included. Discriminative performance of these prediction rules, obtained during the first 24 hours of hospital admission, were computed to predict death or ICU admission. We further exluded patients with therapeutic limitations and reported areas under the curve (AUCs) for 30-day mortality and ICU admission in sensitivity analyses.
Results A total of 2122 patients were included. 216 patients (10.2%) required ICU admission and 303 (14.3%) died within 30 days post admission. 4C score had the best discriminatory performance to predict 30-day mortality (AUC 0.82, 95% CI 0.80 to 0.85), compared with SOFA (AUC 0.75, 95% CI 0.72 to 0.78), qSOFA (AUC 0.59, 95% CI 0.56 to 0.62), CURB-65 (AUC 0.75, 95% CI 0.72 to 0.78) and ROX index (AUC 0.68, 95% CI 0.65 to 0.72). ROX index had the greatest discriminatory performance (AUC 0.79, 95% CI 0.76 to 0.83) to predict ICU admission compared with 4C score (AUC 0.62, 95% CI 0.59 to 0.66), CURB-65 (AUC 0.60, 95% CI 0.56 to 0.64), SOFA (AUC 0.74, 95% CI 0.71 to 0.77) and qSOFA (AUC 0.59, 95% CI 0.55 to 0.62).
Conclusion Scores including age and/or comorbidities (4C and CURB-65) have the best discriminatory performance to predict mortality among inpatients with COVID-19, while scores including quantitative assessment of hypoxaemia (SOFA and ROX index) perform best to predict ICU admission. Exclusion of patients with therapeutic limitations improved the discriminatory performance of prognostic scores relying on age and/or comorbidities to predict ICU admission.
- COVID-19
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Prognostic rules for patients with COVID-19 usually perform better to predict mortality than intensive care unit (ICU) admission.
WHAT THIS STUDY ADDS
In a cohort of 2122 inpatients with COVID-19, prognostic rules including quantitative assessment of hypoxaemia performed best to predict ICU admission.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
Impact studies are required to evaluate if prognostic rules predicting ICU admission allow better patient selection and resource allocation.
Introduction
COVID-19 led to a steep increase in hospital and intensive care unit (ICU) admissions for acute hypoxaemic respiratory failure (AHRF) and overwhelmed hospital and ICU capacity. In Switzerland, a first flare of COVID-19 infections occurred in February 2020 and was followed by an unexpectedly greater second wave in October 2020, compelling authorities to restore a partial lockdown.
The canton of Geneva was particularly affected by the sanitary crisis. Public health authorities decided that non-COVID-related inpatients would be mostly managed in private hospitals and that all patients with COVID-19 would be admitted to Geneva University Hospitals (HUG), the only public hospital of the canton. During the first wave (February–May 2020), 1176 hospitalisations were recorded at HUG with a peak at 399 occupied beds including 58 patients in ICU.1–3 During the second wave (September–December 2020), 2231 hospitalisations occurred with a peak at 642 occupied beds and 32 patients in ICU.1 2
SARS-CoV-2 mainly causes hospitalisation for pneumonitis and AHRF. Most inpatients have a favourable course with non-invasive supplemental oxygen, but 15%–30% of them eventually require invasive mechanical ventilation (IMV).4–8 Limitations in ICU capacity have urged the necessity for hospitals to optimise their ICU admission criteria.9
In the context of shortage of resources and bed capacities, early identification of patients who would benefit most from ICU admission and care is of utmost importance. In this context, multiple prediction rules have been proposed to predict in-hospital mortality or ICU admission with contrasted results.4 10–13 In a study comparing 32 prognostic scores, 19 had a lower discriminative performance than in their original derivation or validation study, and 25 performed better to predict in-hospital mortality than the composite of in-hospital mortality or ICU admission.14 The aim of our study was to externally validate and evaluate the performance of selected prognostic scores in predicting 30-day mortality and ICU admission among patients with COVID-19 admitted for hospitalisation at HUG during the first and second waves of the pandemic. We selected scores routinely used in clinical practice at HUG with easily obtainable parameters.
Methods
Study design and source of data
This study is a retrospective, observational, monocentric cohort study conducted at HUG, a primary and tertiary care hospital in Geneva, Switzerland, during the first (February–May 2020) and second waves (September–December 2020) of the pandemic. All demographic, clinical, biological and outcome data were retrieved in the context of a study aiming to compare the two successive COVID-19 flares at HUG. Patients and the public were not involved in the conduct of the study. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis statement was followed to ensure rigorous analysis and transparent reporting.15 Analyses were performed according to a pre-defined protocol. Data was extracted from a database designed for COVID-19-related data. This database contains all clinical information and general consent information available in HUG for all patients tested for SARS-CoV-2 or flagged as positive or suspect in the electronic health record since the beginning of the pandemic.16
Participants
The study included patients aged 16 and older hospitalised at HUG for acute COVID-19 infection for more than 24 hours. COVID-19 infection was defined as a positive reverse transcription–PCR (RT-PCR) testing on a nasopharyngeal swab or lower respiratory tract sample or clinically confirmed COVID-19 diagnosis. We excluded any nosocomial cases (defined as all patients with COVID-19 for whom the first positive RT-PCR test result occurred on hospital day 3 or later (with day of admission defined as day 1). To exclude nosocomial cases, confirmation of diagnosis of community-acquired cases had to occur within 7 days before and 72 hours after admission.
Outcomes
The main outcomes were 30-day mortality after admission to the hospital ICU admission. Mortality at 30 days was defined as living status at 30 days based on medical records. It was assumed that patients leaving the hospital alive and not readmitted within 30 days of the admission were alive. ICU admission was defined as any transfer to the ICU occurring during hospital stay following hospital admission for acute COVID-19 infection regardless of the treatment administered or ICU length of stay.
Independent predictive variables
Five prognostic scores were selected for the analysis: two general intensive care scores, the Quick Sequential Organ Failure Assessment (qSOFA) score and the Sequential Organ Failure Assessment (SOFA) score17 18; two respiratory scores, the Confusion, Uraemia, Respiratory Rate, Blood Pressure and Age ≥65 (CURB-65) score widely validated for community-acquired pneumonia (CAP)19 and the Respiratory Rate and Oxygenation (ROX) index19 20; and a more recently dedicated COVID-19-mortality score, the Coronavirus Clinical Characterisation Consortium (4C) score.11 We remind that, contrarily to the other scores, a low ROX index means higher risk of poor outcome. The components of the selected scores are provided in table 1. The following values were retrieved on admission: age, gender and Glasgow Coma Scale (GCS). For continuous variables such as creatinine, urea, liver bilirubin, thrombocytes, C reactive protein and 24 hours’ urine output, the initial value during the first 24 hours of hospitalisation was extracted. For respiratory rate (RR) and fraction of inspired oxygen (FiO2), the highest value within first 24 hours of hospitalisation was selected. For the latter, the corresponding percutaneous oxygen saturation (SpO2) value was extracted. For systolic blood pressure, diastolic blood pressure and mean blood pressure, the lowest value during the first 24 hours of hospitalisation was retrieved.
Components of selected prognostic scores
Comorbidities were collected according to a modified Charlson Index.11 21 International Classification of Diseases codes were used to retrieve comorbidities of each patient. Each of the following items scored one point: chronic cardiac disease, chronic respiratory disease (excluding asthma), chronic renal disease, chronic liver disease, dementia, chronic neurological condition, connective tissue disease, diabetes mellitus, HIV/AIDS, malignancy and obesity. The last documented data during hospital stay were used for therapeutic limitation instructions (not to be resuscitated or no ICU admission).
Missing data
Only scores for which data were complete were computed. As GCS score is not routinely documented in the absence of altered consciousness, patients without documented GCS score were considered as having a GCS score of 15. Regarding the PaO2:FiO2 ratio, as arterial blood gas was not routinely performed in all patients, an estimation based on SpO2 was performed for patients without arterial blood gas. The following logarithmic function was used for the conversion: log(Pa/FiO2)=0.48+0.78×log (Sp/FiO2).22 Finally, as urine output was not routinely recorded among inpatients with COVID-19 in the absence of haemodynamic instability and/or renal failure, creatinine value was used for SOFA score calculation in patients without documented urine output (table 1). The study included all consecutive patients fulfilling inclusion criteria admitted to the HUG during the study period (n=2122). No predefined sample-size calculation was performed.
Statistical analysis methods
Continuous variables were reported as means (SD) as required. Categorical variables were described by frequencies and relative proportions. Between-group comparisons were performed using Fisher’s exact test, for categorical variables, and Mann-Whitney non-parametric test, as appropriate, for continuous variables. We computed the area under receiving operator characteristics curves from the logistic regression models for each prognostic model. Between-score comparisons of paired areas under the curve (AUCs) were assessed using non-parametric test.23 The score with the best discriminatory performance was used as the reference for between-score comparisons. We computed sensitivity, specificity, positive and negative predictive values with their 95% CIs at three predefined thresholds for qSOFA (>0, >1 and>2),17 SOFA (>6, >9 and>12),18 CURB-65 (>1, >2 and >3),24 4C (>3, >8 and >14)11 and ROX (<5, <15 and <25). We performed sensitivity analyses after exclusion of patients with therapeutic limitations regarding potential ICU admission. R (cran V.4.1.2) with the ROCR, auROC, prettyR, pROC and g-plot packages was used for all statistical analyses. Significance level was set at 0.05 for all comparisons.
Results
Participants
A total of 2122 patients were included (figure 1). SOFA, qSOFA, 4C and CURB-65 scores could be computed in all patients, while ROX index could be computed for 1998 patients. Characteristics of included patients are provided in table 2. During the study period, infections in Switzerland were caused by COVID-19 claves 19A, 19B, 20A, 20B and 20A (EU1) (https://nextstrain.org/groups/swiss/ncov/switzerland). Two hundred sixteen patients (10.2%) required ICU admission and 303 (14.3%) died within 30 days at the hospital. Age, gender, number of comorbidities, admission vital signs and several biological markers differed significantly between survivors and non-survivors (table 2). Characteristics of patients admitted or non-admitted to the ICU are provided in the online supplemental material (online supplemental appendix table S1).
Supplemental material
Study flow chart.
Baseline characteristics of survivors and non-survivors
External validation and performance of selected scores
The 4C score had the best discriminatory performance to predict 30-day mortality (AUC 0.82, 95% CI 0.80 to 0.85), compared with SOFA (AUC 0.75, 95% CI 0.72 to 0.78), qSOFA (AUC 0.59, 95% CI 0.56 to 0.62), CURB-65 (AUC 0.75, 95% CI 0.72 to 0.78) and ROX index (AUC 0.68, 95% CI 0.65 to 0.72) (table 3).
Discriminatory performance of prognostic scores within validation cohort
The ROX index (AUC 0.79, 95% CI 0.76 to 0.82) had the best discriminative performance regarding ICU admission, compared with 4C score (AUC 0.62, 95% CI 0.59 to 0.66), CURB-65 (AUC 0.60, 95% CI 0.56 to 0.64), SOFA (AUC 0.74, 95% CI 0.71 to 0.77) and qSOFA (AUC 0.59, 95% CI 0.55 to 0.62) (table 3). The corresponding receiver operating characteristic curves are provided in figures 2 and 3.
ROC curve: 30-days mortality outcome. CURB-65, Confusion, Uraemia, Respiratory Rate, Blood Pressure and Age ≥65; qSOFA, Quick Sequential Organ Failure Assessment; ROC, receiver operating characteristic; SOFA, Sequential Organ Failure Assessment.
ROC curve: ICU-admission outcome. CURB-65, Confusion, Uraemia, Respiratory Rate, Blood Pressure and Age ≥65; ICU, intensive care unit; qSOFA, Quick Sequential Organ Failure Assessment; ROC, receiver operating characteristic; SOFA, Sequential Organ Failure Assessment.
Sensitivity analyses after exclusion of 558 patients with therapeutic limitations regarding ICU admission provided higher indices of discriminative capacity to predict ICU admission for CURB-65 (AUC 0.67, 95% CI 0.63 to 0.70) and 4C scores (AUC 0.72, 95% CI 0.69 to 0.75), while the AUC of qSOFA (AUC 0.60, 95% CI 0.57 to 0.64), SOFA (AUC 0.78, 95% CI 0.75 to 0.82) and ROX index (AUC 0.82, 95% CI 0.78 to 0.85) remained unchanged (online supplemental appendix table S2). Discriminative performance to predict 30-day mortality was unchanged (table S3).
The sensitivity, specificity, and positive and negative predictive values of the prognostic scores at various cut-off levels are provided in the online supplemental appendix (table S4 and S5).
Discussion
We conducted an external validation and comparison of five prognostic scores in a retrospective cohort of 2122 patients hospitalised at HUG with confirmed COVID-19 infection. Five scores were assessed: two general scores (qSOFA and SOFA) used at ICU, two respiratory scores (CURB-65 and ROX index) and a recently created COVID-19-specific score (4C). The 4C score performed the best to predict 30-day mortality in our cohort with a discriminative performance similar to previous studies including derivation and internal and external validation cohorts.11 14 25–27 In the original derivation cohort and internal validation cohort, the reported AUCs were 0.79 (95% CI 0.78 to 0.79) and 0.77 (95% CI 0.76 to 0.77), respectively. However, 4C score performance to predict ICU admission was lower and inferior to ROX index or SOFA score in our study. More generally, prognostic scores including age and/or comorbidities (4C and CURB-65) were more discriminant to predict 30-day mortality, while scores including quantitative assesment of hypoxaemia (SOFA and ROX index) were more helpful to predict ICU admission.
These observations deserve several comments: first, and as previously reported by others,4 5 8 age and comorbidities are strong predictors of mortality among patients with COVID-19. However, as previously reported for bacterial CAP, prognostic rules relying heavily on age and comorbidities performed poorly to identify patients requiring treatment intensification compared with scores relying on the severity of the acute infection.28 In our cohort, 26% of patients had therapeutic limitations precluding ICU admission and 84% of study patients who died were not admitted to the ICU. These patients are likely older and comorbid, limiting the predictive value of these variables to predict ICU admission. After exclusion of patients with therapeutic limitations, the discriminative capacity of prognostic scores relying on age and/or comorbidities such as CURB-65 or 4C to predict ICU admission improved but remained inferior to ROX index or SOFA’s discriminative performance for this outcome.
In a recent comparison of 32 COVID-19 prognostic scores in a cohort of 14 343 patients hospitalised in France, the vast majority of scores (78%) performed better to predict mortality alone than the composite outcome of mortality and ICU admission; only two had an AUC of >0.70 predicting death or ICU admission. Unfortunately, the performance of SOFA and ROX index was not evaluated in this comparison.14
Similarly, CURB-65 has been reported to perform well to predict 30-day mortality among patients with COVID-19,11 25 29–34 but results were less consistent regarding ICU admission.25 29–31 33
As severity of COVID-19 is mainly determined by the extent of parenchymal lung injury, it is not surprising that scores quantitatively evaluating the severity of AHRF perform best to predict the need for ICU and invasive ventilation. Moreover oxygen saturation is often maintained in a narrow range in hospitalised patients (92%–94%), and tachypnoea may be absent or delayed in the context of COVID-19 with a pattern of unusual tolerance to hypoxaemia (so-called ‘happy hypoxaemia’).35 Thus, the severity of AHRF is probably best detected by scores taking into account the inspired fraction of oxygen as ROX index or SOFA score rather than scores taking into account RR or oxygen saturation alone.
Although the ROX index was originally validated to predict intubation in patients receiving high-flow nasal oxygen (HFNO) therapy for CAP at 12 hours,20 several studies reported good discriminative performance to predict treatment escalation among patients with COVID-19 receiving HFNO36 37 or conventional low-flow oxygen.12 13
However, the optimal cut-off value to predict HFNO failure in COVID-19 has been debatted, and Vega et al proposed a higher threshold (5.99) for patients with COVID-19 than historically proposed for non-COVID pneumonia (4.9).20 36
In our cohort, a ROX index of 5 was associated with good specificity but poor sensitivity and insufficient positive predictive value to be used as a stand-alone decision tool for ICU admission. As a consequence, and as proposed in the landmark study, low ROX index should prompt timely re-evaluation and close follow-up rather than mandate ICU admission.
In agreement with our analysis, most studies report poor performance of the qSOFA as a mortality prediction tool11 38–45 and as an ICU-admission predictor25 32 43–46 for patients with COVID-19. This may be explained by the fact that, in contrast with bacterial CAP, circulatory failure occurs seldomly in patients with COVID-19, and outcomes are mainly determined by the severity of AHRF, age and comorbid conditions which are poorly taken into account in qSOFA.
Our study has several strengths: first, we included a relatively large number of consecutive patients, allowing us to provide precise estimates of several prediction rules and to perform between-score comparisons for relevant outcomes. Second, we selected widely used prediction rules based on easily available clinical variables in routine practice. Third, we report separate performances for both mortality and ICU admission, which represent different clinical decisions requiring a risk stratification tool, also taking into account the impact of therapeutical limitation decisions on the performance of these models. Finally, all patients of the canton were hospitalised at HUG, which reduces a usually strong selection bias in a public establishment.
Our study also has some limitations: first, it is a monocentric study performed at a large primary care and teaching hospital, which may limit the generalisability of our findings. In particular, our study was conducted before the emergence of COVID-19 variants and before the beginning of vaccination in Switzerland. As these two factors influence the baseline risk of complications among infected patients, the performance of prognostic scores, in particular regarding positive and negative predictive values, may differ among currently infected patients with different viral strains and vaccination status. Second, multiple prediction rules have been proposed for the risk stratification of patients with COVID-19 and were not evaluated in the present work. However, our decision to select a limited number of prediction rules was based on the previous validation of these scores and the easy availability of included variables in clinical practice. Third, we retrospectively extracted comorbid conditions based on ICD-10 (International Statistical Classification of Diseases and Related Health Problems)codes, which may fail to identify some chronic comorbid conditions, and used ICU admission, which represents a medical decision as a proxy for the need for invasive ventilation. In fact, 96% of patients admitted to the ICU received IMV at our institution during the first COVID-19 flare.3 However, the indication for IMV in our institution was based essentially on the inability to maintain peripheral oxygenation superior to 90% and/or respiratory distress despite high FiO2 (>80%), which may explain the better predictive performance of prediction rules based on oxygenation parameters but also constitute some incorporation bias. Finally, our study aimed at evaluating and comparing the general discriminative performance of each score, but we did not assess and compare the calibration of the different models.
In conclusion, scores including age and comorbidities appear to perform better to predict mortality among patients hospitalised for COVID-19, while prediction rules incorporating quantitative assessment of AHRF perform better to predict the need for ICU admission and IMV. Impact studies are required to evaluate if clinical decisions such as ICU admission based on these prognostic models would improve selection of patients requiring IMV and allow a better allocation of resources.
Conclusion
Prognostic scores including age and/or comorbidities (4C and CURB-65) perform better to predict mortality among inpatients with COVID-19, while scores including quantitative assessment of hypoxaemia (SOFA and ROX index) perform better to predict ICU admission. Exclusion of patients with therapeutic limitations improved the discriminative capacity of prognostic scores relying on age and/or comorbidities to predict ICU admission. None of these scores is sufficiently discriminative to be considered as a stand-alone decision for ICU admission but may allow early identification of patients at increased risk of clinical deterioration who may warrant timely re-evaluation or closer monitoring.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by BASEC 2021-00302 Commission Cantonale d’Ethique de la Recherche sur l’être Humain in accordance with the Federal Human Research Act (art. 34). The participants gave informed consent to participate in the study before taking part.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors JM planned the study, performed the data extraction, conducted the analyses and drafted the first version of the manuscript. CG-B participated in study conception, performed the data extraction and participated in interpretation and reporting of the results by critically revising the manuscript for important intellectual content. JS contributed to the study conception, participated in the analysis plan and participated in the interpretation and reporting of the results by critically revising the manuscript for important intellectual content. CL, AG-A, AI, SC, OG, PD-F, AB, AL and J-LR participated in the study conception and reporting by critically revising study protocol, results interpretation and reporting for important intellectual content. CM conceived the study and participated in the interpretation and reporting of the results by critically revising the different versions of the manuscript. CM and JM are responsible for the overall content of the manuscript. CM acts as guarantor.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, conduct, reporting or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.