Ventilator-associated pneumonia (VAP) is the most common Intensive Care Unit (ICU)-acquired infection.1 Rates of VAP have become markers of “quality of care”.2 3

The diagnosis of VAP prompts antibiotic therapy. A diagnostic technique with a high false-positive rate will increase antibiotic use, potentially resulting in adverse outcomes such as Clostridium difficile-associated diarrhoea and selection pressure for multidrug-resistant bacteria.4 5

The diagnosis of VAP depends on a combination of clinical, radiological and microbiological findings.1 Current practice and opinion is divided on the relative merits of non-invasive and invasive (bronchoscopic) techniques in obtaining specimens for diagnosis. A French randomised controlled trial (RCT) compared an invasive, bronchoscopic diagnostic strategy with a non-invasive strategy using tracheal aspirates. Patients in the invasive diagnostic strategy group had more antibiotic-free days and were more likely to survive to 14 days postrandomisation.6 In contrast, a recent Canadian RCT comparing invasive and non-invasive diagnostic strategies found no differences in antibiotic use, or in clinical outcomes.7 One possible explanation for these discordant findings could be that diagnostic information was used differently. A positive diagnosis is more likely using endotracheal aspirate (ETA) than bronchoalveolar lavage (BAL), most probably indicating lower specificity.6 8 9 If antibiotics are continued despite negative BAL cultures, any benefits will be lost. Inappropriate early antibiotic therapy is associated with greater mortality,10 11 which has led to the widespread use of broad-spectrum antibiotics known to cover the major pathogens present in a given ICU.12 13 The consequences of antibiotic overuse, and specifically failure to de-escalate therapy after negative investigations, are less well understood. Changes in clinical outcomes after altering diagnostic techniques are most likely if linked to clinical decision making, especially in relation to the duration and intensity of antibiotic therapy.14 15 Using different diagnostic tests could also affect the reported incidence of VAP. Few studies have systematically evaluated this possibility despite the increasing use of VAP incidence as a marker of patient safety and quality of care.2 3

The objective of our study was to use clinical data from our ICU to model the potential effects of diagnostic technique, specifically ETA- versus BAL-based diagnosis, on the reported incidence of VAP and associated antibiotic use. We then aimed to evaluate our model by comparing VAP incidence and antibiotic use before and after a practice change initiative designed to increase use of BAL.

METHODS

Setting

The study took place in the 18-bed medical–surgical critical care unit of a large Scottish teaching hospital. The ICU admits >1000 patients annually, of whom 50% stay for ⩾48 h. The case mix was 50% surgical in origin, 48% medical and 2% obstetric/gynaecological. A total of 83% of patients received support of two or more organ systems and/or required invasive ventilatory support at some point during their admission. The unit is the Scottish Liver Transplant unit (typically 40–50 transplants annually), and receives trauma cases from the region, although isolated neurotrauma was managed elsewhere. During the entire duration of the study, a seven-element ventilator care bundle was in place and remained unaltered. This consisted of routine stress ulcer prophylaxis, head of bed elevation, nurse-led weaning protocol, sedation protocol and scoring, use of heat and moisture exchangers, no routine ventilator circuit changes and early empiric antibiotic treatment for VAP with rationalisation and de-escalation based on the results of cultures. Selective digestive tract decontamination, oral chlorhexadine and subglottic suction were not used at any point during the study.

Overview of study design

The study had three parts. First, the diagnostic performance of ETA and BAL were compared using paired samples from patients with clinically suspected VAP. Secondly, prospective high quality independently collected infection surveillance data from our ICU were used to model the potential effect on VAP incidence and antibiotic use of exclusively using either ETA or BAL to investigate suspected VAP in our patient population. Thirdly, a practice change initiative was undertaken to increase BAL use. The incidence of VAP and antibiotic use were compared for the 12 months before and after the change using traditional statistical tests, but also statistical process control (SPC) methods.

Comparison of the diagnostic performance of ETA and BAL

As part of an ongoing study of innate immunity in critically ill patients with suspected VAP,16 we collected a series of paired ETA and BAL samples in patients with clinically suspected VAP using a standardised protocol.17 18 “True VAP” was defined as culture of organisms in BAL at >10⁴ colony-forming units (CFU)/ml.19 We compared the diagnostic accuracy of quantitative and non-quantitative cultures of ETA with true VAP. For quantitative ETA a cut-off of >10⁶ CFU/ml20 was used, whilst any growth was considered positive for qualitative ETA.

Modelling of the potential impact of exclusive use of ETA or BAL on reported VAP incidence and antibiotic use

Infection surveillance data

Since 2005, infection surveillance data on VAP using the Hospitals in Europe Link for Infection Control through Surveillance (HELICS) system21 22 has been collected in our ICU by an independent infection control team. These included antibiotic prescriptions and diagnostic method (bronchoscopic, pleural or blood culture, qualitative endotracheal cultures, or clinical diagnosis alone). Data for the 12 months prior to the practice improvement intervention (see below) were analysed. The diagnostic categories used by HELICS are shown in table 1.

Local intensity of antibiotic use for clinically suspected VAP

The “antibiotic load” associated with treating clinically suspected VAP was quantified by calculating the total “antibiotic days”23 used for each suspected VAP episode. For the purpose of this analysis, we defined this as the number of antibiotics multiplied by the duration of treatment. For instance, if a patient received meropenem for 7 days and vancomycin for 4 days this would equate to 11 “antibiotic days”; concurrent use of two antibiotics counted as two antibiotic days per calendar day. As only qualitative ETA and BAL were in routine clinical use, only these two modalities were compared.

Modelling the theoretical effect of using either ETA or BAL in our patient population

Using infection surveillance data, the number of clinically suspected VAPs was determined. Using the diagnostic performance characteristics calculated from the paired ETA and BAL samples (see above), we modelled the proportion of clinically suspected VAPs that would have been confirmed microbiologically if either BAL, or qualitative ETA or quantitative ETA, had, hypothetically, been used exclusively in our ICU. As a sensitivity analysis we used the 95% CIs generated from the paired sample data. Using data on antibiotic prescriptions, we also estimated the theoretical effect of changing from exclusively ETA-based diagnosis to BAL-based diagnosis in our ICU.

Prospective evaluation of the effect of increasing the use of BAL on reported VAP incidence and antibiotic use

We undertook a practice improvement initiative aimed at increasing the use of BAL in our ICU. This included education sessions, improved availability of equipment and expertise for BAL and a case review of VAPs at the weekly grand round. Infection surveillance data were analysed 1 year before and 1 year after the intervention, with a 2-month “run-in period”. Specifically we examined the rates of clinically suspected VAP and the rates of microbiologically confirmed VAP, the proportion of patients undergoing bronchoscopic diagnosis and the total number of “antibiotic days” attributable to each episode of VAP. Additional analysis was undertaken using SPC methodology, which allows time series data to be analysed in the context of background variability.24 To test whether a change results in a meaningful difference, “warning lines” and “control lines” are established, indicating the upper and lower limits of background variation. If the measured variable crosses these lines it indicates a frequency that is outside normal variation.

Consent and ethical approval

For the paired ETA and BAL study, witnessed assent was obtained from a relative or main carer for all patients and the study was approved by the Research Ethics Committee. The collection of infection surveillance data did not require ethical approval. Evaluation of the effect of increasing BAL use was considered quality improvement and did not require ethical approval.

Further details of the methodology can be found in the Supplementary data online.

RESULTS

Diagnostic accuracy of BAL and ETA

Fifty-three patients had paired samples available for analysis (fig 1). With microbiologically confirmed infection (HELICS criteria PN1; for definitions of HELICS criteria see table 1) the true positive rate for VAP (as defined by quantitative BAL cultures) was 21% (n = 11) (95% CI 10% to 32%). Qualitative ETA cultures significantly overdiagnosed VAP compared with BAL; 87% (n = 46) (95% CI 77% to 96%) of all cultures were positive (HELICS criterion PN4). Quantitative ETA cultures also overdiagnosed VAP, 51% (n = 27) (95% CI 37% to 63%) of all cultures being positive by this method. A summary of test performance data for ETA compared with BAL is shown in table 2. Although quantitative ETA cultures (using >10⁶ CFU/ml as the cut-off) had an effect on sensitivity and specificity, they failed to improve diagnostic performance significantly as assessed by either predictive values or likelihood ratios (table 2). There was no difference in ICU mortality between patients with clinically suspected VAP and positive versus negative BAL cultures (36% for both groups). These data suggested that the technique used to diagnose VAP was likely to influence the reported incidence, but did not quantify the likely magnitude of this effect in a clinical population.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Diagnostic tree for ventilator-associated pneumonia (VAP) by endotracheal aspirate (ETA) using quantitative cultures obtained from rigorously standardised bronchoalveolar lavage (BAL) as the reference. (A) demonstrates this for qualitative cultures, and (B) for quantitative ETA cultures taking >10⁶ colony-forming units (CFU)/ml of ETA fluid as positive.

Modelling of the potential impact of exclusive use of ETA or BAL on reported VAP incidence and antibiotic use

In the year prior to the intervention (January–December 2006), 643 patients were admitted to the ICU for >48 h, corresponding to 3771 ventilator days. There were 110 episodes of clinically suspected VAP, based on HELICS criteria. Of these, 68 had positive microbiological cultures. The classification of these VAPs based on the HELICS system is shown in fig 2. The incidence of clinically suspected VAP was 17% (29.1 cases per 1000 ventilator days) and microbiologically confirmed VAP 10.5% (18 cases per 1000 ventilator days). The associated antibiotic use with each mode of diagnosis is shown in table 3. A total of 16% of patients had their antibiotics changed in ICU in the 72 h prior to diagnosis, an acknowledged risk factor for false-negative cultures.1 A total of 89% of patients had received antibiotics prior to diagnosis, 75% within 72 h of diagnosis.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Data from routine infection surveillance. Patients at risk of ventilator-associated pneumonia (VAP) are all those mechanically ventilated for ⩾48 h. PN numbers refer to the criteria laid out in the HELICS (Hospitals in Europe Link for Infection Control through Surveillance) protocol (see table 1). BAL, bronchoalveolar lavage; ETA, endotracheal aspirate.

Modelled effect on reported incidence

Based on the above data and test-specific data from the paired samples study (with 95% CI as sensitivity analysis from fig 1), the potential impact of using exclusively ETA or BAL for diagnosis on the reported incidence of microbiologically confirmed VAP in our ICU is shown in table 4. These data suggested that if BAL were used exclusively together with HELICS surveillance definitions of VAP (PN1), the reported incidence of VAP in our population would be 76% (95% CI 67% to 87%) lower than if non-quantitative ETA were used exclusively (PN4) (relative risk reduction 0.22 (95% CI 0.17 to 0.35)). This estimate did not take into account the small (1%) false-negative rate with qualitative ETA cultures. Use of BAL in preference to a quantitative ETA culture (PN2) would result in a 59% (95% CI 50% to 73%) reduction, but quantitative ETA cultures would have a 6% false-negative rate (and, as can be seen in table 2, quantitative ETA culture did not have significantly better diagnostic performance than qualitative ETA culture).

Modelled effects on antibiotic use

Using the actual rates of antibiotic use associated with different diagnostic techniques (table 3) and the test performance documented in the paired sample analysis (table 4), the hypothetical effects of exclusively using either ETA or BAL on total antibiotic use are shown in table 5. This model suggested that antibiotic use could be reduced by up to 30% (95% CI 20% to 42%) in our ICU by using BAL rather than ETA if clinical use of antibiotics in response to positive and negative ETA and BAL data did not change.

Prospective evaluation of the effect of increasing the use of BAL on reported VAP incidence and antibiotic use

The patient case mix was similar in both the preintervention and postintervention periods (table 6). Following the intervention, the rate of bronchoscopy in suspected VAP increased from 37% to 58% (68% in the final quarter of the analysis period). The overall rate of clinically suspected VAP was similar before and after the intervention (table 6). In contrast, there was a clinically and statistically significant decrease in the reported incidence of confirmed VAP during the 12 months following the practice change, (relative risk reduction 0.61 (95% CI 0.46 to 0.82) p = 0.0012; table 6). Antibiotic use decreased by 21%, which was clinically important (1.9 fewer antibiotic days per episode during the second 12-month period), but did not reach statistical significance (table 6; p = 0.08). There was no difference in the proportion of patients who had antibiotics changed in the 72 h before diagnosis (16%), nor in the proportion receiving antibiotics prior to diagnosis (89%, 70% within 72 h of diagnosis). Mortality for all patients with clinically suspected VAP fell from 37.5% to 17% (p = 0.002), although the median APACHE II (Acute Physiology and Chronic Health Evaluation II) score in the patients with clinically suspected VAP was lower in the postintervention period (23 vs 20 p = 0.02). These data indicated changes consistent with those predicted in our hypothetical model, particularly as the difference in the use of ETA versus BAL was 21% compared with 100% in the model.

The reported incidence of VAP using HELICS surveillance methodology over the entire period of evaluation (January 2006 to May 2008) are shown using SPC charting in fig 3. Of note, there were two time points when the incidence of VAP crossed the upper warning line, indicating variation above normal background variability, during the preintervention period and no episodes following it. Towards the latter part of the postintervention period the incidence dropped below the long-term average for our unit, suggesting a genuine reduction in reported incidence of VAP (for an in-depth review of SPC methodology, see Beneyan et al.24

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Statistical process chart showing incidence of confirmed ventilator-associted pneumonia (VAP) in the preintervention and postintervention periods. Process av, process average; UCL, upper control line; UWL, upper warning line. Lower control line and lower warning lines are omitted for clarity.

DISCUSSION

This study provides the first evaluation of the effect of changing the method of diagnosis on the rate of VAP within the context of normal clinical practice. Using a combination of ongoing surveillance data and direct comparisons of ETA and BAL performance in our own population, we modelled the likely effect of increasing the use of BAL on VAP rates and antibiotic use. We then confirmed this effect using a practice change initiative. Our data suggest a favourable effect on both reported VAP rates and antibiotic use from increasing BAL use in routine clinical practice. The changes were consistent with the effects predicted by the model given that the clinical change in diagnostic method was significantly smaller than the hypothetical model (21% vs 100%).

A strength of our study was the use of high quality infection surveillance data of routine clinical care collected by an independent infection control team, so reducing the risk of bias. The modelling of the potential impact of practice change was conducted using high quality samples from a single experienced bronchoscopist using a highly standardised procedure. All data used in the modelling process were derived from the unit in which the practice improvement occurred, which minimised the number of assumptions that had to be made. By conducting a sensitivity analysis we were able to explore the potential range of effects.

The modelling was tested in the same institution through the use of a quality improvement approach, thus embedding the evaluation within routine practice. Analysis of the results was conducted by classic statistical tests and SPC methodology which both demonstrated similar effects.

Debate continues around the mortality that can be attributed to VAP14 and the value of using this as an outcome measure in studies,15 with a suggestion that other measures such as antibiotic use should be used. The reduction in antibiotics in our study did not achieve statistical significance, although there was a strong trend towards this. The study was not powered to detect a difference in antibiotic prescription, but rather was a pragmatic investigation in the context of normal clinical practice.

Caution should be exercised in extrapolating these data to other populations. The use of a before and after study design is subject to potential time and treatment effect biases, although the finding of similar results to those predicted by the modelling is encouraging. We have compared ETA with a test that is not a universally accepted gold standard (ie, quantitative BAL). In comparison with the proposed, but clinically impractical, gold standard of histology, quantitative BAL may have only moderate predictive ability.26 27 However, our comparison was pragmatic and clinically relevant.

The lack of specificity of ETA cultures has been noted previously8 9 and this is reflected in the higher proportion of diagnoses in patients whose samples are ETA derived.6 Some studies have suggested no difference in sensitivity between the two techniques,28 29 and consequently no impact on treatment or outcomes.7 Although some studies have suggested that quantitative cultures of ETA can be used in place of invasive techniques,20 30 31 we did not demonstrate sufficient diagnostic performance to support this approach, and the 6% rate of false negatives gives cause for concern.

Clinical criteria alone are problematic in the diagnosis of VAP, as a variety of pathologies can mimic pneumonia.32 Attempts to develop more structured clinical tools such as the Clinical Pulmonary Infection Score (CPIS) have not demonstrated significantly improved diagnosis,33 although they may have a role in determining response to therapy.14 Our study confirms the low specificity of clinical criteria.

The finding of reduced reported incidence with BAL-based strategies has two possible interpretations, which are relevant to the conflicting conclusions of the two major RCTs in the field.6 7 First, it may be that quantitative BAL is underdiagnosing true VAP, and that the apparent reduction in incidence and reduction in antibiotic use represent missed VAPs and hence undertreatment. The second interpretation is that ETA overdiagnoses VAP and that use of this method results in overtreatment. Were the former to be correct one might expect a reduction in antibiotic use to be associated with an increased mortality or duration of ventilation. In our study we found a reduction in mortality. There are several possible explanations for this observation. It is possible that the case mix was different, despite comparing all patients fulfilling clinical HELICS VAP criteria (a similar proportion in both cohorts). Illness severity at ICU admission was higher in the preintervention group, as indicated by higher APACHE II scores, which is likely to account for some of the observed effect. However, the observed reduction in mortality associated with clinically suspected VAP is consistent with a previous RCT6 and other studies34 showing associations between inappropriate antibiotic therapy and increased mortality. It is not possible from our before and after study to be sure of the relative contribution, if any, to reduced mortality from changing diagnostic methodology.

Much of the debate concerning methods of diagnosis of VAP has revolved around the impact on patient outcomes.15 This is the first study to look specifically at the effect on incidence rates, which are of crucial importance to intraunit and interunit comparisons. These form the cornerstones of “benchmarking” in the quality improvement processes.2 3 With the potential for the withdrawal of reimbursement by Medicaid and Medicare for episodes of VAP in US hospitals, variations in incidence rates may also have significant financial impact. The variability in incidence demonstrated above leads us to question the usefulness of comparisons of VAP rates for quality control, unless there is considerably more standardisation of diagnostic techniques.

We have demonstrated that, in a unit that is receptive to the idea, the bronchoscopy rate in suspected VAP can be increased and that this increase leads to a significant decrease in reported incidence. Alongside this is a reduction in antibiotic use, suggesting that it is what the clinician does with the information derived from diagnostic testing that is of crucial importance. Despite most infections being monobacterial, especially amongst those who underwent diagnosis by BAL, most patients initially received a combination of antibiotics, with de-escalation once the infective organism was confirmed by microbiological culture. This is in accordance with suggested antimicrobial best practice.12 13 In the future the use of more rapid diagnostic strategies such as PCR testing35 may allow de-escalation to occur more promptly, so further reducing antibiotic exposure. The finding that our unit tended to use shorter courses of antibiotics for negative ETA samples than for negative BAL samples is intriguing. This study was not designed to look at clinicians’ decision-making processes. It is possible that there were differences between patients selected for bronchoscopy and those who were not. Alternatively, clinicians may have greater confidence that a negative ETA, which in our unit meant no bacterial growth, reflected a true negative for pneumonia. A negative BAL, where bacterial growth below the threshold of 10⁴ CFU/ml was classed as negative, may have been associated with greater clinical uncertainty.

A number of interventions have been proposed to reduce rates of VAP, which have been included in “ventilator care bundles”, promoted by a growing number of quality improvement organisations.2 3 The results of this study suggest that increasing the use of bronchoscopy as the preferred diagnostic modality may significantly reduce reported VAP rates. This is clearly relevant to any research in this area, and future trials into diagnostic technique should take into account how the information from these tests is applied by clinicians. It is also vital that comparisons between units take account of differences in diagnostic technique used, and that changes in diagnostic technique be reported alongside any apparent changes in incidence.