Pleural disease•

Pulmonary passport: a service evaluation study of a standardised web-based procedure logbook to aid specialist respiratory training and appraisal

Abstract

Introduction The pulmonary passport (PP) is a secure web-based procedural logbook for specialist respiratory trainees with enhanced functionality that includes automated analysis to provide key performance metrics and in-platform interactions with supervisors.

Methods This service evaluation study used preimplementation and postimplementation online surveys in both trainees and supervisors along with analysis of recorded data within the PP to evaluate the impact of this service on data capture, training, appraisal and quality assurance.

Results From August 2017 to August 2019, 69/73 (95%) specialist respiratory trainees eligible to use the PP across two UK health education deaneries registered with the system and logged 7352 procedures. 3105 thoracic ultrasound procedures identified 2145 pleural effusions and resulted in 1253 pleural procedures of which 96% were successful. 4% of ultrasounds required referral to a more expert sonographer. Iatrogenic bleeding and pneumothorax both occurred in ≤1% of all pleural procedures. 1909 basic diagnostic bronchoscopies were recorded including 1236 bronchial washes, 328 brushes and 221 endobronchial biopsies where definite tumour was identified (biopsy sensitivity 74%). Preimplementation and postimplementation survey data confirmed the PP had increased the consistency of logging procedures by trainees, the depth of data captured, the review of procedural performance metrics in appraisal and the frequency of formal supervisor feedback.

Discussion In this regional project, the implementation of a web-based procedural logbook has been feasible with excellent uptake and has enhanced procedural recording, supervision and appraisal. Furthermore, it provides unprecedented quality assurance at an individual trainee, trust and deanery level and has a number of potential wider applications in the future.

Key messages

  • What has been the impact of a novel, web-based procedural logbook (the pulmonary passport) on data capture, training, appraisal and quality assurance in specialist respiratory trainees?

  • The pulmonary passport has increased the consistency of logging procedures, the depth of data captured, the review of performance metrics in appraisal and the frequency of supervisor feedback.

  • This tool has the potential to drive standardisation of procedural monitoring, audit and performance review for both trainees and consultants within respiratory medicine and in other specialties.

Introduction

The Joint Royal Colleges of Physicians Training Board (JRCPTB) respiratory curriculum mandates satisfactory recording of common respiratory procedures learnt by trainees such as bronchoscopy, thoracic ultrasound and pleural interventions.1 However, the official electronic training portfolio (National Health Service (NHS) ePortfolio) does not provide the ability to log all procedures performed by a trainee, instead educational supervisors and Annual Review of Competency Progression (ARCP) panels are reliant on the individual trainee producing a bespoke logbook or record of practice. This results in variable and inconsistent methods of logging procedures, ranging from maintaining electronic spreadsheets to handwritten notation. These methods are not standardised in format nor data collection and do not provide consistent measurement against established performance metrics. This, together with a lack of meaningful procedural data capture and the lack of quality assurance, made it challenging for supervisors to accurately assess procedural volume, experience and competency. Historically, limited exposure and poor-quality supervision in common respiratory procedures have contributed to trainee dissatisfaction.2 3 Furthermore, a trainee appraisal with an educational supervisor requires an assessment of procedural competency but potentially that supervisor might not perform the procedure in question themselves or may not have supervised the trainee performing the procedure directly. Without a robust record of procedural activity and performance, it is difficult for educational supervisors to undertake a robust evaluation of the trainee’s progress and competency.

A review of published literature relating to bronchoscopy training depicts a clear need for a standardised, competency-based approach to procedural training4 5 and a move away from the historic focus on case numbers.6 There is currently both regional and international variation in the quality and availability of procedural training.7 8 Approaches to defining and achieving competency are described including the use of validated competency assessment tools,9–11 simulation training12 13 and less formal ‘supervisor sign-off. Within the UK, endoscopy training has been supported by an online procedural logbook as part of the Joint Advisory Group Endoscopy Training System. This has been associated with demonstrable improvements in quality of training and quality assurance of clinical endoscopy services.14 An electronic procedural logbook exists for surgical trainees in the UK and Ireland and is endorsed by national training boards.15 However, none existed for respiratory trainees.

The ‘pulmonary passport’ (PP) project was developed across two health education deaneries in the North West of England (North West and Merseyside Deaneries, 21 acute care trusts). The objective of this project was to provide a web-based application for specialist respiratory trainees that allows standardised recording of procedures, input of outcome data, performance analysis and ‘in-platform’ interaction with procedural supervisors. This platform would support high-quality procedural logging, supervision, competency confirmation and quality assurance which in turn may help to improve the overall standard and satisfaction with respiratory medicine training and enhance the appraisal process. In order to meet the project objectives, five critical functional components were identified for implementation within the PP; user interface, logging, performance analysis, training and appraisal. These were further divided into subfunctions (table 1) and embedded within an easy to navigate central dashboard in a secure platform (figure 1). Local charitable funds were secured to develop the PP website and the system was launched across the two deaneries in August 2017 and has been in operation ever since. The ongoing service costs are funded through a small cost that is top sliced from the trainee study budget at £6 per month per trainee.

Figure 1
Request permissionrequest permission icon
Figure 1

Screenshot of The pulmonary passport central dashboard.

Table 1
|
Summary of the pulmonary passport functionality

The aim of this service evaluation study was to understand the mode and depth of procedural recording by specialist respiratory trainees in the two deaneries prior to the launch of the PP and then evaluate the impact of this service on data capture, training, appraisal and quality assurance following implementation. The results of this evaluation could then be used to understand the long term viability and sustainability of the service.

Methods

Baseline trainee data

At any given time there are approximately 50–60 active specialist trainees within the Respiratory Medicine training programmes across the North West and Mersey Deaneries. In August 2020, following a regional call for volunteers to provide baseline information on procedural recording a cohort of 26 trainees were identified and sent a link to an online survey platform. Trainees were asked to provide information on which procedures they kept a logbook for and how they recorded their procedures prior to the PP (or if they were not currently using the PP). They were also asked which specific data fields they collected for selected procedures in which competence is required for completion of the training programme (basic diagnostic bronchoscopy, Seldinger chest drain insertion and thoracic ultrasound). These specific data fields mapped to the data fields collected within the standardised procedural logs in the PP to allow comparison on the depth of data collected between existing procedural logbooks and the PP system. Finally trainees were asked which procedural performance metrics they have previously been able to present within their appraisals prior to the PP (eg, sensitivity of endobronchial biopsies during basic diagnostic bronchoscopy when definite tumour seen). These metrics were also mapped to those routinely provided by the PP in order to compare the provision of these metrics preimplementation and postimplementation.

Evaluating the impact of the PP: user and procedural data

To evaluate the impact of the PP we evaluated uptake (number of trainees and supervisors registered with the website compared with number of trainees it is made available to, number of procedure logs completed and number of outcome logs completed), depth of procedural data captured (number and type of procedures, pathology identified, volume of diagnostic sampling), performance and quality assurance (immediate/late complication rates and diagnostic performance against recognised standards). Data analysis for the purpose of this report was performed on procedures completed between August 2017 and August 2019. In this manuscript, we have focused on the implementation and impact for specialist respiratory trainees and therefore focused on the basic respiratory procedures required for completion of specialist training such as thoracic ultrasound, pleural aspiration, Seldinger chest drain insertion and basic diagnostic bronchoscopy.

Evaluating the impact of the PP: survey data

Within the same survey in August 2020, the same 26 trainees were asked to compare how consistently they recorded performing pulmonary procedures pre and post implementation of the PP using a five-point Likert scale for both time points (very consistently, consistently, somewhere in the middle, inconsistently, very inconsistently). They were also asked which specific functionality features of the passport they used (table 1). At the same time an additional cohort of 33 consultant chest physicians (also identified through a regional call for volunteers) were asked to complete an online survey that sought to evaluate the impact of the PP since implementation from a supervisor perspective. Consultants were asked how frequently they provided formal confirmation or written feedback of procedure supervision pre and post implementation of the PP also using a five-point Likert scale (very frequently, frequently, somewhere in the middle, infrequently, very infrequently) as well as which specific functionality features they used. Both trainees and supervisors were asked about future developments of the PP to enhance the user experience and effectiveness of the service.

Data management/security

All individual logs input to the PP are stored within encrypted PDF format and a secure web server (Microsoft Azure). No personal identifying data is recorded within the passport. Anonymised data can be analysed by a hospital or regional level but not at individual trainee’s level. This dataset is only accessible to the project leads (ME and SG).

There was no specific Patient & Public Involvement (PPI) required for this study centred on evaluating a medical education and training platform.

Results

Baseline trainee data

All 26 trainees completed the pre-implementation online survey questions. There was broad representation across the 5 years of the training programme (year 1, n=8 (30%), year 2, n=6 (23%), year 3, n=5 (19%), year 4, n=8 (30%), year 5, n=3 (12%) and one trainee was ‘Out of Programme’ undertaking research). The proportion of trainees who kept a procedure specific logbook prior to the PP were as follows: thoracic ultrasound 18/26 (69%), diagnostic pleural aspiration 17/26 (65%), therapeutic pleural aspiration 17/26 (65%), Seldinger chest drain 18/26 (69%), blunt dissection chest drain insertion 10/26 (39%), indwelling pleural catheter insertion 15/26 (58%), indwelling pleural catheter removal 5/26 (19%), local anaesthetic thoracoscopy 11/26 (42%), basic diagnostic bronchoscopy 16/26 (62%) and endobronchial ultrasound (EBUS) 16/26 (62%). Prior to the PP 20/26 (77%) trainees used a bespoke database (eg, Microsoft excel), 2/26 (8%) used paper records and 4/26 (15%) did not keep any form of logbook. The proportion of trainees that recorded specific pre-defined data fields within their own logbooks in basic diagnostic bronchoscopy, Seldinger chest drain insertion and thoracic ultrasound are presented in figure 2. Prior to the PP 11/26 (42%) of trainees captured the pathological results of sampling procedures during basic diagnostic bronchoscopy, 17/26 (65%) and 7/26 (27%) recorded immediate and late complications of Seldinger chest drain insertion, respectively, and 9/26 (35%) recorded whether a referral to a more expert sonographer was required for thoracic ultrasound. The proportion of trainees who have been able to present specific performance metrics during their appraisal, prior to the PP, was: referral rate to a more expert sonographer for thoracic ultrasound 0/26 (0%), uses of thoracic ultrasound to guide pleural effusion procedures 9/26 (35%), success rate of ultrasound guided pleural procedures 8/26 (31%), iatrogenic pneumothorax/haemothorax rate for pleural procedures 4/26 (15%), chest drain displacement rate 2/26 (8%), average number of biopsies taken at bronchoscopy when definite tumour seen 8/26 (31%), sensitivity of endobronchial biopsies when definite tumour seen 9/26 (35%) and major complication rate for bronchoscopy 6/26 (23%).

Figure 2
Request permissionrequest permission icon
Figure 2

Proportion of trainees that record specific predefined data fields within procedural logbooks for (A) basic diagnostic bronchoscopy, (B) seldinger chest drain insertion, (C) thoracic ultrasound.

PP uptake

From August 2017 to August 2019, 69/73 (95%) eligible respiratory specialist trainees from the North West and Mersey deaneries registered on the PP system. In the same time frame, 162 chest consultant supervisors registered with the PP. A total of 7352 procedures were logged in this time frame (an average of 107 procedures per trainee). The most common procedures were thoracic ultrasound (n=3105), basic diagnostic bronchoscopy (n=1909) and EBUS (n=728, figure 3). Uptake of outcome logs was lower. For Seldinger chest drain insertion 48/521 (9%) had an outcome log completed to document any late complications. There were 221 basic diagnostic bronchoscopies in which endobronchial biopsies in the presence of definite tumour were performed (12% of all bronchoscopies). The histological results of these biopsies were completed within outcome logs in 63% (139/221) of these cases. In the 1909 diagnostic bronchoscopies recorded in the passport system, 86% (1643/1909) were under either full (direct supervision of the entire procedure at the trainee’s side, 26%, 495/1909) or limited supervision (supervisor present in the room not necessarily at trainee’s side, 61%, 1148/1909), 12% (225/1909) were under distant supervision (supervisor not present in the room but immediately available to attend if required) and 1% (19/1909) were unsupervised.

Figure 3
Request permissionrequest permission icon
Figure 3

Number of procedures logged by type between August 2017 and August 2019.

Depth of procedural data captured

A total of 3105 thoracic ultrasounds were recorded which identified 2145 pleural effusions (60% of scans), 401 cases of consolidated lung (13% of scans) and 222 cases of pleural thickening (7% of scans, table 2). A total of 1253 (40% of scans) ultrasound scans were used to guide a subsequent (immediate) pleural procedure. A total of 521 chest drain insertions using the Seldinger technique were recorded. The drains were inserted for pleural effusions in 373 cases (72%) and for a pneumothorax in 148 cases (28%) with 10 cases (2%) labelled as a hydropneumothorax. In the 1909 recorded basic diagnostic bronchoscopy procedures the most common diagnostic techniques used were bronchial wash (n=1236, 65%), bronchial brush (328, 17%) and endobronchial biopsy in the presence of visible tumour (n=221, 12%, table 3).

Table 2
|
Thoracic ultrasound findings and outcomes recorded in the pulmonary passport
Table 3
|
Immediate and late complication rates in respiratory trainees performing Seldinger chest drain insertion

Performance and quality assurance

In thoracic ultrasound procedures 4% of scans (122/2891, table 1) were referred to more expert sonographer for interpretation and learning. In the 1253 thoracic ultrasounds that resulted in and guided a pleural procedure, 96% (1205/1253) were successful procedures. The immediate complication rate for Seldinger chest drain insertion was 7% (36/521) with the most common immediate complication being failure of insertion at first pass (4%, 21/521). Iatrogenic pneumothorax and iatrogenic bleeding both occurred in <1% of cases (table 2). Only 48 ‘outcome logs’ were recorded for Seldinger chest drain insertion. Of these, 10% (5/48) recorded the drain falling out prior to formal clinical decision to remove. No critical incidents were recorded as either an immediate or late complication. There were 495 diagnostic pleural aspirations recorded and 99% (491/495) were performed using ultrasound guidance. The most common immediate complications were failure of the procedure (4%, 21/495) and iatrogenic bleeding (1%, 5/495). No episodes of iatrogenic pneumothorax and no critical incidents were recorded. There were 443 therapeutic pleural fluid aspirations recorded and 98% (435/443) were performed using ultrasound guidance. The most common immediate complications were failure of the procedure (2%, 8/443), iatrogenic bleeding (0.5%, 3/435) and iatrogenic pneumothorax (0.5%, 3/435). No critical incidents were recorded. In basic diagnostic bronchoscopy procedures, when definite endobronchial tumour was visually identified, the average number of biopsies taken was three. Five or more biopsies were taken in 33% of procedures. The sensitivity for providing tumour within endobronchial biopsies across all trainees was 74%. Comparison between trusts revealed large variability in both the number of recorded biopsy outcomes and the sensitivity for tumour identification (figure 4). Transbronchial lung biopsy (TBLB) was performed in only 2% (35/1909) of procedures. Outcome data were complete in 100% of cases (35/35) and revealed that lung parenchyma was present in 54% (19/35) of biopsies. Only one trainee recorded their outcome for bronchoalveolar lavage so assessment of performance using the proportion of epithelial/other cells was not possible. The most common complications of basic diagnostic bronchoscopy were unexpected hospitalisation (0.6%), severe bleeding (0.3%) and death (0.2%, table 4).

Figure 4
Request permissionrequest permission icon
Figure 4

Number of endobronchial biopsy outcomes in cases of definitive tumour across individual trusts.

Table 4
|
Supervision levels, sampling techniques and complication rates in respiratory trainees performing basic diagnostic bronchoscopy

Evaluating the impact of the PP: survey data

All 26 trainees completed the postimplementation online survey questions. Of 26, 23 (89%) trainees used the PP for their procedural logbooks. The proportion of trainees that reported recording procedures ‘very consistently’ increased from 31% to 50% following the implementation of the PP (figure 5). The proportion of trainees that use specific functions in the PP are as follows: mobile adjusted version 9/26 (35%), procedural logs 23/26 (89%), outcome forms 12/26 (46%), progress checker 5/26 (19%), advanced reports 13/26 (50%), competency certificates 8/26 (31%) and training resources 3/26 (12%).

Figure 5
Request permissionrequest permission icon
Figure 5

(A) Consistency of procedural logging and (B) frequency of providing confirmation of supervision or formal written feedback by consultant chest physicians preimplementation and postimplementation of the pulmonary passport (PP) using a five-point Likert scale.

All 33 consultant chest physicians completed the online survey. Of 33, 24 (73%) were registered supervisors on the PP. The proportion of consultants that reported the frequency with which they provided formal confirmation of procedure supervision or formal written feedback as ‘very frequently’ or ‘frequently’ increased from 21% to 33% pre and post implementation of the PP (figure 5). Of 33, 24 (73%) consultants are educational supervisors and 11/24 (45%) use the PP as part of their appraisal meetings. Of 24, 11 (45%) if educational supervisors use the advanced reports and certificates to facilitate curriculum sign off and 7/24 (29%) use the progress checker as part of the appraisal process.

Discussion

Key findings

We report here the implementation of a novel web-based application which supports the standardised logging of respiratory procedures and summarises key clinical outcomes such as sensitivity of biopsy samples and rates of complications (an example of an advanced report produced by the PP is provided in figure 6). Prior to the PP we estimate, based on survey data, approximately two-thirds of trainees kept dedicated logbooks for basic respiratory procedures for which competency is required for completion of specialist training. This increased to 95% of trainees following implementation of the PP and the depth of data collection was immediately enhanced and standardised for all trainees via the use of this platform. For example, 93% (2891/3105) thoracic ultrasound procedures recorded whether referral to a more expert sonographer was required whereas only an estimated 35% of trainees recorded this metric prior to the PP. One hundred pr cent (521/521) of Seldinger chest drain procedures had immediate complications recorded in the PP whereas only an estimated 65% of trainees recorded this metric prior to the PP. Sixty-three per cent (139/221) of bronchoscopic biopsy procedures in cases of visible tumour had pathology results recorded in the PP whereas an estimated 42% of trainees recorded diagnostic outcomes from bronchoscopy prior to the PP. All trainees registered with the PP have access to advanced reports which analyse individual logbooks and present key performance metrics. Prior to the PP between 0% and 35% of trainees were able to report these performance metrics during an appraisal. Following the implementation of the PP 95% of trainees have access to this function, an estimated 50% of trainees use this functionality regularly and an estimated 50% of supervisors use these reports within appraisals. Preimplementation and postimplementation surveys confirm increased consistency of recording procedures and increased frequency of feedback from supervisors. It is reasonable to conclude from this evaluation study, therefore, that the PP has improved the process of procedural logging, supervision and appraisal. Beyond the benefits to individual trainees the data submitted to the PP can effectively support the quality assurance of procedural training at a hospital and deanery level. It allows inspection of the training opportunities at individual trusts, outcome measures stratified by individual trusts and across the region. This can also highlight areas of good practice and quality attainment but also highlight areas warranting further exploration if any concerns are raised. For example, rates of TBLB (0.9%) and conventional transbronchial needle aspiration (cTBNA, 0.6%) are extremely low. This supports a recent national survey which found that a significant proportion of senior trainees did not feel competent in TBLB.3 Additionally, our outcome recordings showed that in TBLB, lung parenchyma was only present in 56%, indicating poor quality. These results question whether it is appropriate to include TBLB, cTBNA within the core skills and competency framework of a general bronchoscopist in training. The combined data from across the deaneries also helps to define appropriate performance standards based on large scale real-life outcomes across the trainees. These can be combined with existing national standards to produce a portfolio of performance metrics that could act as a framework for appraisal of a trainee’s logbook. A proposal for such a framework is provided in table 5.

Figure 6
Request permissionrequest permission icon
Figure 6

An example of an advanced report for bronchoscopy, summarising a trainee’s log book and calculating key clinical outcomes. TNBA, transbronchial needle aspiration.

Table 5
|
Proposals for quality standards in procedural training in respiratory medicine

Future applications

This web-based system has a number of potential wider applications. The British Thoracic Society (BTS) has recently produced a Thoracic Ultrasound Training Standard.16 This new training pathway aims to create a new pool of thoracic ultrasound operators that can provide thoracic ultrasound in an emergency scenario, for example, out of hours large pleural effusion requiring intervention. This training pathway may be relevant to a broad range of healthcare and specialties that might provide out of hours thoracic ultrasound, for example, acute medicine, general internal medicine, critical care and critical care outreach teams. The training pathway is structured using ‘capabilities in practice’ (CiP), a graded approach to the confirmation of competency.17 An electronic logbook, such as the PP, could facilitate the monitoring and progress of trainees through a thoracic ultrasound CiP. Pleural specialist nurses are an increasingly important part of pleural services but require dedicated pleural training. However, a standardised training framework for pleural specialist nurses does not exist. The BTS training standard for thoracic ultrasound may help address some of this but not for all aspects of pleural training. The PP could facilitate robust and systematic recording of pleural procedures within a training programme for specialist nurses and allow the same quality assurance it affords to respiratory trainees. Finally, the PP could be expanded to include a greater breath of procedures both medical and surgical. The basic template of electronic logging and performance monitoring has been proven to be feasible and acceptable. The addition of further procedures ranging from basic medical procedures in medical school and junior doctor training (eg, venepuncture, cannulation) through to more advanced medical procedures (central line insertion, ascitic drains, lumbar punctures) and complex surgical procedures could all be added to the passport system and provide the same benefits to its users.

Limitations

While the completion of procedural logs has been excellent, the uptake of enhanced functionality has been lower than desired. It is within these enhanced functions (outcome logs, certification, advanced reports) that the true value of the system is to be gained, particularly in the process of appraisal and ARCP panels. These functions represent a potential vehicle to the standardisation of procedural assessment during appraisals and a focus for this project in the future must be to increase the uptake of this functionality. One potential method to achieve this would be to increase the supervisor engagement in the PP and one mechanism for this could be the ability of consultants to log their own procedures in the PP, with the benefits of advanced reports for the purpose of their own appraisal. This was requested frequently by supervisors completing the postimplementation survey in the free-text section for suggested improvements to the PP. While the implementation of a new method for logging respiratory procedures has been proven feasible within two adjacent deaneries this does not confirm widespread adoption would occur at a wider scale. Large-scale implementation would require a clearly defined funding model that would secure ongoing technology support, hosting costs, enhancements and long-term sustainability.

Conclusions

The PP is a novel solution to standardise logging of respiratory procedures and supports procedural supervision and training. It synergises well with the JRCPTB NHS ePortfolio which is designed primarily for the purposes of competency assessment rather than logging. This tool has the potential to drive standardisation of procedural monitoring, audit and performance review for both trainees and consultants within respiratory medicine and in other specialties. National adoption, development and expansion could be explored.

Article metrics
Altmetricopen-url
Dimensionsopen-url