Positive predictive value of stroke identification by ambulance clinicians in North East England: a service evaluation

Introduction/background Accurate prehospital identification of patients who had an acute stroke enables rapid conveyance to specialist units for time-dependent treatments such as thrombolysis and thrombectomy. Misidentification leads to patients who had a ‘stroke mimic’ (SM) being inappropriately triaged to specialist units. We evaluated the positive predictive value (PPV) of prehospital stroke identification by ambulance clinicians in the North East of England. Methods This service evaluation linked routinely collected records from a UK regional ambulance service identifying adults with any clinical impression of suspected stroke to diagnostic data from four National Health Service hospital trusts between 1 June 2013 and 31 May 2016. The reference standard for a confirmed stroke diagnosis was inclusion in Sentinel Stroke National Audit Programme data or a hospital diagnosis of stroke or transient ischaemic attack in Hospital Episode Statistics. PPV was calculated as a measure of diagnostic accuracy. Results Ambulance clinicians in North East England identified 5645 patients who had a suspected stroke (mean age 73.2 years, 48% male). At least one Face Arm Speech Test (FAST) symptom was documented for 93% of patients who had a suspected stroke but a positive FAST was only documented for 51%. Stroke, or transient ischaemic attack, was the final diagnosis for 3483 (62%) patients. SM (false positives) accounted for 38% of suspected strokes identified by ambulance clinicians and included a wide range of non-stroke diagnoses including infections, seizures and migraine. Discussion In this large multisite data set, identification of patients who had a stroke by ambulance clinicians had a PPV rate of 62% (95% CI 61 to 63). Most patients who had a suspected stroke had at least one FAST symptom, but failure to document a complete test was common. Training for stroke identification and SM rates need to be considered when planning service provision and capacity.


AbsTrACT
Introduction/background Accurate prehospital identification of patients who had an acute stroke enables rapid conveyance to specialist units for time-dependent treatments such as thrombolysis and thrombectomy. Misidentification leads to patients who had a 'stroke mimic' (SM) being inappropriately triaged to specialist units. We evaluated the positive predictive value (PPV) of prehospital stroke identification by ambulance clinicians in the North East of England. Methods This service evaluation linked routinely collected records from a UK regional ambulance service identifying adults with any clinical impression of suspected stroke to diagnostic data from four National Health Service hospital trusts between 1 June 2013 and 31 May 2016. The reference standard for a confirmed stroke diagnosis was inclusion in Sentinel Stroke National Audit Programme data or a hospital diagnosis of stroke or transient ischaemic attack in Hospital Episode Statistics. PPV was calculated as a measure of diagnostic accuracy. results Ambulance clinicians in North East England identified 5645 patients who had a suspected stroke (mean age 73.2 years, 48% male). At least one Face Arm Speech Test (FAST) symptom was documented for 93% of patients who had a suspected stroke but a positive FAST was only documented for 51%. Stroke, or transient ischaemic attack, was the final diagnosis for 3483 (62%) patients. SM (false positives) accounted for 38% of suspected strokes identified by ambulance clinicians and included a wide range of non-stroke diagnoses including infections, seizures and migraine. Discussion In this large multisite data set, identification of patients who had a stroke by ambulance clinicians had a PPV rate of 62% (95% CI 61 to 63). Most patients who had a suspected stroke had at least one FAST symptom, but failure to document a complete test was common. Training for stroke identification and SM rates need to be considered when planning service provision and capacity.

InTrODuCTIOn
Approximately two-thirds of patients who had an acute stroke in England are conveyed to hospital by emergency ambulance. 1 Prehospital identification enables earlier access to centralised hyperacute stroke units (HASU) 2 which deliver specialist stroke care and time-critical reperfusion treatments (thrombolysis and thrombectomy). The accuracy of prehospital stroke identification depends on the tools used and the population in question. In the UK, the National Clinical Guidelines encourage ambulance clinicians to use a validated screening tool (eg, Face Arm Speech Test, FAST 3 ) to recognise different possible combinations of symptoms across a broad range of patients. 4 5 The FAST is a simple test looking for facial droop, arm weakness or slurred speech as common symptoms of stroke. It remains the test most commonly used by UK ambulance services 6 due to good sensitivity for anterior circulation stroke (79%-97%), but the specificity is lower (13%-88%). 7 As most symptom checklists do not include symptoms of posterior stroke, such as vertigo or visual deficits, guidelines also recommend that practitioners apply their clinical judgement which is likely to further reduce specificity.
Prioritisation of sensitivity (stroke detection) over specificity (avoiding other conditions being identified as stroke) means that a large number of patients who had a suspected stroke identified by ambulance clinicians receive a final diagnosis other than stroke, that is, stroke mimics (SM). The positive predictive value (PPV) of popular prehospital stroke identification tools ranges from 40% to 94% 7 with an average of 27% of suspected strokes identified in prehospital care subsequently diagnosed as SM. 8 Due to the increasing centralisation of emergency stroke care, including the provision of thrombectomy, patients with SM conditions are becoming an increasingly important consideration as they are likely to be displaced from more appropriate local care in significant numbers and impact on efficient use of specialist resources.
In view of the broad range of prehospital stroke identification performance within the literature and the implications of SM presentations for planning service reconfiguration, the primary objective of this study was to describe the current accuracy of stroke identification by clinicians working for a regional ambulance service in North East England using PPV as a measure of diagnostic accuracy.
The secondary objectives were to describe: (1) FAST documentation by ambulance clinicians and how this impacted on the accuracy of stroke identification; and (2) the proportion and type of 'false positive' SM conditions.

MeThODs study design
A retrospective service evaluation linked routinely recorded ambulance and hospital data sets in order to calculate the PPV of stroke identification by ambulance clinicians.

study setting
The North East Ambulance Service NHS Foundation Trust (NEAS) is the regional ambulance provider for around 2.5 million people in North East England covering Northumberland, Tyne and Wear, County Durham, Darlington and Teesside.
The study included patients who had a suspected stroke conveyed to four hospital trusts with HASUs: County Durham and Darlington NHS Foundation Trust (691 stroke admissions/ year 9 ); Northumbria Healthcare NHS Foundation Trust (1008 stroke admissions/year 9 ); North Tees and Hartlepool Hospitals NHS Foundation Trust (576 stroke admissions/year 9 ); and Newcastle Hospitals NHS Foundation Trust (748 stroke admissions/year 9 ). All figures are for April 2016 to March 2017.

Inclusion/exclusion criteria
A consecutive series of patients across 3 years (1 June 2013 to 31 May 2016) were identified by an electronic search for a documented NEAS clinician impression of suspected stroke (the index test) for any reason (FAST positive or FAST negative/ not recorded). Internal NEAS data show that impression stroke accounts for around 4% of cases where an ambulance attends a patient. Other inclusion criteria were: adult (aged ≥18 years) patients; GCS ≥8 (reflecting the difficulty to assess patients with a low GCS and the population used to develop FAST 3 ); and conveyed to a relevant hospital. Records made by qualified paramedics of all grades and technician personnel were included.
The exclusion criteria were interhospital transfers, GCS <8 and admission other than by emergency ambulance.

Prehospital data
NEAS clinicians routinely record patient details using a portable electronic patient record form (EPRF). A 'clinical impression', selected from predetermined categories including stroke, is recorded at the end of each patient encounter. Clinical impression could include multiple differential diagnoses so stroke may not have been the only impression. Other data on the EPRF are recorded in two formats: (1) structured data on predetermined variables including demographics, common signs and symptoms, physiological observations as well as standard assessments such as FAST, and (2) a free text section recording narrative aspects of the patient encounter including symptoms. These were extracted from the EPRF for all patients who met the inclusion criteria by a combination of automated and manual data extraction according to the nature of the data field. A complete picture of the prehospital data on patients who had a suspected stroke was created by combining any recording of variables of interest in either the structured or free text sections. The results of the data extraction are described in table 1. These data included patients identified by paramedics and non-paramedic clinicians. NEAS clinicians follow the UK National Ambulance guidelines 4 for stroke identification which recommends patients with stroke symptoms, identified using FAST or clinical judgement, starting within the previous 5 hours, are transported directly to the nearest HASU.

hospital diagnosis data
Stroke or SM diagnoses were established according to (1) whether each patient was included in the admitting hospital's Sentinel Stroke National Audit Programme 10 (SSNAP) data set, and (2) primary discharge diagnoses in Hospital Episode Statistics (HES) according to the International Statistical Classification of Diseases and Related Health Problems (ICD-10) codes. SSNAP is a mandatory individual patient audit that measures and regularly reports the quality of stroke care in the National Health Service (NHS) in England, Wales and Northern Ireland using clinical and organisational measures. All patients in SSNAP have a specialist confirmed diagnosis of stroke that is cross-referenced with national HES data through the Office for National Statistics. All participating trusts had high (≥90%) case ascertainment rates for SSNAP, that is, <10% of patients who had an acute stroke treated as inpatients were not listed on the SSNAP database. The reference standard to confirm a final diagnosis of stroke was inclusion in SSNAP, or a local HES discharge diagnosis including ICD-10 codes of I61, I63 and I64 if patients' records could not be confidently matched with SSNAP. Patients with an HES diagnosis including ICD-10 codes G458 or G459 were recorded as transient ischaemic attacks (TIA). TIAs were grouped with patients who had a stroke on the basis that prehospital triage to specialist stroke units would still be appropriate, similar to other prehospital stroke studies. 11 All other diagnoses were recorded as SM (false positives). Where a diagnosis could not be established from SSNAP or HES data it was assumed that the patient had an SM condition.
Ambulance cases were linked to hospital data using a stepwise approach. First, an attempt was made to link NEAS patients who had a suspected stroke with SSNAP data using the ambulance service case number. Where the case number was not recorded, probabilistic matching was used based on admitting hospital, date/time of admission, gender and age. Patients with a potential match (eg, admission time >20 min difference, identical gender but missing age) were re-examined with access to the original NEAS EPRF to identify additional information which might assist with matching such as location of the incident. Discharge diagnoses for patients who could not be linked with SSNAP data were sought from HES using common identifiers within the NEAS data (name, age, date of birth, NHS number, date/time of admission).

Data analysis
All data were analysed in IBM SPSS Statistics V.23. The PPV of ambulance clinician identification of stroke was calculated based on the linked data. The sensitivity of ambulance clinician stroke

Patient and public involvement
There was no patient and public involvement.

resulTs
The study included 5645 'impression stroke' cases transported to the four HASUs identified from the NEAS EPRFs. The results of the data linking process are summarised in figure 1. Half of the patients who had a suspected stroke were confirmed as definite stroke by direct linkage with SSNAP data (n=2828). A further 335 patients were not included in the local SSNAP data set but were confirmed as stroke based on HES data. One thousand and four hundred four of the remaining 2162 patients were linked with HES data and had an SM diagnosis. In 758 patients no positive match could be made with either SSNAP or HES, and were classed as SM. In total, 3163 (56%) patients had a final diagnosis of stroke, 320 (6%) patients were TIA and 2162 (38%) patients were SM.
The mean age of all patients was 73.2 years (SD 14.4) and 48% of patients were male. A formal FAST result was documented in the structured data for 2877 (51%) patients but one or more FAST symptoms were documented for 5244 (93%) overall. Table 1 shows the characteristics of patients extracted from ambulance records according to stroke/TIA and SM categorisation.
Ambulance clinician documentation of 'impression stroke' identified 3483 confirmed cases of acute stroke/TIA out of 5645 total patients, that is, a PPV of 62% (95% CI 61 to 63). Patients with a final stroke diagnosis had higher rates of FAST documentation than patients who had an SM (54% vs 46% in structured data, p<0.001; 96% vs 91% for all documented FAST symptoms, p<0.001). A sensitivity analysis of the differing sources of FAST documentation showed that presence of a structured FAST-positive record had a PPV of 66% (95% CI 65 to 66); any FAST positive (structured FAST test or narrative FAST symptoms) 63% (95% CI 63 to 63); and no FAST symptom documentation 47% (95% CI 44 to 49).
These data were used to estimate the sensitivity of ambulance clinician recognition of stroke. SSNAP data from the four  participating acute trusts included 8538 patients who had a stroke over the study time frame. These records were filtered to include only those conveyed by NEAS, which equalled 6424 (75%) patients who had a suspected stroke. As the total number of NEAS patients who had a suspected stroke subsequently confirmed as correct was 3163 (not including TIA) then the sensitivity of stroke identification based on ambulance clinician impression was 49% (95% CI 48 to 50). When only patients with documented FAST symptoms were included the sensitivity decreased to 47% (95% CI 45 to 48). There were 299 different ICD-10 codes recorded for patients with an SM diagnosis. The 10 most frequently recorded SM ICD-10 codes accounted for 33% of reported SM diagnoses (table 2).

DIsCussIOn
In a large data set linking regional ambulance service data with patient diagnoses from four hospital trusts in North East England, identification of patients who had a stroke by ambulance clinicians had a PPV of 62%. FAST-positive patients were more likely to have a final stroke diagnosis than FASTnegative patients. SM (false positives) accounted for 38% of suspected stroke admissions identified by ambulance clinicians and included a wide range of diagnoses. These real-world performance data have implications for the efficiency of service reconfiguration towards a smaller number of larger HASUs and future provision of mechanical thrombectomy.
Forty factors were identified with statistically (p<0.05) different associations between patients who had a stroke and SM, but it may not be possible to use these to improve the specificity of prehospital stroke identification without increasing the risk that patients who had a genuine stroke do not rapidly access specialist care. Compared with stroke/TIA, patients who had an SM were more likely to be younger and female and less likely to have a history of hypertension. However, the absolute differences between the clinical characteristics of the stroke and SM populations were small. Many of the factors described have been reported by previous studies seeking to identify SM based on analyses of hospital data, 12 13 including: younger age; absence of atrial fibrillation; absence of facial droop; and absence of historical hypertension. Seizures were still evident among the SM group despite being a clinical exclusion from many prehospital stroke pathways, including the NEAS pathway. 14 This may indicate that some types of seizure activity and postictal states are complex presentations to identify in the prehospital setting. Clinical pathway clarification and additional training may be beneficial for these presentations.
Individual FAST symptoms were all significantly (p<0.001) associated with a final diagnosis of stroke, but were also recorded for large numbers of patients who had an SM. The FAST was inconsistently documented by ambulance clinicians with 51% of patients formally designated as FAST positive, whereas 93% of patients had one or more FAST symptoms recorded.
The PPV of 62% calculated in the current study is at the lower end of the range described for prehospital FAST 7 use. Other prehospital services using FAST have reported similar PPV rates, with a PPV of 57% reported in FAST-positive ambulance prealerts in Ireland 15 and a PPV of 68% for stroke/TIA

Original research
prealerts reported in Scotland. 16 The inclusion of TIA and FASTnegative patients might be an explanatory factor for the lower PPV observed, however this appears to increase the sensitivity of prehospital stroke identification. Modifications to FAST such as Balance, Eyes, Face, Arm, Speech, Time 17 have been suggested to better identify patients currently missed by FAST by seeking posterior circulation symptoms such as balance and visual disturbances, but these have been developed from retrospective examination of hospital records and prospective impact during prehospital assessment has not been evaluated. Various scores have also been suggested to identify large vessel occlusive strokes suitable for mechanical thrombectomy, but their ability to exclude SM in real-world populations remains uncertain. 18 Previous smaller studies have reported lower SM rates (22%-23%) than the 38% observed but these were in the context of a rapid ambulance diversion protocol in a single urban unit. 3 19 The high SM rate has implications in terms of planning clinical pathways; organisation of specialist care services; transportation times and ambulance resource availability; and repatriation and travel for relatives when SMs are displaced. The common SMs identified in this study (infection, seizures and migraine) are similar to those reported by two systematic reviews of SM 8 20 and suggest opportunities for development of point-of-care diagnostics. 21 22 Mobile stroke units have already demonstrated improvements in the prehospital treatment of stroke but may only be cost-effective and sustainable in dense urban areas. 23 A more generalisable approach to improve prehospital stroke identification would be including content about common SM presentations and FAST-negative strokes in standardised training for ambulance clinicians. In future, patients who have a suspected stroke may benefit from ambulance clinicians being able to remotely access support from stroke specialists. 24 25 The estimated sensitivity of prehospital stroke identification by ambulance clinicians was 49% (95% CI 48 to 50). This is below the reported figures for FAST sensitivity in prehospital care (79%-97%), 7 so may reflect 'real world' performance or differences in training and data collection.
limitations This is a service evaluation, therefore results cannot be generalised. Ambulance personnel could select more than one 'impression' per patient without indicating the most likely, thereby leading to inclusion of patients where stroke may not have been considered the main problem. Incorporation and workup biases may be present as the hospital response is influenced by the prehospital identification, or lack of identification, of stroke so the index test is not independent of the prehospital actions. Hospital coding is imperfect and the use of single ICD-10-based primary diagnoses does not represent the multiple conditions which some patients present with, but it is assumed that acute stroke would be the primary diagnosis if present. The hospital diagnoses used reflect diagnoses based on specialist input documented at discharge which may have been made after a prolonged admission, so are based on access to more data than were available to the ambulance clinicians. The hospital data set may have included small numbers of inpatient stroke which cannot be distinguished through HES. Narrative data were used, as well as structured clinical data, to describe factors recorded by ambulance clinicians, which have limitations due to the wide variability in documentation. These data made the results more representative of clinical practice, but introduced an element of interpretation. The probabilistic record matching process was also a limitation due to inconsistencies such as misspelling of names, missing data and differences in formatting between prehospital and hospital data sets. Patient diagnoses were established for the majority of patients but the assumption that the 13% of patients unmatched with SSNAP data (and without a confirmed diagnosis in HES) were SM may have been incorrect and led to underestimating the PPV. While PPV is the main measure reported in this study sensitivity has been estimated as well, more robust data on the true sensitivity of prehospital stroke identification would be valuable as this could inform improvements to identification which is a key role of the ambulance services.
In summary, the PPV of prehospital stroke identification by ambulance clinicians in the North East England was 62%, which is below the original performance from FAST validation studies but similar to other prehospital settings. 17 18 SMs continue to make up a sizeable proportion of patients who had a suspected stroke identified by ambulance clinicians. Further training and technological innovation are needed to improve prehospital stroke stratification if services are to achieve optimal efficiency in patient flow and resource utilisation.