Andrew Wing , Barbara Hill-Taylor , Ingrid Sketris , Jeanne Smith , Sam Stewart , Katrina F HurleyABSTRACT
Background:
Research exploring the agreement between traditional medication records and electronic records generated by an automated dispensing device has been limited.
Objective:
To evaluate the extent of agreement between medication administration records written in paper-based emergency department charts and records generated by an automated dispensing device with regard to the presence or absence of a single, prespecified medication.
Methods:
Medication administration records in paper-based emergency department charts and medication dispensation records generated by an automated dispensing device were evaluated for concordance. The primary outcome measure was agreement between the 2 sources with regard to the presence or absence of a record for salbutamol by metered-dose inhaler (MDI) for randomly selected patients who presented to a pediatric emergency department with wheeze-related illness from January 1, 2008, to December 31, 2009.
Results:
In total, 1172 patient visits met the inclusion criteria. Of these, records for 1013 visits showed agreement between the paper-based emergency department chart and the dispensation record of the automated dispensing device (kappa = 0.71, 95% confidence interval 0.67–0.75). This value did not meet the target kappa of 0.80. Stratification by time of day, day of week, month, season, or year of presentation at triage or by triage level or disposition (whether or not the patient was admitted to the hospital ward) did not significantly affect the level of agreement between the 2 sources.
Conclusions:
Agreement between records of salbutamol MDI administration in paper-based charts and dispensation records from an automated dispensing device was substantial, but discrepancies were present. There are significant quality management, legal, clinical, and research reasons to strive for concordance between multiple records with respect to medication use in the emergency department. Data generated by automated dispensing devices have potential value for research, but their strengths and limitations need to be understood.
KEYWORDS: data source , chart review , medical records , automated dispensing device
RÉSUMÉ
Contexte :
Peu de recherches ont étudié la concordance entre les fiches de médicaments traditionnelles et les registres électroniques générés par une armoire de distribution automatisée.
Objectif :
Évaluer le degré de concordance entre les fiches manuscrites d’administration des médicaments dans les dossiers papier du service des urgences et les registres générés par une armoire de distribution automatisée en ce qui a trait à la présence ou à l’absence d’un seul médicament présélectionné.
Méthodes :
Les fiches d’administration des médicaments dans les dossiers papier du service des urgences et les registres d’administration des médicaments générés par une armoire de distribution automatisée ont été évalués au chapitre de la concordance. Le principal paramètre d’évaluation était la concordance entre les deux sources en ce qui a trait à la présence ou à l’absence d’une mention au sujet du salbutamol en aérosol-doseur chez des patients choisis au hasard qui se sont présentés au service de l’urgence pédiatrique pour des symptômes de respiration sifflante, entre le 1 er janvier 2008 et le 31 décembre 2009.
Résultats :
Au total, 1172 visites de patients ont satisfait aux critères d’inclusion. Lors de 1013 de ces visites, les fiches d’administration des médicaments dans les dossiers papier du service des urgences concordaient avec les registres d’administration des médicaments générés par une armoire de distribution automatisée (kappa = 0,71, intervalle de confiance à 95 % 0,67–0,75). Cette valeur n’a pas atteint le kappa cible de 0,80. La stratification selon l’heure du jour, le jour de la semaine, le mois, la saison ou l’année de présentation au triage ou selon le niveau de triage ou l’issue (si le patient était oui ou non hospitalisé) n’a pas eu d’effet significatif sur le degré de concordance entre les deux sources.
Conclusions :
La concordance entre les mentions de l’administration du salbutamol en aérosol-doseur dans les dossiers papier et les registres générés par une armoire de distribution automatisée était considérable, mais des différences ont été observées. Il y a des motifs importants touchant la gestion de la qualité ainsi que les aspects juridiques, cliniques et de recherche qui justifient la concordance de multiples registres servant à consigner l’utilisation des médicaments dans le service de l’urgence. Les données générées par les armoires de distribution automatisée ont une valeur potentielle en recherche, mais leur force et leur faiblesse doivent être comprises.
MOTS CLÉS: source de données , analyse de dossiers médicaux , dossiers médicaux , armoire de distribution automatisée
[Traduction par l’éditeur]
Many hospital departments rely on automated dispensing devices (ADDs) to improve quality of patient care and inventory control.1–3 In a 2007 survey, 82.8% of general and children’s medicosurgical hospitals in the United States reported using ADDs.2 In a 2009/2010 survey of hospital pharmacies in Canada, 54% of respondents reported using ADDs, up from 36% in 2007/2008.3 The Canadian survey also documented that 94% of respondents who were using ADDs employed the devices for medication control in the emergency department, which made the emergency department the most common site for the deployment of an ADD in Canadian hospitals.3
In 2007, an ADD (Pyxis Medstation, Carefusion, San Diego, California) was introduced to the IWK Health Centre in Halifax, Nova Scotia. In the emergency department of this hospital, the ADD allows medical, nursing, and pharmacy staff on-site access to unit-dose and multidose medications stored in drawers within a cabinet. Authorized staff follow a login process employing biometric or password control (with both forms of control in use at the study hospital) before selecting patients’ required medications, as recorded in the paper-based emergency department charts. The machine then permits access to the appropriate medication drawer and generates an electronic medication dispensation record, which facilitates real-time tracking of dispensed medications and ward inventory. At the time of publication, in mid-2012, the ADD was not being used to generate patient-specific medication administration records.
Quality care and drug utilization researchers in emergency care frequently review medical records to obtain secondary data.4–9 Traditional review of medical records, considered the “gold standard” in retrospective patient-related research, is reportedly used in as many as 25% of studies in emergency care.9 However, records in patients’ charts are subject to errors, omissions, and conflicting data.7 Traditional medical record review is also subject to abstraction and interpretation difficulties, and is time-consuming and costly.7
ADDs have the potential to become expedient sources of secondary data. However, research exploring agreement between traditional medical records and ADD dispensation records has been limited. A review of 188 randomly selected ADD transactions in a US tertiary care hospital found a 6.9% discrepancy rate between patients’ medication administration records and the device’s records.10 An attempt to reconcile dispensation records for controlled substances generated by a Pyxis Medstation system with records in an anesthesia information management system used at the case level revealed a 15% discrepancy rate, and significant data-entry errors were noted in both systems.11
Evaluating the extent of agreement between data abstracted from paper-based emergency department charts (i.e., traditional medical record review) and data obtained from medication dispensation records generated by an ADD is important in assessing the value of ADD records as a data source for secondary use. This study was undertaken to evaluate the agreement between these 2 sources of data.
In this retrospective study, paper-based medication administration records (written by nursing staff) for a random selection of pediatric patients with wheeze-related illnesses who presented to the emergency department were reviewed and compared with the results of an audit of medication dispensation records generated by the hospital’s ADD for the same visits. The primary outcome measure was the level and significance of agreement between the 2 data sources with regard to the presence of documentation of salbutamol metered-dose inhaler (MDI), evaluated in terms of Cohen’s kappa statistic.12,13 The assessment of kappa values was further stratified by time of day, day of week, month, season, or year of presentation at triage, triage level, and disposition (whether or not the patient was admitted to the hospital ward).
The emergency department of the study hospital used the Canadian Paediatric Triage and Acuity Scale to assess triage level. This validated scale indicates the severity of a patient’s presenting illness on a 5-point scale, where 1 is for the most seriously ill patients (those requiring resuscitation), 2 = emergent, 3 = urgent, 4 = semiurgent, and 5 = non-urgent.14
Ethics approval was obtained from the Research Ethics Board of the IWK Health Centre.
This study took place in August 2010 at the IWK Health Centre, the tertiary care pediatric facility for maritime Canada.
The emergency department of the IWK Health Centre sees about 29 170 emergency patients per year, approximately 2000 of whom present with acute wheeze-related illnesses.15 This study was limited to visits coded (according to the International Statistical Classification of Diseases and Related Health Problems, 10th Revision16) with a diagnosis of asthma (J45), bronchiolitis (J21), other specified respiratory disorders (J98.8), or wheeze (R06.2). These diagnoses were chosen as being most likely to be associated with administration of salbutamol by inhalation. Visits coded for other diagnoses related to respiratory illnesses, such as upper respiratory tract infection, croup, pneumonia, and cough, which are unlikely to be treated with salbutamol, were excluded.
Salbutamol by MDI was chosen for this study because it is commonly used in this patient population and, at the time of this study, was also being evaluated as part of a quality improvement study.17 Each salbutamol MDI is used for a single patient.
A total of 4140 wheeze-related patient visits between January 1, 2008, and December 31, 2009, met the study inclusion criteria, and 1376 of these visits were randomly selected for the study. Random selection was accomplished in an arms-length fashion using a web-based random sampling program (Research Randomizer, www.randomizer.org).
Cantor’s method was used to calculate the sample size required for significance testing with Cohen’s kappa statistic.12,13 A preliminary review of medical records for 152 randomly selected wheeze-related patient visits from 2006 to 2009 was performed to determine the proportion of patients who had been treated with salbutamol by inhalation (by nebulization or MDI). This proportion was approximately 75%.
Studies evaluating the agreement of medication administration records from patients’ charts with administrative electronic databases have generated results ranging from slight agreement (κ = 0.20) to almost-perfect agreement (κ = 0.86).18–20 For this study, a target kappa of 0.8 was chosen, representing the upper limit of “substantial agreement”13,21 (see Table 1 for descriptors used in interpreting kappa values). A sample size of 1020 patients with wheeze-related illness (510 per year) was calculated to ensure that the study could detect a significant improvement in the kappa statistic of at least 0.05 over the target value of 0.8.
Table 1.
Definition of Kappa Values13
In early 2008, just as this study was beginning, the IWK Health Centre changed its practice for administration of salbutamol, from primarily nebulization to almost exclusive use of MDIs with holding chambers.17 The change was not instantaneous, however, and it could not be guaranteed that patients randomly selected for the study would have been treated with salbutamol by MDI. Therefore, the sample size was increased to 680 patient visits for wheeze-related illness per year (i.e., a total of 1360 patient visits), to account for the potentially lower frequency of salbutamol MDI use in early 2008.
All administration records for salbutamol by inhalation were collected from the paper-based emergency department charts and the ADD. Two researchers (A.W. and K.F.H.) used standardized forms to abstract nursing medication administration records from paper-based emergency department charts. ADD dispensation records were supplied by the Department of Pharmacy.
Analysis of ADD records revealed that salbutamol nebules were often withdrawn from the Pyxis Medstation ADD by the box (20 nebules per box). Nursing staff reported that unused nebules were then stored outside of the ADD for administration to subsequent patients. Therefore, it was not possible to analyze the salbutamol nebule data, as they did not reflect individual patient usage. As such, only MDI data were used for the agreement analysis. For this reason, patient visits for wheeze-related illness and salbutamol inhalation treatments that involved only nebulization records were excluded.
To validate the accuracy of the initial abstraction process, one researcher (B.H.-T.) repeated data abstraction for a randomly selected 10% of the medical records. A 10% sample is an accepted proportion for testing inter-rater reliability.7–9
Of the original sample of 1360 visits, a total of 188 involved nebulization and were therefore excluded, which left 1172 patient visits for analysis (515 from 2008 and 657 from 2009). Almost all of the patients whose visits were included in the study had semiurgent, urgent, or emergent status, and most had a diagnosis of asthma or bronchiolitis (Table 2).
Table 2.
Characteristics of Patients Presenting to the Emergency Department (ED) with Wheeze-Related Illness Whose Records Were Reviewed
Of the 1172 visits for patients with wheeze-related illness who received salbutamol by MDI, 1013 records (86.4%) showed agreement between the medication administration record in the paper-based emergency department chart and the dispensation record generated by the ADD (Table 3). Using chart review of the paper-based patient charts as the “gold standard”,8,9 positive agreement was 81.1%, whereas negative agreement was 97.6%. Of the 1172 patient visits sampled, 150 (12.8%) involved treatment with salbutamol by MDI according to the paper-based charts with no ADD dispensation record for the medication; conversely, 9 (0.8%) of the 1172 visits had an ADD dispensation record for salbutamol by MDI but did not have a paper-based medication administration record.
Table 3.
Agreement* Between Paper-Based Emergency Department Charts and Dispensation Records of Automated Dispensing Device Regarding Documentation of Salbutamol Metered-Dose Inhaler
For evaluation of agreement between the paper-based medication administration records and dispensation records generated by the ADD, Cohen’s κ was 0.71 (95% confidence interval 0.67–0.75). Although this level of agreement fell within the range arbitrarily described as “substantial” (see Table 1), the kappa value did not meet the a priori target level (κ ≥ 0.80).
Stratification by time of day, day of week, month, season, or year of the emergency department visit or by triage level or disposition of the patient did not significantly affect agreement.
Testing of inter-rater reliability showed a significant level of agreement in the data abstraction process (κ = 0.97, 95% confidence interval 0.928–1).
In this study, agreement between medication administration records from paper-based emergency department charts and electronic medication dispensation records from the ADD with regard to ordering of salbutamol by MDI for patients with wheeze-related illness was substantial, yet did not meet the target level of κ ≥ 0.80.
Although no attempt was made to determine the specific reasons for individual discrepancies, the literature offers several possibilities. Vigoda and others11 documented more than 30 reasons for discrepancies between 2 types of electronic medication administration records (Pyxis Medstation and an anesthesia information management system), including medications withdrawn from the ADD under the wrong patient name or medications being removed, not administered, or returned to inventory without a correction in the inventory record. The most frequent errors described by these authors were related to recording of waste in the ADD and recording of incorrect dosages in the anesthesia information management system. Sutter and others10 found that the most common reason for discrepancies was drugs being accessed from the ADD without a written physician’s order.
An ADD may create tension between the need of emergency department staff to have expedient access to medications and the constraints of inventory control strategies, especially in such a demanding environment. Some staff members may not understand the value of the ADD for inventory control (for accountability and quality improvement); rather, they may see the device as an obstacle preventing access to medications, which could lead to actions to circumvent device controls.22
Potential individual, system, and practice factors that might have contributed to nonconcordance at the IWK Health Centre Emergency Department were considered (Table 4). This table, adapted from work by Leape and others,23 was created in consultation with emergency and pharmacy department staff and reflects the current situation at the study site, with the hospital’s specific ADD configuration and the ADD record not being a legal part of the patient’s chart.
Table 4.
Selected Factors That Might Have Contributed to Nonconcordance Between Paper-Based Charts and Dispensation Records from Automated Dispensing Device
In urgent situations, staff may gain access to medications in an ADD by overriding the patient identification entry or by accessing medications using another patient’s name.1,24 In the study emergency department, a “John or Jane Doe” entry was established to allow access to medication for unregistered patients, although staff do have the ability to add a patient directly in case of delayed patient registration. Urgency may reasonably lead to the use of the “John or Jane Doe” option. We used the triage level as an indicator of potential urgency to determine if there was a relationship between urgency and lack of agreement, but found no such relationship.
Patients with asthma who are repeat visitors to the emergency department may be encouraged to bring and use their own salbutamol MDI for treatment in the department. Discussions with nursing staff indicate that this does occur, but staff do not always document this action in the patient chart, as has been recommended in some guidelines.25 The practice of using a patient’s own salbutamol MDIs could explain, at least in part, a positive paper-based medication administration record in the chart with no corresponding ADD record (accounting for 150 of the discrepancies in Table 3, for which potential factors are discussed in Table 4).
There are important clinical, quality improvement, legal, and financial reasons to strive for concordance between multiple sources of information about medication administration. At this institution, the ADD is functioning as an inventory control system. The Department of Pharmacy would like to achieve better agreement between paper-based emergency department charts and ADD dispensation records, as a way to reliably calculate the average cost for patient visits (case-costing) and for inventory control. In other health care systems, accurate medication administration records are also needed for private insurance billing and reimbursement. Although the study institution currently has no plans to incorporate the ADD record as a legal record in the patient chart, it would be crucial to achieve better concordance before that could be considered.
Quality improvement and health policy researchers will benefit from the ability to use electronic medication dispensation records generated by an ADD for research purposes. These records eliminate the potential for error associated with data abstraction and translation, and they greatly diminish the time and cost of accessing paper-based medication administration records. In addition, aggregated, de-identified data would be readily available for analysis, to identify areas for quality improvement. For example, a recent study at the same hospital used aggregated ADD data to evaluate a clinical practice intervention.17
As with any review of medical records, there was potential for error in data abstraction and translation in the review of paper-based emergency department charts.6–9 To minimize this potential, a structured abstraction methodology was adopted and, although researchers were not blinded, a standardized abstraction form was developed to make the data collection explicit.7,8
Salbutamol by MDI was chosen for this study in part because it is a common medication used in pediatric emergency departments. However, the MDI is a multidose device, and we were only able to assess the presence or absence of documentation of the MDI in the medication record. More detailed information on the accuracy of medication dosage was not gathered, but it would be important to study such information in future work. In addition, it would be useful to consider other types of medications used in the emergency department setting. An area of particular interest would be medications for which the cause of any error could have greater legal implications (for example, opioids).
Although this analysis focused on records for salbutamol by MDI, the randomly selected visits were chosen on the basis of diagnosis, not medication administration or dispensation records. Therefore, this evaluation would have missed potential discrepancies for patients who did not meet the selection criteria (i.e., without a diagnosis of asthma, bronchiolitis, wheeze, or reactive airway disease) for whom salbutamol MDI administration records were found in the paper-based charts.
Several factors may reduce the ability to generalize these results to other institutions using an ADD. For example, ADDs may be configured in various ways (e.g., cabinet layout, software, and interface options), and this variability may limit generalizability. Furthermore, the experience, training, practice behaviours, and oversight of individuals using the ADD may vary between departments and institutions.
At the study hospital, substantial agreement existed between the medication administration records for salbutamol MDI use as recorded in the nursing medication administration record in the paper-based chart and the dispensation record generated by a newly installed ADD in an emergency department; however, agreement was not perfect, and discrepancies were found.
ADD-generated data offer a number of real advantages for inventory control and for research into drug utilization and quality care. Such data are already being used for these purposes, and investigators must understand the strengths and limitations of this data source.
Further work is needed to determine reasons for discrepancies between the 2 types of medication records, to examine if similar discrepancies occur with other types of medications, and to identify approaches to improving the accuracy of records on the prescribing, dispensation, and administration of medications.
1.
ASHP Section of Pharmacy Informatics and Technology. ASHP guidelines on the safe use of automated dispensing devices.
Am J Health Syst Pharm
2010;67(6):483–490.
2.
Pedersen CA, Gumpper KF. ASHP national survey of informatics: assessment of the adoption and use of pharmacy informatics in U.S. hospitals—2007.
Am J Health Syst Pharm
2008;65(23):2244–2264.
3. Harding J. Drug distribution systems. In: Babich M, Bornstein C, Bussières JF, Hall K, Harding J, Lefebvre P, et al., editors. Hospital pharmacy in Canada 2009/2010 report . Eli Lilly; 2010 [cited 2011 Sep 15]. p. 21–35. Available from: www.lillyhospitalsurvey.ca/HPC2/Content/2010_report/chapter_c%20.pdf
4.
Badcock D, Kelly AM, Kerr D, Reade T. The quality of medical record review studies in the international emergency medicine literature.
Ann Emerg Med
2005;45(4):444–447.
5.
Gilbert EH, Lowenstein SR, Koziol-McLain J, Barta DC, Steiner J. Chart reviews in emergency medicine research: Where are the methods?
Ann Emerg Med
1996;27(3):305–308.
6.
Gorlick MH, Alpern ER, Singh T, Snowdon D, Holubkov R, Dean JM, et al.; Pediatric Emergency Care Applied Research Network. Availability of pediatric emergency visit data from existing data sources.
Acad Emerg Med
2005;12(12):1195–1200.
7.
Kimberlin CL, Winterstein AG. Validity and reliability of measurements instruments used in research.
Am J Health Syst Pharm
2008;65(23): 2276–2284.
8.
Lowenstein SR. Medical record reviews in emergency medicine: the blessing and the curse.
Ann Emerg Med
2005;45(4):452–455.
9.
Worster A, Haines T. Advanced statistics: understanding medical record review (MRR) studies.
Acad Emerg Med
2004;11(2):187–192.
10.
Sutter TL, Wellman GS, Mott DA, Schommer JC, Sherrin TP. Discrepancies with automated drug storage and distribution cabinets.
Am J Health Syst Pharm
1998;55(18):1924–1926.
11.
Vigoda MM, Gencorrelli FJ, Lubarsky DA. Discrepancies in medication entries between anesthetic and pharmacy records using electronic databases.
Anesth Analg
2007;105(4):1061–1065.
12.
Cantor AB. Sample-size calculations for Cohen’s kappa.
Psychol Methods
1996;1(2):150–153.
13.
Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic.
Fam Med
2005;37(5):360–363.
14.
Gravel J, Manzano S, Arsenault M. Validity of the Canadian Paediatric Triage and Acuity Scale in a tertiary care hospital.
CJEM
2009;11(1):23–28.
15. About us [website]. Halifax (NS): IWK Health Centre; © 2006–2009 [cited 2012 Jun 29]. Available from: www.iwk.nshealth.ca
16. International statistical classification of diseases and related health problems: 10th revision . 2nd ed. Geneva (Switzerland): World Health Organization; 2004 [cited 2010 Oct 25]. Available from: www.who.int/classifications/icd/en/
17. Hill-Taylor B, Hurley KF, Sketris I, O’Connell C, Sinclair D, Wing A. Evaluating a clinical practice intervention to promote delivery of salbutamol by metered-dose inhalers with holding chambers in a pediatric emergency department. CJEM . Forthcoming.
18.
Chen G, Faris P, Hemmelgarn B, Walker RL, Quan H. Measuring agreement of administrative data with chart data using prevalence unadjusted and adjusted kappa.
BMC Med Res Methodol
2009;9:5. 
19.
Humphries KH, Rankin JM, Carere RG, Buller CE, Kiely FM, Spinelli JJ. Co-morbidity data in outcomes research: Are clinical data derived from administrative databases a reliable alternative to chart review?
J Clin Epidemiol
2000;53(4):343–349.
20.
So L, Beck CA, Brien S, Kennedy J, Feasby TE, Ghali WA, et al. Chart documentation quality and its relationship to the validity of administrative data discharge records.
Health Informatics J
2010;16(2):101–113.
21.
Landis JR, Koch GG. The measurement of observer agreement for categorical data.
Biometrics
1977;33(1):159–174.
22.
Krishnan RS. Preventing Pyxis wars: without nurses, inventory control impossible.
Hosp Mater Manage
1999;24(6):10–11.
23.
Leape LL, Bates DW, Cullen DJ, Cooper J, Demonaco HJ, Gallivan T, et al.; ADE Prevention Study Group. Systems analysis of adverse drug events.
JAMA
1995;274(1):35–43.
24. Miller K, Shah M, Hitchcock L, Perry A, Englebright J, Perlin J, et al. Evaluation of medication removed from automated dispensing machines using the override function leading to multiple system changes. In: Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in patient safety: new directions and alternative approaches . Vol 4: Technology and medication safety. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008.
25.
Lummis H, Sketris I, Veldhuyzen van Zanten S. Systematic review of the use of patients’ own medications in acute care institutions.
J Clin Pharm Ther
2006;31(6):541–563.
We acknowledge the assistance of the Decision Support department of the IWK Health Centre. Funding for the primary author was supplied by the Dalhousie University Faculty of Medicine Osman Research Studentship. At the time of the study, Ingrid Sketris held a Chair funded by the Canadian Health Services Research Foundation and the Canadian Institutes for Health Research, cosponsored by the Nova Scotia Health Research Foundation. This chair funded a Drug Use and Management Policy Residency and the continuing work of Barbara Hill-Taylor.
Canadian Journal of Hospital Pharmacy , VOLUME 65 , NUMBER 4 , July-August 2012