Ariane Blanc, Vivian Ho, Jameason CameronDOI: https://doi.org/10.4212/cjhp.3281
ABSTRACT
Background
Pharmacists and allied health researchers need to ensure that their practice is supported by current, evidence-based information. Critical appraisal tools have been developed to aid in this process.
Objectives
To analyze the current landscape of critical appraisal tools and to create an aid for pharmacists and other allied health researchers to use in comparing various tools and choosing the best one for each particular study design.
Data Sources
A literature search of the PubMed, University of Toronto Libraries, and Cochrane Library databases was conducted in December 2021, to generate an up-to-date list of critical appraisal tools. The tools were then summarized in a descriptive table.
Study Selection and Data Extraction
Review articles, original manuscripts, and tool webpages were examined to develop a comparison chart based on the user-friendliness, efficiency, comprehensiveness, and reliability of each tool.
Results
Fourteen tools were found through the literature search. These tools were compared using the findings of included review articles, and a comparison chart was created to aid pharmacists and allied health researchers in selecting the appropriate tool for their practice.
Conclusions
There are many standardized critical appraisal tools that can help in assessing the quality of evidence, and the summary list of tools developed and reported here can help health care researchers to compare among them and choose the best one. No tools were found that have been specifically adapted to serve the needs of pharmacists when assessing scientific articles. Future research should examine how existing critical appraisal tools can better identify common data elements that are essential to evidence-based decision-making in pharmacy practice.
KEYWORDS: critical appraisal tools, risk-of-bias 2 (RoB 2) tool, pharmacist, evidence-based practice, validity
RÉSUMÉ
Contexte
Les pharmaciens et les chercheurs en soins de la santé doivent faire en sorte que leur pratique soit étayée par des informations actualisées et fondées sur des données probantes. Des outils d’évaluation critique ont été développés pour faciliter ce processus.
Objectifs
Analyser le paysage actuel des outils d’évaluation critique et créer une aide que les pharmaciens et les autres chercheurs paramédicaux peuvent utiliser pour comparer divers outils et choisir le meilleur pour chaque conception d’étude particulière.
Sources des données
Une recherche documentaire dans trois bases de données (PubMed, les University of Toronto Libraries et la Cochrane Library) a été menée en décembre 2021 afin de générer une liste actualisée d’outils d’évaluation critique qui ont ensuite été résumés dans un tableau descriptif.
Sélection des études et extraction des données
Des articles de synthèse, des manuscrits originaux et des pages Internet d’outils ont été examinés pour dresser un tableau comparatif basé sur la convivialité, l’efficacité, l’exhaustivité et la fiabilité de chaque outil.
Résultats
Quatorze outils ont été trouvés grâce à la recherche documentaire. Ils ont été comparés à l’aide des résultats des articles de synthèse inclus, et un tableau comparatif a été créé pour aider les pharmaciens et les chercheurs en soins de la santé à sélectionner l’outil approprié pour leur pratique.
Conclusions
De nombreux outils d’évaluation critique normalisés peuvent aider à évaluer la qualité des données probantes, et la liste récapitulative des outils développés et rapportés ici peut aider les chercheurs en soins de santé à les comparer et à choisir le meilleur. Aucun outil spécifiquement adapté pour répondre aux besoins des pharmaciens lors de l’évaluation d’articles scientifiques n’a été trouvé. Les recherches futures devraient se pencher sur la manière dont les outils d’évaluation critique existants peuvent mieux identifier les éléments de données communs qui sont essentiels à la prise de décision fondée sur des données probantes dans la pratique de la pharmacie.
MOTS CLÉS: outils d’évaluation critique, outil Risque de biais 2 (RoB 2), pharmacien, pratique fondée sur des données probantes, validité
Pharmacists regularly use their knowledge and skills to provide patient care, to support decision-making by the health care team, and to conduct research. These activities must be supported by current, evidence-based information, and pharmacists must develop their critical appraisal skills and become experts at synthesizing and interpreting relevant information. National campaigns like Choosing Wisely Canada,1 which aim to reduce unnecessary tests and treatments, encourage clinicians to follow recommendations that have been developed following review of scientific evidence to make informed choices with their patients. Similarly, the National Association of Pharmacy Regulatory Authorities (NAPRA)2 lists expertise in medications and medication use as a requirement for licensed pharmacists practising in Canada. As part of modelling this standard, NAPRA highlights the importance of evidence-based medicine and critical appraisal of the source of information when providing care to patient.2 Critical appraisal is not only a skill important to pharmacy practice, but also part of pharmacy practice standards in Canada.
Critical appraisal is a systematic process that is used to identify credible and relevant evidence to support clinical practice and policy.3 When pharmacists and health care workers read a scientific article, they apply their critical appraisal skills to determine whether the article supports or changes their recommendations and practice. They may look at evidence from randomized controlled trials (RCTs) to understand the efficacy, safety, and appropriateness of new or innovative drugs, disease treatments, and pharmacy interventions.4 Observational studies can be read for evidence of an association of drug exposure or pharmaceutical interventions with unintended effects or other outcomes of interest.4,5 Systematic reviews attempt to uncover “all” of the evidence relevant to a specific question, focusing on research that reports data rather than concepts or theory.6 Systematic reviews are rich resources to gain rapid insight into specific health-related questions in a single document, and they are the pinnacle of evidence synthesis used to create and update guidelines for clinical pharmacists.7
Critical appraisal involves interpreting information in a systematic and objective manner. Critical appraisal tools for all types of research methodologies (e.g., case–control studies, observational studies, RCTs) have been developed for quality appraisal of the literature in a formal and systematic process, each with study-specific applicability.3 As described by Twells,8 the traditional critical appraisal process for scientific articles involves 3 main questions: Are the results of the study valid (internal validity)? What are the results? Are the results applicable or generalizable to my patient population? In terms of the specific lens of pharmacy practice, in addition to these 3 questions, the pharmacist is also concerned with questions for evaluating appropriate drug use in practice, such as the following: What are the study limitations, and will they affect my recommendation in this situation? Will I make this recommendation (i.e., do the benefits outweigh the risks)? Will this study change my practice? With so much research available, pharmacists and allied health workers need to use the appropriate critical appraisal tools to select the highest-quality evidence and to determine if the quality of the research is applicable to their objectives and practice.
Although other health care professionals, such as nurses, have produced guidance on the use of critical appraisal tools,9 to our knowledge there are no similar guidance documents comparing current critical appraisal tools that are specifically directed at pharmacists. Therefore, the goal of this narrative review is to analyze the current landscape of critical appraisal tools and to create an aid for pharmacists and other allied health researchers to use in comparing various tools and choosing the best one for a particular study design.
A literature search was conducted in December 2021 using the PubMed, University of Toronto Libraries, and Cochrane Library databases. The databases were searched for articles compiling and/or reviewing critical appraisal tools. The keywords used in the search were “critical appraisal”, “tools”, “risk of bias”, and “validated”, and the results were restricted to articles published between 2011 and 2021. Relevant articles and their reference lists were examined to obtain a preliminary list of potential critical appraisal tools. Tools that were described for use in critical appraisal, assessments of quality or methodology, and analysis of risk of bias were included. Tools described primarily as reporting guidelines, guides developed with the goal of helping authors know what to include in research reports, and tools described as classifying recommendations or assessing only animal studies were excluded from the preliminary list. The same databases were searched with the additional keywords “pharmacy” and/or “pharmacist” to determine if there were any critical appraisal tool recommendations specific to pharmacy. This initial search process yielded a final list of appraisal tools and where to access them (see Appendix 1, available from https://www.cjhp-online.ca/index.php/cjhp/issue/view/214), at which point the literature was reviewed to determine whether the tools had been validated and compared, and to gauge the frequency of their use in literature reviews.
For each identified tool, the original tool-development manuscript or webpage was reviewed for information. PubMed was also searched with a combination of keywords, including the name of the tool, “critical appraisal”, “reliability”, and “validation or validated”. No publication date filters were applied for this stage of searching, because some of the critical appraisal tools that we identified were developed before 2011.
The final list of appraisal tools was additionally formatted as a comparison chart that could serve as a convenient visual selection aid for pharmacists and allied health researchers. The comparison categories—user-friendliness, efficiency, comprehensiveness, and reliability—were determined through discussion among the authors, and the rating system, from 1 star (lowest rating) to 5 stars (highest rating), was established on the basis of information from the literature search results and an analysis of how the critical appraisal tools compared with each other.
More specifically, user-friendliness was compared to indicate how easy it would be to understand and use the tool without additional training. Tools that reviewers described as requiring extra training, being more complex, and/or being more appropriate for experienced methodologists were given lower ratings. Efficiency was compared to indicate how much time would be required to complete the assessment. Tools with fewer items to complete and those described with words like “convenient” were given higher ratings, whereas tools with many items to complete and those described by reviewers as being “more demanding” or requiring more time than other tools were given lower ratings. Efficiency ratings were also influenced by findings from articles comparing tools, if available. Comprehensiveness was compared to indicate how “complete” the tool was in terms of fulfilling the requirements for the critical appraisal process for scientific articles.
If a tool could be used to assess the 3 main components—internal validity, results, and relevance—and additionally included questions similar to what pharmacists would ask when appraising scientific articles, it was given a rating of 5 stars. Tools allowing assessment of only internal validity were given a rating of 1 star, since all tools included in the review assessed internal validity.
For the last category, reliability, tools with inter-rater reliability testing or other forms of validation were given higher ratings. Tools with criticisms of reliability or limited testing were given lower scores. Tools with unclear results on reliability testing or no reliability testing were given a rating of 1 star.
From the literature search, 5 review articles on critical appraisal tools were identified and examined3,8–11 (for the PRISMA diagram, see Appendix 2, available from https://www.cjhp-online.ca/index.php/cjhp/issue/view/214). One of the 5 articles was written as a resource for registered nurses,9 whereas the other 4 articles were general reviews of the available critical appraisal tools. None of the 5 articles provided recommendations specific to pharmacy, and no pharmacy-specific reviews came up during the database searches. Certain universities, including the University of Waterloo,12 provided links to some critical appraisal resources for pharmacy students. Bashir and Dziemidowicz13 also published an online article discussing the theory of critical appraisal to assist pharmacists in evaluating research, providing links to selected user-friendly critical appraisal tools. In addition, the Canadian Society of Hospital Pharmacists provides a list of critical appraisal resources, but it was last updated in 2011,14 and new tools have been developed since then.
From the 5 review articles and the online article by Bashir and Dziemidowicz,13 a preliminary list of 21 critical appraisal tools was obtained. The National Institutes of Health (US) Study Quality Assessment Tool was excluded because the developer did not consider it to be standardized.15 Three tools—the revised Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2), for diagnostic studies; the Evidence-Based Practice Process Quality Assessment (EPQA), for evidence-based projects that guide nursing practice; and the Physiotherapy Evidence Database (PEDro) scale, for physiotherapy intervention studies—were excluded because they have limited applicability to pharmacy practice. Three additional tools—the Jadad Scale and the Delphi List for RCTs and the Reisch Tool for nonrandomized intervention studies—were excluded because they are no longer commonly used or recommended,11 likely because of development of newer tools for their respective study designs. The Reisch Tool was also criticized as being too complex and specific for general use.10 Finally the JAMA user guide was excluded because another, more recent set of tools, the CASP checklists, was developed using its recommendations. The remaining 14 critical appraisal tools and their strengths and limitations are summarized in Table 1 (where each tool abbreviation is also defined), and the selected tools are compared by category in Table 2.
TABLE 1 Summary of Critical Appraisal Tools Useful for Pharmacists
TABLE 2 Comparison of Critical Appraisal Tools to Help Pharmacists with Evidence-Based Practicea
In this narrative review, we have summarized the current landscape of critical appraisal tools that can be used to assess scientific articles, with a specific focus on the unique needs of pharmacists. This review can serve as an aid for pharmacists and other health care practitioners, helping them to quickly choose an optimal critical appraisal tool for the study design in question. Of the 14 tools listed in Table 1, all contain components that assess internal validity, answering the question “Are the results of the study valid?” Furthermore, 5 of the 14 tools include other components of the critical appraisal process for scientific articles (i.e., answering the questions “What are the results?” and/or “Are the results applicable or generalizable to my patient population?”). None of the tools analyzed in this narrative review included questions specific to pharmacy per se, although the CASP checklists came the closest, including components that assess internal validity, results, risks and benefits, and relevance. The 14 tools may still be incorporated into the critical appraisal process that pharmacists and allied health researchers apply for scientific articles, given that they do provide value for learning to identify and select high-quality scientific articles to support evidence-based practice.11
We have also created an up-to-date comparison chart (Table 2) that will serve as a guide to pharmacists and allied health researchers in selecting the appropriate critical appraisal tool, while acknowledging that these tools do not answer all questions in the critical appraisal process for scientific articles used by these practitioners. More specifically, Table 2 summarizes comparisons among the 14 tools covered in this narrative review based on user-friendliness, efficiency, comprehensiveness, and reliability. Some of the tools are user-friendly, including the NOS scale, which, according to Wells and others,18 was developed as “an easy and convenient tool for quality assessment” and is available in the form of a brief manual that walks the reviewer through each item in the scale. The MINORS tool20 is also user-friendly, formatted as a list with a simple scoring system. The CASP, CEBM, SIGN, JBI, AMSTAR 2, and GRACE tools are all formatted as checklists (Table 2), which makes them easy to understand and would make the appraisal process efficient, allowing users to check off the study criteria as they read a research article. In addition, the CEBM tool, while not allowing in-depth assessment and not frequently used in literature reviews (Table 1), has clear explanations that would make it a good introductory tool for beginners, such as pharmacy students.
The Cochrane Collaboration’s risk-of-bias (RoB) tool requires training to interpret the bias domains16 but can be efficient and easy to present once the reviewer has gained some familiarity. The RoB 2 tool30 is an updated tool that is more comprehensive than the original RoB tool but, as a result, can require more consideration and understanding of the training materials to obtain reliable results. The ROBINS-I and ROBIS tools are the most in-depth tools and are best used by experienced methodologists.31,32 In a study comparing the NOS with the ROBINS-I, both intended for assessing non-RCTs, Zhang and others31 found that the ROBINS-I took more time to complete (3 h versus 30 min to assess a single study), which would limit its use in practice. The AMSTAR 2 and ROBIS tools, which are used for assessing systematic reviews, were compared in another study. According to Perry and others,32 “raters felt AMSTAR-2 was more straightforward and user-friendly than ROBIS” possibly because “it does not require expertise in systematic reviewing … just knowledge of trial design.” Thus, pharmacists may find the AMSTAR 2 tool more practical to use.
In terms of reliability, the NOS, ROBINS-I, AMSTAR 2, and ROBIS tools have demonstrated good inter-rater reliability.18,21,31,32 The Cochrane Collaboration’s RoB tool, while widely accepted (Table 1), has only modest inter-rater reliability because of its emphasis on assessor judgment, nonstandard implementation, and the need for training to interpret the bias domains.16,17 The RoB 2 tool is an improvement over its predecessor, but given its greater complexity, training would still be beneficial to improve reliability in application.17,30 The AGREE II tool is a refinement of the original AGREE tool, intended to improve validity and reliability, but it still requires multiple assessors to achieve the increased reliability.23 The MINORS tool also had limited reliability testing, with only 2 surgeons as reviewers.20 The GRACE checklist was piloted with comparative efficacy studies on drugs, medical devices, and medical procedures, which resulted in good specificity and sensitivity scores relative to other quality assessment methods.35 However, no inter-rater reliability studies have been completed (Table 1). The CASP, CEBM, JBI, and SIGN tools also had no inter-rater reliability testing completed or validation method specified, so they would be less appropriate for use in literature reviews.
Most of the tools identified were developed for conducting research, primarily research to support systematic reviews or clinical guidelines, although tools such as the MINORS, GRACE, JBI, and SIGN tools were developed with clinicians or health care decision-makers as additional end-users (Table 1). The AGREE II tool was developed for health care providers, guideline developers, policy-makers, and educators.23 The AMSTAR 2, CASP, and CEBM tools were developed for educational purposes or for use by consumers of research, such as clinicians (Table 1). The MINORS tool was initially developed for surgical studies,20 whereas the JBI tools are mainly used in nursing.10 None of the tools were developed specifically for pharmacy, although the GRACE checklist was successfully applied to comparative effectiveness studies of drugs.24
As for possible bias and conflicts of interest, it should be noted that the RoB, RoB 2, and ROBINS-I tools were developed by the Cochrane Collaboration, and their criteria are specifically applicable to the development of Cochrane reviews (Table 1). The JBI and SIGN tools were also described as meeting the standards of their respective organizations (Table 1), but these standards were not specified and may not be applicable to other practice settings. The Critical Appraisal Skills Programme25 and the Centre for Evidence-Based Medicine26 provide critical appraisal workshops, so their tools may be adapted to suit an educational setting, which may in turn make the tools more appropriate for students or as introductions to critical appraisal.
Two of the tools have been translated into other languages. The AGREE II tool is available in multiple languages, with more translations in progress and available on request,23 and the CEBM checklists are also available in several languages (Table 2). In terms of formats, all tools are available as printable pdfs, with the exception of the RoB tool, for which the documentation only provides examples of how to format the tool, and the RoB 2 tool, which is available as both a pdf and an Excel (Microsoft) template. High-reliability tools such as RoB 2 (see Table 2) are available for pharmacists to evaluate RCTs for quality of evidence. Although RCTs represent the “gold standard” for experimental design, a trial’s execution and the resulting article’s analysis and reporting can influence the quality of the evidence. By selecting the appropriate tools, pharmacists working in different settings can support their evidence-based practice. Pharmacists who work in a hospital setting often have opportunities to work on collaborative projects spanning all types of evidence-based research (e.g., case–control study, cohort study, RCT, cross-sectional study, meta-analysis). The applicability column of Table 2 shows that the Critical Appraisal Skills Programme and the Joanna Briggs Institute offer tools for several study types based on the same checklist formats, which can be efficient, since pharmacists need to learn how to use only one type of tool.
A pharmacist serving on an internal interprofessional committee may be involved in developing local drug therapy manuals for their organization and would need to examine systematic reviews to find evidence for recommendations. Table 2 includes 2 tools for systematic reviews, AMSTAR 2 and ROBIS. A tool like AMSTAR 2, which was developed for health professionals and policy-makers, would be ideal for this purpose. Alternatively, if the interprofessional team has an experienced methodologist available for consultation, the pharmacist could use the ROBIS tool, and then compare tool outcomes and reach a consensus with the methodologist. In contrast, a pharmacist working in a pediatric neonatal care unit might be caring for a unique patient whose condition has only been described in case reports. In this situation, the pharmacist could review the applicability column of Table 2 and would find that the Joanna Briggs Institute has a checklist for case reports, which can be used to assess the quality of each case report identified.
Pharmacists involved in education initiatives such as journal clubs present and critique new research articles to other pharmacists. For journal club presentations, a pharmacist would likely want to employ a user-friendly and efficient tool. If presenting findings from a novel and timely RCT, for example, the pharmacist could use the RoB assessment tool to easily translate the RCT data into tables or figures for succinct presentation. If presenting findings on a cohort study or case–control study, the pharmacist could use the NOS tool to create pleasing visuals based on a star rating system.
Of the many validated tools available, most address only internal validity, with few asking questions that would provoke judgments of study applicability, limitations, and practice-changing outcomes. Moreover, no tools have been developed specifically for pharmacists, and no literature was found indicating how pharmacists could apply critical appraisal skills in practice or commenting on whether a standardized approach would be beneficial. A potential future project could take an approach similar to that used in the development of the MINORS tool,20 with involvement of pharmacists and pharmacy leaders in the development and piloting of a critical appraisal tool specific to the literature on drugs and pharmacy interventions. In addition, many current tools are available only as pdf files that must be printed for use. Newer tools such as the RoB 2 allow creation of Excel spreadsheet files,17 which can be more efficient but are still not as accessible as applications developed for other uses, such as MDCalc for medical calculations (www.mdcalc.com). It would be interesting to see the development of an accessible application for critical appraisals.
This narrative review had some structural limitations. In particular, it was not a systematic review and did not generate an exhaustive list of all critical appraisal tools currently available. A limitation of the comparison chart (Table 2) is that the rating score was based on findings from the included review articles rather than being determined systematically through expert consensus. The scoring system would benefit from incorporating a survey or results from a study piloting these tools with pharmacists and allied health researchers.
Critical appraisal is an essential skill for pharmacists and health care practitioners alike. Many standardized critical appraisal tools are available that can help in systematically assessing various aspects of the quality of evidence, and the current narrative review summarizes 14 tools useful for pharmacists and allied health care researchers. In examining the current landscape of critical appraisal tools, we found that no tools that have been specifically modified to serve the needs of pharmacists when assessing scientific articles. As such, future research should examine how critical appraisal tools could be improved to better identify common data elements that are essential to evidence-based decision-making in pharmacy practice.
1 Choosing Wisely Canada [website]. Choosing Wisely Canada; 2021 [cited 2022 Jan 6]. Available from: choosingwiselycanada.org
2 Model standards of practice for pharmacists and pharmacy technicians in Canada. National Association of Pharmacy Regulatory Authorities; 2022 [cited 2023 Jan 13]. Available from: https://www.napra.ca/wp-content/uploads/2022/09/NAPRA-MSOP-Feb-2022-EN-final.pdf
3 Haile ZT. Critical appraisal tools and reporting guidelines. J Hum Lact. 2021;38(1):21–7.
Crossref PubMed
4 Awaisu A, Mukhalalati B, Mohamed Ibrahim MI. Research designs and methodologies related to pharmacy practice. In: Babar ZUD, editor. Encyclopedia of pharmacy practice and clinical pharmacy. Elsevier; 2019. pp. 7–21.
5 Bosdriesz JR, Stel VS, van Diepen M, Meuleman Y, Dekker FW, Zoccali C, et al. Evidence-based medicine—when observational studies are better than randomized controlled trials. Nephrology (Carlton). 2020; 25(10):737–43.
Crossref PubMed PMC
6 Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs. 2014;114(3):53–8.
Crossref PubMed
7 Bobo KS, Cober MP, Eiland LS, Heigham M, King M, Johnson PN, et al. Key articles and guidelines for the pediatric clinical pharmacist from 2019 and 2020. Am J Health Syst Pharm. 2022;79(5):364–84.
Crossref
8 Twells LK. Evidence-based decision-making 1: Critical appraisal. Methods Mol Biol. 2021;2249:389–404.
Crossref PubMed
9 Buccheri RK, Sharifi C. Critical appraisal tools and reporting guidelines for evidence-based practice. Worldviews Evid Based Nurs. 2017; 14(6):463–72.
Crossref PubMed
10 Zeng X, Zhang Y, Kwong JS, Zhang C, Li S, Sun F, et al. The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review. J Evid Based Med. 2015;8(1):2–10.
Crossref PubMed
11 Ma LL, Wang YY, Yang ZH, Huang D, Weng H, Zeng XT. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Mil Med Res. 2020;7(1):7.
PubMed PMC
12 Pharmacy: critical appraisal [list of resources]. University of Waterloo; 2021 [cited 2021 Dec 12]. Available from: https://subjectguides.uwaterloo.ca/c.php?g=695509&p=5007334
13 Bashir S, Dziemidowicz K. Critical appraisal: how to evaluate research for use in clinical practice. Pharm J. 2021 [cited 2022 Apr 26];306(7950). Available from: https://pharmaceutical-journal.com/article/ld/critical-appraisal-how-to-evaluate-research-for-use-in-clinical-practice
14 Part 2: how to critically appraise evidence. In: CSHP 2015 toolkit. From paper to practice: incorporating evidence into your pharmacy practice (objective 3.1). Canadian Society of Hospital Pharmacists; 2011 [cited 2021 Dec 12]. Available from: https://cshp-scph.ca/toolkit-from-paper-to-practice
15 Study quality assessment tools [list of resources]. National Heart, Lung, and Blood Institute (US); 2021 [cited 2021 Dec 12]. Available from: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools
16 Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
Crossref
17 Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898.
Crossref PubMed
18 Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute; 2021 [cited 2021 Dec 12]. Available from: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
19 Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.
Crossref PubMed PMC
20 Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J. Methodological index for non-randomized studies (MINORS): development and validation of a new instrument. ANZ J Surg. 2003;73(9):712–6.
Crossref PubMed
21 Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.
Crossref PubMed PMC
22 Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.
Crossref PMC
23 AGREE Next Steps Consortium. AGREE II instrument. AGREE Research Trust; 2017 [cited 2021 Dec 12]. Available from: https://www.agreetrust.org/resource-centre/agree-ii/
24 Dreyer NA, Velentgas P, Westrich K, Dubois R. The GRACE checklist for rating the quality of observational studies of comparative effectiveness: a tale of hope and caution. J Manag Care Spec Pharm. 2014; 20(3):301–8.
PubMed
25 CASP checklists. Critical Appraisal Skills Programme; 2021 [cited 2021 Dec 12]. Available from: https://casp-uk.net/casp-tools-checklists/
26 Critical appraisal tools. University of Oxford, Nuffield Department of Primary Care Health Sciences, Centre for Evidence-Based Medicine; 2022 [cited 2021 Dec 12]. Available from: https://www.cebm.ox.ac.uk/resources/ebm-tools/critical-appraisal-tools
27 JBI critical appraisal tools. Joanna Briggs Institute; 2021 [cited 2021 Dec 12]. Available from: https://jbi.global/critical-appraisal-tools
28 SIGN checklists. SIGN (Scottish Intercollegiate Guidelines Network); 2021 [cited 2021 Dec 12]. Available from: https://www.sign.ac.uk/what-we-do/methodology/checklists/
29 Crowe M. Crowe critical appraisal tool (v1.4). Conchra Research & Technology (Michael Crowe); 2015 [cited 2022 Apr 26]. Available from: https://conchra.com.au/2015/12/08/crowe-critical-appraisal-tool-v1-4/
30 Minozzi S, Cinquini M, Gianola S, Gonzalez-Lorenzo M, Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application. J Clin Epidemiol. 2020;126:37–44.
Crossref PubMed
31 Zhang Y, Huang L, Wang D, Ren P, Hong Q, Kang D. The ROBINS-I and the NOS had similar reliability but differed in applicability: a random sampling observational studies of systematic reviews/meta-analysis. J Evid Based Med. 2021;14(2):112–22.
Crossref PubMed
32 Perry R, Whitmarsh A, Leach V, Davies P. A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2. Syst Rev. 2021;10(1):273.
Crossref PubMed PMC
33 Crowe M, Sheppard L. A general critical appraisal tool: an evaluation of construct validity. Int J Nurs Stud. 2011;48(12):1505–16.
Crossref PubMed
34 Crowe M, Sheppard L, Campbell A. Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs. J Clin Epidemiol. 2012;65(4):375–83.
Crossref
35 Dreyer NA, Bryant A, Velentgas P. The GRACE checklist: a validated assessment tool for high quality observational studies of comparative effectiveness. J Manag Care Spec Pharm. 2016;22(10):1107–13.
PubMed
Competing interests: None declared. ( Return to Text )
Funding: None received. ( Return to Text )
Canadian Journal of Hospital Pharmacy, VOLUME 76, NUMBER 2, Spring 2023