Introduction to Dental Materials In-vitro Studies (DEMIST) appraisal tool
and the Dental Materials in-Vivo Studies Extension (DEVISE) tool

The evaluation of research holds significant importance within evidence-based practice. The realm of evidence synthesis provides a range of methods for qualitative assessment, employed to offer recommendations for clinical guidelines and to conduct systematic reviews. Undoubtedly, these methods are valuable and suited to their intended purposes. However, they cannot be readily employed in the context of dental materials research, as they do not encompass the procedures and frameworks employed for ensuring high-quality research in the field of dentistry.

Dental research encompasses a wealth of information, and it is essential to recognize the potential for bias when drawing conclusions from this body of data. Research into dental materials involves both laboratory-based (in-vitro) and clinical (in-vivo) studies. Established and standardized tools exist for the rigorous evaluation of in-vivo studies, yet the methodology is still evolving for in-vitro studies.

To bridge this gap in evidence-based dentistry, two specific tools have been developed: the Dental Materials In-vitro Studies (DEMIST) appraisal tool and the Dental Materials in-Vivo Studies Extension (DEVISE) tool. DEMIST serves as a comprehensive instrument designed for the meticulous evaluation of in-vitro dental materials studies. On the other hand, DEVISE functions as a complementary extension to existing tools, facilitating the assessment of in-vivo dental materials research studies.

DEMIST (DEntal Materials In-vitro STudies) Appraisal; and DEVISE (DEntal in-VIvo Studies Extension) Tools for the Critical Appraisal of Dental

Materials Research:
Inceptive Development
Anirudha Agnihotry, Karanjot Singh Gill, Richard Gray Stevenson III
Stevenson Dental Research Institute, San Dimas, California

Key words:
Evidence Based Dentistry, Critical Appraisal, Evidence Based Practice, Systematic Reviews, Research Methods, Dental Materials

Graphical Abstract:

Why is it important?
The critical appraisal of research is an integral part of the evidence-based practice. The domain of evidence synthesis offers several tools for qualitative analysis, which are used to provide recommendations on clinical practice guidelines and to conduct systematic reviews. No doubt, these are great tools and suit their purpose well but, they cannot be applied directly to dental materials research as they do not address the processes and systems used for conducting quality research in dentistry. There is vast data in dental research, and it is imperative to identify the risk of bias involved while generating recommendations from this data. Both in-vitro and in-vivo research studies on dental materials exist. Proven standardized tools are present for critical appraisal of in-vivo studies, but the science is still developing for in-vitro studies. To address this gap in evidence-based dentistry, the DEntal Materials In-vitro Studies (DEMIST) appraisal tool and the DEntal materials in-Vivo Studies Extension (DEVISE) tool were created. DEMIST is a complete tool meant for the critical appraisal of in-vitro dental materials studies, while the DEVISE is to be used like an extension with existing tools to evaluate in-vivo dental materials research studies.

Evidence synthesis is incomplete without qualitative analysis. A Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement provides reporting guidelines for write-ups of the systematic reviews. These guidelines clearly state to specify the tools used, the domains assessed, and the overall risk of bias rating given to the included studies1. These systematic reviews are further used to give recommendations in clinical practice guidelines and informing the clinicians for best practices2. After the search and inclusion/exclusion, critical appraisal is the next crucial filter in the process and should be meticulously applied.

All the available critical appraisal tools and checklists used presently were assessed for the risk of bias. The domains and methods for the scoring criteria were also assessed. These included but were not limited to: AMSTAR 1 and 2 (Assessment of Multiple SysTemAtic Reviews 2), Cochrane RoB (Risk of Bias) 1 and 2, ROBINS (Risk of Bias In Non-randomized Studies), SciRAP (Science in Risk Assessment and Policy), SYRCLE RoB tool (SYstematic Centre for Laboratory animal Experimentation Risk of Bias), Checklists by JBI (Joanna Briggs Institute), and the checklists by CASP (Critical Appraisal Skills Program). These tools were assessed for common domains and the differences.
The next step was to evaluate the research that included these tools using inter rater-reliability tests and descriptive feedback from experts on the ease of usability of the tools. Basic research concepts were also referred to when comparing these tools.
Lastly, the domains of these tools were evaluated against the guidance provided by the Academy of Dental Materials (ADM) to conduct dental materials research studies. The new tools for critical appraisal of the dental studies were finally developed keeping basic research methods in perspective and as per ADM guidance.

There are 27 tools and instruments designed to critically appraise the methodological quality of primary and secondary medical studies3. For in-vitro studies, there are 22 tools present for qualitative assessment4. All the domains of these tools were studied and were evaluated in the context of dental studies, referring to ADM Guidance on the ideal methods for minimizing bias when conducting dental materials research. To assess evidence quality, the risk of bias and with an intention of providing a reproducible and consistent framework, we focused on applicability, as well as inter-rater and inter-center reliability, with the aim to maintain ease of usability (like AMSTAR) to facilitate quick appraisals with scoring system of closed ended questions of Yes/No and three levels of evidence (high, moderate, and low) categories5. After all the critical appraisal tools were studied, it was decided to fabricate two tools as follows:

1. DEMIST (DEntal Materials In-vitro STudies) Appraisal Tool was developed to address the in-vitro study methods. This was designed as a complete tool, as no other pre-existing tools could be applied directly to in-vitro dental materials studies. It has 10 questions and 5 domains: sample preparation bias; performance bias; attrition bias; analysis bias and reporting bias. 10 questions with yes/no will give either a score of 1 or 0. Higher the rating, lower the bias: high: 0-3; moderate: 4-6; low: 7-10.

2. DEVISE (DEntal in-VIvo Studies Extension) Tool was not designed as a complete tool, but rather an extension to be used alongside the existing tools such as Cochrane RoB and ROBIS for in-vivo studies. The existing critical appraisal tools for clinical studies do cover the basic methods of in-vivo studies, but do not measure or include the biases mentioned in the ADM guidance. It has 5 questions and 3 domains: sample preparation bias; performance bias and analysis bias. 5 questions with yes/no will give either a score of 1 or 0. Higher the rating, lower the bias: high: 0-1; moderate: 2-3; low: 4-5.

According to a recent methodological evaluation of all systematic reviews of in-vitro dental studies conducted up to January 2022, only a fifth could present a satisfactory technique to assess the risk of bias on the included studies6. Another analysis reported on 51 tools for critical appraisal and quality assessment in systematic reviews of in-vitro studies conducted from 2006 to 2020. Of these 51 tools, 26 were developed by authors, and only 17 of those were related to dentistry4. Overall systematic reviews reveal variability, inconsistency, and inadequacy in the quality assessment of dental studies.
Critical appraisal of dental studies needs to be standardized for quality evidence synthesis from systematic reviews to generate robust recommendations for clinical practice. RoBDEMAT (A risk of bias tool and guideline to support reporting of pre-clinical dental materials research and assessment of systematic reviews) has been a meaningful step in this direction.7 The DEMIST critical appraisal tool and DEVISE Extension tool lay the framework for the same. While RoBDEMAT is developed to perform quality assessment on dental materials preclinical research with scrupulous and extensive investigation; DEMIST and DEVISE are focused on both, preclinical and clinical studies with concise and expeditious scrutiny. Going forward, we do understand that both (DEMIST AND DEVISE) these tools need more research to establish their merit and credibility. Both the tools will be subjected to testing on multiple published manuscripts in dental materials. We will further investigate the validity and suitability of the domains, questions, and scoring systems via Delphi studies with experts, clinicians (general and specialists), academics and, students to make the tool standardized and easier to use. An elaborate explanatory handbook is being prepared to provide the details of each question, per ADM guidance in context. Informed by user feedback and extensive validity studies, this tool will improve the quality of systematic reviews of dental studies and strengthen their recommendations for clinical use. Further application of the concepts of the GRADE approach to the tool, will help generate meaningful clinical recommendations, furthering critical appraisal.8
Our foremost goal is to keep the methodology and results interpretation simple so maximum number of individuals can benefit from using DEMIST and DEVISE.


  1. BMJ (OPEN ACCESS) Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ 2021;372:n160. doi: 10.1136/bmj.n160
  2. World Health Organization. (‎2014)‎. WHO handbook for guideline development, 2nd ed. World Health Organization.
  3.  Ma, LL., Wang, YY., Yang, ZH. et al. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better?.Military Med Res 7, 7 (2020).
  4. Tran, L., Tam, D.N.H., Elshafay, A. et al. Quality assessment tools used in systematic reviews of in vitro studies: A systematic review. BMC Med Res Methodol 21, 101 (2021).
  5. Gates M, Gates A, Duarte G, Cary M, Becker M, Prediger B, Vandermeer B, Fernandes RM, Pieper D, Hartling L. The reliability, usability, and applicability of tools to appraise quality and risk of bias in systematic reviews: a prospective evaluation of AMSTAR, AMSTAR 2 and ROBIS. In: Abstracts of the 26th Cochrane Colloquium, Santiago, Chile. Cochrane Database of Systematic Reviews 2020;(1 Suppl 1)
  6. Hammel, C., Pandis, N., Pieper, D. et al. Methodological assessment of systematic reviews of in-vitro dental studies. BMC Med Res Methodol 22, 110 (2022).
  7. Delgado AH, Sauro S, Lima AF, Loguercio AD, Della Bona A, Mazzoni A, Collares FM, Staxrud F, Ferracane J, Tsoi J, Amato J, Neuhaus KW, Ceballos L, Breschi L, Hannig M, Melo MA, Özcan M, Scotti N, Opdam N, Yamaguchi S, Paris S, Turkun LS, Doméjean S, Rosa V, Palin W, Schwendicke F. RoBDEMAT: A risk of bias tool and guideline to support reporting of pre-clinical dental materials research and assessment of systematic reviews. J Dent. 2022 Dec;127:104350. doi: 10.1016/j.jdent.2022.104350. Epub 2022 Oct 28. PMID: 36341980.
  8. Mustafa RA, Santesso N, Brozek J, Akl EA, Walter SD, Norman G, et al. The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. Journal of clinical epidemiology. 2013;66(7):736-42; quiz 42.e1-5

DEntal Materials In-vitro STudies Appraisal Tool – DEMIST Appraisal Tool

Sample Preparation Bias
• Standard Sample Preparation?
• Sample preparation identical for all groups?

Performance Bias
• Standard tests being used?
• Tests performed with standards?
• Blinding of the examiner/assessor?
• Randomization of samples?

Attrition Bias
• Loss of sample size can be explained without question of bias?

Analysis Bias
• Appropriate Statistical tests used?

Reporting Bias
• Selective Reporting? (Any variables omitted from the analysis that could have been analyzed)
• Results adequately discussed in conclusion?
• Conflicts of Interest?

High – 0 – 3
Moderate – 4 – 6
Low – 7- 10

DEntal materials in-Vivo Studies Extension – DEVISE tool

Sample Preparation Bias
• Standard Sample Preparation?
• Sample preparation identical for all groups?

Performance Bias
• Standard tests being used?
• Tests performed with standards?

Analysis Bias
• Appropriate Statistical tests used?

High: 0 – 1
Moderate: 2 – 3
Low: 4 – 5

About Us:

Stevenson Dental Foundation was founded in 2022 by Dr. Richard Stevenson, a recognized authority on oral health promotion and a 30-year advocate for dental education and care for low-income populations.

Stevenson Dental Foundation’s mission is to provide quality accessible oral care, foster cutting-edge research and support the next generation of dental professionals.

Our Vision:

Improving oral care for all to create a world where everyone smiles with confidence. ❤

In order to serve as many patients in our community as possible, Stevenson Dental Foundation must be able to fund all aspects of a comprehensive, high-functioning program. We hope that you will consider supporting our efforts by making a tax-deductible contribution today to help us make quality, affordable dental care a reality for everyone. Stevenson Dental Foundation commits to maintaining low overhead and administrative costs to ensure that the maximum amount of funds raised go directly to fulfilling our mission.

Our foundation is committed to promoting equal opportunities for all applicants regardless of race, color, religion, gender, national origin, age, disability, or any other protected status. We strive to ensure a fair and unbiased process in awarding oral-care grants and scholarships, fostering inclusivity in our mission to support dental education and patient care.

Stevenson Dental Foundation is a 501(c)(3) public charity. All donations are tax-deductible to the fullest extent of the law.

©2023 Stevenson Dental Foundation | All Rights Reserved