J. Orthod.
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Harrison, J. E.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Harrison, J. E.
Journal of Orthodontics, Vol. 30, No. 4, 309-315, December 2003
© 2003 British Orthodontic Society

Scientific Section

Clinical trials in orthodontics II: assessment of the quality of reporting of clinical trials published in three orthodontic journals between 1989 and 1998

J. E. Harrison

Liverpool University Dental Hospital and School of Dentistry, UK

Jayne E. Harrison, Department of Clinical Dental Sciences, Liverpool University Dental Hospital and School of Dentistry, Pembroke Place, Liverpool L3 5PS, UK. Email: Jayne.Harrison{at}rlbuh-tr.nwest.nhs.uk


   Abstract
Top
 Abstract
Introduction
 Materials and methods
 Results
 Discussion
 Conclusions
 References
 
Aims: To test the hypothesis that the quality of reporting of orthodontic clinical trials is insufficient to allow readers to assess the validity of the trial.

Design: A retrospective observational study.

Setting: The American Journal of Orthodontics and Dentofacial Orthopedics (AJODO), the British Journal of Orthodontics (BJO) and European Journal of Orthodontics (EJO).

Data source: Clinical trials published between 1989 and 1998.

Method: A hand search was performed to identify all clinical trials. The concealment of allocation, whether the trial was randomized, double blind, and whether there was a description of withdrawals and dropouts was recorded.

Results: One hundred and fifty-five trial reports were identified of which 4 (2.6%) were adequately concealed, 85 (54.8%) were described as being randomized, 10 (6.5%) as double-blind, and 44 (28.4%) gave a description of withdrawals and drop-outs from the trial. The type of randomization was considered appropriate in 78 (50.3%) reports and in 57 (36.8%) reports the level of blinding was considered appropriate. When assessed for the risk of bias in the reported trials,1Go one trial (0.6%) had a low risk of bias, 17 (11%) a moderate risk, and 137 (88.4%) a high risk.

Conclusions: In general the quality of reporting orthodontic clinical trials was insufficient to allow readers to assess the validity of the trials. Reporting of clinical trials could be improved by orthodontic journals adopting the CONSORT statement2,Go3Go to ensure that all relevant information is provided.

Key words: Clinical orthodontic research, clinical trials, quality assessment, randomized clinical trials, reporting


   Introduction
Top
 Abstract
 Introduction
Materials and methods
 Results
 Discussion
 Conclusions
 References
 
Well-designed randomized controlled trials (RCTs), confirming the same hypothesis, have, for many years, been recognized as providing the strongest level of evidence of the treatment effect of competing therapeutic interventions. However, with the development of systematic review and meta-analytical techniques, systematic reviews of RCTs can now be seen as providing the best level of evidence.4–Go8Go Over recent years developments in the science of reviewing and summarizing evidence from clinical trials in systematic reviews have highlighted the need for clinical trial reports to contain all the relevant information needed to assess the internal and external validity of a clinical trial. These are the degree of control of factors that could systematically affect the results of a trial and the degree to which the results can be generalized to populations who may receive the interventions.9Go

As a result, the quality of the conduct of controlled trials has been found to systematically influence the results of the trials. Importantly, poorer quality trials tend to over estimate the treatment effects of the interventions being assessed.10,Go11Go Although the quality of reporting is not a direct measure of the inherent quality of a trial, it does provide readers with a useful means of assessing its validity. Also, the published report is often the only information available on how a trial was carried out. Several scales and checklists have, therefore, been developed to help readers assess the quality of a trial report.12Go

The Jadad scale13Go was adopted for use by the Cochrane Collaboration to assess the quality of trials included in Cochrane Reviews. However, recent work assessing the quality of trials using different composite scales showed that the perception of the quality of a clinical trial varies according to which scale is used.14,Go15Go Consequently, the conclusions of a meta-analysis can be affected if trials are excluded/included or weighted according to the results of a summary score of a quality scale.14Go Based on empirical evidence and theoretical considerations it appears that concealment of allocation, blinding and completeness of data are the most likely indicators of trial quality.15Go For these reasons, the validity of trials being considered for inclusion in Cochrane reviews is now assessed in terms of the level of bias that is likely in each trial1Go (see Table 1Go).


View this table:
[in this window]
[in a new window]
  		TABLE 1 Method to summarize the validity of studies
 
Guidelines and checklists, aimed at improving the reporting of clinical trials, have also been published.2,Go3Go These have now been adopted by many journals, including the Journal of Orthodontics,16Go and can be used by authors, referees, and journal editors to ensure that the key pieces of information, needed to assess the internal and external validity of the trial, are reported.

The aim of this study was to test the hypothesis that the quality of reporting orthodontic clinical trials is inadequate to allow readers to assess the validity of the trial.


   Materials and methods
Top
 Abstract
 Introduction
 Materials and methods
Results
 Discussion
 Conclusions
 References
 
Identification of clinical trials
The principal investigator successfully completed the Cochrane Collaboration Oral Health Group hand-searching test search for the identification of randomized controlled trials (RCTs) and controlled clinical trials (CCTs).17Go I hand-searched the American Journal of Orthodontics and Dentofacial Orthopedics (AJO-DO), British Journal of Orthodontics (BJO), and European Journal Orthodontics (EJO) to identify all papers that reported randomized or controlled clinical trials published between 1989 and 1998 inclusive.

Assessments
The following information on each publication was recorded:

  1. The concealment of allocation of treatment according to Cochrane Collaboration Guidelines.1Go
  2. Individual domains of the Jadad scale,13Go i.e. whether
    2.1. the trial was randomized;
    2.2. the trial was double-blind;
    2.3. there was a description of withdrawals and drop-outs;
    2.4. the randomization method was appropriate;
    2.5. the level of blinding used was appropriate.

The studies were then assessed as having a low, moderate, or high risk of bias depending on whether they met all the quality criteria or if one or more criteria was partially met or not met suggested in the Cochrane Handbook1Go (see Table 1Go).

Reliability
I reclassified a random 10% sample of the trials identified in each journal, at a period no less than 3 months after the first classification, to assess the intra-examiner reliability of this classification system.

Statistical analysis
Descriptive statistics were used to assess the reporting characteristics of trials published in each journal. Any differences in categorical data were evaluated with the chi-squared ({chi}2) test. Intra-examiner reliability of the assessments was evaluated with the Kappa statistic18Go and percentage agreement.


   Results
Top
 Abstract
 Introduction
 Materials and methods
 Results
Discussion
 Conclusions
 References
 
Between 1989 and 1998, 155 reports of clinical trials were published in the AJODO, BJO, and EJO, which represents 6.4% (155/2407) of all papers published in these journals over this period. Of these 85 (54.8%) were classified as randomized controlled trials (RCTs) and 70 (45.2%) as controlled clinical trials (CCTs) that used a quasi-random or haphazard control.

Reliability of the assessments
The intra-examiner reliability of the assessments ranged from 81% to 100% with Kappa statistics of 0.54–1.0 (moderate–very good agreement) for the individual domains (Table 2Go).


View this table:
[in this window]
[in a new window]
  		TABLE 2 Reliability of the assessments used to assess clinical trials published in AJODO, BJO, and EJO 1989–1998
 
Quality assessment of the trial reports
Only four trials (2.6%) had adequately concealed allocation of treatment. The concealment was inadequate in 28 trial reports (18.1%) and unclear in over three-quarters (123/155, 79.4%) of the papers (Table 3Go).


View this table:
[in this window]
[in a new window]
  		TABLE 3 Concealment of allocation
 
Over half of the trial reports (85/155, 54.8%) were described as being randomized. However, only 10 (6.5%) were described as double blind and a quarter (44/155, 28.4%) gave a description of withdrawals or drop-outs from the trial (Table 4Go).


View this table:
[in this window]
[in a new window]
  		TABLE 4 Jadad Score domains I: method used in the reports of clinical trials published in AJODO, BJO, and EJO 1989–1998
 
The type of randomization used was considered appropriate in half of the trial reports (78/155, 50.3%), but the level of blinding was considered to be appropriate in only a third (57/155, 36.8%) of trials (Table 5Go).


View this table:
[in this window]
[in a new window]
  		TABLE 5 Jadad Score domains II: appropriateness of randomization and blinding in the reports of clinical trials published in AJODO, BJO, and EJO 1989–1998
 
The assessments for the individual domains were then combined to give an assessment of the risk of bias in each trial. Only one study (0.6%) met all the quality criteria and was assessed as having a low risk of bias. Most of the studies (137/155, 88.4%) had a high risk of bias and 17 (11.0%) a moderate risk of bias (Table 6Go).


View this table:
[in this window]
[in a new window]
  		TABLE 6 Risk of bias in the clinical trials published in the AJODO, BJO and EJO 1989–1998
 

   Discussion
Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
Conclusions
 References
 
Unfortunately, the reports of orthodontic clinical trials, published in the AJODO, BJO and EJO from 1989 to 1998, were generally incomplete and many lacked key information thus making it difficult for readers to judge the validity of the trials. For example, the level of concealment of allocation of treatment was considered adequate in only four (2.6%) papers. Although over half of the trials (85/155, 54.8%) were described as being randomized, only 10 (6.5%) were double-blind and 44 (28.4%) gave a description of withdrawals and drop-outs from the trial. The implications of these findings on the conduct and findings of systematic reviews of orthodontic clinical trials will be discussed.

Assessing the quality of clinical trials
Unfortunately, unless researchers make a direct approach to the authors, the only information about the methodological quality of a trial that is available is what is contained within the trial report. It is important, therefore, to ask what is being assessed when trial quality is assessed from published reports, and distinguish between the methodological quality of the trial and the quality of its report.12,Go19Go

Inadequate concealment of treatment allocation has been identified as the factor that was most likely to affect the assessment of treatment effect.10,Go11Go It has been estimated that, on average, this results in an exaggerated treatment effect of about 40% and may be considered a major problem.15Go

The CONSORT statement2,Go3Go aims to improve the reporting of RCTs by providing a checklist for authors, so that they are prompted to include all the information that is relevant and necessary to assess the quality of trials. The quality of trial reports has started to improve since journals have adopted these guidelines and included them in their ‘Instructions for Authors’.20,Go21Go

Orthodontic clinical trials
The main reasons for orthodontic trials failing to minimize the risk of bias was that they did not adequately conceal the allocation of interventions, the trials were not double blind, and an account of participants who withdrew or were lost to follow-up was not given.

Concealment of allocation
It was disappointing that only four trial reports contained explanations of the method of concealment of allocation that were considered to be adequate, e.g. there was central randomization by telephone or sealed, opaque, sequentially numbered envelopes were used to keep the allocation concealed until after recruitment. In some trials the concealment was clearly inadequate so that it would have been possible for clinicians to predict which intervention a patient or quadrant was to receive. Such methods include alternate patients receiving intervention A and B, or all upper right and lower left quadrants receiving intervention X, and the other quadrants Y. In the remaining trials the method of concealment was unclear and patients were simply randomized to the different interventions.

Blinding
Ideally, the participants, clinicians and assessors involved in a clinical trial do not know which intervention any participant is receiving. In the situation where the clinicians are the assessors the trial is said to be ‘double-blind’, but if the assessors are independent the trials may be ‘triple-blind’. In orthodontics, it is often very difficult to carry out triple or even double-blind trials, because orthodontic appliances and materials often differ in appearance so that participants and/or clinicians are aware of which intervention any participant is receiving. However, it is possible to adopt double or triple blinding strategies in studies that assess different mouthwashes, toothpastes or analgesics that can be prepared and packaged to be identical. In studies in which blinding of the intervention is not possible the records that are used could have all means of identification removed. Furthermore, the data derived from these records could be recorded by an assessor who is independent of the trial and unaware of the group allocation. If guidelines on the design, conduct and reporting of orthodontic clinical trials are to be drawn up it would seem worthwhile to consider what level of blinding of patient/clinician/ assessor would be considered appropriate in the situations where clinical trials are likely to be conducted. For example, in bonding trials is it possible for patients or clinicians to detect which bracket is bonded with which material—are they slightly different colours; were they cured in different ways, e.g. one light cured and the other chemical? If so, who should assess whether a bracket is off or a band lose? In this situation, it may not be possible to achieve any level of blinding, so open trials may be appropriate. For other studies, e.g. comparisons of functional appliances, it is probably not possible to blind patients or the clinicians who are treating them, but records can be assessed independently with the assessor blind to the intervention used. So, for these trials, an appropriate level of blinding would be where records are anonymous and examined by an independent assessor away from the participants. As suggested above, for trials assessing mouthwashes it would seem that a minimum of double blinding would be appropriate.

Withdrawals
For clinical trials to meet the criteria of describing withdrawals and drop-out careful records need to be kept of all trial participants and for their progress to be reported. The CONSORT2,Go3Go statement has gone a long way to help authors disclose this information by incorporating a flow chart of the numbers of participants at each stage of the trial.

Comparison with other specialties
Assessment of the quality of trials carried out in other specialties has been carried out. In general, the quality of reporting clinical trials is low with more than a half of all trials scoring less than half of the points available from the quality scales used.22–Go30Go However, these studies used a variety of scales and checklists that assessed several criteria (range 5–32 criteria), so it is not possible to make a direct comparison with this study.

Implications of poor reporting on evidence based orthodontics
Evidence-based medicine (EBM) has been defined as the process of ‘systematically finding, appraising and using contemporary research as the basis for clinical practice’.31Go This definition can also be applied to dentistry and, in turn, to orthodontics. If orthodontics is to become a specialty where the clinical decisions that we make for our patients are based on sound evidence, all the relevant evidence needs to be available so that it can be found and appraised in a systematic way. Unfortunately, this study has shown that the amount of information provided in reports of clinical trials, in three of the leading orthodontic journals is, in general, inadequate. This will inevitably hamper efforts to bring together the best evidence that is available.

The quality of a trial report has implications for the interpretation of an individual trial, but the effects are compounded in systematic reviews where data from several trials are combined in a meta-analysis.32Go Unless steps are taken to reduce the influence of poor quality trials on the meta-analysis, the overall estimate of treatment effect could be severely biased and over estimated. Due to the problems associated with the validity of poor quality trials,10,Go11Go it is important that details of the methods of the clinical trial in trial reports are comprehensive. This information allows reviewers to assess the quality and make valid judgements as to whether to include trial results in a meta-analysis. If the appropriate details are omitted from the reports of orthodontic clinical trials it is likely that systematic reviews of them may be biased and any estimation of treatment effect would be inaccurate.


   Conclusions
Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 Conclusions
References
 
In general, the quality of reporting orthodontic clinical trials was insufficient to allow readers to assess the internal validity of the trials. This was related to the allocation of treatment not being adequately concealed in many trials, and/or trials not being double or adequately blinded, or lacking a full description of trial participants, withdrawals and dropouts.

Orthodontic journals should adopt or continue to use the CONSORT guidelines3Go for the reporting of randomized controlled trials with the aim of improving the quality of orthodontic trial reports and, in turn, systematic reviews of trials and the design and conduct of new trials.



View larger version (23K):
[in this window]
[in a new window]
  		FIG. 1 The Jadad Scale for the assessment of the quality of RCTs (modified from Jadad, 1996).

 

   Acknowledgments
 
I would like to thank Professors Lennon and Ashby for their supervision of the work that has contributed to this paper. I would also like to thank Anne-Marie Glenny and Helen Worthington for their comments on this work.


   References
Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 Conclusions
 References
 
1 Clarke M, Oxman AD, eds. Assessment of Study Quality. Cochrane Reviewers’ Handbook 4.1.5 (updated April 2002); Section 4, The Cochrane Library, Issue 2. Oxford: Update Software, 2002. Updated quarterly.

2 Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996; 276: 637–639.[CrossRef][Medline]

3 Moher D, Schulz KF, Altman DG; CONSORT GROUP (Consolidated Standards of Reporting Trials). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001; 285:1987–91.[Abstract/Free Full Text]

4 Deeks JD, Sheldon TA. Guidelines for Undertaking Systematic Reviews of Effectiveness, Version 4. York: York Centre for Reviews and Dissemination 1995; 35–42.

5 Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DC. Users’ guide to the medical literature: IX A method for grading healthcare recommendations. JAMA 1995; 274: 1800–4.[CrossRef][Medline]

6 Harrison JE, Ashby D, Lennon MA. An analysis of papers published in the British and European journals of orthodontics. Br J Orthod 1996; 23: 203–9.[Abstract]

7 Greenhalgh T. How to Read a Paper: the basis of evidence based medicine. London: BMJ Publishing Group, 1997.

8 Antczak-Bouckoms AA. The anatomy of clinical research. Clin Orthodont Res 1998; 1: 75–9.

9 The Cochrane Reviewers’ Handbook Glossary (Updated March 2001), The Cochrane Library, Issue 2 Oxford: Update Software, 2002. Updated quarterly.

10 Schultz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408–12.[Abstract]

11 Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352: 609–13.[CrossRef][Medline]

12 Moher D, Jadad AR, Nichol G, Penman M, Tugwell P Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales of and checklists. Controll Clin Trials 1995; 16: 62–73.

13 Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, et al. Assessing the quality of randomized controlled clinical trials: Is blinding necessary? Controll Clin Trials 1996; 17: 1–12.

14 Jüni P, Altman DG, Egger M. Assessing the quality of randomized controlled trials. In: Egger M, Davey Smith G, Altman DG. (eds), Systematic Reviews in Health Care. London: BMJ Publishing Group, 2001: 91–2.

15 Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999; 282:1054–60.[Abstract/Free Full Text]

16 Newcombe RG. Guest editorial: reporting of clinical trials in the JO—the CONSORT guidelines. J Orthodont 2000; 27: 69–70.[Free Full Text]

17 Cochrane Collaboration Oral Health Group (CCOHG). Oral Health Review Group Journal Hand Searchers’ Manual. Manchester: Cochrane Oral Health Group, 1999.

18 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–74.[CrossRef][Medline]

19 Huwiler-Muntener K, Jüni P, Junker C, Egger M. Quality of reporting of randomized trials as a measure of methodological quality. JAMA 2002; 287: 2801–4.[Abstract/Free Full Text]

20 Sánchez-Thorin JC, Cortés MC, Montenegro M, Villate N. The quality of reporting of randomized clinical trials published in Ophthalmology. Ophthalmology 2001; 108: 410–15.[CrossRef][Medline]

21 Moher D, Jones A, Lepage L, and the CONSORT Group (Consolidated Standards of Reporting Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA 2001; 285: 1992–5.[Abstract/Free Full Text]

22 Antczak AA, Tang J, Chalmers TC. Quality assessment of randomized control trials in dental research I: Methods. J Periodont Res 1986; 21: 305–14.[Medline]

23 Antczak AA, Tang J, Chalmers TC. Quality assessment of randomized control trials in dental research II: results: periodontal research. J Periodont Res 1986; 21: 315–21.[Medline]

24 Koes BW, Assendelft WJJ, van der Heijden GJMG, Bouter LM, Knipschild HJ. Spinal manipulation and mobilization for back and neck pain: a blinded review. BMJ 1991; 303: 1298–303.

25 Soloman MJ, Laxamana A, Devore L, McLeod RS. Randomized controlled trials in surgery. Surgery 1994; 115: 707–12.[Medline]

26 Schulz KF, Chalmers I, Grimes DA, Altman DG. Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA 1994; 272: 125–8.[Abstract]

27 Hall JC, Mills B, Nguyen H, Hall JL. Methodological standards in surgical trials. Surgery 1996; 119: 466–72.[CrossRef][Medline]

28 Gluud C, Nickolova D. Quality assessment of reports on clinical trials in the Journal of Hepatology. J Hepatol 1998; 29: 321–7.[CrossRef][Medline]

29 Dickinson K, Bunn F, Wentz R, Edwards P, Roberts I. Size and quality of randomized controlled trials in head injury: review of published studies. BMJ 2000; 320:1308–11.[Abstract/Free Full Text]

30 Thornley B, Adams C. Content and quality of 2000 controlled trial in schizophrenia over 50 years. BMJ 1998; 317: 1181–4.[Abstract/Free Full Text]

31 Rosenberg WC, Donald A. Evidence based medicine: an approach to clinical problem-solving. BMJ 1995; 310: 1122–6.[Free Full Text]

32 Khan KS, Daya S, Jadad AR. The importance of primary studies in producing unbiased systematic reviews Arch Int Med 1996; 156: 661–6.

Received August 14, 2002; accepted May 1, 2003




This article has been cited by other articles:


Home page
 Am J EpidemiolHome page
L. L. Gluud
Bias in Clinical Intervention Research
Am. J. Epidemiol., March 15, 2006; 163(6): 493 - 501.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Harrison, J. E.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Harrison, J. E.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS