| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Scientific Section |
Liverpool University Dental Hospital, Liverpool, UK
Address for correspondence: Jayne E. Harrison, Consultant Orthodontics, Liverpool University Dental Hospital, Liverpool, L3 5PS, UK., Email: jayne.harrison{at}rlbuht.nhs.uk
Received 19 November 2007; accepted 12 October 2008
 |
Abstract |
|---|
 Top Abstract IntroductionMethodResultsDiscussionConclusionsReferences |
|---|
Design: Retrospective, observational study.
Sample: Clinical trials, published in six dental journals. Three that adopted structured abstracts (BDJ, CPJ, JO) and three that remained unchanged (JDR, EJO, AJODO) between January 1995 and December 1998 (extended to December 2002 for the JO).
Intervention: Adoption of a structured abstract format.
Control: Continued use of a non-structured abstract format.
Method: A combination of handsearching and the Cochrane Collaboration Oral H1ealth Group’s Trials Register and/or CENTRAL were used to identify randomised controlled trials (RCTs) and controlled clinical trials (CCTs) over the selected time period. MEDLINE was used to identify clinical trials in the selected journals over the same time period.
Results: There was no significant difference in the sensitivity or yield of clinical trial retrieval in journals with either abstract format over time. However, there was a significant increase in precision in journals that did not change their format (OR=4.96 (95% CI 1.18, 20.86) but not those that did. There was no significant difference in the sensitivity or yield of clinical trial retrieval either before or after the change in format or precision of retrieval before the change. However, in the later period, the precision of retrieval was significantly better in journals with unstructured abstracts compared to those with structured abstracts (OR=0.17 (95% CI 0.04, 0.7).
Conclusion: The use of a structured abstract format does not improve the sensitivity, precision or yield of retrieval of clinical trials from MEDLINE.
Key words: Structured abstract, handsearching, electronic retrieval
|
Introduction |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionsReferences |
|---|
A structured abstract is an abstract that describes a study using specific content headings rather than paragraph format. Structured abstracts were proposed by the Ad Hoc Working Group for Critical Appraisal of the Medical Literature in 1987.1
In March 2004, the editors of the Annals of Internal Medicine proposed a new type of structured abstract – a critical one.2 The reason behind this proposal was a concern that abstracts might give the readers the impression that research has no flaws. A new ‘Limitations’ section, located immediately before ‘Conclusions’, was therefore added (Table 1). Reflecting on the limitations of a study can assist readers in deciding whether results apply to their patient or not.
|
Dental and orthodontic journals have lagged behind in adopting a structured abstract format but a number have implemented this format for abstracts of articles reporting original research or reviews. They include the Journal of Periodontology; Journal of Dentistry; Journal of Oral Pathology and Medicine; Journal of Prosthetic Dentistry; Journal of Orthodontics (formerly British Journal of Orthodontics),4 Clinical Orthodontics and Research,5 Cleft Palate Craniofacial Journal (CPJ),6 and more recently the American Journal of Orthodontics and Dentofacial Orthopaedics (AJODO).7
Figure 1 (Ref. 8) shows the conversion of an unstructured abstract for a randomised clinical trial published in the AJODO prior to the adoption of a structured abstract format and the CONSORT guidelines (The AJODO changed to a structured abstract format in January 2005).
|
The more help authors give by producing good abstracts, the more accurately the article is likely to be indexed and a search will retrieve that article from the database. It is therefore important for editors and publishers to have strict guidelines to ensure that authors are as meticulous in writing these abstracts as they are in conducting the original research.
In view of the limitations of electronic searching, where thorough searching is paramount, electronic searching must be extended to databases beyond MEDLINE and also accompanied by handsearching of journals. Handsearching is the inspection of a journal, page by page, in order to detect published and unpublished CCTs and RCTs, or information on trials such as abstracts and correspondence.
The Cochrane Oral Health Group is one of the specialty-based groups contributing to the International Cochrane Collaboration. The objectives of this group are to create a database of all clinical trials related to oral health and to prepare systematic reviews of topics of interest. The handsearchers send the information to OHG’s Trials Search Coordinator. Any trials which have not been identified previously by electronic searching are downloaded into the OHG’s trials register, which is in turn uploaded into the CENTRAL database in The Cochrane Library.9 Thus the OHG’s Trials Register is a compilation of trials found by handsearching journals and electronic searching several databases.
The objectives of this study were to assess whether structured abstracts improved the sensitivity, precision and yield of retrieving clinical trials, using electronic searches e.g. MEDLINE, from dental journals.
The null hypothesis was that there is no difference in retrieval of articles reporting clinical trials that were published with structured abstracts compared with those published with non-structured abstracts against the alternative hypothesis of a difference.
|
Method |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionsReferences |
|---|
Sample and setting
Clinical trials, published in six dental journals between January 1995 and December 1998 were identified. However, the time period examined for the Journal of Orthodontics was extended from January 1995 to December 2002 due to the delay in implementation of the change in format. These time periods were chosen so that trials published in the issues a year prior to and a year after the date of change in abstract format were included. This also allowed changes over time to be assessed.
The journals selected included three that had adopted a structured abstract format and three that continued to use unstructured abstracts in this period. There were two general dental journals – British Dental Journal (BDJ) and Journal of Dental Research (JDR); three specialist orthodontic journals – Journal of Orthodontics (JO) [formerly the British Journal of Orthodontics], European Journal of Orthodontics (EJO), American Journal of Orthodontics and Dentofacial Orthopedics (AJODO); and a specialist journal related to orthodontics – The Cleft Palate-Craniofacial Journal (CPJ). These journals were selected to represent a range of dental journals. They included journals that were of particular interest to the authors and ones with a high impact factor. The journals that had adopted a structured abstract format included BDJ, CPJ and JO. The journals that continued to use a non-structured abstract format included the EJO, AJODO, and JDR.
Sample size
The sample size was determined by the number of clinical trials published in the relevant journals over the test period.
Methods
All RCTs and CCTs were identified through hand-searching and use of the Cochrane Collaboration Oral Health Group’s Register for each journal over the test period. One author (HAS) handsearched the CPJ (1995–1997, January and September 1998) and the JO (1999–2002) to update the OHG’s Trials Register and include the RCTs and CCTs for the test period of this study. The results for the remaining papers were obtained from the Cochrane Collaboration Oral Health Group’s Register.
MEDLINE was used to identify RCTs and CCTs in the selected journals over the relevant time periods. The searches were limited to identifying studies in humans from 1995–1998 for the BDJ, CPJ, EJO, AJODO, JDR and from 1995–2002 for the JO. The identified literature was then further limited to MEDLINE ‘Publication Type’ – ‘clinical trial’ and then to ‘randomised controlled trial’.
RCTs and CCTs, identified through handsearching, were collated for each journal and the following criteria recorded: Author(s), Title of article, Reference Classification – CCT or RCT, Year, Identifed up by MEDLINE (Yes/No). In addition to this any articles that had been identified by MEDLINE, but not handsearching, were recorded. The full papers of all the aforementioned articles were then obtained and examined in order to determine whether it was a report a clinical trial or if the study had been misclassified.
Reliability
Sixteen clinical trials were reassessed for the reliability study. This represented a random sample of 9% of the total number of clinical trials. A randomisation list was prepared by one author (JEH) and each article was re-evaluated by a second author (HAS), who was blinded to the original findings, at least three months after the initial search.
Statistical analysis
The sensitivity, precision and yield of retrieval of clinical trials by MEDLINE were calculated for each of the six journals before and after the change to a structured abstract format.
Sensitivity is defined as the proportion of positives (clinical trials) that are correctly identified by a method.
 |
Precision is the number of relevant items (clinical trials) retrieved out of the actual number of items (all reports) identified by a method.
 |
Yield looks at how the use of a specific method increases the number of clinical trials retrieved.
 |
Key
A method=handsearching or MEDLINE
CTs=clinical trials
Intra-journal comparison. Odds ratios and 95% confidence intervals were calculated to assess whether there was a statistically significant difference in the sensitivity, precision and/or yield, of retrieval of clinical trials by MEDLINE:
Percentage agreement scores were calculated to assess intra-examiner agreement for the MEDLINE retrieval during the reliability study.
|
Results |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionsReferences |
|---|
Total number of clinical trials
The combined results of handsearching each issue of the six chosen journals and the electronic MEDLINE search, between January 1995 and December 1998 (extended to December 2002 for JO), identified 187 clinical trials.
Handsearching and the OHG register identified 183 trials. The MEDLINE search identified 116 of the 183 (63.4%) trials that had been found by handsearching journals together with an additional 31 citations. The full papers of the 31 unmatched MEDLINE citations were examined and of these 27 (87.1%) were found not to be clinical trials. These included thirteen observational, nine retrospective and five in vitro studies. The remaining four (12.9%) were found to be clinical trials (three RCTs and one CCT; see Figure 2).
|
).
|
].
|
).
|
).
|
]. Finally, there was no significant difference in the yield between journals that changed and those that remained unchanged prior to [odds ratio=0.68 (95% CI 0.21, 2.19] or following the change to a structured abstract format [odds ratio=1.25 (95% CI 0.46, 3.37; Table 7].
|
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionsReferences |
|---|
Limitations of the study
This was a retrospective study, which, by its nature was open to bias. However, all clinical trials were identified by hand searching and electronic searching (using MEDLINE). It is possible that mistakes were made and articles that should have been included in the sample were omitted or not found. Conversely, some studies may have been included that were not clinical trials. However, the likelihood of including non-clinical trials was small, as 97.9% of the clinical trials were identified by handsearching. The remaining 2.1% that were identified by MEDLINE were then handsearched to confirm they were clinical trials.
The study was open to an element of human and computer error. Care was needed when recording the comparisons of the OHG handsearching results with the MEDLINE search results. To minimise such error each assessment was limited to 15 clinical trials at any one time. This was then followed by a rest period.
Computer error was minimised by performing a reliability study on a random sample of 9% of the total number of clinical trials. There was 100% agreement with the original search suggesting 100% reliability of the MEDLINE search strategy. This was carried out at least three months after the original searches to take into account any changes with time in the principal investigator’s searching ability and memory of having identified particular trials previously.
Comment should also be made about the different time period for the JO sample. The original editorial in the JO4 announced the intent to change to a structured abstract but the actual change didn’t occur until September 2001 following a change in Editor. This is in contrast to the editorials in the other journals5–7 that announced the change in format of the abstracts that occurred in that edition of the journal. It is for this reason that the time period for the JO was extended to December 2002.
The sample size was determined by the number of clinical trials published in the relevant journals over the test period. However, the numbers in each journal were relatively small. A study by Markey and Harrison10 highlighted the difficulty in selecting dental journals that contained sufficient numbers of clinical trials for use in research. They revealed the limited number of clinical trials that are available in the various dental journals.
The small sample size in the present study could have resulted in the study having insufficient power to detect a difference, resulting in a type II error. Lack of evidence of a difference in effect does not necessarily mean that that there was no difference in effect.11 This then raises the question, could extending the time period over which the study was set and therefore increasing the sample size, have lead to a significant difference in retrieval of clinical trials between the non-structured and structured abstract groups? To test this theory the number of trials was firstly increased by a factor of 10 and then 20, etc. It was only when increased by a factor of 100 that a significant difference in the yield was found. The number of trials required to turn the non-significant results to significant is very high and is unlikely to be attainable in terms of the number of trials currently available for the post change data because most of the journals that changed to a structured abstract format only did so in the last 5–9 years. This suggests that to date, the results are a true negative, but that in time may become positive in favour of structured abstracts, once significant numbers of clinical trials have been published but this may be several years hence. By undertaking a retrospective power calculation this study had a power of 80% to detect a 20% increase in sensitivity.
The increase in precision of retrieval in the journals that did not change there format over time is an interesting finding. However, the exact reason for this is difficult to explain. There was no change in editor for the respective journals over this time period.
Comparison with other studies
In comparing our findings with those of previous studies there are areas of both agreement and disagreement. Dickersin et al.,12 found that approximately half of relevant controlled trials on a topic may be missed by an electronic search even though most of the missed citations were in the database. Similarly, Bickley and Harrison13 searched four leading orthodontic journals on MEDLINE using the indexing terms, ‘Randomised controlled trial’ or ‘Controlled Clinical Trial’ in the Publication Type field. They then compared the results to handsearching the four journals. They found that MEDLINE picked up 39.5% of the citations that had been found by handsearching together with an additional 12.5% of unmatched records. Of the unmatched records 82.4% were found not to be clinical trials. The remaining 15.8% were clinical trials and had been missed by handsearching. This took the percentage of trials retrieved by MEDLINE to 40.8% and those missed to 59.2%.
The findings of the present study are higher in terms of retrieval of clinical trials by MEDLINE at 64.2% with 35.8% being missed. However, the figures for false positives (not clinical trials) retrieved by MEDLINE in the present study (18.4%) are lower than those found by Bickley and Harrison (82.4%).13
The results of the present study confirm the view that electronic searching alone is likely to miss a substantial proportion of the clinical trials available as well as picking up a high percentage of trials that are not relevant. It also highlights the fact that handsearching, although regarded as the ‘gold standard’ is not 100% effective. Both electronic and handsearching should therefore be used in order to maximise retrieval of all the available evidence when undertaking systematic reviews.
To date, research on structured abstracts has focused on information content and has reported improvements in the information contained in them when compared to traditional abstracts. A recent study by Sharma and Harrison14 found that scores for the abstracts from journals that maintained the use of a non-structured abstract did not increase significantly with time whereas the scores for the abstracts from the journals that adopted the structured abstract format did increase significantly. It was, therefore, likely to be the change in abstract format that had a positive effect on the quality score rather than an improvement occurring over time irrespective of format. They concluded that structuring abstracts does improve their quality.
One of the main reasons for the introduction of a structured abstract format was to aid accurate indexing and retrieval of reports from computerised databases such as MEDLINE and EMBASE.15 However, research assessing a structured abstract format and associated citation retrieval is not abundant.
The character of structured abstracts in biomedical journals indexed in MEDLINE over a three-year period, was studied by Harbourt et al.16 The authors concluded that the presence of structured abstracts may be associated with other article characteristics leading to the assignment of higher numbers of MeSH headings, or may itself contribute to this phenomenon. However, the additional searchable terms are likely to assist in bibliographic retrieval. A further point highlighted by this article was that only two of the six journals required the original format of structured abstracts as published in the Annals of Internal Medicine in 1987. The remaining four specified a modification of this format. The authors felt that variations in the structured abstract formats were probably inconsequential to the reader, but will complicate more sophisticated use of structured abstracts in automated retrieval systems.
Wilczynski et al.17 found improvements in the retrieval characteristics of some MeSH and text-words associated with the use of structured abstracts, but they also found improvements over time. The authors, therefore, concluded that structured abstracts improved the retrieval properties of some, but not all, text-words and medical subject headings.
In comparing the findings of the present study with previous studies, the first point to note is the different method. The previous studies investigated whether structured abstracts improved citation retrieval by assessing the assignment and retrieval performance of MeSH terms within search strategies, whereas the current study assessed the retrieval of clinical trials using the indexing field ‘Publication type’.
The previous studies mentioned did not reach any definite conclusions about citation retrieval and the use of structured abstracts. Both studies16,17 alluded towards an improvement associated with the use of structured abstracts, but were unable to exclude other confounding variables such as changes with time.
Electronic searches mainly rely on two things – the controlled vocabulary (in MEDLINE MeSH) terms assigned to the article by professional indexers and descriptors (text-words) used by the author/s in the title and abstract. By producing informative abstracts containing systematic information authors will assist indexers in assigning the appropriate MeSH that may help to increase citation retrieval. Furthermore, all journals should specifically request the use of the new critical structured abstract format as proposed by the editors of the Annals of Internal Medicine (2004), in their instructions to authors.2
Implications in practice
The results of this study suggest that changing the format of abstracts doesn’t improve the retrieval of clinical trials from Medline which was one of the original reasons for introducing them.1,2,6 Nevertheless, we still advocate the use of structured abstracts because they have been shown to improve the information provided in abstracts14 and therefore make it easier for readers to evaluate whether an article is methodologically sound and applicable to their clinical situation which was another stated reason for introducing them.1,6
|
Conclusions |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionsReferences |
|---|
|
References |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionConclusionsReferences |
|---|
2 The Editors. Editorial. Addressing the limitations of structured abstracts. Ann Intern Med 2004; 140: 480–81.
3 Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomised controlled trials. The CONSORT statement. JAMA 1996; 276(8): 637–39.
4 Jones ML. Editorial. A new look for 1998. Br J Orthod 1998; 25(1) 55–56.
5 Tuncay OC. Instructions to authors. Clin Orthod Res 1998; 1: 74–75.
6 Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ. More informative abstracts revisited. Cleft Palate-Craniofacial J 1996; 33(1): 1–9.[Medline]
7 Turpin DL. Consensus builds for evidence-based methods. Am J Orthod Dentofacial Orthop 2004; 125: 1–2.[CrossRef]
8 This article was published in the American Journal of Orthodontics and Dentofacial Orthopedics 107: Wilson TG, Gregory RL. Clinical Effectiveness of fluoride-releasing. I. Salivary Streptococcus mutans numbers: 293–97, © Elsevier 1995.
9 The Cochrane Oral Health Group’s (OHG) Trials Register, available at: http://www.ohg.cochrane.org/trials.html.
10 Markey S, Harrison JE. An interdisciplinary analysis of papers published in dental journals [Undergraduate elective study]. Liverpool: University of Liverpool, 2001.
11 Altman DG, Bland JM. Statistics notes: Absence of evidence is not evidence of absence. BMJ 1995; 311: 485.
12 Dickersin K, Scherer R, Lefebvre C. Systematic reviews: Identifying relevant studies for systematic reviews. BMJ 1994; 309: 1286–91.
13 Bickley SR, Harrison JE. How to ...find evidence. J Orthod 2003; 30: 72–78.
14 Sharma S, Harrison JE. Structured abstracts: Do they improve the quality of information in abstracts. Am J Orthod Dentofacial Orthop 2006; 103: 523–30.
15 Scherer R, Crawley B. Reporting of randomised clinical trial descriptors and use of structured abstracts. JAMA 1998; 280(3): 269–72.
16 Harbourt AM, Knecht LS, Humphreys BL. Structured abstracts in MEDLINE, 1989–1991. Bull Med Libr Assoc 1995; 83(2): 190–95.[Medline]
17 Wilczynski NL, Walker CJ, McKibbon KA, Haynes RB. Preliminary assessment of the effect of more informative (structured) abstracts on citation retrieval from MEDLINE. Medinfo 1995; 8(pt 2): 1457–61.[Medline]
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
