Journal of Pediatric Psychology, Vol. 28, No. 3, 2003, pp. 159-167
© 2003 Society of Pediatric Psychology
Clinical Trials in the Journal of Pediatric Psychology: Applying the CONSORT Statement
1 University of Toronto, 2 Dalhousie University and IWK Health Centre, 3 Hospital for Sick Children
All correspondence should be sent to Jennifer N. Stinson, Department of Anaesthesia, Hospital for Sick Children, 555 University Avenue, Toronto, Ontario, M5G 1X8. E-mail: jennifer.stinson{at}sickkids.ca.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodResultsReferences |
|---|
Objective To determine how well therapeutic randomized controlled trials (RCTs) in the Journal of Pediatric Psychology (JPP) met the CONSORT criteria as compared to pediatric trials in the Journal of Clinical and Consulting Psychology (JCCP), which served as a control. Methods Nine trials were found in JPP and 19 clinical trials were retrieved from JCCP. The modified Consort Checklist and Flow Chart was applied to each trial by two unblinded independent raters. Results The number of CONSORT items not reported between the two journals was strikingly similar. One half (11/22) of the CONSORT items were reported less than 25% of the time. Information provided on the flow of participants in the trials was similar across the two journals; however, JCCP more often reported on those assessed to be eligible overall,
2 = 5.241, p < .05. JPP reported on
recruitment 33.3% of the time, while it was not reported at all in
JCCP, 2 = 7.093, p < .05.
Conclusions Although many of the items in the CONSORT statement were
not adhered to by the two psychological journals, most of the CONSORT items
can clearly be applied to psychosocial trials. Key words: CONSORT statement; clinical trials.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodResultsReferences |
|---|
Randomized clinical trials (RCTs) are a powerful way of scientifically evaluating treatment effectiveness. If appropriately conducted, analyzed, and reported, the RCT can give clinicians valuable information about how to select the best treatment for individuals. The first RCT reputedly was conducted by James Lind in 1753. Lind's classic therapeutic experiment on sailors with scurvy provided clear evidence of the curative value of oranges and lemons (antiscorbutics) and was the first example of a therapeutic controlled clinical trial using human subjects.
Psychology has made considerable strides in using evidence to evaluate
treatments. The American Psychological Association developed a set of criteria
for considering psychological treatments as empirically supported, or
evidence-based. These criteria are often referred to as the Chambless criteria
after the chair of the Task Force on the Promotion and Dissemination of
Psychological Procedures
(1995
) and involve the
demonstration of statistical significance either in RCT group designs or a
large series (n 
9) of single case experiments, the development
of treatment manuals, clear specification of client characteristics, and the
demonstration of effects by at least two different investigators or
investigative teams (Chambless et al.,
1996,
1998). Beginning in 1999, the
Journal of Pediatric Psychology (JPP) published a series of
review articles on empirically supported treatments
(Spirito, 1999). Each review
is a systematic analysis of the literature using a variation of the Chambless
criteria and is accompanied by commentaries. While these reviews have been
widely cited, Drotar (2002)
argues that such scientific reviews could be further enhanced by inclusion of
information concerning effect sizes, theory specification, and clinical
significance. Moreover, while the RCT is the prime basis for determining
whether a treatment in psychology is deemed effective, the criteria for a
reasonable randomized trial for psychosocial interventions remain sketchy.
In the medical literature, there has been a concerted effort to develop
standards for improving the reporting of clinical trials because reviews had
shown that trials were inadequately reported, which further compounds the
problems arising from poorly conducted studies
(Schulz, Chalmers, Grimes, & Altman,
1994; Schulz, Chalmers, Hayes,
& Altman, 1995). Standards of reporting trials are intended to
provide guidance for the conduct of trials as well as the reporting of them
for publication. The most recent evidence-based effort to improve the
reporting of randomized clinical trials is the Consolidated Standards of
Reporting Trials (CONSORT) statement first published in 1996
(Begg et al., 1996). The
original CONSORT statement and its accompanying checklist and flow diagram of
participant progress through the stages of the trial were developed following
the merger of two different groups who had independently developed checklists
(Standards of Reporting Clinical Trials Group 1994;
Asilomar Working Group on Recommendations
for Reporting of Clinical Trials in Biomedical Literature, 1996).
The CONSORT statement has subsequently been adopted by most of the major
medical journals including the British Medical Journal
(Altman, 1996), Canadian
Medical Association Journal (Hutson & Hoey, 1996), Annals of
Internal Medicine, Journal of the American Medical Association, and
The Lancet (Rennie,
2001). The checklist applies to randomized trials of two parallel
groups, with modifications required for other designs such as crossover trials
and those with more than two treatment groups (Altman). There is also some
evidence that, subsequent to the adoption of the CONSORT statement, the
quality of reporting of trials improved
(Moher, Jones, & Lepage,
2001). After 5 years of experience with the CONSORT statement, it
was recently modified and includes definitions and rationale for each
checklist item, as well as published exemplars of reporting
(Altman et al., 2001). Readers
are directed to the associated Web site for the statement and explanatory
document
(www.consort-statement.org).
The CONSORT Checklist and the CONSORT Flow Diagram are contained in
Table I and
Figure 1.
|
|
Psychological journals have not yet adopted the CONSORT statement, perhaps because elements of it may not be appropriate for psychosocial interventions. We undertook this study to determine how well articles in the defined journals met the CONSORT criteria and to discern if specific criteria might not be appropriate for the studies reported in these journals.
|
Method |
|---|
|
TopAbstractIntroductionMethodResultsReferences |
|---|
One of the authors (Stinson) independently searched all issues of JPP and Journal of Clinical and Consulting Psychology (JCCP) between 1998 and 2001. All pediatric randomized controlled trials were selected. This recent sampling was taken to reflect the current state of pediatric clinical trial methodology in psychology. JCCP was chosen as the comparator as it is widely viewed as the most prestigious psychology journal publishing intervention trials and has a high citation impact factor of 4.390. The citation impact factor for JPP is 1.541 (Institute for Scientific Information, 2000
). A randomized controlled trial was defined as a group design study in which children or their parents were randomly allocated to receive one of several preventative or therapeutic clinical interventions compared to a control condition. A total of 9 (64%) therapeutic clinical trials were found in JPP out of a total of 14 RCTs, and 19 (16%) pediatric clinical trials were retrieved from JCCP out of a total of 118 randomized controlled studies. However, when you compare the number of pediatric clinical trials to the total number of published studies in both journals, they are strikingly similar (9 out of 166 [5.4%] in JPP compared to 19 out of 406 [4.7%] in JCCP). The denominator in this instance excluded special series articles, commentaries, and editorials.
Table II lists the retrieved
articles and describes their content for the two journals. The trials covered
a range of clinical problems and interventions. Each article was evaluated
independently by two of the authors (Stinson and Yamada), who were not blinded
to the journal type. Interrater reliability for assessment of each item in the
revised CONSORT Checklist and CONSORT Flow Chart
(Altman et al., 2001) was
determined for 10 randomly chosen articles using the kappa statistic. Many of
the individual items in the CONSORT Checklist have multiple criteria. For
these questions, we assessed each criterion within the item separately.
Disagreements were resolved by consensus. Each item was assigned a yes or no
response depending on whether the authors reported it. We compared overall
completeness of reporting between the two journals using Fisher's exact
test.
|
|
Results |
|---|
|
TopAbstractIntroductionMethodResultsReferences |
|---|
The number of pediatric clinical trials found between the two journals was similar. Twenty trials in JCCP were pediatric-focused clinical trials; however, one trial was excluded as two studies reported on the same clinical intervention with one reporting immediate outcomes and the other on long-term follow-up results. Of the 14 randomized trials in JPP, only 9 met the inclusion criteria, with the remaining 5 being explanatory experimental randomized trials. Although clinical trials are more often submitted to JCCP due to its high citation impact factor, the small number of trials in the pediatric age range may reflect the general paucity of clinical trials on children. Interrater reliability for the subset of 10 randomly chosen papers was
= .90 ( values ranging from .72 to
1.00). Over half of the trials (17/28) were parallel two-group designs. The
majority (24/28) of trials used standard or alternative care control groups
rather than placebos. The proportion of trials that adhered to each individual
item in the CONSORT statement is listed in
Table III.
|
The true benefit of a well-conducted randomized trial can be realized only when it has been reported clearly and comprehensively. Well-reported studies enable clinicians to adequately assess the validity of the published trials before deciding whether to apply their results to clinical practice (Bhandari, Guyatt, Lochner, Sprague, & Tornetta, 2002). We have shown that most reports of randomized trials in clinical psychology fail to provide the information necessary for the reader to judge study validity and to effectively apply the results to patient care. Fifty percent of the published trials met fewer than half of the CONSORT criteria. Even more striking was the finding that 20 of the 43 items were reported in less than 25% of the articles (see Table II).
The number of CONSORT items consistently not reported between the two
journals was strikingly similar; however, there were two exceptions. First,
there was a significant difference found in reporting of information on
recruitment. JPP reported on recruitment 33.3% of the time, while it
was not reported at all in JCCP, = 7.093, p < .05.
Knowing when a study took place and over what period participants were
recruited and followed not only provides a historical context but also aids in
generalizability of trial results. Second, there was a significant difference
in the overall reporting of those assessed for overall eligibility.
JCCP more often reported on those found to be eligible overall in
terms of participant flow, = 5.241, p < .05. Although this
number is not relevant for assessing the internal validity of a trial, it is
useful for estimating generalizability (Egger, Juni, & Bartlett,
2001).
Fewer than 15% of the trials that we examined provided details on the
method used to generate random allocation sequence. This finding is even lower
than that reported in published specialty journals and general medical
journals (Altman et al., 2001).
Moreover, only one of the trials provided details that allowed readers to
ascertain whether randomization was effectively concealed. Studies tend to
show larger treatment effects when randomization is not concealed
(Rennie, 1996). In addition,
unblinded studies also tend to show larger treatment effects. We found that
fewer than 21.4% of the trials in JPP provided details allowing the
readers to determine whether all of the relevant goups (e.g., patients,
clinicians, and those involved in ascertaining outcome status) were blind with
regard to allocation. While blinding of clinicians and patients is often
impossible in most psychological trials, blinded assessment of outcomes is
almost always feasible. Second, trials in JCCP more often reported on
statistical methods to compare groups for primary outcome(s) and additional
analyses. This finding may be due to the fact that the plan for analyses was
commonly interwoven throughout the Results section in JPP, making it
difficult to decipher planned analyses from additional subgroup or adjusted
analyses.
The final difference relates to the use of the CONSORT Flow Diagram. The
CONSORT Flow Diagram showing the flow of participants from enrollment to
analysis is an important element of the CONSORT standards for reporting of
clinical trials. The flow diagram is intended to provide readers with a clear
picture concerning the progress of all participants from the time they are
randomized until the end of their involvement in the trial
(Moher, 1998). We found that
the majority of trials did not include a flow diagram. The two journals were
similar in the degree to which they provided information on the progress of
participants through the trial. However, this information was not always
explicit or clearly outlined in the Methods section. JCCP provided
more information regarding the numbers of participants lost to follow-up,
those excluded from the analysis, and those included in the main analysis.
This finding is concerning because the latter count is essential for
appraising whether a trial has been analyzed by intention-to-treat.
Furthermore, Egger et al. (2001) have shown in a study of RCTs published in
five general and internal medicine journals that CONSORT flow diagrams were
associated with improved quality of reporting of RCTs.
Although many of the items in the CONSORT statement were not adhered to by
the two psychology journals, most of the CONSORT items can be applied to
psychosocial trials. The primary reporting omissions centered on bias
reduction methods, such as randomization sequence generation, allocation
concealment, and implementation and blinding. Similarly, our findings confirm
a concern raised by others that discussion sections of RCTs are not sufficient
in their consideration of the other extant RCTs
(Hawkins, 1999;
Moher, Jones, & Lepage,
2001). Although the majority of trials studied here structured the
discussion section as recommended in the CONSORT statement, none included a
systematic review that combined the results of the current study with results
of all previous relevant studies. Moreover, none of the studies reported on
adverse events or side effects in each intervention group. Whereas
psychological therapies are assumed to be benign, most interventions have
unintended side effects that can range from being overly burdensome to
participants (e.g., often a reason for dropping out) to harmful
(Imrie et al., 2001).
Several items, including the use of only two-group parallel designs, the
blinding of patients and clinicians, and incorporating a systematic review in
the interpretation of the findings, are likely to be difficult for many
clinical trials of psychosocial interventions. Moreover, although Item 4 of
the CONSORT statement requires the reporting of the "precise details of
the interventions intended for each group and how and when they were actually
administerd," more reporting detail is needed for psychological
interventions to ensure intervention fidelity and replication in clinical
practice. This information was provided in over half of the articles but is
not specified in the checklist and would be useful to the reader. For example,
42% of the trials in JCCP report on treatment fidelity, while it was
reported only 11% of the time in JPP. The Evidence-Based Behavioral
Medicine (EBBM) Committee of the Society of Behavioral Medicine has proposed
five additional CONSORT items for the review and reporting of psychosocial
interventions (Davidson et al., in
press). These include training of treatment providers, supervision
of treatment providers, patient and therapist allegiance or preference, manner
of testing, and success of treatment provider fidelity and patient adherence.
In addition, the use of placebo control groups remains a challenge for
psychological interventions. The majority of trials used standard or
alternative care control groups possibly due to difficulties developing
credible placebo (e.g., attention) control groups. Finally, another area that
the CONSORT statement fails to address is the reporting of clinical
significance. Clinical significance goes beyond statistical significance to
identify whether the statistically significant difference is large enough to
have implications for patient care (Sloan
et al., 2002). The journals only reported on clinical significance
25% (7/28) of the time. Therefore, in addition to adopting EBBM criteria, it
would be important to include an additional item on the reporting of clinical
significance for psychological trials.
In conclusion, the reporting of critical details of RCTs in pediatric psychology, while only marginally below that of the major clinical psychology journal, could be improved. Although the CONSORT statement was devised for medical interventions, it may help to ensure consistent and full reporting of the results of psychological RCTs. Standardizing the information that allows the reader to decide if an RCT has merit can only improve our field. These CONSORT-based recommendations hold immense value to advancement of the identification and dissemination of evidence-based pediatric interventions. However, the CONSORT statement as it currently stands is not a panacea that will cure all the ills of the design, reporting, and review of RCTs in psychological journals. Further modification of the CONSORT criteria to include EBBM items on treatment fidelity and clinical significance would be important for the reporting of psychosocial interventions. Joining the evidence-based movement, with the CONSORT guidelines as a first step, will ultimately improve the health of children and their families.
|
Acknowledgments |
|---|
Jennifer Stinson is funded by Canadian Nurses Foundation/Hospital for Sick Children's Foundation/Canadian Institutes of Health Research Fellowship and the Hospital for Sick Children Clinician Scientist Fellowship. Patrick McGrath is funded by a Canadian Institutes of Health Research Distinguished Scientist Award.
Received July 19, 2002; revision received October 24, 2002; accepted November 7, 2002
|
References |
|---|
|
TopAbstractIntroductionMethodResultsReferences |
|---|
Altman, D. C. (1996). Better reporting of randomised controlled trials: The CONSORT statement. BMJ, 313, 570-571.
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F.,
Elbourne, D., et al. (for the CONSORT Group). (2001). The revised
CONSORT statement for reporting randomized trials: Explanation and
elaboration. Annals of Internal Medicine,
134, 663-694.
The Asilomar Working Group on Recommendations for Reporting of
Clinical Trials in Biomedical Literature. (1996). Checklist of
information for inclusion in reports of clinical trials. Annals of
Internal Medicine, 124,
741-743.
Begg, C., Cho, M., Eastwood, S., Horton, R., Moher, D., Olkin, I.,
et al. (1996). Improving the quality of reporting of randomized
controlled trials: The CONSORT statement. Journal of the American
Medical Association, 276,
637-639.
Chambless, D. L., Baker, M. J., Baucom, D. H., Beutler, L. E., Calhoun, K. S., Crits-Christoph, P., et al. (1998). Update on empirically validated therapies, II. Clinical Psychologist, 51(1), 3-16.
Chambless, D. L., Sanderson, W. C., Shoham, V., Bennet Johnson, S., Pope, K. S., Crits-Christoph, P., et al. (1996). An update on empirically validated therapies. Clinical Psychologist, 49, 5-18.
Davidson, K., Goldstein, M., Kaplan, R., Kaufmann, P., Genell Knatterud, G., Orleans, T., et al. (In press). Evidence-based behavioral medicine: What it is, and how do we get there? Annals of Behavioral Medicine.
Drotar, D. (2002). Enhancing reviews of psychological
treatments with pediatric populations: Thoughts on next steps.
Journal of Pediatric Psychology,
27(2),
167-176.
Hawkins, B. S. (1999). The CONSORT statement: Will it
lead to improved reporting of clinical trials in ophthalmology?
Archives of Ophthalmology, 117,
677-680.
Huston, P., & Hoey, J. (1996). CMAJ endorses the CONSORT statement. Consolidation of standards for reporting trials. Canadian Medical Association Journal, 155, 1277-1282.[Medline]
Imrie, J., Stephenson, J. M., Cowan, F. M., Wanigaratne, S.,
Billington, A. J. P., Copas, A. J., et al. (for the Behavioural Intervention
in Gay Men Project Study Group). (2001). A cognitive behavioural
intervention to reduce sexually transmitted infections among gay men:
Randomized trial. British Medical Journal,
322,
1451-1454.
Institute for Scientific Information. (2000). Journal Citation Reports: Science Edition. Philadelphia, PA: Author.
Lind, J. (1753). A treatise of the scurvy. Edinburgh: Sands, Murray & Cochran.
Moher, D. (1998). CONSORT: An evolving tool to help
improve the quality of reports of randomized controlled trials. Consolidated
Standards of Reporting Trials. Journal of the American Medical
Association, 279(18),
1489-1491.
Moher, D., Jones, A., & Lepage, L. (2001). Use of
the CONSORT statement and quality of reports of randomized trials: A
comparative before-and-after evaluation. Journal of the American
Medical Association, 285(15),
1992-1995.
Rennie, D. (1996). How to report randomized controlled
trials. The CONSORT statement. Journal of the American Medical
Association, 276(8),
649.
Rennie, D. (2001). CONSORT revised—Improving the
reporting of randomized trials. Journal of the American Medical
Association, 285(15),
2006-2007.
Schulz, K. F. (1996). Subverting randomization in controlled trials. Journal of the American Medical Association, 274(18), 1456-1458.
Schulz, K. F., Chalmers, I., Grimes, D. A., & Altman, D. G.
(1994). Assessing the quality of randomization from reports of
controlled trials published in obstetrics and gynecology journals.
Journal of the American Medical Association,
272(2),
125-128.
Schulz, K. F., Chalmers, I., Hayes, R. J., & Altman, D. G.
(1995). Empirical evidence of bias: Dimensions of methodological
quality associated with estimates of treatment effects in controlled trials.
Journal of the American Medical Association,
273(5),
408-412.
Sloan, J. A., Cella, D., Frost, M., Guyatt, G. H., Sprangers, M.,
Symonds, T., et al. (2002). Assessing clinical significance in
measuring oncology patient quality of life: Introduction to the symposium,
content overview and definition of terms. Mayo Clinic
Proceedings, 77(4),
367-370.
Spirito, A. (1999). Introduction to series. Journal of Pediatric Psychology, 24, 87-90.
Standards of Reporting Trials Group. (1994). A
proposal for structured reporting of randomized controlled trials.
Journal of the American Medical Association,
272(24),
1926-1931.
Task Force on Promotion and Dissemination of Psychological Procedures. (1995). Training in and dissemination of empirically validated psychological treatments: Report and recommendations. Clinical Psychologist, 48, 3-23.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  D. Drotar Editorial: How to Write an Effective Results and Discussion for the Journal of Pediatric Psychology J. Pediatr. Psychol., May 1, 2009; 34(4): 339 - 343. [Full Text] [PDF]
|
|
|
|
|
|
D. Drotar Editorial: How to Report Methods in the Journal of Pediatric Psychology J. Pediatr. Psychol., April 1, 2009; 34(3): 227 - 230. [Full Text] [PDF]
|
|
|
|
 |
|
 T. Bustos, T. Jaaniste, K. Salmon, and G. D. Champion Evaluation of a Brief Parent Intervention Teaching Coping-Promoting Behavior for the Infant Immunization Context: A Randomized Controlled Trial Behav Modif, July 1, 2008; 32(4): 450 - 467. [Abstract] [PDF]
|
|
|
|
|
|
T. Wysocki Editorial: Journal of Pediatric Psychology Statement of Purpose Section on Randomized Trials J. Pediatr. Psychol., January 1, 2008; 33(1): 12 - 15. [Full Text] [PDF]
|
|
|
|
|
|
R. T. Brown Journal of Pediatric Psychology (JPP), 2003 2007: Editor's Vale Dictum J. Pediatr. Psychol., December 11, 2007; (2007) jsm111v1. [Full Text] [PDF]
|
|
|
|
 |
|
 K. Whittaker, C. Sutton, and C. Burton Pragmatic randomised controlled trials in parenting research: the issue of intention to treat. J Epidemiol Community Health, October 1, 2006; 60(10): 858 - 864. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. P. Tercyak, J. R. Donze, S. Prahlad, R. B. Mosher, and A. T. Shad Identifying, Recruiting, and Enrolling Adolescent Survivors of Childhood Cancer into a Randomized Controlled Trial of Health Promotion: Preliminary Experiences in the Survivor Health and Resilience Education (SHARE) Program J. Pediatr. Psychol., April 1, 2006; 31(3): 252 - 261. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. Drotar Commentary: Randomized Controlled Trials of Psychological Interventions with Pediatric Populations: The Time has Come and the Journal of Pediatric Psychology is Ready J. Pediatr. Psychol., July 1, 2005; 30(5): 409 - 412. [Full Text] [PDF]
|
|
|
|
|
|
L. J. Stark, D. M. Janicke, A. M. McGrath, L. M. Mackner, K. A. Hommel, and D. Lovell Prevention of Osteoporosis: A Randomized Clinical Trial to Increase Calcium Intake in Children with Juvenile Rheumatoid Arthritis J. Pediatr. Psychol., July 1, 2005; 30(5): 377 - 386. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. F. Patenaude and M. J. Kupst Psychosocial Functioning in Pediatric Cancer J. Pediatr. Psychol., January 1, 2005; 30(1): 9 - 27. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. T. Brown Editorial: A General Approach to Publication in the Journal of Pediatric Psychology: From the Process of Preparing Your Manuscript to Revisions and Resubmissions J. Pediatr. Psychol., January 1, 2004; 29(1): 1 - 5. [Full Text] [PDF]
|
|
|
|
|
|
P. J. McGrath, J. Stinson, and K. Davidson Commentary: The Journal of Pediatric Psychology Should Adopt the CONSORT Statement as a Way of Improving the Evidence Base in Pediatric Psychology J. Pediatr. Psychol., April 1, 2003; 28(3): 169 - 171. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||