Journal of Pediatric Psychology Advance Access originally published online on December 27, 2007
Journal of Pediatric Psychology 2008 33(7):688-693; doi:10.1093/jpepsy/jsm130
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This article appears in the following Journal of Pediatric Psychology issue: Series on Methodology Article [View the issue table of contents]
Brief Report: Reporting Practices of Methodological Information in Four Journals of Pediatric and Child Psychology
University of Kansas
All correspondence concerning this article should be addressed to Michael C. Roberts, PHD, ABPP, University of Kansas, Clinical Child Psychology Program, 2010 Dole Human Development Center, 1000 Sunnyside Ave., Lawrence, KS 66045-7555, USA. E-mail: mroberts{at}ku.edu
 |
Abstract |
|---|
 Top Abstract MethodResultsDiscussionReferences |
|---|
Objective To replicate Sifers, Puddy, Warren, and Roberts (2002
) examining reporting rates of demographic, methodological, and ethical information in articles published during 1997, and to compare these rates to those found in articles published during 2005, in order to determine whether and how reporting practices of these variables have changed over time. Methods We examined reporting demographic, methodological, and ethical information in articles in four journals: Journal of Pediatric Psychology, Journal of Clinical Child and Adolescent Psychology, Journal of Abnormal Child Psychology, and Child Development. Reporting rates during 2005 were compared to articles published during 1997. Results These four journals improved on many of the 23 variables compared to Sifers et al. including increases in the reporting of ethnicity, attrition, child assent procedures, socioeconomic status, reliability, and reward/incentive offered to participants. Conclusions Improvements in descriptive information have implications for interpretation, replication, and generalizability of research findings. Key words: demographics; ethics; journal content analyses; research methodology.
Information regarding the demographic characteristics of a research population is extremely important to the interpretation and generalization of findings. Providing this information to other professionals allows for the evaluation and generalization of scientific studies, which in turn furthers the knowledge and applicability of that field. Further, it is essential that consumers of research be aware of and understand for whom the findings of a study apply. For example, it is important to know the ethnic composition of a participant pool, in order to determine if a specific intervention would be effective for minority clients. Additionally, ethical considerations, such as the protection of participants and consumers, as well as methodological information, such as how participants are recruited and contacted for participation, are important aspects to report (Weil, Nelson, & Ross, 2002
). The collection and dissemination of ethical and methodological procedures allows other professionals to make informed decisions regarding the integrity and validity of research findings, and helps set standards and guidelines for subsequent research to follow, thus ensuring high-quality practices and results in future research.
The American Psychological Association (APA) holds that participant information is a vital component of research studies and publications. Specifically, the Publication Manual of the American Psychological Association (APA, 2001) requires that the methods sections of manuscripts include major demographic and methodological variables, such as the number of participants, method of selecting participants, agreements made, payments made, and attrition. Ethnicity, level of education, and geographic location are recommended but not stated as required. The APA Publication Manual best summarizes the importance of including this information by stating:
Appropriate identification of research participants and clientele is critical to the science and practice of psychology, particularly for assessing the results (making comparisons across groups); generalizing the findings; and making comparisons in replications, literature reviews, or secondary data analyses. The sample should be adequately described, and it should be representative (if it is not, give the underlying reasons). (p. 13)
With regard to the reporting of important demographic, methodological, and ethical variables, Sifers, Puddy, Warren, and Roberts (2002) stated that "it is worthwhile to periodically examine psychology publications for compliance to these standards of scientific communication" (p. 20). To examine this issue, Sifers et al. investigated the degree to which four major psychological journals complied with the established methodological standards as put forth by the APA Publication Manual. According to Sifers et al., many of the published articles neglected to include important methodological information in 1997, such as child assent procedures, parental consent procedures, attrition rates, the use of rewards or incentives, location of the study, and total contact time with participants. Other characteristics that also exhibited low rates of reporting included ethnicity, socioeconomic status, exclusion/inclusion criteria, and methods of contacting participants.
The present study re-examined the reporting rates of significant methodological, demographic, and ethical variables in the four journals previously examined by Sifers et al. (2002). Re-examination of the reporting rates of these variables is important in determining whether positive changes have been made over the past 8 years to rectify the low reporting rates.
|
Method |
|---|
|
TopAbstractMethodResultsDiscussionReferences |
|---|
Database
The database included all empirical research articles published during 2005 in Journal of Pediatric Psychology (JPP, 43 articles), Journal of Clinical Child and Adolescent Psychology (JCCAP, 42 articles), Journal of Abnormal Child Psychology (JACP, 46 articles), and Child Development (CD, 75 articles). Articles not of an empirical nature (i.e., review articles, commentaries, editorial articles, addresses, case studies, and studies that did not include human participants) were excluded from the database. In total, 206 research articles were coded and included in this study.
Coding Procedure
To ensure consistency with Sifers et al. (2002), the current study focused on the rates at which articles in the research literature reported the same methodological, ethical, and demographic variables, including the use of rewards/incentives, exclusion and inclusion criteria, attrition, consent and assent procedures, age, gender, ethnicity, and socioeconomic status (SES). Three additional variables were added to provide more detailed information regarding methodological and ethical information. These included whether a control group was utilized within the study, whether approval by an IRB or other institutional board was reported, and whether the validity of the dependent measures was reported.
The coding procedure was based on the methods used by Sifers et al. (2002). The coding procedure included the development and use of a 23-item checklist (see Table I for a list of these variables). Three coders used this checklist to identify which key components were included or described within the research article. Interrater reliability was calculated on 
10% (n = 22) of the articles. Percent agreement between coders ranged from 77% to 100% across the 23 items with a mean of 93%; 
-interrater reliability coefficients were also calculated, ranging from .45 to 1.0 with a mean of .81. Interrater reliability (;% agreement) for each coded variable is presented subsequently: (a) population (e.g., general/school children, medical condition: 1.0; 100%); (b) control group used (1.0; 100%); (c) number of participants (1.0; 100%); (d) ages (1.0; 100%); (e) gender (0.65; 95%); (f) ethnicity (0.65; 95%); (g) socioeconomic status (0.70; 86%); (h) location (urban or rural: 0.90, 95%); (i) location (within/outside United States: 1.0; 100%); (j) setting of sample (e.g., school, psychological clinic, hospital: 0.45; 91%); (k) identification/selection of sample (e.g., requested, teacher/staff recommended, records: 0.65; 95%); (l) method of contacting participants (e.g., information mailed home or sent home with child, advertisement, phone contact: 0.61; 77%); (m) reward/compensation offered for participation (1.0; 100%); (n) consent rate for participation (0.61; 82%); (o) child assent (1.0; 100%); (p) exclusion criteria (0.56; 77%); (q) inclusion criteria (0.72; 86%); (r) number of contacts requested (0.58; 86%); (s) total contact time (0.90; 95%); (t) attrition rate (0.82; 91%); (u) reliability of dependent measures used in study (1.0; 100%); (v) validity of dependent measures used in study (0.73; 86%); and (w) IRB approval (1.0; 100%).
|
|
Results |
|---|
|
TopAbstractMethodResultsDiscussionReferences |
|---|
The frequency and percentage of articles reporting each variable were calculated for each of the four journals individually, as well as together (see Table I). The results of the total percentages across all journals yielded a wide range in reporting practices of methodological and demographic information. Perhaps not surprisingly (Sifers et al., 2002
), information regarding the type of population sampled, the number of participants in the sample, and the ages of the participants were consistently reported in all four journals. In comparison, Sifers et al. reported rates of 99.2 and 98.5% for ages and populations of participants, respectively, across all four journals. Representing relatively high (but not uniform) reporting practices were variables such as gender, methods of identifying and selecting the sample, ethnic distribution, setting and location of the research project, reliability of the dependent measures, and the number of contacts requested. The remaining variables represented somewhat lower reporting practices, with the use of a reward/incentive, child assent procedures, the use of a control group, total contact time, and approval from an IRB or other research institution anchoring the bottom of the distribution. Overall, all four journals exhibited an increase (from 1997 to 2005) in reporting practices with regard to almost every demographic, methodological, and ethical variable examined. Decreases were noted in only two categories: the setting of the sample (2.9% decrease) and exclusion criteria (12.6% decrease). Across all journals, the largest increases in reporting rates were found with regard to the reliability information of dependent measures (24.2% increase), ethnic distribution of the sample (23.3% increase), reward/incentive offered (16.3% increase), child assent procedures (14.7% increase), attrition rate (12.2% increase), and SES (10.8% increase).
In terms of individual journals, reporting rates in 2005 declined compared to 1997 with regard to the setting of the sample (12.9% decrease) and identification/selection of the sample (12.3% decrease) in JPP; the total contact time (15.1% decrease), number of contacts requested (13.1% decrease), parental consent rate (1.0% decrease), and setting of the sample (0.9% decrease) in CD; exclusion criteria (10.0% decrease), setting of the sample (1.8% decrease), method of contacting participants (1.0% decrease), and gender (0.5% decrease) in JCCAP; and total contact time (1.8% decrease) and parental consent rate (1.6% decrease) in JACP.
|
Discussion |
|---|
|
TopAbstractMethodResultsDiscussionReferences |
|---|
Overall, the results of this brief report indicate that there has been an increase over time in the reporting of important demographic variables, such as ethnicity. This was found to be true across multiple variables and four journals. Sifers et al. (2002
) found that age was reported at a much higher rate in journals pertinent to pediatric and child psychology. This rate was also found to be high in this study. Again, the high incidence of age being reported may be attributed to the developmental manner by which researchers and journals view the study of pediatric and child psychology. Within the present study, ethnicity was also found to be one of the more frequently reported variables, reported more frequently in 2005 than 1997. As Sifers et al. noted, greater efforts have been made to increase the reporting of ethnicity in child and pediatric psychology research. Our results suggest similar strides over the past 5 years. For example, in contrast to the Sifers et al. study, where much of the ethnicity information was reported by stating the percentage of the sample belonging to the majority group, many of the articles reviewed here reported a minimum of two to three ethnic groups. This observation suggests that researchers may be placing more importance on the impact that different ethnicities may have on the results of a study by collecting and reporting this information. Alternatively, editors and reviewers may have placed more emphasis on the methodological importance and are rejecting more papers that inadequately describe ethnicity. As methodologies improve and as emphases in the discipline shift, changes in journal policy, and content would be expected.
Previous studies have also alluded to the impact that ethnicity may have on research outcomes. For example, Miller and Cross (2006) found that ethnicity had a significant effect on outcomes of child maltreatment. Other findings may suggest that this pattern is not true for all areas of research in psychology. For example, Ram, Starek, and Johnson (2004) found that the rate of reporting ethnicity for adults was as low as 19.86% in the field of exercise and sport psychology. Based on this information, it is possible that the importance placed on the identification of ethnicity for demographic purposes may be limited to specific fields of research. However, the clear impact of ethnicity and the continued underreporting of it in some journals indicate the necessity for continued studies to examine again the rate of reporting ethnicity. Ethnicity, as a stand alone variable, has the possibility to show significant effects in future research (Ram et al., 2004). However, we find it difficult to definitively state what constitutes appropriate ethnic information necessary for publication. It is reasonable, however, to require investigators to provide at least an overview of the ethnic make-up of the sample, in order to publish in professional journals (Jackson, 2003).
Other important variables to report include the rate of consent and/or participation. This information allows the reader to determine how representative the sample may be of the overall population from which it was extracted. Additionally, the inclusion of consent/assent for all participants is an ethically and legally important aspect of an article submitted for publication in recognition of the special vulnerabilities of children as research participants (Weil et al., 2002). We found that rates of reporting for consent and/or participation information provided within more recent journal articles to be higher than those found in the research articles from 1997. Compared to other variables, however, reporting rates of consent and assent procedures was relatively low across all journals. It is possible that information pertaining to child assent, as with other variables, may have been provided to the editors of the journals and omitted from the final draft due to space constraints or editor preferences (Roberts & Buckloh, 1996; Sifers et al., 2002). However, the "Instructions to Authors" for JPP specifically state that authors must report "how informed consent was obtained and report the approval of the study by the appropriate Institutional Review Board." Acknowledgment of the importance of this information, as well as improved reporting practices, allows readers to make better informed judgments regarding the degree to which the study adheres to ethical and legal standards, and sample represents the general population. These practices also enable psychologists to verify and/or replicate previous studies. Psychologists are required by the APA ethical code (2002) to share data for the purposes of verification (8.14(a)). This is to ensure that psychological studies can be replicated and/or evaluated by third parties. An important advantage of electronic communication enables psychologists to obtain this information directly from researchers more easily and quickly than in the past. This increased accessibility to data and other information may provide researchers with an alternate way to share demographic or methodological information (and therefore comply with the APA ethics code) without inclusion of the data within the actual publication.
Attrition rates are beneficial in publications to aid in understanding how and why participants may have discontinued their participation in a study. Reasons such as fatigue, minimal interest, or feeling that the intervention is adequate are all important considerations, as they would all be potentially applicable for future clinical use of the intervention. On average, this study showed a 12% improvement in the frequency of reporting attrition rates from the Sifers et al. (2002) study.
Although one might think that having submission requirements and guidelines for inclusion of specific pieces of information would result in compliance by submitting authors, the fact appears to be that some of the variables investigated here have relatively low occurrence (even though emphasized in the APA Publication Manual and in journal instructions to contributors). Authors sometimes do not comply and editors/reviewers are not enforcing the instructions uniformly. Some journals place page limits on submission, so demographic and procedural information as well as more comprehensive referencing may be the first pieces omitted. Fitting to page requirements may be a more rigidly enforced rule than are, apparently, those for including certain information. As Sifers et al. (2002) established, the present authors do not wish to imply that the nonreporting of variables is unethical or unprofessional. It may simply be that researchers do not feel that reporting a particular demographic, methodological, or ethical variable is important for the understanding and interpretation of particular studies. Similarly, researchers may have decided to omit the collection of these data from their study, or simply to exclude the data from further analyses or publication. However, the inclusion of this information for the reader's consideration would allow the reader to determine the relevance of that data for him or herself, which may be more appropriate (Sifers et al.).
Although the present study revealed valuable cross-sectional information about reporting rates, some limitations must be considered when interpreting the results. First of all, inferential statistics were not used to determine the statistical significance of differences found between reporting rates in 1997 and those in 2005. In consideration of the nature of the data (i.e., frequencies of reporting specific information across four journals in 1 year), we provided descriptive, qualitative information regarding changes over time. In addition to periodic "snapshots" of reporting practices such as this, a longitudinal investigation of trends in reporting information (e.g., reporting rates of variables across several years within a set of journals) would provide important information as well.
Another apparent limitation is the presence of several seemingly low -coefficients and interrater reliability rates on categories that should be objective and stable across raters. Frequency of interrater reliability ranged from 77% to 100%, with -coefficients ranging from 0.45 to 1.0. Some variables had lower rates of interrater reliability due to clarity issues within the studies. For example, some investigations included vague descriptive information (e.g., method of contacting participants), while others allowed the readers to draw conclusions about methods based on implied or contextual information (e.g., location of the study). The difficulty and ambiguity associated with collecting this information indicates that authors should consider presenting that information in a more explicit and identifiable manner. In keeping with the importance of complete reporting among empirical articles, we encourage authors to carefully consider the clarity of their reporting of methodological and demographic information.
Despite the increase in reporting rates, as illustrated by this study, it is important for the science and practice of psychology to continue to strive for more uniform and complete reporting of all demographic, methodological, and ethical variables. The addition of the CONSORT and TREND reporting standards for clinical trials in JPP (Brown, 2003; Des Jarlais, Lyles, Crepaz, & the TREND Group, 2004) appears to us to be worthwhile to adopt for all empirical research. This information is critical for replication of a study and for the important issue of generalizability. For example, determining the exact composition of the population on which an intervention was tested helps in the judgment that the intervention may be beneficial for use with another population.
Improvements have been made, but as scientist–practitioners, there is always room for continued improvement and evaluation. The reporting of demographics and methodology is one such area in need of continued monitoring to ensure that all journal readers are provided with the information necessary to make informed decisions regarding the interpretation and application of research outcomes.
Conflicts of interest: None declared.
Received April 20, 2007; revision received November 21, 2007; accepted December 8, 2007
|
References |
|---|
|
TopAbstractMethodResultsDiscussionReferences |
|---|
American Psychological Association. Publication manual of the American Psychological Association (2001) 5th. Washington, DC: Author.
American Psychological Association. Ethical standards of psychologists and code of conduct. American Psychologist (2002) 57:1060–1073.[CrossRef][Medline]
Brown RT. Editorial: The Journal of Pediatric Psychology will support the publication of clinical trials. Journal of Pediatric Psychology (2003) 28:173.
Des Jarlais DC, Lyles C, Crepaz N, & the TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. American Journal of Public Health (2004) 94:361–366.
Jackson Y. Research in ethnic minority communities: Cultural diversity issues in clinical psychology. In: Handbook of research methods in clinical psychology—Roberts MC, Ilardi SS, eds. (2003) Malden, MA: Blackwell Publishing Ltd. 376–393.
Miller AB, Cross T. Ethnicity in child maltreatment research: A replication of Behl et al.’s content analysis. Child Maltreatment (2006) 11:16–26.
Ram N, Starek J, Johnson J. Race, ethnicity, and sexual orientation: Still a void in sport and exercise psychology? Journal of Sport and Exercise Psychology (2004) 26:250–268.
Roberts MC, Buckloh LM. Five points and a lament about Range and Cotton's "Reports of assent and permission in research with children: Illustrations and suggestions." Ethics and Behavior (1996) 5:333–344.
Sifers SK, Puddy RW, Warren JS, Roberts MC. Reporting of demographic, methodological, and ethical procedures in journals in pediatric and child psychology. Journal of Pediatric Psychology (2002) 27:19–28.
Weil E, Nelson RM, Ross LF. Are research ethics standards satisfied in pediatric journal publications? Pediatrics (2002) 110:364–370.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  D. Drotar Editorial: How to Report Methods in the Journal of Pediatric Psychology J. Pediatr. Psychol., April 1, 2009; 34(3): 227 - 230. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||