Journal of Pediatric Psychology, Vol. 27, No. 1, 2002, pp. 19-25
© 2002 Society of Pediatric Psychology
Reporting of Demographics, Methodology, and Ethical Procedures in Journals in Pediatric and Child Psychology
University of Kansas
All correspondence should be sent to Michael C. Roberts, Clinical Child Psychology Program, 2006 Dole Human Development Center, University of Kansas, 1000 Sunnyside Avenue, Lawrence, Kansas 66045. E-mail: mroberts{at}ukans.edu .
 |
Abstract |
|---|
 Top Abstract IntroductionMethodResultsDiscussionReferences |
|---|
Objective: To identify potential problems in methodology reporting that may limit research interpretations and generalization.
Methods: We examined the rates at which articles in four major journals publishing research in pediatric, clinical child, and child psychology report 18 important demographic, methodological, and ethical information variables, such as participants' gender, socioeconomic status, ethnicity, inclusion/exclusion criteria, and consent and assent procedures.
Results: Overall, participants' ages, genders, and ethnicity were reported at moderate to high rates, whereas socioeconomic status was reported less often. Reports of research methodology frequently did not include information on how and where participants were recruited, the participation/consent rates, or attrition rates. Consent and assent procedures were not frequently described.
Conclusions: There is wide variability in articles reporting key demographic, methodological, and ethical procedure information. Necessary information about characteristics of participation samples, important for drawing conclusions, is lacking in the flagship journals serving the child psychology field.
Key words: methodology; generalization; ethics; demographics; journal content analyses.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
Among the hallmarks of the sciences, including the science of psychology, are an objective perspective and the ability to evaluate and replicate research methodology. Inherent in these is the comprehensive and accurate description of the research sample, the population from which it is drawn, and the methodology used to gather the data. In recognition of the communication requirements for science, the Publication Manual of the American Psychological Association (APA, 1994
) states that, when the participants in a research study are
human, certain information should be presented in the Method section of a
manuscript considered for publication in a journal (see Section 1.09, pp.
12-15). This necessary information includes details regarding major
demographic variables, the number of participants, method of selecting
participants, assignment to groups, agreements made, payments made, and the
number of participants who withdrew from the study and why. Additionally, this
information may include, but is not limited to, ethnicity, level of
educational attainment, and type of geographic area in which the participants
reside. The APA Publication Manual has assumed a leading position in dictating publication standards, not only for the primary APA journals but also for the numerous other journals in psychology and related fields. As representative of agreed-upon standards, the Publication Manual presented the reasons for fully describing the research participants:
Appropriate identification of research participants and clientele is critical to the science and practice of psychology, particularly for assessing the results (making comparisons across groups), generalizing the findings, and making comparisons in replications, literature reviews, or secondary data analyses. The sample should be adequately described. (p. 13)
Furthermore, the precise reporting of methods and demographics is especially important when determining the generalizability of research findings with children and adolescents. This is particularly important because psychologically manifested differences as a result of gender, development, or other factors may be more prominent in children and adolescents. Lack of adequate information is a methodological weakness placing considerable constraints on interpretation and conclusions in pediatric and clinical child psychology.
The Publication Manual also states that, in order to be published
in an APA journal, either the manuscript or a cover letter to the editor of
the journal should indicate that the researchers followed all ethical
standards set forth in the APA Ethical Principles of Psychologists and
Code of Conduct (APA,
1992). These guidelines seek to ensure the protection of the
interests of the participants, as well as providing important information to
the consumers of the research.
Given the importance for published articles to present demographic,
methodological, and ethics-related information, it is worthwhile to
periodically examine psychology publications for compliance to these standards
of scientific communication. A few previous reports have provided some
information related to the completeness of research articles in describing the
characteristics of the sample and the ethical procedures used in the study
(e.g., Bernal & Enchautegui-de-Jesus,
1994; Betan, Roberts, &
McCluskey-Fawcett, 1995;
Graham, 1992;
Park, Adams, & Lynch,
1998; Phares & Compas,
1992; Ponterotto,
1988). These reports indicate that there is considerable neglect
of methodological information in published articles, with some discrepancy
depending on the variable and the specialty. The research methodology
literature has long called for comprehensive description of research samples
(e.g., Bordens & Abbott,
1996; Hersen & Bellack,
1984).
Content analyses of journals help discern patterns in the development of a
field or subdiscipline and provide objective "snapshots" useful in
evaluating its science (Elkins &
Roberts, 1988; Peterson,
1996; Roberts, McNeal,
Randall, & Roberts, 1996). They provide the field with an
additional tool for assessing its past and current status. This examination is
important because it allows for self-correction when oversights are detected,
as well as the opportunity to set new directions. Consequently, we applied the
technique of journal content analysis to determine the presence of and utility
of comprehensive information reported in four publication outlets in pediatric
and clinical child psychology. This study focused on the rates at which
articles reported key demographic, methodological, and ethical variables such
as number of participants, age, gender, ethnicity, socioeconomic status (SES),
location of participants, rewards given to participants, exclusion and
inclusion criteria, attrition, and consent and assent procedures.
|
Method |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
Database
The database included all empirical research articles published during 1997 in Journal of Pediatric Psychology (JPP, 58 articles), Journal of Clinical Child Psychology (JCCP, 52 articles), Child Development (CD, 94 articles), and Journal of Abnormal Child Psychology (JACP, 56 articles). Review articles, editorial articles, addresses, case studies, and studies that did not include human participants were excluded from this review. In total, 260 articles were coded and included in this study.
Coding Procedure
The coding procedure was based on the procedure used by previous content
analyses (Betan et al., 1995;
Elkins & Roberts, 1988;
Roberts, 1992). For articles
containing more than one study, the studies were coded separately. Four
graduate students read and coded all the articles. Interrater reliability was
calculated on over 10% of the articles. Each article was coded using a
checklist with 18 items regarding characteristics of the study and its
participants. Kappa interrater reliability coefficients are presented, as well
as the percent agreement between rates for each coded variable: (1) ages (1.0;
100%), (2) gender (.47; 92%), (3) ethnic distribution (.92, 96%), (4) SES
(.55; 77%), (5) identification/selection of sample (e.g., requested, teacher
recommended, records:.38, 81%), (6) population (e.g., general/school children,
physical disability: 1.0; 100%), (7) setting of sample (e.g., school,
psychological clinic, hospital:.29; 77%), (8) method of contacting
participants (e.g., via mail, information sent via child:.57; 81%), (9) number
of contacts requested (.36; 77%), (10) total contact time (.77; 89%), (11)
exclusion/inclusion criteria (.34; 69%), (12) attrition (.35; 81%), (13)
reliability of dependent measures used in study (.43; 73%), (14) number of
participants (1.0; 100%), (15) location (geographically where sample was
recruited:.58; 85%), (16) reward offered for participation or time/expense
(1.0; 100%), (17) consent rate (after solicitation, participants who agreed to
study versus those who did not: (.74; 88%), and (18) child assent (.47; 92%).
Lower kappa coefficients were observed on a number of variables in part due to
the lack of variability observed within some variables. (A copy of the
decision rules for coding can be obtained from Michael Roberts.)
|
Results |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
The frequency and percentage of articles reporting the variables were calculated for each journal individually, as well as for the journals overall (see Table I). As can be seen in the total column, some variables tend to get reported at a fairly high rate across journals. The number of participants is reported in all the journals at a 100% level. The participants' ages are included very close to that perfect mark, as are the types of population from which the sample is drawn. Also reported at a high rate are setting of the research, the gender of the participants, and the methods of identifying and selecting participants. At the middle levels of reporting overall are characteristics such as participants' ethnicity, SES, exclusion/inclusion criteria used, reliability reporting, number of contacts requested, and methods of contacting participants. Low rates of reporting were found overall for child assent, parent consent, attrition rates, whether rewards were used, the location of the research project, and total contact time.
|
Within these overall trends, the journals varied as to whether each included or omitted some of the information components. One-sample t tests were conducted for each journal to determine if the frequency with which each reported study variable differed significantly from the other journals. The mean of the four journals on each variable was used as the test value (all analyses used two-tailed levels of significance). Based on these analyses, JPP reported identification/selection methods (t [57] = -3.061, p =.003), setting of the sample (t [57] = -3.952, p <.001), method of contacting participants (t [57] = -3.258, p =.002), exclusion/inclusion criteria (t [57] = -2.475, p =.016), and parental consent (t [57] = -2.548, p =.014) significantly more frequently than the other journals. JCCP reported gender (t [51] = -6.202, p <.001), ethnicity (t [51] = -4.263, p <.001), and child assent procedures (t [51] = -2.356, p =.022) significantly more frequently than the other journals, whereas it reported significantly less frequently information on total contact time (t [51] = 2.833, p =.007). CD reported the number of contacts requested (t [93] = -3.791, p <.001) and total contact time (t [93] = -2.638, p =.010) significantly more frequently than the other journals, whereas it reported information less frequently than the other journals on ethnicity (t [93] = 2.114, p =.037), identification/selection methods (t [93] = 3.245, p =.002), setting of the sample (t [93] = 2.573, p =.012), method of contacting participants (t [93] = 2.575, p =.012), parental consent (t [93] = 3.092, p =.003), and child assent procedures (t [93] = 3.597, p =.001). The rates of reporting variables in JACP did not differ significantly from the other journals.
A basic demographic description of the participants' gender ranged from a low of 80.4 % (JACP) to a high of 98.1% (JCCP). The percentage of articles that described the ethnic distribution of the participants varied greatly from a low 52.1% (CD) to a high of 84.6% (JCCP). CD reported the low of 43.6% of the participants' SES while JPP reported the high of 51.7%. The rate at which articles in this study indicated the geographic location of the sample ranged from the low 31% of JPP to a high of 42.6% of CD. The percentage of studies reporting whether a reward was offered was small and varied from 13.5% (JCCP) to 25.9% (JPP). The percentage of articles reporting consent rate differed from the low of 27.7% (CD) to 58.6% (JPP). The rate of reporting child assent also was low and significantly varied 8.5% (CD) to 34.6% (JCCP). Although just over half of the articles reported inclusion/exclusion criteria, the rate varied significantly from 48.1% (JCCP) to 70.7% (JPP). Across journals, just over a fourth of the articles reported causes for attrition, although rates differed from 19.6% (JACP) to 36.2% (JPP).
|
Discussion |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
In general, the results of this study suggest wide variability in the percentage of articles that reported key demographic, methodological, and ethical procedure items. This variability was observed across journals and across variables. The conclusion seems clear that, in general, articles published in flagship journals serving the pediatric and child psychology field do not provide needed information about characteristics of their participation samples. These journals ostensibly adhere to the APA Publication Manual for manuscript preparation, which calls for authors to include this detailed information.
The participants' ages and gender tend to be reported at a fairly high
rate. This rate for age is higher than for "adult" research
journals such as Health Psychology
(Park et al., 1998) likely
because these four journals have more of a developmental focus. Ethnicity
information, although left out of many articles, seems to be higher than found
in previous content analyses and in other specialties in psychology. Ethnicity
description is likely present in these later reports because of the many
efforts to enhance recognition of diversity issues in psychology research
(e.g., Iijima Hall, 1998). Of
course, the overall percentage of 63.1% indicates only that this information
was reported in some form, even if only a general statement of predominant
ethnicity, not specific breakdowns. Such global information does not indicate
anything about the ethnic representativeness of the sample to the larger
population, degree of acculturation, or other aspects, for example. Similarly,
the SES information was provided in about half of the studies. Age, gender,
ethnicity, and SES are demographic characteristics important to most of the
psychological variables under study in these research articles. The omission
of even this basic or minimal information restricts the research consumers'
ability to draw proper conclusions.
How to report ethnicity and cultural variables for research publications
requires further clarification by the field, given the complexities inherent
in these phenomena. Our analyses indicated only whether some information was
presented, not the precision with which the information was reported. When
ethnicity of the participants was indicated in the articles we analyzed, what
typically was included was a general statement about race (i.e., African
American, Asian American, Hispanic American, Native American, Caucasian, or
Euro-American). Unfortunately, the majority of the articles did not describe
elements usually included in the concept of ethnicity, such as language,
religion, degree of acculturation, and nation of origin. Overall, the field
likely would need a minimal standard of reporting ethnicity and culture
established by a consensus or editorial degree. When ethnicity might be
conceived as a major influence or related to other psychological variables
under study, then more elaborate conceptualizations would be needed.
Psychologists might benefit from conceptualizations arising from controversies
on ethnicity and culture in anthropology and sociology
(Jenkins, 1997;
Malik, 1996;
Solomos & Back, 1996).
Knowing the rates of consent/participation helps us to discern the representativeness of the sample from the overall population and to draw generalizable conclusions. A low rate of participation may or may not be a problem, depending on the circumstances of recruitment and the psychological variables under study. Too few research articles included this information, which is needed to form any consensus for acceptable ranges of participation.
The attrition rates were significantly under-reported. This information is important in determining the representativeness of the sample. Attrition may indicate whether the procedures biased the results, for example, because participants could not complete all aspects of an experiment or data gathering through fatigue, lack of interest, or alienation. Knowledge of attrition is also critical for evaluating clinical interventions. Essentially, differential attrition can bias results and invalidate research findings or mislead consumers of the research.
Reporting of parental permission/consent and child assent procedures as
ethical information remains relatively low, despite the fact that two of the
journals (JPP and JCCP) have instructions to include these
procedures. Some information on how consent/assent procedures were handled may
have been conveyed in a submission letter to the editor, and for two of the
journals, authors of manuscripts accepted for publication sign an ethics
compliance form indicating all procedures comply with the APA ethical code. In
no way do we want to imply that these investigators were unethical in their
research practices by omitting reports of consent and/or assent
(Roberts & Buckloh, 1995),
and much of the research in the United States has been reviewed by
institutional review boards. We can conclude only that the authors did not
report this information. Certainly, in the case where journals report research
with infants (e.g., CD), child assent would be inappropriate. Even
though avowal of proper use of consent is required to be published in these
journals, reporting consent/assent procedures explicitly models ethical
practices in research for fellow scientists and symbolizes adherence to
ethical practices in research. Furthermore, perhaps more than the presence of
a consent/assent form should be reported. Perhaps researchers should include
relevant information such as the power differential between researcher and
participant or the information the participant was actually given about the
study.
Researchers experience the "judgment calls" of editors and reviewers (and usually make such calls themselves when roles are reversed) when manuscripts are reviewed for acceptance/rejection. Such judgments may include decisions about whether a participant sample is adequate from which to draw conclusions. Missing data about a sample, however, may not be caught in the editorial review and, as evidenced here, articles will be published without important pieces of information. Of course, including some information about the sample may affect submission/publication because aspects of the report sample seemingly fall short of some ill-defined criteria (e.g., about what constitutes a currently acceptable return rate of participation or about what is a necessary ethnic distribution of participants). At this time, there is currently not enough information in the literature on which to make this type of judgment.
In the interest of fairness in the publication process, but more important, for the advancement of the science in pediatric and child psychology, we suggest that all manuscripts be held to a standard of comprehensive reporting. If this happens, the field eventually will have a more complete picture from which to draw conclusions about psychological phenomena.
Commentary on the rates at which articles within a journal report the variables explored in this study is not meant to be a judgment of the quality of the research, journal, or editor. Although not reporting the variables considered in this study does restrict the reader's ability to evaluate articles, justification for these omissions may be reasonable. For example, the researchers may believe that some variables were not crucial to understanding their study. Nonetheless, if a standard of comprehensive reporting were used, then consumers of research would be able to judge for themselves the value of these variables. Furthermore, the journals' submission requirements or editorial review might not encourage the reporting of such variables. On the other hand, the researchers may believe that the editor's or the consumer's perception of the worth of their study may be negatively affected by reporting demographic characteristics that are not consistent with those found in the population of interest or by describing less than ideal methodological variables. These variables may then be submerged or obfuscated through global statements.
A couple of examples may illustrate best the deficiencies of reporting even basic information. One coded article on the psychometric development of a screening instrument for young children failed to report anything on the variables of gender, ethnicity, SES, location, identification/selection of the sample, consent rates, attrition, or reliability. This article passed the editorial review, but we question the use of the measure when the consumer has no knowledge about the group on which it was normed. An article on cross-cultural comparison of a widely used behavior problem checklist failed to indicate the ethnicity of the sample and provided no information on location, SES, attrition, and inclusion/exclusion criteria. As the results indicate, we could describe many articles in which critical information was lacking. Although these articles seem particularly egregious, there may be some benign omissions of information. However, an article author may not know how future researchers and clinicians might use the research findings since they will lack critical aspects of a study. Interpretation of findings is limited by this lack of information.
A primary consequence of research articles failing to report demographic
and methodological variables is that consumers are not able to estimate
whether the sample is representative of the population of interest or if
procedures were adequate. As suggested by Betan et al.
(1995), the more representative
a sample is of the population being studied, the more likely the findings will
generalize to the desired population. We were interested here in determining
generally whether information on those characteristics is being reported;
therefore, dichotomous coding of presence or absence of information was used
in this study, not assessing the level of detail or meticulousness. Based on
these overall results, greater precision could be employed in future work to
consider the actual degree of representativeness for one or more of the
variables in particular lines of research.
Based on our findings, like others before us, we suggest that journal editors and reviewers require, and researchers follow through by including, more demographic and methodological information in their articles. Describing this information in journal publications would enhance the scientific development of the field and the clinical applicability of the research. We hope that, in the future, the reporting of this demographic and methodological information will attain a 100% level for reporting the key variables needed for conclusions and interpretation in pediatric and child psychology research. The "gold standard" of reporting all these variables in all journal articles is a lofty goal and would necessitate changes in common writing and reviewing processes. As certain issues are highlighted in contemporary research, needs for reporting information may change over time. For example, the recent trend toward ensuring explicit accountability in ethical procedures provides pressure to report practices that might not have been the focus of such attention in the past. Similarly, issues of culture and ethnicity in research seem to assume greater emphasis more recently. Such a higher standard of reporting would require effort on the part of researchers to include such information and diligence on the part of reviewers and editors to ensure such information is included. This time and effort seems small in comparison to other resources invested in the research enterprise, yet it has such potential for advancing scientific rigor within research in pediatric and child psychology.
|
Acknowledgments |
|---|
This article is based on a poster presentation at the Kansas Conference in Clinical Child Psychology, Lawrence, in October 1998.
Received June 7, 1999; revision received May 1, 2000; accepted October 30, 2000
|
References |
|---|
|
TopAbstractIntroductionMethodResultsDiscussionReferences |
|---|
American Psychological Association (APA) (1992). Ethical principles of psychologists and code of conduct. American Psychologist, 12, 1597 -1611.
American Psychological Association (APA) (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
Bernal, G., & Enchautegui-de-Jesus, N. (1994). Latinos and Latinas in community psychology: A review of the literature. American Journal of Community Psychology, 22, 531-558.[ISI][Medline]
Betan, E. J., Roberts, M. C., & McCluskey-Fawcett, K. (1995). Rates of participation for clinical child and pediatric psychology research: Issues in methodology. Journal of Clinical Child Psychology, 24, 227 -235.
Bordens, K. S., & Abbott, B. B. (1996). Research design and methods: A process analysis (3rd ed.). Mountain View, CA: Mayfield.
Elkins, P. D., & Roberts, M. C. (1988).
Journal of Pediatric Psychology: A content analysis of articles over
its first 10 years. Journal of Pediatric Psychology, 13, 575-594.
Graham, S. (1992). "Most of the subjects were White and middle class": Trends in published research on African Americans in selected APA journals, 1970-1989. American Psychologist, 47, 629 -639.
Hersen, M., & Bellack, A. S. (1984). Research in clinical psychology. In A. S. Bellack & M. Hersen (Eds.), Research methods in clinical psychology (pp. 100 -138). New York: Pergamon.
Iijima Hall, C. C. (1998). Cultural malpractice: The growing obsolescence of psychology with the changing U.S. population. American Psychologist, 52, 642 -651.
Jenkins, R. (1997). Rethinking ethnicity: Arguments and explorations. London: Sage.
Malik, K. (1996). The meaning of race: Race, history and culture in Western society. New York: New York University Press.
Park, T. L., Adams, S. G., & Lynch, J. (1998). Sociodemographic factors in health psychology research: 12 years in review. Health Psychology, 17, 381 -383.[ISI][Medline]
Peterson, L. (1996). Establishing the study of development as a dynamic force in health psychology. Health Psychology, 15, 155 -157.[ISI][Medline]
Phares, V., & Compas, B. E. (1992). The role of fathers in child and adolescent psychopathology: Make room for daddy. Psychological Bulletin, 111, 387 -412.[ISI][Medline]
Ponterotto, J. G. (1988). Racial/ethnic minority research in the Journal of Counseling Psychology: A content analysis and methodological critique. Journal of Counseling Psychology, 35, 410 -418.
Roberts, M. C. (1992). Vale dictum: An editor's view
of the field of pediatric psychology and its journal. Journal of
Pediatric Psychology, 17, 785
-805.
Roberts, M. C., & Buckloh, L. M. (1995). Five points and a lament about Range and Cotton's "Reports of assent and permission in research with children: Illustrations and suggestions." Ethics & Behavior, 5, 333 -344.
Roberts, M. C., McNeal, R. E., Randall, C. J., & Roberts, J. C.
(1996). A necessary reemphasis on integrating explicative
research with the pragmatics of pediatric psychology. Journal of
Pediatric Psychology, 21, 107
-114.
Solomos, J., & Back, L. (1996). Racism and society. New York: St. Martin's Press.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  J. M. Raad, S. Bellinger, E. McCormick, M. C. Roberts, and R. G. Steele Brief Report: Reporting Practices of Methodological Information in Four Journals of Pediatric and Child Psychology J. Pediatr. Psychol., August 1, 2008; 33(7): 688 - 693. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 L. D Henley and D. M Frank Reporting Ethical Protections in Physical Therapy Research Physical Therapy, April 1, 2006; 86(4): 499 - 509. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. E. Kazak Journal of Pediatric Psychology (JPP), 1998-2002: Editor's Vale Dictum J. Pediatr. Psychol., December 1, 2002; 27(8): 653 - 663. [Full Text] [PDF]
|
|
|
|
 |
|
 E. Weil, R. M. Nelson, and L. F. Ross Are Research Ethics Standards Satisfied in Pediatric Journal Publications? Pediatrics, August 1, 2002; 110(2): 364 - 370. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||