Journal of Pediatric Psychology, Vol. 25, No. 6, 2000, pp. 403-414
© 2000 Society of Pediatric Psychology
Translation and Linguistic Validation of a Disease-Specific Quality of Life Measure for Cystic Fibrosis
1 University of Florida, 2 Research Triangle Institute, 3 Genentech, Inc., 4 Wayne State University, 5 Indiana University, 6 Arcos, Arcos, France
All correspondence should be sent to Alexandra L. Quittner, Dept. of Clinical & Health Psychology, University of Florida, 1600 SW Archer Rd., DG-98, Gainesville, Florida 32610-0165. E-mail: Aquittne{at}hp.ufl.edu .
| Abstract |
|---|
|
|
|---|
Objective: To develop a conceptually and semantically valid English version of a French disease-specific measure of quality of life for children, adolescents, and adults with cystic fibrosis (CF).
Methods: Following a backward and forward translation of the measure, 60 participants, including 20 children, 20 parents, and 20 adolescents/young adults completed the Cystic Fibrosis Questionnaire (CFQ) and a series of cognitive probes evaluating their understanding of the items and response choices.
Results: Semantic and conceptual problems with the items were identified and modified for the second set of cognitive interviews. Response distributions across items and ages were adequate, and the predicted associations between disease severity and quality of life were obtained.
Conclusions: The English version of the CFQ appears to be a linguistically valid measure of quality of life for patients with CF. A national validation study is now under way to test the psychometric properties of the measure.
Key words: cystic fibrosis; quality of life; methodology; pediatric psychology.
| Introduction |
|---|
|
|
|---|
The measurement of health-related quality of life (HRQOL) provides unique information about the impact of an illness and the effectiveness of various treatments, particularly for chronic conditions, such as cystic fibrosis (CF), in which treatment aims to improve daily functioning and overall well-being (Drotar, 1998
Significant progress has been made over the last two decades in defining
and measuring HRQOL (Guyatt et al.,
1997
; Spilker,
1996
). Although initially considered vague and difficult to
define, HRQOL has become a multidimensional construct that includes several
core domains, such as physical functioning and symptoms, psychological and
emotional state, and activities of daily living. Furthermore, it reflects an
individual's subjective evaluation of his or her daily functioning and
well-being because it is generally centered on the patient rather than the
physician (Levi & Drotar,
1998
; Spilker,
1996
). Rigorous standards for the development and psychometric
evaluation of QOL measures have been published
(Juniper, Guyatt, & Jaeschke,
1996
), and reliable and valid measures of HRQOL for several
chronic conditions, such as asthma, are now widely used
(Juniper, Guyatt, Ferrie, & Griffith,
1993
).
There are several reasons for developing a HRQOL tool for CF. First, the
discovery of the genetic defect for CF in 1989 has led to dramatic advances in
our understanding of the pathophysiology of the disease. This, in turn, has
led to the development of new treatments that have shown promise for
increasing life span and improving patients' quality of life. For example, new
drug treatments, such as rhDNASE and inhaled antibiotics, have recently been
evaluated in a series of short-term safety and efficacy trials, with some
indication of improvement in pulmonary functioning and HRQOL
(Fuchs, Borowitz, & Christiansen,
1994
; Ramsey, Astley, &
Aitken, 1993
). In these studies, however, ad-hoc QOL items were
administered in the absence of a well-validated measure of HRQOL for CF,
making it difficult to ascertain how these treatments affected quality of
life.
Another reason for assessing HRQOL is that for chronic diseases, in
particular, some treatments may yield benefits in activities of daily living
that are not reflected in conventional medical outcomes. Anecdotal reports
from families in our nutrition intervention study, aimed at increasing calorie
intake and weight gain in young children with CF, have suggested that the
children's increased energy level and ability to participate in sports was as
exciting for families as the changes in weight and height percentiles
(Stark et al., 1998
). Further,
a patient's perception of improvements in daily functioning may play a
critical role in increasing adherence to complex and time-consuming treatment
regimens (Abbott, Bilton, Dodd, & Webb,
1994
; Quittner et al.,
2000
).
Finally, the assessment of HRQOL provides another basis for evaluating the
effectiveness of different treatments. Many of the new therapies being
introduced for CF are time-consuming and use significant health care
resources. The QOL data may be useful for comparing the cost-effectiveness of
different treatments (e.g., shorter hospital stays), which may influence
health care policy and reimbursement
(Feeney, Torrance, & Labelle,
1996
; Kaplan et al.,
1989
). In a recent review of outcome measures for clinical trials
in patients with CF, the Consensus Group recommended that HRQOL measures be
incorporated into phase 3 clinical trials for both children and adults with CF
(Ramsey & Boat, 1994
).
More than a decade ago, the National Institutes of Health sponsored a
conference on the psychological and behavioral aspects of CF and recommended
that a disease-specific measure of HRQOL be developed and included in future
studies along with conventional health outcomes
(Eigen, Clark, & Wolle,
1987
). Despite this recommendation, efforts to develop a reliable
and valid CF-specific measure have just begun
(Henry, Aussage, Grosskopf, & Launois,
1996
; Henry, Grosskopf,
Aussage, Goehrs, & Launois, 1997
;
Quittner, Sweeny, Watrous, Munzenberger,
& Henry, 1998
).
Prior studies of HRQOL in CF have relied on generic measures, which ask
general questions about symptoms and functioning that can be completed by
patients with a variety of chronic conditions. Two major types of generic QOL
measures have been used with CF patients: (1) utility measures, such as the
Quality of Well-Being Scale (Kaplan et al,
1989
; Orenstein & Kaplan,
1991
; Orenstein, Nixon, Ross,
& Kaplan, 1989
; Orenstein,
Pattishall, Nixon, Ross, & Kaplan, 1990
), and (2) health
profiles, such as the Nottingham Health Profile
(Caine et al., 1991
;
Dennis et al., 1993
;
Buschbach, Horiks, van den Bosch, de la
Riviere, & de Charro, 1994
). Although a complete review of
these measures is beyond the scope of this article, their major limitation is
a lack of sensitivity to the specific challenges posed by CF
(Czyzewski, Mariotto, Bartholomew,
LeCompre, & Sockrider, 1994
; Munzenberger, Van Wagner,
Abdulhamid, & Walker, 1999), which makes it more difficult to quantify
changes in HRQOL that result from new interventions (e.g., lung
transplantation), or the natural progression of the disease (see
Quittner, 1998
, and
Tullis & Guyatt, 1995
, for
reviews).
Another significant limitation of current HRQOL research is the relatively
small number of measures for children and adolescents
(Landgraf, Abetz, & Ware,
1996
; Varni, Seid, & Rode,
1999
). Instruments designed for these age groups must be
developmentally appropriate and, therefore, must take into account the
psychosocial and cognitive differences between age groups. Health-related
quality of life measures for children, for example, should include modules
that assess daily functioning at school and social relationships with peers
(Quittner et. al., 1996
). The
QOL measures designed for adolescents should contain items that reflect
developmental shifts in independence (e.g., planning for college) and
increased responsibility for disease management (e.g., remembering nebulizer
treatments) (DiGirolamo, Quittner,
Ackerman, & Stevens, 1997
;
Quittner et al., 2000
). In
sum, there is a great need for a reliable and valid CF-specific HRQOL measure
that can be used with children, adolescents, and adults.
To date, the work of Henry et al. on the Cystic Fibrosis Questionnaire
(CFQ) in France represents the only published effort to develop a
disease-specific HRQOL measure for patients with CF
(Henry, Aussage, et al., 1996
;
Henry, Aussage, Grosskopf, & Goehrs,
1998
; Henry, Aussage, Staab,
Prados, Grosskopf, & Goehrs, 1997
;
Henry, Grosskopf, et al.,
1997
). A major advantage of the CFQ is its developmental approach,
which has yielded three separate QOL measures, one for school-age children 6
to 13 years, one for parents of school-age children, and one for adolescents
and adults (ages 14 and older). Initial item development began with an
extensive review of the literature, interviews with experts and CF
associations, and qualitative interviews with 44 children and adults with CF
and their parents (Henry, Aussage, et al.,
1996
). Five generic (e.g., physical symptoms, role functioning)
and four CF-specific (e.g., eating disturbances, treatment burden) HRQOL
domains were identified. Three initial questionnaires were constructed: a
child version with 20 items, a parent version with 81 items, and an
adolescent/adult version with 90 items. In all three versions of the measure,
higher scores indicate better quality of life. Pilot testing with 30
participants led to minor wording changes and the addition of several new
items of the child version of the CFQ
(Henry, Aussage, et al.,
1998
).
In the item-reduction phase, 534 questionnaires were completed at 24
hospital sites in France with patients whose illness severity differed.
Descriptive statistics and factor analyses provided support for the initial
scale structure. Internal consistency coefficients of.46 (social) to.71
(physical) were reported for the CFQ-Child version,.60 (body image) to.89
(physical) for the CFQ-Parent version, and.61 (embarrassment) to.92 (physical)
for the CFQ-14+. Moderate convergence was found between the CFQ 14+ and the
Nottingham Health Profile (correlations ranging from.43 for social limitations
to.75 for physical functioning). Significant associations were also found
between greater illness severity, as measured by Schwachman scores, and lower
scores on the CFQ-14+. Short-term test-retest reliability (7 to 10 days)
yielded significant intraclass correlation coefficients ranging from.66 to.97
(Henry, Aussage, et al., 1998
;
Henry, Grosskopf, et al.,
1997
).
The CFQ is currently undergoing extensive development and psychometric
evaluation in other countries. Because CF is found throughout the world, and
the medical and behavioral treatment approaches are fairly consistent and
shared in several conference and publication venues, it is important to have a
well-validated, international measure of HRQOL for CF. As new medical
treatments become available, such as gene therapy, and others, such as lung
transplantation, become more widely available, it would be extremely
beneficial to have an international tool to evaluate the impact of these
treatments on HRQOL. Psychometric validation of the French CFQ is nearly
complete, and the German and Spanish translations of the CFQ are now being
evaluated with larger samples (Henry,
Aussage, Staab, et al., 1997
). More recently, analyses testing the
conceptual equivalence of the items across all three languages (French,
German, and Spanish) using Rasch modeling techniques were presented at a
national conference (Henry, Staab, et al.,
1998
).
Given the importance of developing a disease-specific measure of HRQOL for CF, the strong psychometric properties of the original French CFQ and its potential as an internationally validated tool, the central aim of this study was to develop a conceptually equivalent English version of the French CFQ. The specific objectives of this study were (1) to complete independent forward and backward translations of the French CFQ into English following international guidelines; (2) to complete an initial qualitative evaluation of the English versions of the CFQ with a small sample of patients with CF and parents to identify semantic, conceptual, and psychometric problems with the initial translations; (3) to modify the measures based on results of the first evaluation, and then conduct a second round of qualitative interviews to identify any remaining semantic or conceptual problems with the items; and (4) to produce English versions of the CFQ that are conceptually and semantically valid and can be tested in a national psychometric evaluation.
| Translation Process |
|---|
|
|
|---|
For researchers developing and validating quality of life measures in different countries, a consensus has emerged that four levels of cross-cultural equivalence must be achieved: (1) conceptual equivalence, (2) construc or item equivalence, (3) operational equivalence, and (4) metric equivalence (Anderson, Aaronson, Leplegè, & Wilkin, 1996
International guidelines have been established for translating quality of
life measures from one language into another and then demonstrating
cross-cultural validity (Bullinger,
Anderson, Cella, & Aaronson, 1993
;
Ware, Gandek, Keller, & the IQOLA
group, 1996
). This is an iterative process that began in this
study with conference calls between the French developer (B. Henry) and the
U.S. research teams to clarify important concepts and medical terms. Next, two
independent forward translations were conducted with native French speakers
living in the United States (i.e., from Wayne State University and Research
Triangle Institute). Once the translations from French to English were
complete, discussions were held between the translators to resolve semantic
and conceptual discrepancies and to develop a consensus forward translation.
The English versions of the measure were sent to France for backward
translation (English to French), with Dr. Henry as the consultant. Conference
calls between the French and U.S. research teams focused on the conceptual and
rating scale issues that had been identified (e.g., what is the equivalent of
"pleine forme" in English?). Following these discussions, a
consensus version of the three CFQ measures in English was finalized
(Quittner et al., 1998
). These
measures were then qualitatively evaluated and modified in two separate
studies to assess their conceptual equivalence.
| Study 1: Initial Qualitative Evaluation: Method |
|---|
|
|
|---|
Participants
Thirty participants were recruited for the initial qualitative evaluation: 10 children with CF ages 8 to 13, 10 parents of the children completing the CFQ-Child version, and 10 adolescents and adults with CF ages 14 and older. Participants were recruited from CF Centers in Indiana (Riley Hospital for Children) and North Carolina (University of North Carolina at Chapel Hill and Duke University). Patients with CF and their parents were contacted prior to a routine clinic visit and asked if they would be willing to complete a quality of life measure along with follow-up questions. Efforts were made to recruit even numbers of male and female patients, and a representative range of illness severity as measured by FEV1 (forced expiratory volume in one second).
In study 1, the mean age of children completing the CFQ-Child was 10.56
(SD = 1.67); more female than male children participated in the
CFQ-Child interviews (70%), but an even number of boys and girls completed the
CFQ-14+. Mean age of participants completing the CFQ-14+ interviews was 23.3
(SD = 8.17). The majority of CFQ-Parent respondents were mothers
(70%); however, two fathers and one grandmother, who were primary caregivers,
also completed the interviews. An adequate range of illness severity was
represented, with most younger children falling into the mildly ill category
(range of FEV1 73% to 101%; M = 89.67, SD =
10.31), and adolescents and adults falling into the mildly to moderately ill
categories (range of FEV1 59% to 93%; M = 69.71,
SD = 11.63) (FEV1 values for the three children and adults
in study 1 were not available.) Normal pulmonary functioning values range from
80% to above 100% predicted (Knudson,
Slatin, Lebowitz, & Burrows, 1976
). One child's protocol was
not used because it became apparent during the interview that she was
developmentally delayed and could not understand the questions.
Measures
Cystic Fibrosis Questionnaire-Child Version (CFQ-C;
Henry, Grosskopf, et al.,
1997
; Quittner et
al., 1998
). CF-specific quality of life was measured using
the CFQ-Child version. This measure is administered directly to children ages
6 to 13 and consists of 36 items. Twenty-one items require a frequency
response ranging from "always" to "never" on a 4-point
scale and 15 items require a true-false rating on a 4-point scale. These items
assess the following seven QOL generic and disease-specific domains: physical
functioning (eight items), energy and wellbeing (six items), emotional state
(five items), social limitations (four items), body image (three items),
eating disturbances (three items), and treatment constraints (two items). Two
additional symptom scales, respiratory (four items) and digestive (one item),
are also included (see Table
1). Two practice questions are used with colored rating cards to
train the child on the two types of response choices. This procedure ensures
that the child understands the response choices. The cards are used as
reminders during administration of the questionnaire. To further evaluate the
child's understanding of the frequency and true-false scales, analogue
thermometer scales were constructed to represent the response choices. After
choosing a response (e.g., "always" to "never" or
"very true" to "very false"), the child was instructed
to show the interviewer where that response would fall on the thermometer.
These selections were noted on the protocol. When not accompanied by
additional qualitative questions, the CFQ-C takes 15-20 minutes to
complete.
|
Cystic Fibrosis Questionnaire-Parent Version (CFQ-P;
Henry, Grosskopf, et al.,
1997
; Quittner et
al., 1998
). Parents' report of their child's disease-specific
quality of life was measured using the CFQ-Parent version, a self-report
measure consisting of 44 items. Items require either a frequency response on a
4-point scale ("all the time" to "never"), a
difficulty rating on a 4-point scale ("a lot of difficulty" to
"no difficulty"), a true-false rating on a 4-point scale, or the
selection of a statement that describes the child (on a 4- or 5-point scale).
These items assess the following seven QOL domains: physical functioning (nine
items), energy and well-being (six items), emotional state (four items),
performance at school (four items), body image (three items), eating
disturbances (two items), and treatment constraints (two items). Three symptom
scales, respiratory (seven items), digestive (two items), and weight (one
item), and an overall health perception scale (four items) are also included
(see Table 1). This measure
takes approximately 15-20 minutes to complete on its own.
Cystic Fibrosis Questionnaire-14+ (CFQ-14+;
Henry, Grosskopf, et al.,
1997
; Quittner et
al., 1998
). An adolescent/adult form of the CFQ was developed
to measure perceptions of quality of life for teens and young adults with CF.
This self-report measure consists of 48 items. Items are rated in terms of
frequency, difficulty, true-false categories, or weighted statements using the
same response choices described for the CFQ-P form above. In this version,
however, nine QOL dimensions are assessed: physical functioning (eight items),
energy and well-being (four items), emotional state (five items), social
limitations (four items), role limitations (three items), body image (three
items), eating disturbances (two items), treatment constraints (two items),
and embarrassment (three items). Three symptom scales, respiratory (seven
items), digestive (two items), and weight (one item), and an overall health
perception scale (four items) are also included (see
Table 1). The CFQ-14+ takes
15-20 minutes to complete.
Procedure
The study's procedures were reviewed and approved by an institutional
review board; consent was obtained from parents and adults; assent was
obtained from children. The study protocol consisted of face-to-face
interviews with each participant, during which they completed the CFQ with a
series of cognitive probes following each question. The interviews were
conducted in families' homes or in a private room adjoining the clinic and
lasted approximately 45 minutes for children and between 60 and 90 minutes for
parents and adolescent/adults. Children were reimbursed $30 for their time and
effort; parents were compensated $50 for their time and effort. Recent
pulmonary functioning test results were obtained, when possible, from medical
charts.
Four sets of cognitive probes were developed, using well-validated
cognitive testing procedures (Lessler
& Forsyth, 1996
), to evaluate the participant's understanding
of the items and response choices. The first set of questions asked whether
the directions for completing the measure were clear. The second set followed
each item and served to clarify what the item meant to the participant, how
the response choice was made, and what circumstances would require a different
response. For example, on the CFQ-Child, after the item "You felt full
of energy," the child was asked the following questions: (1) What did
you think of when answering this question? (2) How do you act when you are
feeling this way? (3) How did you choose your answer? (4) What would have made
you choose a different answer? (5) Are there other words that mean the same
thing, but are clearer? Probes for the CFQ-P also targeted the parent's
reference point when answering the questions (e.g. "How well do you
think your answer matches what your child might say?"). The third set of
probes appeared after each major section and consisted of open-ended comments
about the previous set of items, questions about the difficulty and relevance
of the previous questions and answer choices, and comments on any difficulties
the respondent had in distinguishing among the response choices. Finally, the
interview ended with a series of open-ended questions assessing their overall
evaluation of the measure, suggestions for topics that were not covered, and
comments about items that seemed irrelevant or were particularly difficult to
answer. On the CFQ-C, specific probes were also inserted periodically to
determine whether children were attending to the 2-week time frame in choosing
their responses. (Cognitive protocols are available upon request from the
authors).
The qualitative interviews for each version of the CFQ were transcribed from audiotapes or session notes and collated across participants. In a series of group meetings that included several of the authors, as well as graduate and undergraduate students, the responses and comments to each item were read and discussed in an effort to identify ambiguities in the wording of items, trends in the distribution of responses, and common themes that emerged from the open-ended comments (e.g., meaning of the item, relevance). These qualitative data were then collated and tabulated across participants for each version of the measure.
| Study 1: Results |
|---|
|
|
|---|
The analyses of the qualitative interviews had four primary foci: (1) to examine the range of responses for each item; (2) to identify semantic problems with the items, (3) to identify conceptual problems with the items, and (4) to evaluate the scope and relevance of the measures through responses to the open-ended interview prompts.
First, we checked the numeric range of responses to each item for adequacy of the distribution. Given the small sample size of this study, a liberal criterion was set for the response distribution. Items that did not elicit responses in three of the four rating categories were flagged and reworded, as necessary.
Second, we examined semantic problems with the three versions of the CFQ. Two types of wording and language issues were documented. First, we noted problems with the translation of medical terms from French into English. As can be seen in Table II, two French terms, when translated literally into English (i.e., "sprays" and "spit"), were reported to be confusing and awkward for respondents on all three versions of the CFQ. Participants were asked to generate alternative terms for these items, and the consensus across groups was to change "spray" to "aerosol or mist treatments" and "spit" to "cough up mucus." Participants also had difficulty rating two of the items that contained double-barreled adjectives. Parents on the CFQ-P noted that they would rate the two adjectives "worried and anxious" quite differently. Similarly, adolescents and adults stated that they would give different ratings to the adjectives "sad and depressed." For both of these items, a restricted range of responses was noted (i.e., only "sometimes" and "never" were endorsed). The less intense adjective was chosen for both items (i.e., "worried," "sad").
|
Conceptual problems with the items (interpreting their meaning) were also identified in study 1 (see Table II). The French phrase, "pleine form," was translated into English as "felt very fit." However, the cognitive testing revealed that participants interpreted this as exclusively referring to physical fitness, rather than the broader French meaning of "well-being" as a whole (physical, social, and emotional). After considerable discussion among the French and U.S. teams, the item was changed to "felt well emotionally and physically." A different problem with the phrase "full of energy" emerged in all three versions of the CFQ. Both children, parents, and adults interpreted this item as meaning too much energy, as in "bouncing off the walls," behaving in a hyperactive manner. This item consequently was changed to "energetic." Intensity was also a problem in the item "forced to eat" in the CFQ-C and was changed to "pushed to eat." The words "short-tempered" were difficult for younger children, so this item was changed to "felt mad." Cultural differences were noted in items that reflected the socialization patterns in France and the United States. The item "visit my friends a lot" generated responses indicating that "visiting" is considered a more formal activity here than in France, and that the phrase "get together" better reflected a casual, American approach to socializing. In addition, "spend a lot of time at home" (reading and watching TV) was considered a social preference, rather than a consequence of having a chronic illness; therefore, this item was changed to "stay home more than I want to." One item on the CFQ-P was interpreted to mean how well the child performed in school, in terms of grades and achievements, rather than more general functioning. Consequently, two items reflecting typical school functioning were written and tested in the second cognitive evaluation.
Finally, all of the children reported that the directions for the CFQ-C were clear; several children liked the practice instructions, and none of the children had suggestions for additional items. On the CFQ-P, 90% of the parents reported that the instructions were clear and the items were highly relevant. Additional items for the CFQ-C were suggested to measure the child's acceptance by peers, comfort in telling others about CF, and stigma associated with taking enzymes, as well as items covering sexuality, fertility issues, and job limitations for the CFQ-14.
| Study 2: Second Qualitative Evaluation: Method |
|---|
|
|
|---|
Participants
An additional 30 participants were recruited from the same locations and in a similar manner for the second qualitative evaluation: 10 children with CF ages 8 to 13, 10 parents of those children, and 10 adolescents and adults with CF ages 14 and older. The mean age of the children completing the CFQ-C was 10.1 (SD = 2.03); more female than male children participated in the CFQ-Child interviews (70%); however, slightly fewer females (40%) than males completed the CFQ-14+. Mean age of participants completing the CFQ-14+ interviews was 22.9 (SD = 6.24). All of the parent respondents were mothers. In the child sample, illness severity as measured by FEV1 ranged from 75% to 104% predicted (M = 90.4, SD = 9.29), with most younger children falling into the mildly ill category. Illness severity in the adolescent/adult sample ranged from 23% to 135% predicted (M = 79.3, SD = 35), representing mildly to severely ill patients.
Procedures
The revised Cystic Fibrosis Questionnaires (CFQ-Child, CFQ-Parent, CFQ-14+;
Henry, Grosskopf, et al.,
1997
; Quittner et al.,
1998
) were administered along with a shorter list of cognitive
probes. Follow-up questions were written to specifically address the
problematic items or response formats identified in the first evaluation. One
additional item was added to the CFQ-P in order to test the phrasing of a
school performance question. Parents responded to both versions of the item
and were then queried about their understanding of the item and how their
response might differ between the two choices.
| Study 2: Results |
|---|
|
|
|---|
The goal of the second round of qualitative interviews was to identify any remaining problems with the measure. Thus, we conducted the following analyses: (1) identification of remaining semantic and conceptual problems with the revised versions of the questionnaires; (2) inspection of the response distributions across participants in studies 1 and 2; (3) scoring of the CFQ measures across both studies, with an analysis of the means and standard deviations of the scales; (4) correlational analyses between illness severity and quality of life scores for the CFQ-C and CFQ-14+; and (5) review of the clarity, scope, and relevance of the measures via open-ended comments elicited at the end of the interviews.
First, the item-specific revisions of medical terms ("aerosol or mist treatment," "cough up mucus") were reviewed. Responses across all three versions of the CFQ indicated that these changes were beneficial, with little ambiguity or confusion reported. The only problem that remained was the description of mucus; several respondents noted that they may cough up blood in the absence of green mucus; thus, this item was changed back to "traces of blood" (see Table III).
|
Second, the changes made to clarify the semantic and conceptual meaning of the items following the first evaluation appeared to resolve many of the problems. An item that remained problematic referred to emotional and physical health as a whole ("pleine forme" in French). This item was still interpreted as referring primarily to physical health and probably reflects the cultural value in the United States placed on physical fitness. After consulting with Dr. Henry, a decision was made to revise this item to "felt well" for subsequent studies. Factor analytic procedures with larger samples will be needed to determine which scale this item loads on, Physical Functioning or Energy and Well-Being. Rewording the item "full of energy" to "energetic" worked well for the parent and adult samples but was too difficult a word for children. Therefore, this item for the CFQ-C was changed to "felt you had enough energy."
All of the children and most of the adolescents, adults, and parents reported that the instructions and items were clear and that the items were highly relevant and easy to understand. Two parents commented on the redundancy of some items, and two adults found the true-false rating scale difficult to complete. A few of the questions related to school functioning were also described as problematic, since two of the children (20% of the sample) were home-schooled. Additionally, one parent noted that it was difficult to answer questions about her child's school behavior since she is not there to observe it. Suggestions were made by both parents and adolescents/adults for additional items. These included sibling issues, teacher lack of awareness, concerns about prognosis and mortality, marital relationships, and financial difficulties.
The samples from both evaluations were combined to examine the response distributions. In general, a majority of items on all three versions of the CFQ elicited responses in all four response categories (all categories endorsed for 50%-69% of the items, three categories endorsed for 24%-45.5% of the items, two categories endorsed for 4.5%-7% of the items). One other item distribution issue was noted. A gender difference was found on both the CFQ-C and CFQ-14+ for one item appearing on the Body Image scale. Girls tended to rate the statement "you think you're too thin" as "false" or "very false," whereas boys tended to endorse this as "true" or "very true."
Next, the three versions of the CFQ were scored for the respondents in both
studies. Given the small sample size, these results should be considered
descriptive and preliminary. Means and standard deviations were calculated,
and those that appeared in all three versions of the CFQ are shown in
Figure 1. As expected, because
illness severity is related to older age in CF, adolescents and adults had
lower HRQOL scores than children. Significant differences were found on the
Energy and Well-Being (t [37] = 3.50, p <.01), Social
Limitations (t [37] = 2.15, p <.05)(child and adult
versions only), and Treatment Constraints (t [37] = 2.0, p
=.05) scale, and a marginal difference was found on the Respiratory Scale
(t [37] = 1.86, p <.07). Consistent with the literature
on emotional adaptation to CF (DiGirolamo
et al., 1997
; Quittner et al.,
1996
), we found no differences between children and
adolescents/adults on the Emotional State scale.
|
Significant associations were also expected between illness severity (FEV1 values) and quality of life scores on the CFQ-14+. Correlational analyses revealed positive associations between pulmonary functioning values (e.g., better lung function) and higher HRQOL on the Physical Functioning (r [17] =.62, p <.01), Body Image (r [17] =.56, p <.05), and Emotional State (r [17] =.54, p <.05) scales. (We did not compute correlations between FEV1 scores and the CFQ-Child version because of a restricted range of illness severity in this group.)
| Discussion |
|---|
|
|
|---|
In general, the major objective of this study, to conduct a linguistic validation of a French, disease-specific measure of HRQOL for children, parents, and adults with CF was met. The systematic forward and backward translation process yielded measures that were generally clear to the respondents and highly relevant to their assessment of quality of life. Most important, children as young as 7 years of age were able to complete the measure with little difficulty, and adults at the older end of the developmental spectrum (41 years of age) found the CFQ-14+ to be face valid and applicable to their daily functioning. Results from the first qualitative evaluation, however, did reveal problems with the wording of certain items, particularly medical terms, and minor problems with conceptual equivalence. Corrections of terminology (e.g., "sprays" to "aerosol treatments") were fairly easy to make, and the second phase of cognitive testing indicated that these problems were resolved.
Most of the semantic and conceptual problems were also easy to address by slightly modifying the wording of specific items and retesting them in the second qualitative evaluation. One phrase in French (i.e., "pleine forme"), which likely reflects a culturally specific view of health as both physical and emotional, remained difficult to convey in English. Although this item loaded on the Energy and Well-Being scale in the French version, feedback during both phases of cognitive testing indicated that it may load more strongly on the Physical Functioning scale in the English version. Factor analyses with larger samples will be needed to test this.
The response categories were also reported to be clear and easy to use. Adequate response distributions were found in the combined study samples across all three versions of the measure. One gender difference was noted for an item on the Body Image scale. Females tended to rate "thinness" as less of a problem than males. This is consistent with cultural pressures for females in the United States to be thin and has been noted clinically as problematic in the CF literature. This potential gender difference will have to be re-examined in a larger sample.
Although the combined samples sizes for the two studies were small, results on the scaled scores were in the expected direction. Higher quality of life scores were obtained for children, who are healthier and have fewer medical problems, than for adolescents and adults. Significant associations were also found between pulmonary function scores and the CFQ. Ratings of HRQOL were also similar for parents and their children, although a statistical test of this association will require a greater number of parent-child dyads.
Two methodological limitations should be noted. First, a small number of children, parents, and adults participated in both rounds of cognitive testing. Although this was appropriate for the purposes of evaluating the adequacy of the translation and identifying problems with conceptual equivalence, it precludes the use of most traditional psychometric tests (e.g., internal consistency). In addition, we recruited samples of volunteers who were willing to participate in a lengthy and intensive interview process, and these samples may not be representative of the larger pool of patients with CF and their family members. Efforts were made, however, to recruit participants from several CF centers to increase the diversity and generalizability of the sampled responses.
We encountered several challenges in conducting a multinational measurement
study of this type that are worth noting. First, the research team represented
several different disciplines (e.g., psychology, pharmacy, health economics),
who brought different views and training backgrounds to the research
enterprise. This was a strength in terms of adding the perspectives of
different social science fields, but it added a layer of complexity to our
communication and decision-making process (e.g., traditional rules of survey
development vs. observations of how individuals responded to the items).
Second, we experienced difficulties in trying to balance our goal of producing
a "pure" translation of the original French measure into English
that still captured the substance and meaning of the items for patients and
family members living in the United States. These tensions are not unique to
our development of the English CFQ and have been well-documented in other
efforts to develop multinational measures
(Cella, Lloyd, & Wright,
1996
). The extent of our success in reaching this goal will be
evident once we complete the national psychometric validation of the CFQ in
this country. Finally, there were pragmatic difficulties involved in
communicating with multiple investigators in the United States and France. We
were able to arrange two long-distance conference calls to discuss potential
changes in wording and question formats. This level of communication was
necessary to maintain the integrity of the process, but posed significant
financial and practical difficulties.
Future Directions
In the United States, the linguistic validation of the CFQ has now been
completed, and the next step is to conduct a large-scale, psychometric
evaluation of the measure. Funding has recently been obtained to conduct a
national assessment of the measure at 22 different CF centers. One hundred and
fifty respondents will complete each version of the measure. This will allow
us to apply classical test theory to examine internal consistency, convergent
and discriminant validity, and factor structures
(Quittner et al. 1999
).
Convergent validity will also be calculated between CFQ scores for
parent-child dyads and with a generic HRQOL measure. Finally, in a subset of
patients, a short-term (7 to 10 days) test-retest reliability study will be
completed. Once this national database is available and analyzed, a manual
will be developed containing the three CFQ measures, means and standard
deviations, and other psychometric data. This manual will be made available to
researchers and clinicians in the CF community.
| Acknowledgments |
|---|
This research was supported in part by Genentech, Inc., and by a National Institutes of Health Research Career Development Award (K04#HL02892) to A. Quittner. We thank the participants, who were generous with their time and effort.
Received February 28, 1999; revision received September 4, 1999; accepted November 4, 1999
| References |
|---|
|
|
|---|
Abbott, J., Bilton, D., Dodd, M., & Webb, A. (1994). Treatment compliance in adults with cystic fibrosis. Thorax, 49, 115-120.
Acquadro, C., Jambon, B., Ellis, D., & Marquis, P. (1996). Language and translation issues. In B. Spilker (Ed.), Quality of life and pharmacoeconomics in clinical trials (2nd ed., pp. 575-586). Philadelphia: Lippincott-Raven.
Anderson, R. T., Aaronson, N. K., Leplegè, A. P., & Wilkin, D. (1996). International use and application of generic health-related quality of life instruments. In B. Spilker (Ed.), Quality of life and pharmacoeconomics in clinical trials (2nd ed., vol. 67, pp. 613-633). Philadelphia: Lippincott-Raven.
Brice, T. W., & Kalimo, E. (1971). Comparisons of health-related attitudes: A cross national factor analytic study. Social Science Medicine, 5, 283-318.
Bullinger, M., Anderson, R., Cella, D., & Aaronson, N. (1993). Developing and evaluating cross-cultural instruments from minimum requirements to optimal models. Quality of Life Research, 2, 451-459.
Buschbach, J., Horiks, P., van den Bosch, J., de la Riviere, A.,
& de Charro, F. (1994). Measuring the quality of life before
and after bilateral lung transplantation in patients with cystic fibrosis.
Chest, 105,
911-917.
Caine, N., Sharples, L., Smyth, R., Scott, J., Hathaway, T., Higenbottam, T., & Wallwork, J. (1991). Survival and quality of life of cystic fibrosis patients before and after heart-lung transplantation. Transplant Procedures, 23, 1203-1204.
Cella, D. F., Lloyd, S. R., & Wright, B. D. (1996). Cross-cultural instrument equating: Current research and future directions. In B. Spilker (Ed.), Quality of life and pharmacoeconomics in clinical trials (pp. 707-716). Philadelphia: Lippincott-Raven.
Czyzewski, D., Mariotto, M., Bartholomew, K., LeCompre, S., & Sockrider, M. (1994). Measurement of quality of well-being in a child and adolescent cystic fibrosis population. Medical Care, 23, 965-972.
Dennis, C., Caine, N., Sharples, L., Smyth, R., Higenbottam, T., & Stewart, S. (1993). Heart-lung transplantation for end-stage respiratory disease in patients with cystic fibrosis at Papworth Hospital. Journal of Heart Lung and Transplant, 12, 893-902.
DiGirolamo, A. M., Quittner, A. L., Ackerman, V., & Stevens, J. (1997). Identification and assessment of ongoing stressors in adolescents with a chronic illness: An application of the Behavior Analytic Model. Journal of Clinical Child Psychology, 26, 53-66.[Web of Science][Medline]
Drotar, D. (1998). Measuring health-related quality of life in children and adolescents: Implications for research and practice. Mahwah, NJ: Lawrence Erlbaum.
Eigen, H., Clark, N., & Wolle, J. (1987). NHLBI Workshop summary: Clinical behavioral aspects of cystic fibrosis. Directions for future work. American Review Respiratory Disease, 136, 1509-1513.
Feeny, D., Torrance, G., & Labelle, R. (1996). Integrating economic evaluations and quality of life assessments. In B. Spilker (Ed.), Quality of life and pharmacoeconomics in clinical trials (2nd ed., pp. 85-97). Philadelphia: Lippincott-Raven.
Fuchs, H., Borowitz, D., & Christiansen, D.
(1994). Effect of aerosolized recombinant human DNase on
exacerbations of respiratory symptoms and on pulmonary function in patients
with cystic fibrosis. New England Journal of Medicine,
331, 637-642.
Guyatt, G., Naylor, D., Juniper, E., Heyland, D., Jaeschke, R., & Cook, D. (1997). Users' guides to the medical literature. Journal of American Medical Association, 227, 1232-1237.
Henry, B., Aussage, P., Grosskopf, C., & Goehrs, J. M. (1998). Evaluating quality of life (QOL) in children with cystic fibrosis (CF): Should we believe the child or the parent? [Abstract]. Pediatric Pulmonology, 26(suppl).
Henry, B., Aussage, P., Grosskopf, C., & Launois, R. (1996). Constructing a disease-specific quality of life questionnaire for children and adults with cystic fibrosis. Isreali Journal of Medical Science, 32(suppl), S181.
Henry, B., Aussage, P., Staab, D., Prados, C., Grosskopf, C., & Goehrs, J. M. (1997). Assessing cross-cultural validity of the Cystic Fibrosis Questionnaire. [Abstract]. Pediatric Pulmonology, 24(suppl 17).
Henry, B., Grosskopf, C., Aussage, P., Goehrs, J. M., Launois, R., & the French CFQoL Study Group. (1997). Construction of a disease-specific quality of life questionnaire for cystic fibrosis. Pediatric Pulmonology, 13(suppl), 337-338.
Henry, B., Staab, D., Prados, C., Aussage, P., de Ponthbrune, S., Grosskopf, C., & Goehrs, J. M. (1998). How to measure quality of life in Cystic Fibrosis (CF) patients across countries and cultures: The Cystic Fibrosis Questionnaire (CFQ) [Abstract]. Pediatric Pulmonology, 17(suppl), 392.
Hui, C., & Triandis, H. (1985). The instability of
response sets. Public Opinion Quarterly,
49, 253-260.
Juniper, E., Guyatt, G., Ferrie, P., & Griffith, L. (1993). Measuring quality of life in asthma. American Review of Respiratory Disease, 147, 832-838.[Web of Science][Medline]
Juniper, E., Guyatt, G., & Jaeschke, R. (1996). How to develop and validate a new health-related quality of life instrument. In B. Spilker (Ed.), Quality of life and pharmacoeconomics in clinical trials (2nd ed., pp. 49-56). Philadelphia: Lippincott-Raven.
Kaplan, R., Anderson, J., Wu, A., Mathews, C., Kozin, F., & Orenstein, D. (1989). The Quality of Well-Being Scale. Medical Care, 27(suppl), 27-43.
Knudson, R. J., Slatin, R. C., Lebowitz, M. D., & Burrows, B. (1976). The maximal expiratory flow-volume curve. American Review of Respiratory Disease, 113, 587-600.[Web of Science][Medline]
Landgraf, J., Abetz, L., & Ware, J. E. (1996). Child Health Questionnaire (CHQ): A user's manual. Boston: The Health Institute, New England Medical Center.
Lessler, J. T., & Forsyth, B. H. (1996). A coding system for appraising questionnaires. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative process in survey research (pp. 259-292). San Francisco: Jossey-Bass.
Levi, R., & Drotar, D. (1998). Critical issues and needs in health-related quality of life assessment in children and adolescents with chronic health conditions. In D. Drotar (Ed.), Measuring health-related quality of life in children and adolescents (pp. 3-24). Mahwah, NJ: Lawrence Erlbaum.
Munzenberger, P., Van Wagnen, C., Abdulhamid, I., & Walker, P. (1999). Quality of life as treatment outcome in patients with cystic fibrosis. Pharmacotherapy, 19, 393-398.[Web of Science][Medline]
Orenstein, D., & Kaplan, R. (1991). Measuring the
quality of well-being in cystic fibrosis and lung transplantation.
Chest, 100,
1016-1018.
Orenstein, D., Nixon, P., Ross, E., & Kaplan, R.
(1989). The quality of well-being in cystic fibrosis.
Chest, 95,
344-347.
Orenstein, D., Pattishall, E., Nixon, P., Ross, E., & Kaplan,
R. (1990). Quality of well-being before and after antibiotic
treatment of pulmonary exacerbation in patients with cystic fibrosis.
Chest, 98,
1081-1084.
Quittner, A. L. (1998). Measurement of quality of life in cystic fibrosis. Current Opinion in Pulmonary Medicine, 4, 326-331.[Medline]
Quittner, A. L., Drotar, D., levers-Landis, C., Seidner, D., Slocum, N., & Jacobsen, J. (2000). Adherence to medical treatments in adolescents with cystic fibrosis: The development and evaluation of family-based interventions. In D. Drotar (Ed.), Promoting adherence to medical treatment in childhood chronic illness: Interventions and methods (pp. 383-407). Hillsdale, NJ: Lawrence Erlbaum.
Quittner, A. L., Espelage, D. L., Watrous, M. L., FitzSimmons, S., Munzenberger, P., Davis, M. A., Fisher, L. A., & Henry, B. (1999). U.S. Validation of a disease-specific quality of life measure for cystic fibrosis: The Cystic Fibrosis Questionnaire (CFQ). Paper presented at the meeting of the 13th Annual North American Cystic Fibrosis Conference, Seattle, WA.
Quittner, A. L., Sweeny, S., Watrous, M., Munzenberger, P., & Henry, B. (1998). Initial US validation of a disease-specific quality of life (QOL) measure for cystic fibrosis: The Cystic Fibrosis Questionnaire [Abstract]. Pediatric Pulmonology, 27 (suppl 17).
Quittner, A. L., Tolbert, V. E., Regoli, M. J., Orenstein, D.,
Hollingsworth, J. L., & Eigen, H. (1996). Development of the
Role-play Inventory of Situations and Coping Strategies (RISCS) for parents of
children with cystic fibrosis. Journal of Pediatric
Psychology, 21,
209-235.
Ramsey, B., Astley, S., & Aitken, M. (1993). Efficacy and safety of short-term administration of aerosolized recombinant human deoxyribonuclease in patients with cystic fibrosis. American Review of Respiratory Disease, 148, 145-151.[Web of Science][Medline]
Ramsey, B., & Boat, T. (1994). Outcome measures for clinical trials in cystic fibrosis. Journal of Pediatrics, 124, 177-192.[Web of Science][Medline]
Spilker, B. (1996). Introduction. In B. Spilker (Ed.), Quality of life and pharmacoeconomics in clinical trials (2nd ed., pp. 1-10). Philadelphia: Lippincott-Raven.
Stark, L., Quittner, A. L., Opipari, L., Jelalian, E., Jones, E., Poweres, W., Higgins, L., Maguiness, K., Seidner, D., Hoover, J., Eigan, H., Duggan, C., & Stallings, V. (1998, October). The contribution of behavior therapy to enhancing adherence in school age children with CF: The example of diet. Paper presented at the 12th Annual North American Cystic Fibrosis Conference, Montreal, Canada.
Tullis, E., & Guyatt, G. (1995). Quality of life in cystic fibrosis. Pharmacoeconomics, 8, 23-33.[Web of Science][Medline]
Varni, J. W., Seid, M., & Rode, C. A. (1999). The PedsQoL: Measurement model for the Pediatric Quality of Life Inventory. Medical Care, 37, 126-139.[Web of Science][Medline]
Ware, J. D., Gandek, B. L., Keller, S. D., & Group, I. P. (1996). Evaluating instruments used cross-nationally: Methods from the IQOLA Project. In B. Spilker (Ed.), Quality of life and pharmacoeconomics in clinical trials (2nd ed., vol. 71, pp. 681-693). Philadelphia: Lippincott-Raven.
![]()
CiteULike
Connotea
Del.icio.us What's this?
This article has been cited by other articles:
![]() |
H A Hutchings, P Upton, W-Y Cheung, A Maddocks, C Eiser, J G Williams, I T Russell, S Jackson, and M E M Jenney Adaptation of the Manchester-Minneapolis Quality of Life instrument for use in the UK population Arch. Dis. Child., October 1, 2007; 92(10): 855 - 860. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Rosenfeld An Overview of Endpoints for Cystic Fibrosis Clinical Trials: One Size Does Not Fit All Proceedings of the ATS, August 1, 2007; 4(4): 299 - 301. [Abstract] [Full Text] [PDF] |
||||
![]() |
C. H. Goss and A. L. Quittner Patient-reported Outcomes in Cystic Fibrosis Proceedings of the ATS, August 1, 2007; 4(4): 378 - 386. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. A. Riekert, S. J. Bartlett, M. P. Boyle, J. A. Krishnan, and C. S. Rand The Association Between Depression, Lung Function, and Health-Related Quality of Life Among Adults With Cystic Fibrosis Chest, July 1, 2007; 132(1): 231 - 237. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Schnabel, C. Grasemann, D. Staab, H. Wollmann, F. Ratjen, and for the German Cystic Fibrosis Growth Hormone Stud A Multicenter, Randomized, Double-Blind, Placebo-Controlled Trial to Evaluate the Metabolic and Respiratory Effects of Growth Hormone in Children With Cystic Fibrosis Pediatrics, June 1, 2007; 119(6): e1230 - e1238. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. P. Agans, N. Deeb-Sossa, and W. D. Kalsbeek Mexican Immigrants and the Use of Cognitive Assessment Techniques in Questionnaire Development Hispanic Journal of Behavioral Sciences, May 1, 2006; 28(2): 209 - 230. [Abstract] [PDF] |
||||
![]() |
D. J. Serisier, J. K. Shute, P. M. Hockey, B. Higgins, J. Conway, and M. P. Carroll Inhaled heparin in cystic fibrosis Eur. Respir. J., February 1, 2006; 27(2): 354 - 358. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. R. Elkins, M. Robinson, B. R. Rose, C. Harbour, C. P. Moriarty, G. B. Marks, E. G. Belousova, W. Xuan, P. T.P. Bye, and the National Hypertonic Saline in Cystic Fibrosis A Controlled Trial of Long-Term Inhaled Hypertonic Saline in Patients with Cystic Fibrosis N. Engl. J. Med., January 19, 2006; 354(3): 229 - 240. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. H. Donaldson, W. D. Bennett, K. L. Zeman, M. R. Knowles, R. Tarran, and R. C. Boucher Mucus Clearance and Lung Function in Cystic Fibrosis with Hypertonic Saline N. Engl. J. Med., January 19, 2006; 354(3): 241 - 250. [Abstract] [Full Text] [PDF] |
||||
![]() |
A. L. Quittner, A. Buu, M. A. Messer, A. C. Modi, and M. Watrous Development and Validation of the Cystic Fibrosis Questionnaire in the United States: A Health-Related Quality-of-Life Measure for Cystic Fibrosis Chest, October 1, 2005; 128(4): 2347 - 2354. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. G. Liou, F. R. Adler, and D. Huang Use of Lung Transplantation Survival Models to Refine Patient Selection in Cystic Fibrosis Am. J. Respir. Crit. Care Med., May 1, 2005; 171(9): 1053 - 1059. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. H. C. Klijn, A. Oudshoorn, C. K. van der Ent, J. van der Net, J. L. Kimpen, and P. J. M. Helders Effects of Anaerobic Training in Children With Cystic Fibrosis: A Randomized Controlled Study Chest, April 1, 2004; 125(4): 1299 - 1305. [Abstract] [Full Text] [PDF] |
||||
![]() |
A. C. Modi and A. L. Quittner Validation of a Disease-Specific Measure of Health-Related Quality of Life for Children with Cystic Fibrosis J. Pediatr. Psychol., December 1, 2003; 28(8): 535 - 546. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. Saiman, B. C. Marshall, N. Mayer-Hamblett, J. L. Burns, A. L. Quittner, D. A. Cibene, S. Coquillette, A. Y. Fieberg, F. J. Accurso, and P. W. Campbell III Azithromycin in Patients With Cystic Fibrosis Chronically Infected With Pseudomonas aeruginosa: A Randomized Controlled Trial JAMA, October 1, 2003; 290(13): 1749 - 1756. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. Goldbeck, T. G. Schmitz, G. Henrich, and P. Herschbach Questions on Life Satisfaction for Adolescents and Adults With Cystic Fibrosis: Development of a Disease-Specific Questionnaire Chest, January 1, 2003; 123(1): 42 - 48. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. T. Britto, U. R. Kotagal, R. W. Hornung, H. D. Atherton, J. Tsevat, and R. W. Wilmott Impact of Recent Pulmonary Exacerbations on Quality of Life in Patients With Cystic Fibrosis Chest, January 1, 2002; 121(1): 64 - 72. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. G. Liou, F. R. Adler, B. C. Cahill, S. C. FitzSimmons, D. Huang, J. R. Hibbs, and B. C. Marshall Survival Effect of Lung Transplantation Among Patients With Cystic Fibrosis JAMA, December 5, 2001; 286(21): 2683 - 2689. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||










