In epidemiological surveys, the prevalence of asthma and other wheezing disorders is usually assessed by questionnaires. Because objective tests are difficult to perform, especially in young children, and their sensitivity and specificity for the diagnosis of asthma are questionable,1 the design of appropriate questionnaires is crucial. Basically, a new instrument needs to demonstrate repeatability and validity. This paper focuses on test–retest repeatability, a measure of the consistency of the performance of a questionnaire when used under similar circumstances.

Standard respiratory questionnaires for schoolchildren have shown good or satisfactory repeatability2^–6 but may not be appropriate for infants and preschool children, where symptoms may differ due to developmental changes and rely exclusively on proxy reports. For instance, limitation of speech with severe wheeze and exercise-induced symptoms cannot be assessed in infants unable to talk and run. Moreover, different clinical phenotypes are thought to co-exist within the large group of preschool children suffering from wheeze.7^–9 To distinguish these phenotypes, symptoms must be assessed in detail. Therefore, standard questionnaires developed for schoolchildren cannot be applied automatically for very young children. Few targeted questionnaires exist. Their repeatability has been tested in small numbers of children recruited in neonatal units or outpatient clinics10 or excluding ethnic minority groups.11 They are therefore not necessarily generalisable.

We developed a new postal questionnaire for use in a large population-based cohort study of respiratory symptoms in preschool children in Leicestershire.12^–16 This paper describes the short-term repeatability of this questionnaire when it was reapplied to parents of 1-year-old children after a 3-month interval.

METHODS

Setting and study population

In 1998, we used the Leicestershire Child Health Database, which includes the birth notification with mother’s self-reported ethnic origin, religion, country of birth and language, to select a random sample of 3500 white (mother self-identified as British Isles or European) and 800 south Asian (mother self-identified as Indian, Pakistani or Bangladeshi) children aged 1 year (ie, in their second year of life at recruitment) with complete birth records. South Asians, the largest ethnic minority group, accounted for 14% of 1-year-old children resident in the county at that time.

Questionnaire and mailings

We developed a short four-page questionnaire that could be posted to families with young children, for self-administration by parents (the questionnaire is available on the ADC website at http://adc.bmj.com/supplemental). It contained sections on upper and lower respiratory symptoms and diagnoses, healthcare utilisation and treatments for wheeze, environmental exposures including indoor air pollutants, breastfeeding, pets, nursery care, number of siblings and other household members, parental history of atopic diseases and ethnicity, language and social conditions. Some questions were derived from an earlier questionnaire used locally11 or from the International Study of Asthma and Allergy in Childhood (ISAAC).6 Others were newly developed. The sources of the different questions are summarised in the online table 1 (available at http://adc.bmj.com/supplemental). All questionnaires were printed in English, with an accompanying letter translated into the four main local south Asian languages offering translation services. We sent this questionnaire to all 4300 families, with a reminder letter to non-responders 6 weeks later (first mailing in April, reminder in May, baseline survey). Within 3 months, 3194 (74%) questionnaires had been returned. An identical questionnaire with an explanatory letter was then sent exactly 3 months later (in July, repeat survey) to a random sample of responders stratified by ethnic group (600 white and 200 south Asian families).

Data analyses

Statistical analyses were performed with Stata, version 8.2 (Stata, Austin, TX). Symptom prevalence at baseline was calculated separately for participants and non-participants of the repeat survey, and comparison between these groups was assessed by χ² tests (dichotomous variables) and tests for trend (categorical variables). To assess repeatability, Cohen’s kappa (κ) coefficients were calculated.17 κ compares the observed agreement between two assessments made on two different occasions, with the agreement that would be expected simply by chance. Because the κ coefficient is sensitive to the population prevalence of responses and also to asymmetrical imbalance in marginal totals, we also present the percentage of observed total agreement (number of positive and negative answers to both questionnaires divided by the total), and the separate proportions of positive and negative agreement (number of answers in positive agreement divided by the average number of positive answers; number of answers in negative agreement divided by the average number of negative answers). These values help in understanding individual results.18 19 For ordinal and quantitative variables, agreement was assessed as intraclass correlation coefficients (ICC). Landis and Koch20 have suggested that κ coefficients ⩽0.4 indicate poor agreement, values of 0.41–0.60 moderate agreement, 0.61–0.80 good agreement and >0.8 excellent agreement. To examine which factors might influence agreement, we stratified analyses by household language (English, other), ethnicity (white, south Asian), the Townsend local area deprivation score (low, medium, high), interval between baseline and repeat survey (<3 months, ⩾3 months), domicile (urban, rural) and respondent (same respondent, different respondent to repeat questionnaire, and mother vs father). The equality of subgroup agreement measurements was tested using the methods of Donner et al.21 22 Missing values were coded as “no” because sensitivity analyses showed no difference in results.

RESULTS

Response rates

The response rates were 74% (3194/4300) in the baseline survey and 58% (460/800) in the repeat survey, and higher for white (n = 368/600, 61%) than for south Asian children (n = 92/200, 46%, p<0.001). Most baseline questionnaires were completed in April or May (n = 2691/3194, 84%), repeat questionnaires were mostly completed in July and August (n = 451/460, 98%). The median (interquartile range) age of the children was 17.7 (14.8–20.7) months at the baseline survey and 20.5 (17.7–23.2) months at the repeat survey. For test–retest analyses, we included the 413 children with the same respondent (mother or father) in both occasions, as conventional.2 However, results were very similar in a sensitivity analysis including all 460 questionnaires.

Prevalence of respiratory symptoms

The 413 study participants who replied to the repeat questionnaire had a lower prevalence of wheeze, shortness of breath and cough in the baseline survey than the children who failed to respond (table 1). Among 413 participants in the repeat survey, the overall prevalence of respiratory symptoms was similar on the two occasions.

Repeatability

The repeatability of the most important symptoms are shown in table 2 (see also online tables 2 and 3 at http://adc.bmj.com/supplemental which give detailed results for all symptoms, family history and environmental exposures). In general, agreement was excellent (κ >0.8) for family history of atopic diseases and environmental exposures (smoking, breastfeeding, pets, cooking fuel). Questions on the frequency and severity of wheeze, asthma diagnosis, inhaler treatment and healthcare utilisation had good repeatability (κ 0.61–0.80). Repeatability was moderate for questions on triggers of wheeze and cough (κ 0.49–0.72). In contrast, agreement for questions on chronic cough, upper respiratory symptoms and skin problems were (with the exception of a diagnosis of eczema) only moderate (κ 0.39–0.68). Stratified analyses did not show any significant differences in agreement between subgroups, with one exception: repeatability of several questions was poorer when both questionnaires had been completed by the fathers (n = 39) compared to mothers (n = 374). Low p Values were found for questions on night cough (κ 0.10 vs 0.44 for fathers and mothers, respectively, p = 0.003) and cough apart from colds (κ 0.28 vs 0.56, p = 0.09), snoring (κ 0.51 vs 0.67, p = 0.08), eczema (κ 0.22 vs 0.70, p = 0.003), paternal smoking (κ 0.53 vs 0.88, p = 0.001), number of physician visits (ICC 0.55 vs 0.76, p = 0.02), posseting (ICC 0.36 vs 0.70, p = 0.002) and duration of breastfeeding (ICC 0.85 vs 0.96, p<0.001).

DISCUSSION

This paper presents the repeatability of a short respiratory questionnaire designed for self-completion by parents of 1-year-old children. Repeatability was excellent for sections on family history and environmental exposures, good for questions on wheeze, asthma, treatment and healthcare utilisation over the past 12 months, and moderate for upper respiratory symptoms and cough.

One strength of our study compared to most others is the relatively large sample size, providing increased precision (reasonable confidence intervals around κ coefficients). We had a well-defined population-based study group, including British south Asians, the main local ethnic minority group, in a proportion similar to that of the general population (14%). The only other published repeatability studies for preschool questionnaires of which we are aware included few and/or selected children: 72 hospital-based children10 and <100 (exact number not quoted) children of white ethnicity.11 The response rate in the repeat survey (58%) compares well with response rates of 47–50% in other repeatability studies (table 3).

As in other surveys,2 respondents in the repeatability study were less symptomatic than non-respondents. κ depends on the marginal observed prevalence.18 With a fixed agreement rate, κ is maximal for a prevalence of 0.5 and decreases if the prevalence approaches 0 or 1.23 In our study, the prevalence of wheeze in the last 12 months in the participants of the repeatability study was 25%, with a resulting κ of 0.69 and an agreement rate (P_O) of 88%. For a prevalence of 35%, as in the total study population, κ would have been 0.74 (assuming the same agreement rate). Thus we may have underestimated κ values. On the other hand, respondents may be more reliable than non-respondents, leading to an overestimation of κ. Overall, the resulting bias is likely to be small. Other characteristics of the study, such as the age of the children, the time period over which the questionnaire extends, who responded to the questionnaire and the interval between repeat surveys, will have a bigger impact on κ values.23 These differences in methodology have to be taken into account when comparing different studies (table 3). For instance, short intervals between the two measurements4 10 24 give the parents less time to forget previous answers and the children less opportunity to change their true symptom status. Also, we need to be cautious when extrapolating our results from 1-year-old children to older toddlers. For instance, repeatability of the question on duration of breastfeeding is likely to be poorer if the question is asked at an older age.

It is reassuring that repeatability did not differ much between subgroups defined by language, ethnicity or social class, although power for these comparisons was limited. Where fathers responded, significantly lower repeatability was found for questions on current symptoms and infant care, confirming clinical observations. In most families, mothers spend more time at home with young children than fathers and are more likely to take time off work when the children are ill. Therefore, mothers usually provide more accurate reports on children’s health status. The relatively poor repeatability we found for the questionnaires completed by fathers might, however, not have a large impact for the interpretation of questionnaire surveys, because most questionnaires are usually completed by mothers.

Although the absence of objective measurements of atopy and environmental exposures such as parental smoking in epidemiological studies is often criticised, the high repeatability of parental answers gives confidence in these questionnaires. When interpreting the results for symptoms over the past 12 months, it has to be kept in mind that perfect repeatability can never be obtained because the time windows do not completely overlap. We sent the second questionnaire 3 months after the first one, so that new symptoms will have developed in some of the children and remitted in others.

This is not the only cause for imperfect agreement, as shown for the symptom “wheeze ever” which, illogically, was reported in the first but not the repeat questionnaire by about 5% of parents (as also in Brunekreef et al2) (table 2). Poorer repeatability for cough and upper respiratory symptoms compared to wheeze has also been noted in other studies. This may be partly explained by high short-term variability in these symptoms, making it likely that symptoms are reported if they have occurred recently, but not if the last episode happened many months previously. As all questionnaires were sent at the same time of the year, we could not compare repeatability between different seasons of the year. In contrast, wheeze, shortness and breath and inhaler use might be recalled more consistently because parents are more concerned. For skin problems, repeatability was significantly better for the question on diagnosis (“eczema”) than for ISAAC questions on symptoms (“itchy rash”) (table 2 and online table 2).

Finally, we want to stress that having shown a good repeatability does not allow the conclusion that our questionnaire has also a good validity. While repeatability refers to the reproducibility of a measurement, validity refers to whether the questionnaire measures what it intends to measure. The best way to assess validity is to compare answers to the questionnaire with objective measurements. These might include repeated respiratory sound recordings to validate reports of wheeze and cough, comparison with hospitalisation or GP records to validate health care utilisation and inhaler use, or measurement of urinary cotinine to validate exposure to environmental tobacco smoke. It was not possible for us to do this.

In conclusion, this short postal respiratory questionnaire developed for cross-sectional and longitudinal studies of preschool wheeze and other respiratory symptoms has a repeatability, in 1-year-old children, similar to standard respiratory questionnaires for older children, and could therefore be recommended for further use in community-based studies in this age group.