These chapters were published with modifications by Oxford University Press, 2004

Reporting and Review of Human Genome Epidemiology Studies
Julian Little

Tables | Box | References

Introduction
The recent completion of the first draft of the human genome sequence (1,2,16) and advances in technologies for genomic analysis are generating tremendous opportunities for epidemiologic studies to evaluate the role of genetic variants in the etiology of human disease (3). The basis of this evaluation will be identification of the allelic variants of human genes, description of the frequency of these variants in different populations, identification of diseases influenced by these variants and assessment of the magnitude of the associated risk, and identification of gene-environment and gene-gene interactions. The process of identifying DNA variation that may be associated with disease is under way through the cataloguing and mapping of single nucleotide polymorphisms (SNPs) throughout the genome. The analysis of genotype data on SNPs may aid in the identification of DNA alterations that result in or contribute to disease states.

Not surprisingly, the number of published human genome epidemiologic studies has increased rapidly (4). Therefore, integration of evidence will become increasingly important as a means of dealing with potentially unmanageable amounts of information. Heterogeneity between studies can be assessed, and when this occurs, attempts can be made to explain it. So far, few gene-disease associations have been replicated (5-7). This is also true for gene-environment and gene-gene interaction (8,9). It is important to determine how far methodologic issues may account for differences between studies. This requires that the studies are adequately reported and appraised. Investigation of heterogeneity between studies can lead to the formulation of new hypotheses.

In this chapter, we consider the reporting and systematic review of human genome epidemiologic studies. Systematic reviews differ from traditional reviews in that systematic reviews are supported by evidence that is integrated in explicitly defined stages (see below). Meta-analyses form a subset of systematic reviews in which quantitative methods are obtained to obtain an overall measure of effect across different studies or detect and explain heterogeneity between studies. Pooled analysis of data on individual subjects from multiple studies has many features in common with systematic reviews, but involves obtaining and re-analyzing the primary data, as distinct from aggregating published information (10)

Critical appraisal and integration of evidence require that the evidence be adequately reported. Brief checklists or guidelines for reporting gene-disease associations have been proposed (11,12). In this chapter, we present a more detailed overview of issues in the critical appraisal of studies of genotype prevalence, gene-disease associations, and gene-environment interactions, based in large part on the deliberations of an expert panel workshop convened by the Centers for Disease Control and Prevention and the National Institutes of Health in January 2001 (4,13). A checklist intended to guide investigators in preparation of manuscripts, to guide those who need to appraise manuscripts and published papers and to be useful to journal editors and readers is presented in Table 10-1. It should not be regarded as an exhaustive list of points that have to be presented in all journal articles. Addressing all of the considerations, for example in studies of rare conditions in clinical settings, may not always be feasible.

Reporting and Appraisal of Single Studies

Hypothesis specification
Associations between several genes and a disease can be tested according to a priori hypotheses based, for example, on a documented biologic mechanism of these genes in determining the disease. For example, the associations between a number of gene variants whose products are thought to influence the metabolism of folate and related nutrients and colorectal neoplasia have been investigated, because of the roles of folate in methylation and DNA synthesis (14). It is becoming usual practice in human genome epidemiology studies to initiate a study to test hypotheses that are current at that time and to establish a resource to test additional hypotheses proposed later on the basis of knowledge external to the resource. These are all a priori hypotheses. Hypothesis-testing is important to distinguish from hypothesis-generation.

In gene-disease association studies and studies of the prevalence of allelic variants, it has been suggested that data on genotypes are presented, because it is the genotype that determines risk (13). A point to consider in appraising studies is the choice of categories. In a two-allele system, for example, justification would be sought for the decision to consider heterozygotes separately, include them in the reference category with homozygotes for the common variant, or group them with homozygotes for the rarer variant(s). This is more complex for multi-allelic systems.

In studies of gene-environment and gene-gene interactions, many hypotheses of interaction can potentially be tested. The distinction between a priori hypotheses and hypothesis generation is, again, important. Even in the simplest case of a dichotomous genotype and dichotomous exposure, genotype and environment can interact in six ways (15). Many more can be defined if more categories are introduced. For instance, Taioli et al. (16) have proposed a model in which an effect of the genotype is apparent at low environmental exposures but is not apparent at high exposures. Once multiple categories of dose are defined for the environmental variable, many different dose-response models can be tested in the data. Clearly, model specification becomes more difficult as more environmental factors (and levels of exposure), and more genes (and alleles), are included.

Design
In appraising studies, it is important to consider design as this affects the biases that may occur and generalizability (17,18). Most studies of gene-disease associations and gene-environment and gene-gene interactions for late-onset diseases have used the case-control design. Much of the discussion therefore focuses on this design. However, DNA samples are being collected in a number of ongoing cohort studies. Compared with case-control studies, cohort studies have a number of advantages, including the capacity to examine age-at-onset distributions and multiple-disease outcomes (19-21). The use of case-cohort and nested case-control analysis of archived samples that are suitable for genotypic analysis potentially can minimize the disadvantages of the cost of genotyping an entire cohort. A major advantage of the case-cohort design for studies in which use of expensive assays is planned is that the same comparison group can be used for several different disease outcomes. Therefore, this design is likely to be used increasingly. Because the detection of gene-environment and gene-gene interaction is particularly challenging, novel study designs, most notably the case-only design and multistage designs, have been proposed (17). Concern about the possible impact of population stratification has stimulated the development of family-based case-control designs; these are discussed briefly in the section on population stratification.

Issues that are particularly important in the appraisal of studies of genotype prevalence, gene-disease associations, and gene-environment and gene-gene interactions include the analytical validity of genotyping, selection of subjects, confounding (especially as a result of population stratification), statistical power, and multiple statistical comparisons. In addition, exposure assessment is an important issue in the appraisal of studies of gene-environment interaction. Because many methodologic issues are common to the three types of study, these are discussed in parallel.

Assessment of genotypes
The definition of the genotype(s) investigated should be clearly presented. The validity of grouping genotypes on the basis of putative functional effects depends on the availability and quality of functional studies of gene variants, and information on functional effects is likely to change over time. For multi-allelic systems, genotypes have been grouped according to functional effects in some investigations. For example, grouping according to inferred rapidity of acetylation has been done for the NAT2 polymorphisms (8).

True functional variants are important to distinguish from markers associated with a disease only because they are in linkage disequilibrium with a functional variant. Typing several polymorphisms throughout a candidate gene may be useful in order to construct haplotypes, which could then be tested for association with the phenotype of interest. The increasing availability of mapped SNP markers (20-24) offers the opportunity for such an approach and presents methodologic challenges (see below).

Other factors affecting the analytical validity of genotyping, including the types of samples and timing of collection, the method used for genotyping, and quality control procedures are summarized in Table 10-1. These issues are discussed in Chapter 5 and Little et al. (13).

Assessment of exposures
Not surprisingly, exposure assessment is important in studies of gene-environment interaction. Points that need to be considered are the method of exposure assessment, and its validity and reproducibility. Exposure misclassification can bias the estimation of an interaction effect, the magnitude of which depends on the prevalence of the misclassified exposure and on the interaction model (Chapter 8, 25). If interaction is defined as lack of fit to a multiplicative model, a test for interaction will be conservative (26). In theory, case-control studies are more vulnerable to differential misclassification than are cohort studies (and the related case-cohort and nested case-control designs). However, provided that the extent of misclassification of exposure does not vary by genotype, differential misclassification between cases and controls is not a serious problem for the detection of departures from a multiplicative gene-environment joint effect (26).

Selection of subjects
Evaluation of potential selection bias requires consideration of study design and fieldwork. It is important to distinguish between studies that aim to detect an association from those that aim to estimate the magnitude of an association. In the former situation, cases may be “overselected” from multiplex families to increase the power to detect an association; presenting the measure of association as an estimate of population association would be inappropriate. In the latter situation, the principles underlying study design are essentially the same as for the investigation of the magnitude of association with environmental risk factors, including the minimization of the potential for selection bias emphasized in many epidemiologic textbooks (27-30). In a number of studies the selection of cases has not been well described (31). In a review of type1 diabetes and HLA-DQ polymorphisms, the authors noted that many studies were based on convenience samples of cases in which persons with type 2 diabetes who used insulin in their treatment regimen had been included (32). In several studies of cancer, prevalent cases have been included to varying extents (33). In these studies, bias would occur if the genotype affected survival or if genotypes were assayed by a phenotypic test that was influenced by disease progression or treatment.

A recurrent problem in case-control studies of gene-disease associations with unrelated controls has been that the controls were not selected from the same source population as the case-subjects (8, 9, 31, 32). The potential problem of selecting controls who do not represent the population from which case-subjects arise is illustrated by the divergence in odds ratios for the association between colorectal cancer and the GSTT1 null genotype (34), when the different control groups were analyzed (9). In regard to genotype prevalence, many early studies were based on convenience samples and not infrequently, little information was given about how the samples were selected (8, 9, 31, 35).

Population stratification
Concern has been raised about the possible effects of population stratification on the results of population-based case-control studies (36-41). Population stratification includes differences between groups in ethnic origin and can arise because of differences between groups of similar ethnic origin but between which there has been limited admixture, such as in isolated populations. For example, a population might comprise the descendants of waves of immigrants from the same source but differ generally because of founder effects. The differences may then be apparent because insufficient time has elapsed for mixture between the groups. In an exploration of the possible degree of bias from population stratification in U.S. studies of cancer among non-Hispanic Americans of European descent, this bias was considered unlikely to be substantial when epidemiologic principles of study design, conduct, and analysis were rigorously applied (42). A similar conclusion was reached using data from case-unrelated control studies of non-Hispanic U.S. whites with hypertension or type 2 diabetes, and Polish subjects with type 2 diabetes (43). Variations in the frequency of certain genotypes in African Americans appear to be much wider than those observed in persons of European origin and therefore the possibility of stratification may be higher (44). Evidence was weak for an effect of population stratification in data from a case-unrelated control study of hypertension in African Americans, but this was no longer apparent when the study was restricted to persons with U.S.-born parents and grandparents (43).

Concern about the possible effects of population stratification has stimulated development of family-based case-control designs, which essentially eliminate potential confounding from this source (45, 46). The most commonly used examples of such designs involve the use of siblings or parents as controls. Sibling controls are derived from the same gene pool as cases. However, selection bias could result because a sibling may not be available for every case–bias would arise if determinants of availability (e.g., sibship size) were associated with genotype. In addition, compared with a study in which unrelated controls were used, a study using an equivalent number of sibling controls has less statistical power because of over-matching on genotype (47). This loss of power generally does not occur for case-parental control studies (46), which have been advocated for the identification of modest gene-disease associations (48). However, the need to obtain samples from parents is a practical problem limiting the applicability of the design for diseases of late onset. Clearly, the study design is appropriate to consider in assessing the possible impact of population stratification.

Another approach proposed to minimize the potential problem of population stratification when unrelated controls are used is to measure and adjust for genetic markers of ethnicity that are not linked to the disease under investigation (49-52). This would be expected to control for ethnic variation in disease risk attributable to genetic factors. However, residual confounding from other sources of ethnic variation in disease risk would be a potential issue. A single measure is unlikely to capture the important sources of ethnic variation (53). In appraising case-unrelated control studies, or cohort studies, points to consider are the adequacy of matching for ethnicity or adjusting for it in analysis.

Confounding from other sources
Confounding of a gene-disease association, and of gene-environment and gene-gene interactions, potentially could result from linkage disequilibrium. Linkage disequilibrium depends on population history and on the genetic make-up of the founders of that population (7, 54). Linkage disequilibrium varies between populations (54) and may in part account for the variable results of studies of gene-disease associations (7). In a correctly designed association study, except for allelic associations that extend for a short genomic region from the locus under investigation, the comparison of groups of individuals defined by genotype could be equivalent to a randomized comparison (26). However, so far data on linkage disequilibrium for SNPs show that the extent of linkage disequilibrium varies by region of the genome, and that its variation at all distances is great (54). Moreover, studies of microsatellite polymorphisms have shown linkage disequilibrium between a few loci that are separated by many megabases (>= 1 cM) (55).

In studies of gene-environment interaction, confounding of exposures is a potential problem. The principles regarding the control of confounding are the same as those for studying the relation between exposure and disease. In practice, the use of biomarkers of exposure may need care in interpretation because the genotype may influence the presence or level of the biomarker. Rothman (56) noted that an extraneous risk factor is a confounder only if its effect becomes mixed with the effect under study. For example, an exposure may cause altered physiology, which in turn causes disease. A biomarker of the altered physiology is a risk factor for the disease and is unrelated to exposure because it results from exposure. It is not confounding because the effect of the exposure is mediated through the effect of the altered physiology, and therefore no effects are mixed. However, decisions about whether a biomarker represents an intermediate factor in aetiology or is a potential confounder are difficult when uncertainties exist about the mechanism of effect of exposure. This would also apply to genes.

In case-cohort studies, controls are a random sample of the cohort, and the effect of age, which is the key time variable, is controlled for in the analysis only. In more traditional nested case-control designs, controls are selected to match the cases on a temporal factor, such as age, and the main comparisons are within the time-matched sets (60). In appraising case-cohort studies, the method of age adjustment and, in appraising nested case-control studies, details of the matching on age or other temporal factors are important to consider.

Statistical issues
In appraising studies, the main statistical issues are study power, multiple testing and method of analysis.

Power
A small study size is a limitation of many studies testing a priori hypotheses about gene-disease associations, (e.g., references and 9 and 58). This problem is exacerbated in studies of gene-environment and gene-gene interactions. To test for departures from multiplicative effects, it has been noted that study size should be at least four times larger than needed to detect only the main effects of the individual factors (59). In studies of modest gene-environment interactions, the sample size requirement is of the order of a thousand cases or more. When non-differential misclassification of exposure is taken into account, many thousands of cases may be needed (25). The biggest problem facing the field of gene-environment interaction is that almost no published studies have these sample sizes. A possible solution is pooled analysis (see below).

Multiple testing
One proposed research strategy is large-scale testing by genomewide association mapping (48, 60-62). This strategy is hypothesis-generating rather than hypothesis-based and thus may require additional safeguards against type 1 error. For example, Risch and Merikangas (48) suggested specifying a higher significance level. However, increasing the significance level will increase the number of subjects required to have adequate statistical power, although this may not make studies unfeasible (48).

In the analysis of gene-environment interactions, a large number of potential interactions could be tested for in a typical data set. Current data sets often already have several dozen genotypes determined, and many dozen, or even hundreds, of different environmental variables may be determined for each person in the data set (e.g., a typical food-frequency questionnaire will measure intake of more than 100 foods and permit estimation of more than 50 nutrients). Moreover, it is important to know whether there was an a priori choice of categories or scale used to quantify the amount of exposure, because this will give insight as to whether multiple testing is an issue for interpretation. In addition, the interaction model must be specified (see earlier discussion of the many models of gene-environment interaction). An approach of assessing interaction of every genotype with every environmental variable under every possible interaction model would generate a large number of false-positive results. Increasing the significance level is unlikely to solve the multiple comparisons problem in this context. The limited power to detect even established interactions at the p<0.05 level in most studies (because of modest effects and limited sample sizes) means that adjusting for multiple comparisons would be equivalent almost never declaring statistical significance for “true” interactions. In other words, reducing the nominal p value would mitigate the false-positive problem by creating a potentially unacceptably high false-negative rate.

Methods of analysis
Well-established methods exist for describing the prevalence of exposure and for measuring associations (27, 28). These can be applied to describing genotype prevalence and assessing gene-disease associations. In regard to trend-tests for gene-disease associations, even in the case of a single gene with two alleles, a decision is needed about whether to treat genotype as a trichotomous variable in which heterozygotes are categorized separately (i.e., assuming co-dominance) or to combine them with one of the two groups of homozygotes (i.e., assuming a dominant or recessive model). A problem in the choice of such models is the lack of functional information. There can be substantial loss of statistical power when a test suitable for one mode of inheritance is used where another mode is the true one (63).

Methodologic issues relating to haplotype analysis are still under development. In particular, in studies based on unrelated persons, haplotypes can be estimated only probabilistically on the basis of allele frequencies. If external estimates of haplotype frequency in the population are applied, inference may be affected by the quality and availability of the data on haplotype frequencies in the relevant population. As more SNP loci are identified, the number of possible haplotypes can become huge, in turn raising the issues of multiple comparisons and sparse data for many haplotypes (60, 64). A potential limitation of the approach of constructing haplotypes is that the effect of a true functional variant might be diluted when haplotypes rather than loci are the units of analysis.

The methods of assessing gene-environment and gene-gene interaction are less established. Three common methods have been used to assess the statistical significance of gene-environment interactions, when defined as departures from multiplicative effects. First, an interaction term is introduced into a logistic model and the Wald p value for the coefficient of the interaction term reported. In the case of multiple ordered categories of the environmental variable entered as an ordered categorical variable, the interaction term tests whether the linear trend in the environmental variable is significantly different between the dichotomous categories of genotype. Second, a cross-product “dummy” term is introduced into the logistic model for each combination of genotype and environment category (omitting the combination for the reference category). The p value for interaction is then given as the difference in the log-likelihood between this model and the model containing the main effect estimates for the genotype and environment variables. When both genotype and exposure are dichotomous, then these two tests are equivalent. However, when there is more than one category, they test different models. In this situation, a point to appraise is whether the model of interaction was specified a priori. A potential problem with the likelihood ratio test for interaction is that it does not directly test for trend. In situations in which the data depart from an ordered trend, the likelihood ratio test may give a significant result because the cross-product terms improve the fit of the model to the data. Therefore, assessing gene-environment interaction solely by screening for level of significance of a formal test for interaction should be avoided.

Third, estimates of environmental effects are compared between genotype strata. However, the finding of a significant effect in one or more strata but no significance in at least one other stratum does not constitute statistical evidence of interaction. Often such a pattern has been observed when inadequate power exists in one of the strata. Whether a formal test of statistical interaction has been performed to assess the strength of the evidence for interaction should be considered.

Analytic methods to test for gene-environment and gene-gene interactions are still under development. For example, the application of hierarchical models is being explored (65, 66). Little work has been done on testing for departures from additive models of genetic and environmental effects (26, 67).

Systematic Review
The stages involved in systematic review are (1) specification of the issue for which integrated evidence is needed; (2) identification of studies; (3) critical appraisal of studies; (4) abstraction of data; and (5) synthesis.

Specification of the issue
Typically, the need exists to specify the allelic variant, then consider one or more of questions relating to its frequency (at an early stage in research), its variation in frequency (as data accumulate), its relations with specific diseases, and whether it modifies the effect of exposures that are etiologically important (and vice versa).

Identification of studies
A comprehensive search is one of the key differences between a systematic review and a traditional review (68). Typically, the strategy used to identify relevant papers for a systematic review involves specifying the search terms, the time period of publication, the databases searched and software used to do this (69). Because problems may exist with the indexation of papers, hand-searches of the reference lists of relevant papers identified from the original search and of key journals are common practices. Thus, for example, in a review of the association between glutathione S-transferase polymorphisms and colorectal cancer, Medline and EMBASE were searched using the MeSH heading "glutathione transferase" and the textwords "GST" and "glutathione S transferase" for papers published between 1993 and 1998 (9). The CDC Office of Genomics and Disease Prevention Medical Literature Search was also searched and reference lists in published articles were hand-searched.

A further issue is the possible inclusion of unpublished sources, including abstracts, technical reports, and non-English journals (70) that may not be identified by electronic searches, as a means of minimizing the potential impact of publication bias (see below). However, this material should be treated with caution because it may not be peer-reviewed and may be subject to modification and revision. In addition, information on study methods may be insufficient to assess study quality.

Several instances have occurred of sequential or multiple publications of analyses of the same or overlapping datasets. For example, in studies of CYP1A1 polymorphisms and breast cancer, substantial overlap between the studies of Ambrosone at al. (71) and Moysich et al. (72), and between that of Taioli et al. (73) and Taioli et al. (74), is likely. An aid to identifying this problem is to organize evidence tables (see below) first by geographic area and then by study period within a specified area. If the reports clearly relate to the same or overlapping datasets, then a consistent method of dealing with this should be adopted, such as including data only from the largest or most recent publication. Under these circumstances, details of the methodology may be described in greater detail in an earlier publication. If so, the reference to the earlier publication should be given with the reference to the publication from which the data were abstracted in the evidence tables.

Critical Appraisal
Issues in the appraisal of single studies have been discussed above. A number of reports have been published about the rating of the quality of analytical observational studies. Several relate to case-control studies (27, 29, 75-79). Some (75, 76) are part of a series of articles documenting the deficiencies of epidemiologic research; they have been challenged on the grounds of technical errors, failure to distinguish important from unimportant biases, and ignoring the need to weight the totality of the evidence about a relation (80,81). Other issues include possible over-emphasis of the potential problems of case-control studies in comparison with cohort studies (78) and difficulty in assessing differences between methods applied in the case and control groups, or between different exposure (prognostic) groups (79,82).

Several authors have proposed quantitative quality scoring systems for critical appraisal (82). Other schemes have been developed for meta-analyses in which an attempt has been made to assess the importance of study quality in accounting for heterogeneity of results between studies (83-85). This type of assessment also has been considered for pooled analysis (86, 87). Certain features of the assessment schemes are specific to the disease or the exposure under consideration, and each aspect of the study is given equal weight. Thus, summation of points might result in worse quality scores for a study with several minor flaws than for a study with one major flaw. Although empirical studies on a large number of primary investigations might suggest an overall relation between a specific aspect of study design and the reported results, this relation is ecologic and may not be true for a specific investigation. Therefore, specific non-causal factors, which might affect the interpretation of a single investigation, are difficult to isolate. Jüni et al. (88) observed that the use of scores to identify clinical trials of high quality is problematic and recommended that relevant methodologic aspects should be assessed individually and their influence on the magnitude of the effect of the intervention explored. Similar caution in consideration of studies of gene-disease associations is likely to be justified. As in clinical trials, multidimensional domains may be more appropriate to consider than a single grade in the integration of evidence from observational studies.

Little or no empirical evaluation exists of the quality scoring of association studies. However, many users of data on genotype prevalence and gene-disease associations need a robust means of grading evidence. This approach has been proposed by the Scottish Intercollegiate Guidelines Network (89). In this approach, studies of gene-disease association in which all or most of the criteria specified as appropriate to a research question are satisfied would be graded as “++.” Criteria that have not been fulfilled would not affect the grade if the conclusions of the study were considered very unlikely to be affected by their omission. Studies in which some of the criteria have been fulfilled, and criteria that were not fulfilled considered unlikely to alter the conclusions would be graded as “+.” Studies in which few or no criteria were fulfilled and the conclusions of the study considered likely or very likely to be altered by multiple omissions in required criteria for an acceptable study, would be graded as “-.”

Abstraction of Data
Specific forms are often used for this purpose, for example, that used for the Human Genome Epidemiology Network’s
e-journal reviews (90). The form should be piloted to ensure a consistent approach to data abstraction. Ideally, this would be done by two independent reviewers and discrepancies resolved, but resources may not permit this (91,92). Typically, such forms include reference details, information about study eligibility, study methods, and study results.

Synthesis of the Evidence
The first steps include describing the volume of evidence and preparing evidence tables that summarize the basic characteristics of the studies, factors relating to study quality, measures of association (with indicators of precision), and the reference. On this basis, consideration is given to combining results. The simplest way of combining results is counting the number of studies showing positive, negative and inverse associations. However, this approach is very limited as no account is taken of study quality or of the magnitude of the association. Other approaches take account of these issues.

Hierarchy of Evidence
In many schemes of qualitative synthesis of evidence, a hierarchy exists whereby certain study designs are considered inherently superior to others. In general, analytical epidemiologic designs are stronger than ecologic designs and studies of case series or reports. Although cohort studies may be less subject to bias than case-control studies, important issues exist about quality of follow-up and case-ascertainment. Therefore, it seems more rigorous to weight the evidence from specific studies of these types on the basis of a full critical appraisal rather than solely on the basis of general design.

Quantitative synthesis
There are two types of quantitative synthesis of evidence: (1) meta-analysis of the results of studies and (2) pooled analysis of data on individual subjects obtained in several studies. The validity of meta-analysis of observational studies has been debated (69, 93). On the one hand, meta-analysis may indicate a “spurious precision” and either meta-analysis of observational studies should be abandoned altogether (94) or possible sources of heterogeneity between studies should be considered (95). On the other hand, meta-analysis can help clarify whether an association exists and indicate the quantitative relation between the dependent and independent variables (96). The indication of the quantitative relation, although potentially biased, may be valuable in considering public health effects of interventions based on knowledge of the genetic factor or its interactions.

Pooled analysis requires data on individual subjects. This approach offers many advantages over the meta-analysis of the results of studies, including standardization of definitions of cases and variables, better control of confounding, and consistent determination of subgroup effects (10, 86). For example, this approach has been used successfully to study the effect of chemokine and chemokine receptor alleles on HIV-1 disease progression (97) . Nevertheless, pooling approaches require much greater resources (98). Interestingly, the results of meta-analyses of the glutathione S-transferase M1 polymorphism and cancer of the lung (99) and bladder (67) were similar. Pooled analysis is preferred to meta-analysis of the results of studies when a high degree of accuracy of the measures of effect is required. However, stratification by original study may still be important, to allow for and elucidate causes of heterogeneity among the data sets being pooled.

Interpretation
The main issues appear to be consideration of possible publication bias and application of guidelines for causal inference.

  Publication Bias. Publication bias is the selective publication of studies on the basis of the magnitude and direction of their findings (100). Research with statistically significant results has long been accepted to be more likely to be submitted and published than work with null or non-significant results (101), and this has led to a preponderance of false-positive results in the literature (102). Therefore, publication bias is a potentially serious problem for the integration of evidence on gene-disease associations (6,7), especially in relation to gene-environment and gene-gene interactions. In addition to the larger number of potential comparisons implicit in the concept of multiple interacting variables, authors face the problem that large tables of gene-environment interaction estimates are very cumbersome and difficult to assemble in publishable format. This inevitably increases the potential for publication bias.

In other fields, quantitative and qualitative methods of detecting publication bias have been used, such as the fail-safe technique where the number of new studies averaging a null result needed to bring the overall effect to non-significance is calculated (69, 103). Then a judgment can be made as to whether it is realistic to assume that such a number of studies have been unpublished in the field of investigation. If the assumption were realistic, then the validity of conclusions based on published evidence would be doubtful. Other quantitative and qualitative methods have been reviewed by Sutton et al. (92) and by Thornton and Lee (104). In general, all the methods have limitations. Therefore, it seems appropriate to account for the possibility that the evidence base may be skewed toward positive results in drawing conclusions about causal relations.

Another potential method of identifying publication bias is to search research registers such as CRISP (105) and the Directory of On-going Studies in Cancer Prevention (106). Administering research registers on studies of genotype prevalence and gene-disease associations is challenging because data for each additional allele genotyped would need to be added to the database. It is even more difficult for studies of gene-environment and gene-gene interactions, because of the diversity of joint effects which can be investigated.

  Causal inference. Well-established guidelines exist for causal inference (Box 10.1). However, in practice, only limited subsets of these tend to be used (110). For example, in cancer epidemiology, the guidelines most often applied are consistency, strength, dose-response, and biologic plausibility.

  Consistency: In relation to consistency of gene-disease associations and gene-environment and gene-gene interactions, differences between studies in distributions of subjects by age and sex are sources of heterogeneity. For example, hormonal alterations can affect ligand binding, enzyme activity, gene expression, and the metabolic pathways influenced by gene expression. In particular, some inconsistency between the results of gene-disease association studies may be secondary to variation among studies in the prevalence of interacting environmental factors that have not been assessed. Testing a priori hypotheses about differences in gene-disease associations and genotype frequencies between studies that may arise from these sources would be appropriate.

In relation to interactions, heterogeneity may occur if the allele under study is associated with disease due to linkage disequilibrium with a gene that is truly causal. Such a “marker” allele may behave differently in populations with different genetic backgrounds resulting from differences in the extent of the linkage disequilibrium, even if the “causal gene” has the same effect in the different populations. Differences between populations in allele prevalence may result in differences between studies in the statistical power to detect both the main effect of the genotype and gene-environment interactions. Similarly, the prevalence of exposure, or variability of exposure, may influence whether an interaction exists or is detectable.

  Strength: As noted by Rothman (56), the strength of an association is not a biologically consistent feature but rather a characteristic that depends on the relative prevalence of other causes. In studies of the general population, the associations between disease and biomarkers of susceptibility are not likely to be strong. In particular, many of the genetic variants so far identified as influencing susceptibility to common diseases are associated with a low relative and absolute risk (108). Therefore, exclusion of non-causal explanations for associations is crucial. In this situation, an interaction between a gene and exposure (or another gene) would be expected.

  Dose-response: In the context of gene-disease associations, the value of considering dose-response relations will depend on information about the functional effect(s) of the relevant gene. As already noted, in the particular instance of gene-environment interaction, when multiple categories of dose are defined for the exposure, then many different dose-response models can be tested in the data, and tests for interaction can be applied to the trends across strata. Consequently, false-positive results are likely to be a problem.

  Biologic plausibility: This is a particularly important issue in the evaluation of gene-disease associations, gene-gene, and gene-environment interactions. For example, in investigations of associations with genetic polymorphisms of carcinogen metabolism and DNA repair, many genotypes have been assessed without data on their functional significance. Investigations confined solely to genotypes potentially would lead to numerous false-positive associations. Consideration of biologic plausibility involves determining (1) whether a known function of the gene product can be linked to the observed phenotype; (2) whether the gene is expressed in the tissue of interest; and (3) temporal relations, including the time window of gene-expression in relation to age-specific gene-disease relations. Thus, the gene should be in the disease pathway and/or involved in the mechanism that is responsible for the development of the disease. If not, then the effect of the gene may be indirect. In studies of cancer in young persons, maternally mediated effects of the maternal genotype and parental imprinting also may be relevant to consider. As an example of the need for careful interpretation, N-acetyltransferases have been considered to be important in detoxification. However, NAT has been observed to catalyse O-acetylation (109). O-acetylation is thought to be an activating step.

  Specificity: Although specificity has been included as a criterion of causation, it may be inappropriate in relation to the effects of complex exposures that may influence several outcomes such as tobacco smoking, or genetic variants that may influence the metabolism of a variety of exposures, such as cytochrome P450 gene variants. For example, CYP1A1 gene variants have been investigated in relation to a variety of types of cancer (33), macular degeneration (110), Parkinson’s disease (111), endometriosis (112), primary dysmenorrhea (2) and orofacial clefts (113).

  Temporality: Although a correct time relation is specified in many methodologic texts, it seems to be seldom used in causal inference (107). In the situation of gene-disease associations, the disease could influence the result of a phenotypic assay of the genotype under investigation. This should not be a problem with PCR methods. If data were available on the time window of gene-expression, it would be relevant to consider this in relation to age-specificity of gene-disease relations. As a perhaps extreme example, if an association existed between a type of cancer in infants and the CYP1A1 or CYP1A2 genotype of the index child, this probably would be indirect (e.g., reflecting an effect of maternal genotype) because the enzymes coded by these genes are not expressed in the fetal liver (114, 115).

  Experimental support: In the context of gene-disease associations, experimental support is most likely to be derived from studies of gene expression in knockout or other experimental animals, from in vitro data on gene function, or from experimental interventions based on clinical trials of interventions aimed at normalizing the function or levels of a product regulated by the gene. For example, initially transgenic mouse models appeared to support a role for certain genes in the etiology of orofacial clefts (116). It is now apparent that clefts often occur in knockout and insertion experiments, and that gene expression at a critical time and in a tissue relevant to development of the lip and palate should also be taken into account. An example of in vitro investigation on gene function is an investigation of the effect of the MTHFR C677T polymorphism on folic acid deficiency-induced uracil incorporation into human lymphocyte DNA (117). In regard to trials of interventions aimed at normalizing a gene product, trials of the drug CPX are under way (118). CPX acts by binding to the mutant channel protein, helping it to mature and gain access to the plasma membrane, and it is thought that repair of the defect in trafficking to the membrane helps suppress the high level of synthesis and secretion of IL-8 that is involved in pathogenesis.

  Coherence: This criterion has been defined as being satisfied when an association being consistent with the state of knowledge of the natural history and biology of the disease (119). In practice, this criterion has been little used, perhaps because it has been considered equivalent to biologic plausibility (107). Elwood (29) defines an association as being coherent “if it fits the general features of the distribution of both the exposure and the outcome under assessment.” He notes that the concept holds only if a high proportion of the outcome is caused by the exposure, and if the frequency of the outcome is fairly high in those exposed. An additional constraint on the use of this criterion arises when information about the distribution of the relevant exposure and outcome is inadequate. Information about the distribution of many biomarkers is limited. In the situation of gene-disease associations, the “exposure” would be the genotype being investigated.

Conclusion
There has been a tremendous increase in the number of published human genome epidemiologic studies, and this increase is set to continue. So far, few gene-disease associations, gene-environment or gene-gene interactions have been replicated. This may in part be due to methodologic issues. Methodologic issues that are particularly important include the assessment of genotypes, selection of subjects, confounding, statistical power and multiple statistical testing. In the assessment of gene-environment interaction, assessment of exposure is also an important issue. It is hoped that the checklist presented in this chapter will be useful to investigators preparing manuscripts, to those who need to appraise manuscripts and published papers, and to journal editors and readers. In regard to the integration of evidence, established principles of systematic review should be applied. Meta-analysis and pooled analysis can help address concerns about statistical power and provide a formal means of investigating possible heterogeneity between studies. Pooled analysis is labor intensive. It is preferred to meta-analysis when a high degree of accuracy of the measures of effect is required. In interpreting this evidence, the potential for publication bias is an important consideration. To address the problem of publication bias, a register of research is needed that would include negative findings. In terms of specifying hypotheses to be tested and interpretation of the biologic plausibility of study findings, inter-disciplinary collaboration in this fast expanding field is crucial.

Acknowledgments
Much of this chapter is the result of discussions at an expert panel workshop convened by the Centers for Disease Control and Prevention and the National Institutes of Health in January 2001. We thank the following contributors for comments: Linda Bradley, Molly S Bray, Daniel Burns, Mindy Clyne, Gwen W. Collman, Janice Dorman, Darrell L. Ellsworth, James Hanson, Robert A. Hiatt, David J. Hunter, Muin J. Khoury, Joseph Lau, Thomas R O’Brien, Nathaniel Rothman, Donna Stroup, Emanuela Taioli, Duncan Thomas, Harri Vainio, Sholom Wacholder, Clarice Weinberg, Paula Yoon.

1. McPherson JD, Marra M, Hillier L et al. A physical map of the human genome. Nature 2001;409:934-941.
2. Venter JC, Adams MD, Myers EW et al. The sequence of the human genome. Science 2001;291:1304-1351.
3. Shpilberg O, Dorman JS, Ferrell RE et al. The next stage: molecular epidemiology. J Clin Epidemiol 1997;50:633-638.
4. Khoury MJ. Commentary: epidemiology and the continuum from genetic research to genetic testing. Am J Epidemiol 2002;156:297-299.
5. Dunning AM, Healey CS, Pharoah PD et al. A systematic review of genetic polymorphisms and breast cancer risk. Cancer Epidemiol Biomarkers Prev 1999;8:843-854.
6. Ioannidis JP, Ntzani EE, Trikalinos TA et al. Replication validity of genetic association studies. Nat Genet 2001;29:306-309.
7. Hirschhorn JN, Lohmueller K, Byrne E et al. A comprehensive review of genetic association studies. Genet Med 2002;4:45-61.
8. Brockton N, Little J, Sharp L et al. N-acetyltransferase polymorphisms and colorectal cancer: a HuGE review. Am J Epidemiol 2000;151:846-861.
9. Cotton SC, Sharp L, Little J et al. Glutathione S-transferase polymorphisms and colorectal cancer: a HuGE review. Am J Epidemiol 2000;151:7-32.
10. Ioannidis JP, Rosenberg PS, Goedert JJ et al. Commentary: meta-analysis of individual participants' data in genetic epidemiology. Am J Epidemiol 2002;156:204-210.
11. Weiss ST. Association studies in asthma genetics. Am J Respir Crit Care Med 2001;164:2014-2015.
12. Cooper DN, Nussbaum RL, Krawczak M. Proposed guidelines for papers describing DNA polymorphism-disease associations. Hum Genet 2002;110:207-208.
13. Little J, Bradley L, Bray MS et al. Reporting, appraising, and integrating data on genotype prevalence and gene-disease associations. Am J Epidemiol 2002;156:300-310.
14. Khoury MJ, Adams MJ Jr, Flanders WD. An epidemiologic approach to ecogenetics. Am J Hum Genet 1988;42:89-95.
15. Taioli E, Zocchetti C, Garte S. Models of interaction between metabolic genes and environmental exposure in cancer susceptibility. Environ Health Perspect 1998;106:67-70.
16. National Human Genome Research Institute, National Institute of Health, Department of Health and Human Services and Office of Science, U.S. Department of Energy. International Consortium Completes human Genome Project. Accessed May 15, 2003, from http://www.genome.gov/11006929
17. Dean M, Carrington M, Winkler C et al. Genetic restriction of HIV-1 infection and progression to AIDS by a deletion allele of the CKR5 structural gene. Hemophilia Growth and Development Study, Multicenter AIDS Cohort Study, Multicenter Hemophilia Cohort Study, San Francisco City Cohort, ALIVE Study. Science 1996;273:1856-1862.
18. Michael NL, Chang G, Louie LG et al. The role of viral phenotype and CCR-5 gene defects in HIV-1 transmission and disease progression. Nat Med 1997;3:338-340.
19. Langholz B, Rothman N, Wacholder S et al. Cohort studies for characterizing measured genes. J Natl Cancer Inst Monogr 1999;39-42.
20. Sachidanandam R, Weissman D, Schmidt SC et al. A map of human genome sequence variation containing 1.42 million single nucleotide polymorphisms. Nature 2001;409:928-933.
21. Reich DE, Cargill M, Bolk S et al. Linkage disequilibrium in the human genome. Nature 2001;411:199-204.
22. Altshuler D, Pollara VJ, Cowles CR et al. An SNP map of the human genome generated by reduced representation shotgun sequencing. Nature 2000;407:513-516.
23. Gray IC, Campbell DA, Spurr NK. Single nucleotide polymorphisms as tools in human genetics. Hum Mol Genet 2000;9:2403-2408.
24. Porter CJ, Talbot CC, Cuticchia AJ. Central mutation databases--a review. Hum Mutat 2000;15:36-44.
25. Garcia-Closas M, Rothman N, Lubin J. Misclassification in case-control studies of gene-environment interactions: assessment of bias and sample size. Cancer Epidemiol Biomarkers Prev 1999;8:1043-1050.
26. Clayton D, McKeigue PM. Epidemiological methods for studying genes and environmental factors in complex diseases. Lancet 2001;358:1356-1360.
27. Breslow N, Day N. Statistical methods in cancer research. Volume 1. The analysis of case-control studies. 1980. Lyon, IARC.
28. Kelsey JL, Whittemore AS, Evans AS et al. Methods in observational epidemiology. Oxford: Oxford University Press, 1996.
29. Elwood M. Critical appraisal of epidemiological studies and clinical trials. Oxford: Oxford University Press, 1998.
30. dos Santos Silva I. Cancer epidemiology: Principles and methods. Lyon: IARC, 1999.
31. Botto LD, Yang Q. 5,10-Methylenetetrahydrofolate reductase gene variants and congenital anomalies: a HuGE review. Am J Epidemiol 2000;151:862-877.
32. Dorman JS, Bunker CH. HLA-DQ locus of the human leukocyte antigen complex and type 1 diabetes mellitus: a HuGE review. Epidemiol Rev 2000;22:218-227.
33. d'Errico A, Malats N, Vineis P et al. Review of studies of selected metabolic polymorphisms and cancer. In: Vineis P, Malats N, Lang M et al., eds. Metabolic polymorphisms and susceptibility to cancer. IARC Scientific Publications No. 148. Lyon: IARC, 1999: 323-393.
34. Chenevix-Trench G, Young J, Coggan M et al. Glutathione S-transferase M1 and T1 polymorphisms: susceptibility to colon cancer and age of onset. Carcinogenesis 1995;16:1655-1657.
35. Wang SS, Fernhoff PM, Hannon WH et al. Medium chain acyl-CoA dehydrogenase deficiency human genome epidemiology review. Genet Med 1999;1:332-339.
36. Knowler WC, Williams RC, Pettitt DJ et al. Gm3;5,13,14 and type 2 diabetes mellitus: an association in American Indians with genetic admixture. Am J Hum Genet 1988;43:520-526.
37. Gelernter J, Goldman D, Risch N. The A1 allele at the D2 dopamine receptor gene and alcoholism. A reappraisal. JAMA 1993;269:1673-1677.
38. Khoury M, Beaty TH, Cohen BL. Fundamentals of genetic epidemiology. New York: Oxford University Press, 1993.
39. Caporaso N, Rothman N, Wacholder S. Case-control studies of common alleles and environmental factors. J Natl Cancer Inst Monogr 1999;25-30.
40. Thomas DC, Witte JS. Point: population stratification: a problem for case-control studies of candidate-gene associations? Cancer Epidemiol Biomarkers Prev 2002;11:505-512.
41. Wacholder S, Rothman N, Caporaso N. Counterpoint: bias from population stratification is not a major threat to the validity of conclusions from epidemiological studies of common polymorphisms and cancer. Cancer Epidemiol Biomarkers Prev 2002;11:513-520.
42. Wacholder S, Rothman N, Caporaso N. Population stratification in epidemiologic studies of common genetic variants and cancer: quantification of bias. J Natl Cancer Inst 2000;92:1151-1158.
43. Ardlie KG, Lunetta KL, Seielstad M. Testing for population subdivision and association in four case-control studies. Am J Hum Genet 2002;71:304-311.
44. Garte S. The role of ethnicity in cancer susceptibility gene polymorphisms: the example of CYP1A1. Carcinogenesis 1998;19:1329-1332.
45. Teng J, Risch N. The relative power of family-based and case-control designs for linkage disequilibrium studies of complex human diseases. II. Individual genotyping. Genome Res 1999;9:234-241.
46. Witte JS, Gauderman WJ, Thomas DC. Asymptotic bias and efficiency in case-control studies of candidate genes and gene-environment interactions: basic family designs. Am J Epidemiol 1999;149:693-705.
47. Gauderman WJ, Witte JS, Thomas DC. Family-based association studies. J Natl Cancer Inst Monogr 1999;31-37.
48. Risch N, Merikangas K. The future of genetic studies of complex human diseases. Science 1996;273:1516-1517.
49. Devlin B, Roeder K. Genomic control for association studies. Biometrics 1999;55:997-1004.
50. Pritchard JK, Stephens M, Rosenberg NA et al. Association mapping in structured populations. Am J Hum Genet 2000;67:170-181.
51. Reich DE, Goldstein DB. Detecting association in a case-control study while correcting for population stratification. Genet Epidemiol 2001;20:4-16.
52. Satten GA, Flanders WD, Yang Q. Accounting for unmeasured population substructure in case-control studies of genetic association using a novel latent-class model. Am J Hum Genet 2001;68:466-477.
53. Lin SS, Kelsey JL. Use of race and ethnicity in epidemiologic research: concepts, methodological issues, and suggestions for research. Epidemiol Rev 2000;22:187-202.
54. Ardlie KG, Kruglyak L, Seielstad M. Patterns of linkage disequilibrium in the human genome. Nat Rev Genet 2002;3:299-309.
55. Pritchard JK, Przeworski M. Linkage disequilibrium in humans: models and data. Am J Hum Genet 2001;69:1-14.
56. Rothman KJ. Modern Epidemiology. Boston/Toronto: Little, Brown and Company, 1986.
57. Wacholder S. Practical considerations in choosing between the case-cohort and nested case-control designs. Epidemiology 1991;2:155-158.
58. Boffetta P, Pearce N. Epidemiological studies on genetic polymorphisms: study design issues and measures of occurrence and association. In: Vineis P, Malats N, Lang M et al., eds. Metabolic polymorphisms and susceptibility to cancer. IARC Scientific Publications No. 148. Lyon: IARC, 1999:97-108.
59. Smith PG, Day NE. The design of case-control studies: the influence of confounding and interaction effects. Int J Epidemiol 1984;13:356-365.
60. Schork NJ, Fallin D, Lanchbury JS. Single nucleotide polymorphisms and the future of genetic epidemiology. Clin Genet 2000;58:250-264.
61. Morton NE, Collins A. Tests and estimates of allelic association in complex inheritance. Proc Natl Acad Sci U S A 1998;95:11389-11393.
62. Risch N, Teng J. The relative power of family-based and case-control designs for linkage disequilibrium studies of complex human diseases I. DNA pooling. Genome Res 1998;8:1273-1288.
63. Freidlin B, Zheng G, Li Z et al. Trend tests for case-control studies of genetic markers: power, sample size and robustness. Hum Hered 2002;53:146-152.
64. Fallin D, Cohen A, Essioux L et al. Genetic analysis of case/control data using estimated haplotype frequencies: application to APOE locus variation and Alzheimer's disease. Genome Res 2001;11:143-151.
65. Aragaki CC, Greenland S, Probst-Hensch N et al. Hierarchical modeling of gene-environment interactions: estimating NAT2 genotype-specific dietary effects on adenomatous polyps. Cancer Epidemiol Biomarkers Prev 1997;6:307-314.
66. Witte JS. Genetic analysis with hierarchical models. Genet Epidemiol 1997;14:1137-1142.
67. Engel LS, Taioli E, Pfeiffer R et al. Pooled analysis and meta-analysis of glutathione S-transferase M1 and bladder cancer: a HuGE review. Am J Epidemiol 2002;156:95-109.
68. Oxman, A. D. The Cochrane Collaboration Handbook: preparing and maintaining systematic reviews . 1992. Oxford, Cochrane Collaboration.
69. Stroup DF, Berlin JA, Morton SC et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000;283:2008-2012.
70. Gregoire G, Derderian F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995;48:159-163.
71. Ambrosone CB, Freudenheim JL, Graham S et al. Cytochrome P4501A1 and glutathione S-transferase (M1) genetic polymorphisms and postmenopausal breast cancer risk. Cancer Res 1995;55:3483-3485.
72. Moysich KB, Shields PG, Freudenheim JL et al. Polychlorinated biphenyls, cytochrome P4501A1 polymorphism, and postmenopausal breast cancer risk. Cancer Epidemiol Biomarkers Prev 1999;8:41-44.
73. Taioli E, Trachman J, Chen X et al. A CYP1A1 restriction fragment length polymorphism is associated with breast cancer in African-American women. Cancer Res 1995;55:3757-3758.
74. Taioli E, Bradlow HL, Garbers SV et al. Role of estradiol metabolism and CYP1A1 polymorphisms in breast cancer risk. Cancer Detect Prev 1999;23:232-237.
75. Feinstein AR. Methodologic problems and standards in case-control research. J Chronic Dis 1979;32:35-41.
76. Horwitz RI, Feinstein AR. Methodologic standards and contradictory results in case-control research. Am J Med 1979;66:556-564.
77. Kopec JA, Esdaile JM. Bias in case-control studies. A review. J Epidemiol Community Health 1990;44:179-186.
78. Crombie IK. The pocket guide to critical appraisal. London: BMJ Publishing Group, 1996.
79. Liddle J, Williamson M, Irwig L. Method for evaluating research and guideline evidence (MERGE). Sydney: NSW Health Department., 1996.
80. Savitz DA, Greenland S, Stolley PD et al. Scientific standards of criticism: a reaction to "Scientific standards in epidemiologic studies of the menace of daily life," by A.R. Feinstein. Epidemiology 1990;1:78-83.
81. Weiss NS. Scientific standards in epidemiologic studies. Epidemiology 1990;1:85-86.
82. Dixon RA, Munro JF, Silcocks PB. The evidence based medicine workbook. Critical appraisal for clinical problem solving. Oxford: Butterworth-Heinemann, 1997.
83. Longnecker MP, Berlin JA, Orza MJ et al. A meta-analysis of alcohol consumption in relation to risk of breast cancer. JAMA 1988;260:652-656.
84. Longnecker MP, Orza MJ, Adams ME et al. A meta-analysis of alcoholic beverage consumption in relation to risk of colorectal cancer. Cancer Causes Control 1990;1:59-68.
85. Berlin JA, Colditz GA. A meta-analysis of physical activity in the prevention of coronary heart disease. Am J Epidemiol 1990;132:612-628.
86. Friedenreich CM. Methods for pooled analyses of epidemiologic studies. Epidemiology 1993;4:295-302.
87. Friedenreich CM, Brant RF, Riboli E. Influence of methodologic factors in a pooled analysis of 13 case- control studies of colorectal cancer and dietary fiber. Epidemiology 1994;5:66-79.
88. Juni P, Witschi A, Bloch R et al. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282:1054-1060.
89. SIGN. SIGN 50: A Guideline Developer’s Handbook. 2001. Edinburgh, UK, Scottish Intercollegiate Guidelines Network .
90. HuGE. Human Genome Epidemiology Network e-journal club. (http://www.cdc.gov/genomics/hugenet/ejournal.htm). accessed October 7, 2002.
91. Deville WL, Buntinx F, Bouter LM et al. Conducting systematic reviews of diagnostic studies: didactic guidelines. BMC Med Res Methodol 2002;2:9.
92. Sutton AJ, Abrams KR, Jones DR et al. Systematic reviews of trials and other studies. Health Technol Assess 1998;2:1-276.
93. Blettner M, Sauerbrei W, Schlehofer B et al. Traditional reviews, meta-analyses and pooled analyses in epidemiology. Int J Epidemiol 1999;28:1-9.
94. Shapiro S. Meta-analysis/Shmeta-analysis. Am J Epidemiol 1994;140:771-778.
95. Egger M, Schneider M, Davey Smith G. Spurious precision? Meta-analysis of observational studies. BMJ 1998;316:140-4.
96. Doll R. The use of meta-analysis in epidemiology: diet and cancers of the breast and colon. Nutr Rev 1994;52:233-237.
97. Ioannidis JP, Rosenberg PS, Goedert JJ et al. Effects of CCR5-Delta32, CCR2-64I, and SDF-1 3'A alleles on HIV-1 disease progression: An international meta-analysis of individual- patient data. Ann Intern Med 2001;135:782-795.
98. Steinberg KK, Smith SJ, Stroup DF et al. Comparison of effect estimates from a meta-analysis of summary data from published studies and from a meta-analysis using individual patient data for ovarian cancer studies. Am J Epidemiol 1997;145:917-925.
99. Benhamou S, Lee WJ, Alexandrie AK et al. Meta- and pooled analyses of the effects of glutathione S-transferase M1 polymorphisms and smoking on lung cancer risk. Carcinogenesis 2002;23:1343-1350.
100. Stroup DF, Thacker SB. Meta-analysis in epidemiology. In: Gail MH, Benichou J, eds. Encyclopedia of epidemiologic methods. Chichester //New York : Wiley & Sons Publishers, 2000:557-570.
101. Easterbrook PJ, Berlin JA, Gopalan R et al. Publication bias in clinical research. Lancet 1991;337:867-872.
102. Begg CB, Berlin JA. Publication bias and dissemination of clinical research. J Natl Cancer Inst 1989;81:107-115.
103. Rosenthal R . The file drawer problem and tolerance for null results. Psychological Bulletin 1979;86:638-641.
104. Thornton A, Lee P. Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol 2000;53:207-216.
105. CRISP. Computer Retrieval of Information on Scientific Projects. (http://crisp.cit.nih.gov). accessed October 7, 2002.
106. Sankaranarayannan, R., Becker, N., and Démaret, E. Directory of on-going research in cancer prevention
     (http://www-dep.iarc.fr/direct/prevent.htm). 2000. Lyon, IARC.
107. Weed DL, Gorelic LS. The practice of causal inference in cancer epidemiology. Cancer Epidemiol Biomarkers Prev 1996;5:303-311.
108. Caporaso N. Selection of candidate genes for population studies. IARC Sci Publ 1999;23-36.
109. Hein DW. Acetylator genotype and arylamine-induced carcinogenesis. Biochim Biophys Acta 1988;948:37-66.
110. Kimura K, Isashiki Y, Sonoda S et al. Genetic association of manganese superoxide dismutase with exudative age-related macular degeneration. Am J Ophthalmol 2000;130:769-773.
111. Chan DK, Mellick GD, Buchanan DD et al. Lack of association between CYP1A1 polymorphism and Parkinson's disease in a Chinese population. J Neural Transm 2002;109:35-39.
112. Hadfield RM, Manek S, Weeks DE et al. Linkage and association studies of the relationship between endometriosis and genes encoding the detoxification enzymes GSTM1, GSTT1 and CYP1A1. Mol Hum Reprod 2001;7:1073-1078.
113. van Rooij IA, Wegerif MJ, Roelofs HM et al. Smoking, genetic polymorphisms in biotransformation enzymes, and nonsyndromic oral clefting: a gene-environment interaction. Epidemiology 2001;12:502-507.
114. Cresteil T. Onset of xenobiotic metabolism in children: toxicological implications. Food Addit Contam 1998;15 Suppl:45-51.
115. Sonnier M, Cresteil T. Delayed ontogenesis of CYP1A2 in the human liver. Eur J Biochem 1998;251:893-898.
116. Schutte BC, Murray JC. The many faces and factors of orofacial clefts. Hum Mol Genet 1999;8:1853-1859.
117. Crott JW, Mashiyama ST, Ames BN et al. Methylenetetrahydrofolate reductase C677T polymorphism does not alter folic acid deficiency-induced uracil incorporation into primary human lymphocyte DNA in vitro. Carcinogenesis 2001;22:1019-1025.
118. Eidelman O, Zhang J, Srivastava M et al. Cystic fibrosis and the use of pharmacogenomics to determine surrogate endpoints for drug discovery. Am J Pharmacogenomics 2001;1:223-238.
119. Surgeon General (Advisory Committee) . Smoking and health. 1964. Washington DC, US Department of Health, Education and Welfare.
120. Hill AB. The environment and disease: association or causation? Proceedings of the Royal Society of Medicine 1965;58:295-300.
121. Schlesselman JJ. "Proof" of cause and effect in epidemiologic studies: criteria for judgment. Prev Med 1987;16:195-210.