When Bad Study Design Comes Back to Bite You

A recent study published by Séralini et al. (Séralini study) about the long term toxicity of Roundup® herbicide and Roundup-tolerant genetically modified (GM) maize NK603 in feed of rats1 has caused quite a stir. Opponents of genetically modified crops have seized upon the results, which showed that rats fed a diet containing GM maize NK603 or given water containing Roundup® at residue levels permitted in the United States, died earlier than rats fed a standard diet. Others, including the European Safety Authority (EFSA) have issued statements about the scientific merit of the study due to serious design flaws.2,3 Criticisms from EFSA include unclear objectives, insufficient numbers of animals, use of an inappropriate rat strain and statistical methods, absence of suitable control groups for all treatment groups and information about feed composition or intake.2,3 The design flaws are so significant that the authors have been accused of deliberately planning the study to obtain the desired outcome.4  The Séralini study is a great case study of the importance of proper study design. If the study design was virtually bulletproof, and the same results were found, the impact of this study could have been entirely different. The authors could have been basking in the limelight of an important discovery that could have influenced regulation of GM maize NK603 and testing requirements for GM foods. Instead, the study will go down in history as “rubbish”.5 If the investigators could turn back the clock and perform the study again, they could do the following to help gain acceptability by the scientific community:

1. Have clear objectives. The authors of the study did not state any objectives. “Research objectives define crucial factors such as the study design, correct sample size and the statistical methods used to analyze the data – all of which have a direct impact on the reliability of the findings.”3 Well said, EFSA.

1. Perform the study according to an established guideline for chronic (e.g. OECD 451, FDA Redbook Section IV.C.5a or comparable), carcinogenicity (e.g. OECD 451, FDA Redbook Section IV. C. 6 or comparable) or combined chronic/carcinogenicity (e.g. OECD 453 or comparable) studies in rats. The authors designed the study based on guidelines for subchronic toxicity studies and in a subsequent publication acknowledged that the study design was not suitable to assess long term carcinogenicity.6 The conclusions about carcinogenicity of the test materials that were original publication were effectively nullified by this admission.

1. Use an appropriate number of animals. OECD and FDA Redbook guidelines specify that a minimum of 20 rats/sex/group should be used for chronic studies and 50 rats/sex/group for carcinogenicity studies. Only 10 rats/sex/group were used in the Séralini study. EFSA has concluded that the number of animals used was insufficient to distinguish between effects that occurred by chance, rather than treatment effects.2 While 10 rats/sex/group may be acceptable for subchronic studies, it is not for longer term studies.

1. Use an appropriate animal model. The strain of rat used in the study was the Sprague-Dawley, which is well known to develop spontaneous tumors when fed ad lib (up to 87% of females and 73% of males).7 EFSA concluded that the observed frequency of tumors in the Séralini study was influenced by the natural incidence of tumors typical of the strain, regardless of any treatment. Fischer 344 rats are commonly used in carcinogenicity studies due to the low rate of development of spontaneous tumors and favorable survivability over a two year period.8 Séralini should have used Fischer 344 rats or some other strain of rat that is not known for its high rate of spontaneous tumors.

1. Use appropriate controls. There were nine different treatment groups in the study. Rats were provided standard feed containing 11%, 22% or 33% GM maize NK603 (with or without application of Roundup®) or standard feed and drinking water containing three different concentrations of Roundup®. The only control was a group fed a diet containing 33% non-GMO maize. Any comparisons that were made to the control group are not valid, as the control was not identical to the treatment group in all aspects excepting the treatment itself. If the experiment is repeated, the same concentrations of maize that are used in the treatment groups should be used in the control groups.

1. Use an appropriate feed. The protein, carbohydrate, lipid, mineral and vitamin composition of the diet was not provided. The feed was not analyzed for residual levels of Roundup (or other pesticides) or mycotoxins, many of which are carcinogenic. Therefore, it is altogether possible that the results were confounded by nutritional imbalances or toxic concentrations of pesticides or mycotoxins commonly found in corn (e.g. alflatoxin, which is carcinogenic). The authors should have provided more details about feed composition to support their conclusion that the effects were due to the test substance.

1. Measure food intake and dietary studies and water intake in drinking water studies. These endpoints were not measured in the Séralini study. Therefore, the actual doses of GM maize NK603 or Roundup® that the animals were exposed to cannot be determined. One cannot assume that the dose of test material increases linearly with increasing concentration in food or water. It is altogether possible that the concentrations of test materials used in the study could have caused feed or water attraction or aversion (animals were fed and watered ad libitum). Interestingly, the results of the study showed that lower levels of GM maize NK603 (with or without Roundup®) in the diets or Roundup® in water were more toxic to the animals than higher levels. Because feed and water intake were not measured, the authors were left with an inexplicable lack of effect of concentration.

1. Use conventional, preplanned statistical methods. Clinical chemistry and hematology data were analyzed using an orthogonal partial least squares discriminant analysis (OPLS-DA) and not by a conventional test (e.g. analysis of variance (ANOVA)). Regression coefficients for these indices were reported, rather than means and standard deviations. Tumors were not graded for severity and tumor multiplicity was not reported. Further, there were no comparisons of relative frequencies and total tumor burden, no adjustment for survival times, no analysis of cumulative tumor risks relative to survival duration, no analysis of time to tumor and no discussion of test facility historical control tumor incidence data.9 Independent statistical analyses performed by others on the data presented in the Séralini study did not support the conclusions drawn by the investigators.2 Therefore, it was difficult for evaluators to assess whether the results were actually due to treatment or artifacts based on the statistical method employed to analyze the data.

Perhaps it is unfair to single out the Séralini study as an example of poorly conducted studies, as results of studies with less than sterling designs are published every day. Possible reasons for improper design include financial constraints, pressures to obtain interesting findings or human error. Some studies are well planned, but are plagued with problems in execution. But, if a study is well planned and performed according to an established guideline or protocol that is appropriate for the study, minor problems in execution may have little impact on the overall outcome. Experiments should be designed to test a hypothesis (e.g. Will this substance cause toxicity?) rather than to prove a hypothesis (e.g. This substance will cause toxicity). If one starts with a valid hypothesis to be tested, the appropriate design should naturally follow. Proper planning is essential, or one may be left with a worthless study, or tarnished reputation. Stay tuned for more on this subject in the next edition of the Burdock Group Advisor.

References

1. Séralini, G.E., Clair, E., Mesnage, R., Gress, S., Defarge, N., Malatesta, M., Hennequin, D., and Spiroux de Vedômois, J (2012). Long term toxicity of a Roundup herbicide and a Roundup-tolerant genetically modified maize. Food Chem.Toxicol. 50: 4221-4231.

2. EFSA (2012). Final review of the Séralini et al. (2012a) publication on a 2-year rodent feeding study with glycophosate formulations and GM maize NK603 as published online on 19 September 2012 in Food and Chemical Toxicology. EFSA Journal 10(11):2986. Available at http://www.efsa.europa.eu/en/efsajournal/doc/2986.pdf.

3. EFSA (2012). Press release dated October 4, 2012. Available at http://www.efsa.europa.eu/en/press/news/121004.htm.

4. Clapp, S (2102). Fallout continues from controversial French rat study. Food Chemical News, September 28, 2012, p. 6.

5. Statement by Catherine Geslain-Lanélle, executive director of EFSA in Food Chemical News, October 26, 2012, p.6.

6. Séralini, G.E., Mesnage, R., Gress, S., Defarge, N., Hennequin, D.,Clair, E., Malatesta, M., and Spiroux de Vedômois, J (2012). Answers to critics: Why there is a long term toxicity due to a Roundup tolerant genetically modified maize and to a Roundup herbicide. Food and Chemical Toxicology to a Roundup- tolerant genetically modified maize and to a Roundup herbicide. Food Chem. Toxicol., in press ( http://dx.doi.org/10.1016/j.fct.2012.11.007).  Available at http://gmoseralini.org/wp-content/uploads/2013/01/Seralinial-AnswersCritics-FCT_2013.pdf.

7. Keenan, K. P., Soper, K. A.,Smith, P.F., Ballam, G.C. and Clark, R.L. (1995). Diet, overfeeding and moderate dietary restriction in control Sprague-Dawley rats.I. Effects on spontaneous neoplasms. Toxicol. Pathol. 23(3): 269-86.

8. Hayes, A. W., Dayan, A. D., Hall, W. C., Kodell, R.L., Williams, G.M., Waddell, W.D. , Slesinski, R. S. and Kruger, C. L. (2011). A review of mammalian carcinogenicity study design and potential effects of alternate test procedures on the safety evaluation of food ingredients. Regul. Toxicol. Pharm. 59:142-175.

9. Berry, C. (2013). Letter to the editor. Food. Chem. Toxicol. 53: 445-446.