Wrong Nutrition Facts

Geschatte leestijd: 5 minuten Healthy nutrition is a complex subject. Even with the best intentions of researchers, there can be a lot of confusion about “the best diet” and the best ways to get someone to eat such a diet. ‘Nutritional facts’ should therefore always be approached with caution. When scientists start manipulating figures or poorly controlling them, it doesn’t get any better. As recently proven once again.

A Case of Stapel

Does the name Diederik Stapel ring a bell? The Professor at Tilburg University was fired in 2011 for fabricating data [1]. In a later interview, he justified this by saying that he wanted to create clarity in the chaos of data. “Undesirable” data from assistants was erased, numbers were manipulated and fabricated [2]. This resulted in remarkably large effects in his research, results that seemed too good to be true, and indeed were. It is often studies that demonstrate a large effect that reach the wider public. Or studies that can be of immediate practical use in our daily lives. Or simply studies that can provide a catchy title, both academically and above an article. Recently, another “Stapel case” was discovered. A study that globally had people stick stickers of cartoon characters on apples turned out to be one of many with dubious and incorrect figures.

Elmo’s Apple or a Cookie?

In 2012, a study conducted by researchers from Cornell received a lot of attention [1]. The researchers wondered if clever branding could lead to a healthier school lunch. The study found a simple way to get children aged 8 to 11 to choose an apple instead of a cookie: Stick a sticker of a popular character on the apple. In the case of the study, it was Elmo from Sesame Street. The results were published by JAMA Pediatrics[3]. It was one of the publications that contributed to the recognition and career of lead author Brian Wansink, the director of Cornell Food and Brand Lab. Prior to the intervention (putting stickers on the apples), 20.7% of the children chose an apple and 80% chose the cookie. When the researchers provided the apples with Elmo’s image, more than a third (30.8%) of the children chose the apple. That is certainly a significant increase, for the small price of a sticker. Especially when you make this finding and those from similar studies available in book form, many parents are eager to try out the scientifically supported methods. However, if you look up the source of the third reference, you will see a large warning in red at the top. After receiving critical questions, the publication in JAMA Pediatrics was retracted. The researchers from Food and Brand Lab then posted a replacement on September 21, addressing the errors in the original publication. It turned out that the description of how the children were allowed to choose and what they were compared to (Elmo on apple, Elmo on cookie, or unknown sticker) was incorrect. The numbers of participants were incorrectly noted, and the statistical calculations used had to be explained as they were performed differently than usual. But apart from that, everything in the study was correct, according to director Wansink:

We confirm that there are no other errors or omissions in the original article.

And then it got really embarrassing. On October 20, the article was completely retracted. There was still a ‘small’ mistake discovered: The study was not conducted among 8-11 year olds, but among 3-5 year olds [5,6]. Most parents will agree that this is quite a difference.

P-hacking, Reverse Science

Brian Wansink is also known as “the Sherlock Holmes of food.” He has co-authored more than 200 scientific publications and has been cited more than 2800 times. Not exactly a lightweight, you might think. However, earlier this year, other researchers discovered discrepancies in some of his studies. Numbers didn’t add up, such as the number of participants in different studies based on the same dataset. Wansink had challenged a student to get more out of a dataset that had yielded ‘nothing’ for him. Wansink had hypothesized that people eat less at a buffet when it’s cheaper. However, based on his data, it didn’t matter. “A waste of that data,” thought Wansink. All those costs incurred to gather information in an Italian restaurant. He then had a student work with the data. “Plan A didn’t work, maybe plans B, C, and D are an option” (his words, not mine [4]). So, Wansink actually says: “I had a hypothesis, and for that, I invested time, money, and effort in a study to prove it. Now my hypothesis turns out to be false, and I’ve actually wasted that time, money, and effort.” In my opinion, this is where Wansink goes wrong. His study was not worthless; he has shown that there is no connection. That is a result. It’s just maybe a less interesting result for the public than confirming the hypothesis would be. “Elmo sticker on apple has no effect on children’s lunch,” may not attract much attention. But “Children eat 50% healthier thanks to Elmo sticker,” that scores. So now it’s up to the student to see if other hypotheses can be confirmed with the same data from the Italian restaurant. Wansink apparently had three alternative hypotheses in mind (B, C, and D), but the student found enough connections in the data to write no less than five scientific papers in half a year. This last point raised many eyebrows among colleagues. “Could this be a case of P-hacking (or data dredging, data fishing)?”. Diving into data without a hypothesis and just seeing which connections you come across, to then draw a conclusion? Would Wansink have delved deeper into the data if his initial hypothesis had been confirmed (p-value less than 0.05)?

“Pizza Studies”

The Italian restaurant yielded, among others, the following studies [7-10]:

1. Do men eat more in the company of a woman?
2. To what extent does price determine how you rate an eaten pizza, and which slice of pizza has the greatest influence?
3. Lower buffet prices lead to a lower feeling of satisfaction
4. Does the level of guilt after eating at a buffet depend on the price?

All ‘blog-worthy’ studies. Those popular science topics, click bait in 500 words and a nice headline. For example, the conclusion of the second study was that your rating of a $4 pizza depends mainly on the last piece you taste. With an $8 pizza, it would depend on your rating of the last piece. That kind of world-changing science obtained from Aiello’s Pizza Emporium in New York. In these four studies, researchers from Leiden and Groningen universities, among others, found no less than 150 inconsistencies [11]!

Elmo’s Apple, Urban Legend

Just like previous misconduct in the Netherlands with final theses, it shows the need for thorough peer reviews. Just like the case of Diederik Stapel, it shows that even renowned names need these stringent checks. Such cases are of course very damaging to science. Especially in an area where commerce spends billions to influence us. Where the food industry relies on large marketing budgets, science relies on its reputation. We can’t use Elmo’s apples and urban legends there.

References

1. rtlnieuws.nl/economie/valse-data-hoogleraar-diederik-stapel-uit-tilburg-ontslagen
2. rtlnieuws.nl/nieuws/binnenland/de-acht-opvallendste-bevindingen-van-het-stapel-onderzoek
3. Wansink B, Just DR, Payne CR. Can Branding Improve School Lunches?. Arch Pediatr Adolesc Med.2012;166(10):967–968. doi:10.1001/archpediatrics.2012.999
4. retractionwatch.com/2017/02/02/backlash-prompts-prominent-nutrition-researcher-reanalyze-multiple-papers/
5. buzzfeed.com/stephaniemlee/who-really-ate-the-apples-though?utm_term=.jq3ZpWDa4#.ukLwkYgO6
6. jamanetwork.com/journals/jamapediatrics/fullarticle/2659568
7. Just, D. R., Sı ̆gırcı, ̈O., & Wansink, B. (2014). Lower buffet prices lead to less taste satisfaction.Journal of SensoryStudies,29(5), 362–370. doi:10.1111/joss.121172.Just, D. R., Sı ̆gırcı, ̈O., & Wansink, B. (2015).
8. Peak-end pizza: Prices delay evaluations of quality.Journal ofProduct & Brand Management,24(7), 770–778. doi:10.1108/jpbm01-2015-08023.Kniffin, K. M., Sı ̆gırcı, ̈O., & Wansink, B. (2016).
9. Eating heavily: Men eat more in the company of women.Evolutionary Psychological Science,2(1), 38–46. doi:10.1007/s40806-015-0035-34.Sı ̆gırcı, ̈O., & Wansink, B. (2015).
10. Low prices and high regret: how pricing influences regret at all-you-can-eatbuffets.BMC Nutrition,1(1), 36. doi:10.1186/s40795-015-0030-
11. van der Zee T, Anaya J, Brown NJL. (2017) Statistical heartburn: An attempt to digest four pizza publications from the Cornell Food and Brand Lab. PeerJ Preprints 5:e2748v1https://doi.org/10.7287/peerj.preprints.2748v1