A Guide to Social Media Science

By Naomi Kritzer ’95
If you use social media, you probably spot dozens of science or sciencey articles on your feed daily, from clickbait about weight loss to headlines that tease “Coffee: Does It Cure Basically Everything?” Sometimes a headline is catchy enough that people share the article without even reading it.

One of my specialties as a writer is tackling viral science stories. On occasion, they contain genuinely exciting information about new discoveries. But more often, they don’t. Or the thing that was discovered isn’t the bit that everyone’s talking about. Here’s what I ask when I dive in.

Naomi Kritzer ’95Naomi Kritzer ’95 Photo: Sara Rubinstein ’98
Why We Don’t Give Flu Shots at the Company Holiday Party

The headline says “Drinking Alcohol Can Help You Fight Off Colds,” which is such an appealing take-home message that it’s tempting to stop right there. But here’s what you’ll find if you keep reading or look up the study:

The study was on vaccine response, not colds.

Specifically, it was on vaccine response in rhesus monkeys.

The monkeys in the study were given free access to booze. The chronically drunk monkeys had a markedly worse vaccine response than either the control group or the moderate alcohol consumers. The article gives only glancing mention to the fact that serious inebriation will do you no favors.

That said, the authors seem to define moderate drinking in monkeys differently from how it’s defined in humans. In humans, it’s moderate if it’s no more than one drink per day for women and two drinks per day for men. The monkeys were considered moderate drinkers if they had a blood alcohol content of less than .08 most of the time. I haven’t tested this experimentally, but I’m pretty sure that to even register as a drinker in a study like this, I’d need more than one drink per day.

I’m not convinced the researchers established causality here. I mean, to be fair, probably the causality is that mild drunkenness enhances the monkey immune response while serious drunkenness harms it, but it’s possible that having a crappy immune system makes monkeys want to drink heavily while having a strong immune system makes them want to drink more lightly.

They don’t mention in the research abstract how many monkeys were in the study, and how many of them were heavy drinkers versus how many were moderate drinkers. They also don’t mention whether they observed other differences in the monkeys that drank a lot. There surely were some differences, though, and those differences might also affect immune response.

The authors note in a follow-up article that the people with the worst response to vaccines are infants, the elderly, and people who are immune-compromised. “Drink a shot of whiskey before your flu shot” is not considered a good recommendation for any of these groups. Especially infants.

Does this article present any real evidence?

Plenty of articles with sciencey headlines are nonscience, pseudoscience, or only barely science. An article that makes the rounds endlessly promises whiskey will cure your cold. The text explains that alcohol dilates your blood vessels, which may help slightly to relieve your symptoms. (I categorize that one as “barely science.”)

Is the expert legit?

Sometimes the article is about a person with a PhD or an MD who has an opinion about something. In that case, I question whether the credential is relevant in any way. In general, this is easy to check. I Google the person’s name (and institution affiliation if it’s mentioned). Most academics post their curriculum vitae online, so I can pull that up and see what their research is in, what they’ve published, whether any of it is recent, and whether it’s remotely relevant.

Is the organization’s name deceptive?

The American College of Pediatricians is not the American Academy of Pediatrics. It’s a small splinter group with a conservative agenda. The National Vaccine Information Center is an antivaccination group. There are many similar examples.

If it’s about a medical breakthrough, was the research conducted with humans, animals, or cells?

I see articles every few weeks about breakthroughs in Type I diabetes and Alzheimer’s research, and there is some genuinely exciting work being done. But we’ve cured these diseases in mice multiple times, so I don’t get excited about mouse studies on either one.

I’ve determined that it’s an actual study conducted by a real academic at a legit institution. Now what?

It’s rare that we can read a just-published study for free online. (Carleton alumni can access JSTOR through Gould Library, which is awesome, but I find it to be hit-and-miss when it comes to anything recent.) Abstracts are always available, though, and can be pretty informative. If it’s a medical study, you should be able to find it here: www.ncbi.nlm.nih.gov/pubmed.

For any other sort of study, if I know where it was published, I should be able to find the journal and abstract online.

And once I’ve located the study, I ask more questions. Is there other research on this subject? What does it say?

If there have been 20 studies showing one thing and this study found something else, does the researcher have a theory about why his or her results are different? There’s a reason why meta-analysis is considered a powerful tool: it involves looking at all the different studies that have been done to find an overall consensus.

What was the sample size?

If this was a medical study, was it double-blind with a placebo or sham control? How did they manage the placebo? I once found a study in which participants were not told whether the baked goods they were eating did or did not contain gluten. If you’ve ever eaten a gluten-free muffin, you will understand my skepticism.

Are the researchers saying they found causation or just correlation?

Could the causation run the other way, or is there some other obvious path for a misleading correlation? This is one of the biggest errors I see in science articles in the press. Even when the research study is clearly just reporting a correlation, the articles making the rounds will claim causation.

What is the confidence interval?

Medical studies often report a hazard ratio and it will look something like this: HR: 0.25; 95% CI: 0.10 to .47. A small hazard ratio is good: a hazard ratio of .25 means that this thing is one-fourth as hazardous as the default option they’re comparing to. The CI is the confidence interval: here, they are saying that they are 95 percent certain that the .25 hazard ratio they’re giving you falls between .10 and .47. (So maybe it’s actually one-tenth as hazardous as the default option, or maybe it’s only one-half as hazardous.) The larger the confidence interval, the more meaningless the number they’re giving you. And even if it’s small, if it’s something like .85 to 1.2, that means they’re not sure if this thing is bad for you, or good for you.

Finally, is there actionable information here and, if so, what is it?

Sometimes new studies provide important guidance. For years, new parents have been advised to keep their babies away from peanuts until they were at least two years old in order to prevent allergies. In the past year, however, several studies have shown that this is terrible advice and parents are now being advised to expose their babies to peanuts to avoid allergies.

Sometimes, new studies just provide you with new things to worry about. If you rely on a daily medication and a study finds that this medication has long-term risks, that isn’t helpful information if you need that medication. Sometimes you can mitigate your risks in some way, and certainly it can be worthwhile to talk with your doctor about the study, but much of the time there’s nothing you can do.

And, of course, new studies are being published every day reiterating things we already know: exercise is good for you, getting enough sleep is good for you, friendship is good for you. These are all actionable, but you knew them already. Will one more study on the benefits of exercise get me up out of my chair? Still waiting to find out!

Naomi Kritzer ’95Naomi Kritzer ’95 Photo: Sara Rubinstein ’98

What Doesn’t Kill You Might Make You Stronger. Or Just Kill You.

With most things, we understand the value of moderation. Light drinking makes you healthier; heavy drinking will make you less healthy. Calcium may give you strong bones, but too much may give you kidney stones. Even eating too much kale has risks.

But jogging. Surely there can’t possibly be such a thing as too much jogging, right?

A study published in February claimed that running too much was as bad for you as not exercising at all, which seems like a self-evidently absurd claim, unless people are running on the shoulders of interstate highways or through toxic waste dumps.

And, having scrutinized the study, I am dubious. Bear with me, because this is going to require a statistics explanation.

From the study itself: “The joggers were divided into light, moderate, and strenuous joggers. The lowest HR for mortality was found in light joggers (HR: 0.22; 95% CI: 0.10 to 0.47), followed by moderate joggers (HR: 0.66; 95% CI: 0.32 to 1.38) and strenuous joggers (HR: 1.97; 95% CI: 0.48 to 8.14).”

HR stands for hazard ratio. You want your HR to be small. All the joggers were compared to a group of sedentary nonjoggers: being a couch potato had a hazard ratio of 1. So a hazard ratio that’s in the form of a decimal is a good thing. The researchers calculated that light joggers had a hazard ratio of 0.22, and moderate joggers had a hazard ratio of 0.66. Strenuous joggers had a hazard ratio of 1.97, which is shockingly bad.

But, OK, the second set of numbers in the write-up is the confidence interval. For light joggers, they say, “95% CI: 0.10 to 0.47.” That means they came up with the number 0.22 and, based on their statistical black magic, they are 95 percent sure that the real number is somewhere between 0.10 and 0.47. For moderate joggers, they are 95 percent sure that the real hazard ratio is somewhere between 0.32 and 1.38.

If your eyes glazed over, here’s what you need to know: a small confidence interval is like saying, “We’re pretty sure the dude we’re looking for is somewhere in the building.” A large confidence interval is like saying, “We’re pretty sure the dude we’re looking for is somewhere in the neighborhood.”

For the strenuous joggers, the confidence interval is 0.48 to 8.14. “That dude we’re looking for? We think there’s a 95 percent chance he’s somewhere in New York.” In the actual paper, the authors draw it out for you, and the confidence intervals for the moderate joggers fall almost entirely inside the confidence interval for the strenuous joggers. So, maybe strenuous jogging is worse for you, but that 95 percent confidence level they’re talking about should definitely not be applied to that 1.97 hazard ratio.

The researchers are uncertain because there weren’t many strenuous joggers in the study—there were 40—and, over the course of the study, 2 of them died. No matter how good something is for you, it’s no guarantee that you won’t get hit by lightning or cancer.

On the other hand, “too much running might not actually be good for you” is actually not a new finding. A 2011 study found at least temporary heart damage following marathons. A study published in 2014 found an increase in coronary artery plaque among marathon runners. One of the authors, James H. O’Keefe, was also involved in the February paper. It’s probably fair to say that the risks of excessive exercise are a personal hobbyhorse for him. And, in 2013, a study on heart-attack survivors found that more running or walking was beneficial until participants hit 7.1 kilometers of running or 10.7 kilometers of walking daily, at which point more exercise created worse outcomes. And that paper was from James O’Keefe’s apparent professional nemesis, Paul Williams, who has spent decades researching how more exercise is better than whatever you’re doing right now.

The bottom line here is that we don’t really know. The latest study might show that too much running (by which O’Keefe and company mean more than four hours a week) is bad for you. Or maybe it just shows that two of their most avid runners were unlucky. Lots of running may give you artery plaque and wider arteries, making the plaque a nonissue.

We do know for sure that exercising is a whole lot better for you than not exercising. So if the prospect of running a marathon is what motivates you to be active? Do it. And if you’d rather stick with an easygoing quarter-mile daily jog, do that instead. You can find some science on your side either way. There is no science out there that will back your decision to stay on the couch. At least not this week.

Follow Naomi Kritzer’s blog at naomikritzer.livejournal.com.


  • September 12 2016 at 2:23 pm
    Jacque Harper

    Great article. I'm sending a link to the teachers of sciences at my daughter's high school. I think this article should be read by all students in high school!

Add a comment

The following fields are not to be filled out. Skip to Submit Button.
(This is here to trap robots. Don't put any text here.)
(This is here to trap robots. Don't put any text here.)
(This is here to trap robots. Don't put any text here.)