The internet loves theanine. This is an amino acid analog that’s naturally found in tea, but now sold as a nutritional supplement for anxiety or mood or memory. Many people try theanine and report wow or great for ADHD or cured my (social) anxiety or changing my life. And it’s not just the placebo enthusiast community. This hacker news thread is full of positive reports, and gwern uses it regularly. But does it really work? Biologically speaking, it’s plausible. Theanine is structurally related to the neurotransmitter glutamate (theanine = C₇H₁₄N₂O₃, glutamate = C₅H₈NO₄-). For some reason, everyone is obsessed with stupid flashy dopamine and serotonin, and no one cares about glutamate. But it’s the most common common neurotransmitter and theanine is both metabolized into glutamate and seems to itself have various complicated effects on glutamate receptors. Of course, there are lots of supplements that could act on the brain, but are useless when taken orally. That’s because your brain is isolated from your circulatory system by a thin layer of cells that are extremely picky about what they let through. But it appears that theanine can get through these cells and into the brain. So that sounds good. But do these low-level effects actually lead to changes in mood in real humans? When I looked into the academic research, I was surprised by how weak it was. Personally, on these kinds of issues, I find the European Food Safety Authority to be the single most trustworthy scientific body. They did an assessment in 2011 and found: Claim Result Improvement of cognitive function cause and effect relationship has not been established Alleviation of psychological stress cause and effect relationship has not been established Maintenance of normal sleep cause and effect relationship has not been established Reduction of menstrual discomfort cause and effect relationship has not been established Examine is an independent website that’s respected for summarizing the scientific literature on health and supplements. They looked into if theanine helped with various things, like alertness, anxiety, and attention. In all cases found low quality evidence for near zero effect. A 2020 review of eight randomized double-blind placebo controlled trials found that theanine might help with stress and anxiety. While this review seems generally good, I found it to be insufficiently paranoid. One study they review found that theanine worked better than alprazolam (xanax) for acute anxiety. The correct response would be, “That’s impossible, and the fact that normal scientific practices could lead to such a conclusion casts doubt on everything.” But the review sort of takes it at value and moves on. After 2020, the only major trial I could find was this 2021 study that took 52 healthy older Japanese people and gave them theanine (or placebo) for 12 weeks. They tested for improvements in a million different measures of cognitive functioning and mostly found nothing. Why I did this I’ve long found that tea makes me much less nervous than coffee, even with equal caffeine. Many people have suggested theanine as the explanation, but I’m skeptical. Most tea only has ~5 mg of theanine per cup, while when people supplement, they take 100-400 mg. Apparently grassy shade-grown Japanese teas are particularly high in theanine. And I do find those teas particularly calming. But they still only manage ~25 mg per cup. (Maybe it’s because tea is better than coffee?) Still, I’ve supplemented theanine on and off for more than 10 years, and it seems helpful. So after seeing the weak scientific evidence, I thought: Why not do a self-experiment? Theanine seems ideal because it’s a supplement with short term effects. So you can test it against placebo. (Try that with meditation.) And you can build up a large sample using a single human body without waiting weeks for it to build up in the body before each measurement. Everyone agrees theanine is safe. It’s biologically plausible. While academic studies haven’t proven a benefit, they haven’t disproven one either. Given the vast anecdotal evidence, I saw a chance to stick it to the stodgy scientific establishment, to show the power of internet people and give the first rigorous evidence that theanine really works. Stockholm, prepare thyself. What I did First, I needed placebos. This was super annoying. The obvious way to create them would be to buy some empty capsules and fill some with theanine and others with some inert substance. But that doesn’t sound fun. Isn’t the whole idea of modernity that we’re supposed to replace labor with capital? So I went searching for a pair of capsules I could buy off the shelf, subject to the following constraints: Capsule A contains 200 mg of theanine. Capsule B contains something with minimal effects on anxiety, stress, memory, concentration, etc. Capsule B contains something I don’t mind putting into my body. Both capsules are exactly the same size and weight. Both capsules are almost but not quite the same color. Both capsules are made by some company with a history of making at least a modest effort to sell supplements that contain what they say they contain, and that don’t have terrifying levels of heavy metals. The capsules themselves aren’t made from the skin and bones and connective tissues of dead animals (personal preference). After a ludicrous amount of searching, I found that NOW® sells these veggie capsules: Capsule A: 200 mg L-Theanine Capsule B: 25 mcg (1,000 IU) Vitamin D These are exactly the same size, exactly the same weight, exactly the same texture, and very close in color. They’re so close in color that under warm lighting, they’re indistinguishable. But under cold/blue lighting, the vitamin D capsules are slightly more yellow. For dosing, I decided to take a capsule whenever I was feeling stressed or anxious. Some people worry this invalidates the results. Not so! I’m still choosing randomly, and this better reflects how people use theanine in practice. Theanine is often recommended for reducing anxiety from caffeine. While I didn’t explicitly take caffeine as part of this experiment, I had almost always taken some anyway. Statistically, it would have been best to randomize so I had a 50% chance of taking theanine and a 50% chance of taking vitamin D. But I decided that would be annoying, since I was taking these capsules when stressed. So I decided to randomize so I got theanine ⅔ of the time and vitamin D ⅓ of the time. Randomization was very easy: I took two theanine capsules and one vitamin D capsule and put them into a little cup. I then closed my eyes, shook the cup around a bit and took one. I then covered the cup with a card. This picture shows one vitamin D capsule (top) and two theanine capsules. For each trial, I recorded my subjective starting stress level on a scale of 1-5, then set an alarm for an hour, which is enough to reach near-peak concentrations in the blood. After the alarm sounded (or occasionally later, if I missed it) I recorded the end time, my end stress level, and my percentage prediction that what I’d taken was actually theanine. Then, and only then, I looked into the cup. If the two remaining pills were different colors, I’d taken theanine. If not, it was vitamin D. After ~14 months, I got frustrated by how slowly data was coming in. This was the first time in my life, I’ve had too much chill. At that point, I decided to start taking the capsules once or twice a day, even if I wasn’t stressed. I’ll show the transition point in the graphs below. Ultimately, I collected 94 data points, which look like this: Date Start time Start stres End time End stress Prediction Result Nov 18, 2023 9:38 AM 3.5 10:45 AM 2.2 80% T Nov 19, 2023 9:40 AM 2.8 10:41 AM 2.9 75% T ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ Feb 28, 2025 4:58 PM 2.1 5:58 PM 1.8 75% D Jan 3, 2025 6:12 PM 2.1 7:12 PM 2.0 61% T What are the results? Bad. Here are the raw stress levels. Each line line shows one trial, with the start marked with a tiny horizontal bar. Note the clear change when I started dosing daily: Alternatively, here’s the difference in stress (end - start) as a function of time. If “Δ Stress” is negative, that means stress went down. Here are the start and end stress levels for each trial, ignoring time. The dotted line shows equal stress levels, so anything below that line means stress went down: Finally, here are the probabilities I gave that each capsule was in fact theanine. Thoughts Ooof. My stress level did usually go down, at least provided I was stressed at the start. But it went down regardless of if I took theanine or not. And I was terrible at guessing what I’d taken. Why did my stress decrease when I took vitamin D? Maybe it’s the placebo effect. But I suspects it’s mostly reversion to the mean: If you mark down the times in your life when you’re most stressed, on average you’ll be less stressed an hour later. You can see evidence for this in the stress tended to decrease more when it started at a higher level. So, eyeballing the above figures, theanine doesn’t appear to do anything. (We can argue about statistics below.) Why? I think these are the possibilities: Theanine works, but I got fake theanine. Theanine works, but vitamin D works equally well. Theanine works, but I was unlucky. Theanine works, but I’m disembodied and unable to report my internal states. Theanine works on some people, but not me. Theanine doesn’t work. It’s hard to disprove the idea that theanine works. But I tell you this: I expected it to work. And I really tried. For almost 100 trials over 16 months, I paid attention to what I was feeling and tried to detect any sign that I’d taken theanine, even if it wasn’t a change in stress. I could detect nothing. Even after months of failure, I’d often feel confident that this time I could tell, only to be proven wrong. So, cards on the table, here are my made-up probabilities for each of the possible explanations: Explanation belief Fake theanine 3% D equally good 1% Unlucky 6% Disembodied 15% Not on me 20% Doesn’t work 55% Should I have been surprised by these results? Well, the scientific literature on theanine hasn’t found much of an effect. And the only other good self-experiment on theanine I’ve found is by Niplav, who found it did slightly worse than chance and declared it a “hard pass”. What about other blinded self-experiments with other substances? They’re surprisingly scarce, but here’s what I could find: author substance result Niplav caffeine positive Gwern amphetamines positive Gwern lithium no effect Gwern LSD microdose no effect Gwern ZMA inconclusive Slatestarcodex sleep support no effect Stimulants work! But for everything else… I particularly encourage you to read the sleep support post. He was confident it worked, he’d recommended it to lots of friends, but it totally failed when put to the test. I’ve seen many other self-experiments (including for theanine), but they’re non-blinded and I’d be doing you a disservice if I liked to them. People often mention that hypothetically this means the results aren’t scientific, but treat it like a small niggling technicality. It’s not. So I propose a new rule: Blind trial or GTFO. I know many people reading this probably use and like theanine. Maybe it works for you! But given the weak academic results, and given the fact that I actually did a blinded experiment, I think you now have the burden of proof. Doing this kind of test isn’t hard. If you’re sure theanine (or anything else) works, prove it. Appendix: OK fine let’s argue about statistics Do you demand p-values? Are you outraged I just plotted the data and then started talking about it qualitatively? I think faith in statistics follows a U-shaped curve. By default, people don’t trust them. If you learn a little statistics, they seem great. (Particularly if you’re part of a community that’s formed a little cult around one set of statistical practices and convinced each other that they’re more reliable than they are.) But if you learn a lot of statistics, then you realize all the assumptions that are needed and all the ways things can go wrong and you become very paranoid. If you want p-values, I’ll give you p-values. But first let me point out a problem. While I was blinded during each trial, I saw the theanine/D result when I wrote it down. Over time I couldn’t help but notice that my stress dropped even when I took vitamin D, and that I was terrible at predicting what I’d taken. So while this experiment is randomized and blinded, the data isn’t independent or identically distributed. If I did this again, I’d make sure I couldn’t see any outcomes until the end, perhaps by making 100 numbered envelopes, putting three capsules in each, and only looking at what was left at the end. But if you want to compute p-values anyway, OK! Here are the basic numbers for the trials when I took theanine: Variable Substance Mean 95% C.I. p start stress theanine 2.480 (2.361, 2.599)   end stress theanine 2.181 (2.104, 2.258)   Δ stress theanine -0.299 (-0.392, -0.205) 2.00×10⁻⁸ Predicted T theanine 68.4% (66.2%, 70.5%)   Stress went down, p < .0000001. But here are the the numbers for vitamin D: Variable Substance Mean 95% C.I. p start stress vitamin D 2.350 (2.173, 2.526)   end stress vitamin D 2.025 (1.936, 2.114)   Δ stress vitamin D -0.325 (-0.453, -0.197) 2.44×10⁻⁵ Predicted T vitamin D 72.9% (69.7%, 76.1%)   Stress also went down. Finally, here’s the difference between theanine and vitamin D, computed with a two-sided t-test with unequal variance: Variable Substance Mean 95% C.I. p start stress theanine - D 0.130 (-0.095, 0.354) 0.254 end stress theanine - D 0.156 (0.0165, 0.296) 0.029 Δ stress theanine - D -0.026 (-0.201, 0.148) 0.764 Predicted T theanine - D -4.5% (-8.5%, -0.5%) 0.029 Technically, I did find two significant results. But the second row says that end stress was slightly higher with theanine than with vitamin D, and the last row says that I gave slightly higher probabilities that I’d taken theanine when I’d actually taken vitamin D. Of course, I don’t think this means I’ve proven theanine is harmful. I just think this confirms my general paranoia. To a first approximation, if it ain’t visible in the raw data, I ain’t going. Speaking of raw data, you can download mine here.

GLP-1 drugs are a miracle for diabetes and obesity. There are rumors that they might also be a miracle for addiction to alcohol, drugs, nicotine, and gambling. That would be good. We like miracles. We just got the first good trial and—despite what you might have heard—it’s not very encouraging. Semaglutide—aka Wegovy / Ozempic—is a GLP-1 agonist. This means it binds to the same receptors the glucagon-like peptide-1 hormone normally binds to. Similar drugs include dulaglutide, exenatide, liraglutide, lixisenatide, and tirzepatide. These were originally investigated for diabetes, on the theory that GLP-1 increases insulin and thus decreases blood sugar. But GLP-1 seems to have lots of other effects, like preventing glucose from entering the bloodstream, slowing digestion, and making you feel full longer. It was found to cause sharp decreases in body mass, which is why supposedly 12% of Americans had tried one of these drugs by mid 2024. (I’m skeptical that of that 12% number, but a different survey in late 2024 found that 10% of Americans were currently taking one of these drugs. I know Americans take more drugs than anyone on the planet, but still…) Anyway, there are vast reports from people taking these drugs that they help with various addictions. Many people report stopping drinking or smoking without even trying. This is plausible enough. We don’t know which of the many effects of these drugs is really helping with obesity. Maybe it’s not the effects on blood sugar that matter, but these drugs have some kind of generalized “anti-addiction” effect on the brain? Or maybe screwing around with blood sugar changes willpower? Or maybe when people get thinner, that changes how the brain works? Who knows. Beyond anecdotes, are some observational studies and animal experiments suggesting they might help with addiction (OKeefe et al. 2024). We are so desperate for data that some researchers have even resorted to computing statistics based on what people say on reddit. So while it seems plausible these drugs might help with other addictions, there’s limited data and no clear story for why this should happen biologically. This makes the first RCT, which came out last week, very interesting. This paper contains this figure, about which everyone is going crazy: I admit this looks good. This is indeed a figure in which the orange bar is higher than the blue bar. However: This figure does not mean what you think it means. Despite the label, this isn’t actually the amount of alcohol people consumed. What’s shown is a regression coefficient, which was calculated on a non-random subset of subjects. There are other figures. Why isn’t anyone talking about the other figures? What they did This trial gathered 48 participants. They selected them according to the DSM-5 definition of “alcohol use disorder” which happens to be more than 14 drinks per week for men and 7 drinks per week for women, plus at least 2 heavy drinking episodes. Perhaps because of this lower threshold, 34 of the subjects were women. The trial lasted 9 weeks. During it, half of the subjects were given weekly placebo injections. The other half were given weekly injections of increasing amounts of semaglutide: 0.25 mg for 4 weeks, then 0.5 mg for 4 weeks, and then 0.5 or 1 mg in the last week, depending on a doctor’s judgement. Outcome 1: Drinking The first outcome was to simply ask people to record how much they drank in daily life. Here are the results: If I understand correctly, at some point 6 out of the 24 subjects in the placebo group stopped providing these records, and 3 out of 24 in the semaglutide group. I believe the above shows the data for whatever subset of people were still cooperating on each week. It’s not clear to me what bias this might produce. When I first saw that figure, I thought it looked good. The lines are going down, and the semaglutide line is lower. But then I checked the appendix. (Protip: Always check the appendix.) This contains the same data, but stratified by if people were obese or not: Now it looks like semaglutide isn’t doing anything. It’s just that among the non-obese, the semaglutide group happened to start at a lower baseline. How to reconcile this with the earlier figure? Well, if you look carefully, it doesn’t really show any benefit to semaglutide either. There’s a difference in the two curves, but it was there from the beginning. Over time, there’s no difference in the difference, which is what we’d expect to see if semaglutide was helping. The paper provides other measurements like “changes in drinking days” and “changes in heavy drinking days” and “changes in drinks per drinking day”, but it’s the same story: Either no benefit or no difference. So… This is a small sample. It only lasted nine weeks, and subjects spent many of them on pretty small doses. But this is far the miracle we hoped for. Some effect might be hiding in the noise, but what these results most look like is zero effect. Outcome 2: Delayed drinking There are also lab experiments. They did these at both the start and end of the study. In the first experiment, they basically set each subject’s favorite alcoholic drink in front of them and said them, “For each minute you wait before drinking this, we will pay you, up to a maximum of 50 minutes.” How much were they paid, you ask? Oddly, that’s not specified in the paper. It’s also not specified in the supplemental information. It’s also not specified in the 289 page application they made to the FDA to be able to do this study. (Good times!) But there is a citation for a different paper in which people were paid $0.24/minute, decreasing by $0.01 / minute every five minutes. If they used the same amounts here, then the maximum subjects could earn was $9.75. Anyway, here are the results: So… basically nothing? Because almost everyone waited the full 50 minutes? And they did this for only $9.75? Seems weird. I don’t really see this as evidence against semaglutide. Rather, I think this didn’t end up proving much in either direction. Outcome 3: Laboratory drinking So what’s with that initial figure? Well, after the delayed drinking experiment was over, the subjects were given 2 hours to drink as much as they wanted, up to some kind of safe limit. This is what led to the figure everyone is so excited about: When I first saw this, I too thought it looked good. I thought it looked so good that I started writing this post, eager to share the good news. But at some point I read the caption more carefully and my Spidey sense started tingling. There’s two issues here. First of all, subjects were free to skip this part of the experiment, and a lot did. Only 12 of the 24 subjects in the placebo group and 13 of 24 in the semaglutide group actually did it. This means the results are non-randomized. I mean, the people who declined to do this experiment would probably have drunk different amounts than those who agreed, right? So if semaglutide had any influence on people decision to participate (e.g. because it changed their relationship with alcohol, which is the hypothesis of this research) then the results would be biased. That bias could potentially go in either direction. But basically this means we’re sort of working with observational data. The second issue is that what’s being show in this plot is not data. I know it looks like data, but what’s shown are numbers derived from regression coefficients. In the appendix, you can find this table: Basically, they fit a regression to predict how much people drank in this experiment at the end of the study (“g-EtOH”) based on (a) how much they drank during the same experiment at the start of the study (“Baseline”) (b) their sex, and (c) if they got semaglutide or not (“Condition”). Those coefficients are in the B column. How exactly they got from these coefficients to the numbers in the figure isn’t entirely clear to me. But using a plot digitizer I found that the figure shows ~59.9 g for the placebo group and ~33.3 g for the semaglutide group, for a difference of 26.6 g. I believe that difference comes from the regression coefficient for “Condition” (-25.32) plus some adjustments for the fact that sex and baseline consumption vary a bit between the two groups. So… that’s not nothing! This is some evidence in favor of semaglutide being helpful. But it’s still basically just a regression coefficient computed on a non-randomized sample. Which is sad, since the point of RCTs is to avoid resorting to regression coefficients on non-randomized samples. Thus, I put much more faith in outcome #1. Discussion To summarize, the most reliable outcome of this paper was how much people reported drinking in daily life. No effect was observed there. The laboratory experiment suggests some effect, but the evidence is much weaker. When you combine the two, the results of this paper are quite bad, at least relative to my (high) hopes. Obviously, just because the results are disappointing does not mean the research was bad. The measure of science is the importance of the questions, not what the answers happen to be. It’s unfortunate that a non-randomized sample participated in the final drinking experiment, but what were they supposed to do, force them? This experiment involved giving a synthetic hormone and an addictive substance with people with a use disorder. If you have any doubts about the amount of work necessary to bring that to reality, I strongly encourage you to look at the FDA application. OK, fine, I admit that I do feel this paper “hides the bodies” slightly too effectively, in a way that could mislead people who aren’t experts or that don’t read the paper carefully. I think I’m on firm ground with that complaint, since in the discussions I’ve seen, 100% of people were in fact misled. But I’m sympathetic to the reality that most reviewers don’t share my enlightened views about judging science, and that a hypothetical paper written with my level of skepticism would never be published. (People think the problem with science is that it’s too woke. While I don’t really disagree, I still think the bigger problem is screwed up incentives that force everyone oversell everything, because that’s what you have to do to survive. But that’s a story for another time.) Anyway, despite these results, I’m still hopeful that GLP-1 drugs might help with addiction. This is a relatively small study, and it only lasted 9 weeks. I’m don’t think we can dismiss the huge number of anecdotes yet. And the laboratory experiment was at least a little promising. Given how destructive addictions can be, I vote for more research in this direction. Fortunately, given the billions of dollars to be made, that’s sure to happen. But given just how miraculous semaglutide is for obesity, and given the miraculous anecdotes, I don’t see how to spin this paper as anything but a letdown. It provides weak evidence for any effect and comes close to excluding the possibility of another miracle. If you’ve forgotten what miracles look like, here is the figure for body weight:

Some of my favorite internet people sometimes organize little community experiments. Like, let’s eat potatoes and see if we lose weight. Or, let’s try take some supplements and see if anxiety goes down. I’ve toyed with doing one myself, to see if theanine (a chemical in tea) really helps with stress. But sometimes, when everyone is having fun, some very mean very bad people show up and say, “HEY! YOU CAN’T DO THAT! THAT’S HUMAN SUBJECTS RESEARCH! YOU NEED TO GET APPROVAL FROM AN INSTITUTIONAL REVIEW BOARD!” So I wondered—is that right? Who exactly actually needs to get approval from an institutional review board (IRB)? More than a year later, I’m now convinced that: No single source on the internet actually answers that question. The answer is absurdly complex. The reason it’s so complex is that IRB rules are an illegible mishmash of things, some of which themselves have near-fractal complexity. If you stare at this long enough, it’s impossible not to question the degree to which we actually have “laws”. In this post, I’ll give the answer and then explain—in maddening detail—why I think that answer is right. But first I want to tell a little story. I used to live in an apartment with an extremely steep driveway. When I had visitors, I’d tell them, “At the bottom of the driveway you must slow to ⅒ of normal speed or your car will scrape the ground. I tell this to everyone and they only slow to ½ of normal speed and scrape their car on the ground. Don’t do that!” Then they’d only slow to ½ of normal speed and scrape their car on the ground. Why? I think because my visitors simply couldn’t believe my driveway was as stupid as it was. When I explained the rules of safe driveway usage, they mentally substituted the closest version of those rules that would be correct in a sane universe, one without driveways that form sudden 30° angles with the road. I suspect this is a general cognitive bias. So, who exactly needs IRB approval? Here are some myths: Myth: IRB approval is only needed for medical research. Myth: IRB approval is only needed for federally funded research. Myth: IRB approval is only needed for research at “institutions”. Myth: IRB approval is only needed if you want to publish in a journal. Wrong. But this is also a myth: Myth: IRB approval is needed for all research involving human subjects. Also wrong. As far as I can tell, after months of research, here is how it works: Now, I know what you’re thinking: “That can’t be right! The government can’t possibly claim to regulate what me and my roommates eat at home! That would be stupid!” Yes, it would be stupid. But who says the world makes sense? Now, would you actually be prosecuted for violating those rules? Unlikely. If a prosecutor went after you and you fought them in court, you might even be able to get some of these rules declared unconstitutional. But those are the rules as written. Caution: I estimate a 85% chance that this post contains at least one minor error and a 40% chance of a significant error. I’ve gone to insane lengths to try to get things right—I regret ever becoming interested in this topic—but this stuff is insanely convoluted. Past experience says that someone reading this knows much better than I do. If that’s you, let me know about any errors. Reminder of how law works in the US If you’re a sane person, and you want to know when IRB approval is needed, you might think something like, “I know! I’ll go find the IRB law and read it. Then I’ll know what’s legal and what isn’t. Yay!” Hahaha, no. This won’t work, for several reasons. For one, few federal rules actually come from laws. Usually politicians pass some broad law that says, “Human research is hereby regulated, details to come!” And then government employees write regulations that contain all the details, and those regulations have the power of law. And then the government employees issue “clarifications” that supposedly don’t change anything, but everyone treats like new laws. Also, the US constitution theoretically imposes severe limits on the power of the federal government. Article 1 section 8 says the federal government has the power to (1) tax and spend, (2) regulate commerce, (3) control citizenship, (4) create post offices, (5) protect IP, (6) make treaties and (7) do war. And the 10th amendment says that’s it: everything else is left to the states. The federal government doesn’t like those limits, so it evades them in various ways. The classic trick is to declare that everything is related to commerce and therefore almost anything can be considered “regulating commerce”. For example, it’s OK to make growing and eating wheat on your own property illegal, because growing and eating wheat is “commerce”. Another trick is to write rules that sort of say, “If your state/organization wants back any of the money it’s paying in taxes, then you must create and enforce the following rules: […]”. For example, in 1984, the US government decreed that all states must raise the legal age to buy alcohol to 21 or they would lose federal funding for highways. Finally, the limits of federal power are only clarified when someone fights them in court. Did you know that at some point, bureaucrats decided that your boss must guess your race and report it to the government? Is that an illegal overreach? Is it “regulating commerce” to make personal consumption of cannabis illegal in states that have legalized it? The way you find out is you break the rules, get prosecuted, and spend millions fighting in court. (For cannabis, the answer is yes, that is regulating commerce.) If you win, OK, no refunds on your legal fees. If you lose, you pay massive fines or go to prison. Also, courts have constantly changing judges and opinions. So what happens in practice is politicians write a vague law. Bureaucrats turn that law into very detailed (but often still vague) specific rules. Those rules might or might not be “legal”, but nobody want to risk fighting them in court. If the regulations are particularly ridiculous or likely to be overturned if challenged, prosecutors may quietly stop bringing cases. But the regulations still sit there on the books. And people still usually pay attention to them, because why risk it? OK! Where do IRB regulations come from? The Common Rule In 1932, the United States government and Tuskegee University began studying 600 poor black sharecroppers in Macon Country Alabama, ⅔ of whom had latent syphilis. The idea was to observe the progression of the disease if left untreated. By the late 1940s, syphilis was easily treatable with penicillin, but the men were never informed of their condition, never treated, and were even given fake treatments and diagnoses. In 1965, Peter Buxton joined the Public Health service and soon learned of the experiment. He was horrified and filed several official protests, but all were all rejected. Finally, in 1972, Buxton leaked the details to Jean Heller of the Associated Press which led to this front-page article on July 26, 1972. The led to widespread outrage and cancellation of the study. By this time, dozens of the men had died from syphilis and many of their wives and children had also become infected. It also led, in early 1973, a series of congressional hearings led by Ted Kennedy of Massachusetts. These hearings eventually produced the National Research Act. Several people voted for the first version of this bill and then against the second version, including one Joseph Biden of Delaware, though I can’t find any record of why. Anyway, it was signed into law by Richard Nixon on July 12, 1974. Formally speaking, the National Research Act did was to create a commission to develop guidelines for ethical research in human subjects, including how IRBs should work. But in reality, it also sent a message that federal agencies should use their existing authority to regulate research. Even before the Act was passed, the Department of Health and Human Services had started amending the federal code to regulate human research. In 1978, the commission issued a report on IRBs: The Commission believes that the rights of subjects should be protected by local review committees operating pursuant to federal regulations and located in institutions where research involving human subjects is conducted. […] The Commission further believes that institutions receiving federal support for the conduct of research involving human subjects should be governed by uniform federal regulations applicable to the review of all such research, whether it is supported by one federal department or another, or is not federally supported. Informed by this report, lots of different agencies issued different regulations. This continued until 1991, when 15 federal agencies harmonized on what is known as the Common Rule, because it is “common” to different federal agencies. This is now codified as Title 45 Part 46 of the federal code. Since 1991, this has been continuously updated and various other agencies have joined, notably the department of Labor. There are still a couple holdouts including the Nuclear Regulatory Commission, the National Endowment for the Humanities. The CIA is an odd case—they never issued any formal regulations, but supposedly apply the Common Rule because of a cryptic executive order from 1981. The FDA hasn’t joined because they have their own regulatory structure. So what does the Common Rule say? In principle, you can go here and read it. Good luck with that. It’s exactly as readable as “a series of regulations amended by many agencies over many decades” sounds like it would be. But basically (1) It requires informed consent. (2) It requires that risks are reasonable in relation to expected benefits, not including any long-range benefits from new knowledge. (3) It requires participants are selected fairly and equitably (e.g. not all poor black sharecroppers) Finally, (4) It requires that federally funded research with human subjects is reviewed by an IRB. Now, that might sound simple, but what exactly is “research”? This is defined by 45 CFR 46.102(l) to be a “systematic investigation designed to develop or contribute to generalizable knowledge.” The term “research” in this context, is defined by 45 CFR 46.102(l): It is a “systematic investigation designed to develop or contribute to generalizable knowledge.” It’s best to understand this with an example. Say you’re a teacher and you want to test if the cognitive bias of anchoring applies to your class. Is this research? If you’re doing it as part of normal “teaching stuff”, to demonstrate the effect to your students, then no. But if you’re doing it in hope of learning if the anchoring bias is real, then that is research. To some degree, the truth lies in your heart. (The definition of a “human subject” is also somewhat convoluted: What if you’re using data someone else gathered? What if you’re just using some human tissue samples or nonviable human embryos? But never mind.) Note that this applies to all research. The people who made these rules were almost all medical researchers thinking about serious medical risks. But they wrote the rules to apply to all research, including research where you might just ask people some questions. There are some narrow categories of research that are “exempt” from IRB review, e.g. research on standard educational methods or food taste evaluations, or certain benign behavioral interventions . Theoretically, an institution could allow investigators to decide for themselves if they are exempt, but this is not recommended, and in practice, essentially all institutions require you to apply for a determination of exemption. In reality, no research is really exempt, it’s just that certain categories of research have a somewhat more lightweight application process. To get a feel for how much friction this adds, here’s U. Michigan’s application for exemption. Up until the 1990s, IRBs at universities tended to be staffed with faculty “volunteers”. These were sympathetic to the needs of their colleagues, had little interest in reading (unreadable) federal regulations, and probably just wanted to get back to teaching and research. So IRBs were often pretty loose. But in 1996, Hoiyan Wan died after taking part of an experiment on the effects of smoking and air pollution the University of Rochester. And in 1999, Jesse Gelsinger died after taking an experimental gene therapy at the University Of Pennsylvania. Federal agencies cracked down and sent out hundreds of enforcement letters. Universities nervously responded with “hypercompliance”, leading to IRB process everyone loves today. Why is the Common Rule so common? The 1978 commission recommended that any institution accepting federal funding must apply IRB rules to all research, regardless of funding source. But the written regulations did not follow that recommendation. Theoretically, a university could accept federal funding for some research and still allow other research on human subjects without any IRB approval. But in practice, no university ever does that. Why? Well, before giving any federal funding to an institution, the government requires the institution to file a “Federalwise Assurance”. In this, the institution must promise to obey some “statement of ethical principles”. Supposedly this could be anything, but it appears that the only correct answer is “The Belmont Report”, an ethics statement issued by the same commission from 1978. Then there is a section where the institution may “optionally” promise to apply the common rule to all research, regardless of funding source. The government has a database of these “assurances”, but you can’t actually read them. Still, I found that a few prominent institutions that had published theirs. They all promised to abide by the Belmont report. Some (NYU, U. C. Irvine) promised to apply the Common Rule to all research, but most (Mayo Clinic, U. Florida, Emory, U. Michigan, MIT) do not. So if you’re a researcher at MIT, can you do non-federally funded research without an IRB? Nope. MIT rules still say you need an IRB for all research with human subjects. So do the Mayo Clinic’s. So do U. Florida’s rules. So do Emory’s rules. So do U. Michigan’s rules. What’s going on here? Throughout the 1970s, there was a debate about if IRB rules should apply to social science at all. The OHRP (which oversees the Common Rule) gradually asserted more and more broad IRB requirements. Ithiel de Sola Pool, a professor at MIT, protested this growing power. Apparently, a colleague had sought IRB approval to interview Boston anti-busing activists, and been rejected because this research could be used against them. Pool argued that much of social science should be exempt and got support from professional societies. But no matter. Here’s Charles McCarthy, director of the OHRP from 1977 to 1993, bragging about subverting the Carter and the Reagan administrations to add more regulations during the transition in 1980: But we were trying to decide when is the appropriate time to get the Secretary to sign off on the new regulations?  They had already been proposed, we had got comments from the public.  We’d incorporated the comments.  We wrote the preamble, it was ready to go.  And then when is the best time, ‘cause Harris could care less whether they every got published.  So we talked to the transition team, and they said Reagan will never allow new regulations to see the light of day, so if you’re going to get those out at all, you’d better get them out before he takes over. So we’re struggling with this problem, and the way we worked it out was that’s when we wrote, overnight one night–brilliant inspiration–we wrote–everybody stayed in the office, and we worked all night long, and we wrote the exemptions to the regulations, and we also wrote expedited review neither of which had ever been addressed by the Commission.  And that’s where they came from. Then we went to the transition team, and we said would the transition team endorse regulations that are less stringent than the previous regulations?  And, of course, they weren’t, but they looked like they were because we wrote some exceptions.  And so when we sent the package down to Harris, we said “Diminished Regulations for the Protection of Human Subjects.”  And that was the title.  And, of course, we knew nobody down there in the last weeks of the Harris administration getting ready to leave office would actually read it.  So they didn’t know what all that was about, but they could read the title. And so Secretary Harris signed that at her farewell party.  She put down her glass of champagne and signed the regulations on January 19, 1980, and went out of office on the 20th.  So we squeaked by.  Those were some of the adventures we had that were kind of harrowing for regulators, and that’s why some of the language in those exemptions is so convoluted because it was really a first draft.  But it survived and it’s still in there.  And it makes some sense, but if we could rewrite the regulations, we would run that through three or four more drafts and make them crystal clear.  They’re less than elegant writing, whereas if you read the rest of the regulation, I think you’ll find sub-part A is very well-written if you leave out the part about the exemptions. That’s a real quote that you can go read on the hhs.gov website today. The OHRP sent out “non-binding guidance” that suggested all research should be subject to IRB review. According to Hamburger (2007) in the 1990s, the Federalwise Assurance form implied that institutions had to apply the Common Rule to all research, despite the fact that no federal regulation to this effect existed. I can’t find the old form, but according to “persons who were in a position to know”, at one point all but about five institutions in the entire country had made this promise. By 2000-2005, the form made clear that promising to apply the Common Rule (and thus IRBs) to non federally-funded research was optional, and many institutions decline to do this. But they are only declining to make this assurance. Apparently, after years of applying the Common Rule to all research, it had risen to legally qualify as the “standard of care”, meaning that anyone that failed to apply it risked being legally “negligent “ under state law. So, having established the Common Rule as standard through questionably legal tactics, the government could now relax and rely on state tort laws. Now, say you’re affiliated with a university. Federal funded research without IRB approval is out. Any research at the university is out. But say you want to do research in your spare time, at home, without using any university resources. Is that OK? Policies here seem to vary. Some are clear that this is not OK. U.C. San Francisco says: UCSF faculty, staff, or students or researchers at UCSF-affiliated institutions conducting human subjects research require IRB approval before initiating the study. IRB approval is required regardless of the site of the study or the source of funding (if there is funding). Others are a bit ambiguous. MIT says: Any faculty member, employee or student at MIT who conducts human subjects research must apply [to the IRB] if the research involves any form of MIT involvement or support, including funding, personnel, facilities, academic credit or access to experimental subjects. But how could a MIT employee do research without involving MIT personnel, when they themselves are personnel? The closest I can find to anyone saying it’s OK to do independent research without an IRB is Columbia University which says an IRB is needed if: the research is conducted by or under the direction of any employee or agent (faculty/student/staff) of Columbia, in connection with his or her institutional responsibilities What exactly constitutes a “connection”? I guess you get to argue about the meaning of that term if/when you’re investigated for research misconduct. Journals OK! So if you’re doing any research at any federally funded institution, you need an IRB. If you work at such an institution, you could maybe do research at home, as long as it’s unrelated to your job and you’re willing to live dangerously. But suppose you’re not affiliated with any institution. Can you do research without an IRB? One additional barrier is that if you want to publish your results in a journal, almost all journals that publish human subjects research require IRB approval. For example, all Nature journals, Springer journals , and Taylor and Francis journals require you to identify the ethics committee and give a reference number. Science journals require IRB approval don’t seem to require details unless an editor requests them. (Those are publisher requirements. Specific journals can add additional restrictions.) Now, I’m not sure how carefully these journals actually check. My impression is that some do and some don’t. Maybe you could lie and get away with it, but papers do in fact retracted when a lack of IRB approval is revealed. State laws Another barrier is that some states have their own laws. In 1975, New York passed a law that required IRB approval for all human subjects research: Each person engaged in the conduct of human research or proposing to conduct human research shall affiliate himself with an institution or agency having a human research review committee, and such human research as he conducts or proposes to conduct shall be subject to review by such committee in the manner set forth in this section. Virginia law passed a very similar law in 1979. These go further than requiring IRB approval. They make independent human subjects research fully illegal! (Read that quote!) In most states, a random person could submit their research plan to an independent IRB. Not in New York or Virginia. In 2002, Maryland passed house bill 917. I find this text extremely confusing, but everyone seems to agree that that it means that all human subjects research requires IRB approval. I’ve seen many people claim that California also requires IRB approval for all human subjects research. While it’s true that California has some extra regulations, as far as I can tell none of these require IRB approval. FDA regulations Have you found all this to be too simple? Too straightforward and legible? The FDA is here to help. They have an independent set of IRB regulations. While the Common Rule IRB regulations are in Title 45 Part 46 (45 CFR 46), the FDA IRB regulations are in 21 CFR 56 and also 21 CFR Part 50. Whereas the Common Rule derives its constitutional authority from the idea that the government can decide how it wants to give out money, the FDA regulations seem derive their constitutional authority from the idea that it is regulating commerce. So FDA regulations apply regardless of where you work or how you’ve gotten funding. The FDA operates according to the 1938 Food, Drug, and Cosmetics Act and its many amendments, most notably the 1994 Dietary Supplement Health and Education Act. So what do the FDA regulations say? I found them to be much more painful to read than the (already extremely painful) Common Rule regulations. But after much thrashing and screaming, I think the core regulation is 21 CFR 56.103(a): […] any clinical investigation which must meet the requirements for prior submission (as required in parts 312, 812, and 813) […] shall not be initiated unless that investigation has been reviewed […] by an IRB meeting the requirements of this part. 21 CFR § 56.103 gives a two cases where you’re exempt from the above rule, but these aren’t much help. The first (21 CFR § 56.104) is that you’re exempt if you meet the conditions of which basically are that (1) you started your investigation in 1981 or earlier (2) it’s an emergency and you get IRB approval within 5 days, or (3) “taste and food quality evaluations and consumer acceptance studies if wholesome foods without additives are consumed or if a food is consumed that contains a food ingredient at or below the level and for a use found to be safe”. The second (21 CFR § 56.105) is that you’re exempt if you apply to the FDA to be exempt and they waive the requirements. Good luck with that. 21 CFR § 56.103 (b) says that if you apply to the FDA to get a drug approved, they may ignore your data if you should have gotten an IRB but you didn’t. But this an additional thing to worry about, not a waiver to do whatever you want if you don’t care about FDA approval. So what does this mean? Well, a “clinical investigation” is defined by 21 CFR 56.102(c) to be anything involving human subjects and a “test article”. And a “test article” which is defined in turn by 21 CFR 56.102(l) to be any “drug” for human use. So if you’re doing anything with a “drug” that requires “prior submission”, then you need an IRB. What do those terms mean? 21 CFR Part 312 makes clear that “prior submission” is referring to an investigational new drug (IND) application to the FDA. (Caution: An “IRB” and an “IND” are totally different things. An IRB is what this article is about. An IND is the kind of application that pharmaceutical companies make to the FDA before they can start clinical trials.) So, at first, this seems like good news: As long as you don’t need to do a (gigantic, extremely expensive) IND application to the FDA before starting your research, then FDA regulations don’t force you to use an IRB. Whew! Except… When are you supposed to submit an IND? Surely that’s not some incredibly broad set of circumstances that could apply to almost anyone doing almost anything, is it? Behold: This is a supposedly “nonbinding” document that that supposedly just explains old rules. Supposedly the FDA is just publishing this as a helpful service to all the mouth-breathers who can’t understand the federal regulations. But everyone in industry behaves as if it’s a new law. Which is understandable, since the written regulations are so confusing and ambiguous. If you read this guidance, you will discover that an IND is needed if: The research involves a “drug” The research is a “clinical investigation” The research is not exempt. If you want to avoid an IND, then 2. and 3. are little help. The term “clinical investigation” in is defined as anything where a “drug” is given to human subjects. And the exemptions are extremely narrow: They are only for marketed drugs where the drug is being used for the labeled purpose as directed. Testing something already approved for a disease? Not exempt. Testing if normal food helps with some new disease? Not exempt. You’re only exempt if you’re doing research on something the FDA has already accepted to be true. “Do I need an IND if I use a lawfully marketed drug for an unlabeled indication? If you are a health care provider and you prescribe a marketed drug to treat a patient for an unlabeled indication (also referred to as off-label use), an IND is not required because this use is considered to be within the scope of medical practice and not a clinical investigation. However, if you use the marketed drug for the same purpose in a clinical investigation intended to evaluate the drug’s ability to treat a disease or condition, an IND is required under part 312 unless the clinical investigation meets the criteria for an exemption for studies of lawfully marketed drugs (see 21 CFR 312.2(b) and section IV.A of this guidance.” “If a product containing substances generally recognized as safe (GRAS) for use in food is administered to subjects in a study intended to evaluate the effect of the substance on the pathogenesis of a human disease, is an IND required? Substances designated as GRAS for use in food are generally not approved as drug products. A clinical investigation of a GRAS substance that is intended to evaluate the product’s ability to diagnose, cure, mitigate, treat, or prevent disease requires an IND under part 312, unless the substance to be studied is also a lawfully marketed drug and the clinical investigation meets the criteria for exemption under 21 CFR 312.2(b).” If you want to avoid an IND, your best hope is that your research doesn’t involve a “drug”. Unfortunately, the term “drug” is defined extremely broadly, way beyond the colloquial meaning of “drug” in English. A drug is defined to include: articles intended for use in the diagnosis, cure, mitigation, treatment, or prevention of disease It also includes: articles (other than food) intended to affect the structure or any function of the body Get that? Here’s a little picture: Now, I know what you’re thinking. “That can’t be right! That would mean that if I wanted to study if some normal food reduced the odds of, say, diabetes, then I wouldn’t just need IRB approval, I would also need to submit a freaking IND! That would be stupid!” But the FDA is quite clear that this is right: As is the case for a dietary supplement, a food is considered to be a drug if it is “intended for use in the diagnosis, cure, mitigation, treatment, or prevention of disease,” except that a food may bear an authorized health claim about reducing the risk of a disease without becoming a drug (see section VI.D.3). Therefore, a clinical investigation intended to evaluate the effect of a food on a disease would require an IND under part 312. For example, a clinical investigation intended to evaluate the effect of a food on the signs and symptoms of Crohn’s disease would require an IND. FDA also says: Does a physician, in private practice, conducting research with an FDA regulated product, need to obtain IRB approval? Yes. The FDA regulations require IRB review and approval of regulated clinical investigations, whether or not the study involves institutionalized subjects. FDA has included non-institutionalized subjects because it is inappropriate to apply a double standard for the protection of research subjects based on whether or not they are institutionalized. An investigator should be able to obtain IRB review by submitting the research proposal to a community hospital, a university/medical school, an independent IRB, a local or state government health agency or other organizations. At first, I thought this couldn’t possibly be true. In 2017, it apparently cost $619,200 to submit an IND application to the FDA. However, in practice it appears that there are many exceptions and that there are two categories of INDs: Commercial and research. I assume the FDA doesn’t actually try to charge researchers $619,200 to review their research projects, but I don’t really know. The FDA’s IND application page is here. Have fun. I’m not sure how seriously people take these FDA regulations in practice. People do seem to get IND approval for research on actual (chemical) drugs. But does anyone actually seek IND approval if they want to study if diet influences obesity? Do they carefully avoid talking about disease, so they can fall under the looser “structure and function of the body”? And is private research on normal food really a form of inter-state commerce? I’m just not sure. What I’ve learned Mostly I’ve learned that the set of abstractions I was using to think about the world were wrong. I assumed that there were “laws” that someone had written, and that someone could read to understand what is legal and what isn’t. But in reality, politicians write vague laws, and then agencies write vague sprawling regulations, and then those agencies issue vague “clarifications”. Sometimes these might be unconstitutional or illegal, but no one knows unless they want to fight things in court. But courts are unpredictable, and even going to court is ruinously expensive. Trying to comply with the the written law is like trying to comply with a giant mound of potatoes. So people basically operate as if the law doesn’t exist. The “law” is what the people who enforce the law choose to enforce. If you need to live in this world, that’s what you pay attention to. The law is what you can get away with. What I would change All the IRB rules were designed by medical doctors to regulate the research of other medical doctors, thinking about incidents like the Tuskegee syphilis study. But here’s an opinion: Needing to submit an application before you can ask people to fill out some simple online survey is stupid. Needing to submit an application before you can interview a bunch of people is stupid. I like incremental change, so why don’t we start by making “exempt” research actually exempt? For example, 45 CFR 46.104(d)(3)(ii) exempts “benign behavioral interventions” from IRB review. But you still need to apply to prove you’re exempt. Why? As an analogy, driving a car is dangerous. Whenever I drive, I could easily kill someone. But the government doesn’t force me to submit a driving plan any time I want to go somewhere. Instead, if I misbehave, I am punished in retrospect. Why don’t we apply the same policy to research? Notes I found AI to be completely useless at parsing all these federal regulations. Maybe you need AI that’s specifically trained to be good at parsing legal text? Say I want to feed potatoes to my friends and see if that reduces diabetes. Maybe that’s dumb. Fine. But the FDA seems to be claiming that it is constitutionally permitted to regulate this because by doing this I would be engaging in “interstate commerce”. Really? Maybe the Chevron decision might have some implications for all this? As far as I can tell, no one has ever been prosecuted anyone for failure to comply with IRB requirements. It’s not even clear what the penalty would be. In practice, rules are enforced on institutions—if one failed to comply, then their “assurance” might be cancelled, meaning they couldn’t take any federal funding for research. Or, if they wanted to sell something, the FDA might refuse to approve it for sale. Are you chaotic good? Then you may find it interesting that it seems to be relatively easy to form a new IRB. (HHS rules / FDA rules). And that while all federally funded institutions require IRB approval for all research, only some require approval from their IRB. You can probably see where this is going. Further reading Philip Hamburger: Getting Permission (2007) IRB Licensing (2015) My Bookshelf Runneth Over: A history of US IRBs Ethical Imperialism: IRBs and the Social Sciences, 1965–2009 Many thanks to Willy Chertman.

Some time ago—I’m not sure when exactly—my car started rattling. It would only rattle: When the engine was on, sitting idle, or When accelerating with just the right amount of throttle. This rattle, I did not like it. It sounded like a tiny spoon in a garbage disposal. Which can’t be good, can it? But I exist only in the world of ideas and couldn’t summon the executive function to do anything about it. Eventually, the future Dynomight biologist rode in the car, and we had this conversation: Dynomight biologist: What’s that sound? Dynomight: Rattling! Dynomight biologist: (Pause.) Huh. (In the “Huh”, I could sense overtones of, “How interesting that you would choose to live like this.”) Time went by. I kept reminding myself that selfhood doesn’t exist and therefore we all have a moral responsibility to be kind to our future selves and that future me wouldn’t be any more enthusiastic having this rattle situation dumped on them than I was. So I spent many irreplaceable hours reading about the many, many possible causes of rattling. Eventually, I came to the conclusion that it wasn’t rattling, but rather incomplete fuel combustion. I put in high-octane petrol, convinced that would make the sound would go away. But it didn’t. So I spent more hours reading. Maybe it was a problem with the catalytic converter? Rod bearings? Heat shield? Maybe it was incomplete combustion, but I’d let it go on so long that the car was damaged? Nothing seemed to exactly fit the symptoms. Finally, after several years, I decided try something crazy: I started the car, lay on the ground, and tried to look for where the rattling noise was coming from. When I did that, I immediately saw an extremely rusted piece of metal dancing around on a pipe. I pulled on it, and this thing fell off the car: And the rattling stopped. What is this disease? (Cf. my stupid noise journey.) Maybe it’s that if you spend all your time trying to understand complex systems, then when you face a complex system that isn’t behaving like you want, you naturally… try to understand it. But that’s not necessarily smart. Often, “understanding” is weak. Thinking is weak. The world is chaotic and not easy to simulate inside a brain. Often, you want to resist the urge to understand and simply gather more information. Instead of thinking, look. Maybe that’s the bitter lesson for real life.