GLP-1 drugs are a miracle for diabetes and obesity. There are rumors that they might also be a miracle for addiction to alcohol, drugs, nicotine, and gambling. That would be good. We like miracles. We just got the first good trial and—despite what you might have heard—it’s not very encouraging. Semaglutide—aka Wegovy / Ozempic—is a GLP-1 agonist. This means it binds to the same receptors the glucagon-like peptide-1 hormone normally binds to. Similar drugs include dulaglutide, exenatide, liraglutide, lixisenatide, and tirzepatide. These were originally investigated for diabetes, on the theory that GLP-1 increases insulin and thus decreases blood sugar. But GLP-1 seems to have lots of other effects, like preventing glucose from entering the bloodstream, slowing digestion, and making you feel full longer. It was found to cause sharp decreases in body mass, which is why supposedly 12% of Americans had tried one of these drugs by mid 2024. (I’m skeptical that of that 12% number, but a different survey in late 2024 found that 10% of Americans were currently taking one of these drugs. I know Americans take more drugs than anyone on the planet, but still…) Anyway, there are vast reports from people taking these drugs that they help with various addictions. Many people report stopping drinking or smoking without even trying. This is plausible enough. We don’t know which of the many effects of these drugs is really helping with obesity. Maybe it’s not the effects on blood sugar that matter, but these drugs have some kind of generalized “anti-addiction” effect on the brain? Or maybe screwing around with blood sugar changes willpower? Or maybe when people get thinner, that changes how the brain works? Who knows. Beyond anecdotes, are some observational studies and animal experiments suggesting they might help with addiction (OKeefe et al. 2024). We are so desperate for data that some researchers have even resorted to computing statistics based on what people say on reddit. So while it seems plausible these drugs might help with other addictions, there’s limited data and no clear story for why this should happen biologically. This makes the first RCT, which came out last week, very interesting. This paper contains this figure, about which everyone is going crazy: I admit this looks good. This is indeed a figure in which the orange bar is higher than the blue bar. However: This figure does not mean what you think it means. Despite the label, this isn’t actually the amount of alcohol people consumed. What’s shown is a regression coefficient, which was calculated on a non-random subset of subjects. There are other figures. Why isn’t anyone talking about the other figures? What they did This trial gathered 48 participants. They selected them according to the DSM-5 definition of “alcohol use disorder” which happens to be more than 14 drinks per week for men and 7 drinks per week for women, plus at least 2 heavy drinking episodes. Perhaps because of this lower threshold, 34 of the subjects were women. The trial lasted 9 weeks. During it, half of the subjects were given weekly placebo injections. The other half were given weekly injections of increasing amounts of semaglutide: 0.25 mg for 4 weeks, then 0.5 mg for 4 weeks, and then 0.5 or 1 mg in the last week, depending on a doctor’s judgement. Outcome 1: Drinking The first outcome was to simply ask people to record how much they drank in daily life. Here are the results: If I understand correctly, at some point 6 out of the 24 subjects in the placebo group stopped providing these records, and 3 out of 24 in the semaglutide group. I believe the above shows the data for whatever subset of people were still cooperating on each week. It’s not clear to me what bias this might produce. When I first saw that figure, I thought it looked good. The lines are going down, and the semaglutide line is lower. But then I checked the appendix. (Protip: Always check the appendix.) This contains the same data, but stratified by if people were obese or not: Now it looks like semaglutide isn’t doing anything. It’s just that among the non-obese, the semaglutide group happened to start at a lower baseline. How to reconcile this with the earlier figure? Well, if you look carefully, it doesn’t really show any benefit to semaglutide either. There’s a difference in the two curves, but it was there from the beginning. Over time, there’s no difference in the difference, which is what we’d expect to see if semaglutide was helping. The paper provides other measurements like “changes in drinking days” and “changes in heavy drinking days” and “changes in drinks per drinking day”, but it’s the same story: Either no benefit or no difference. So… This is a small sample. It only lasted nine weeks, and subjects spent many of them on pretty small doses. But this is far the miracle we hoped for. Some effect might be hiding in the noise, but what these results most look like is zero effect. Outcome 2: Delayed drinking There are also lab experiments. They did these at both the start and end of the study. In the first experiment, they basically set each subject’s favorite alcoholic drink in front of them and said them, “For each minute you wait before drinking this, we will pay you, up to a maximum of 50 minutes.” How much were they paid, you ask? Oddly, that’s not specified in the paper. It’s also not specified in the supplemental information. It’s also not specified in the 289 page application they made to the FDA to be able to do this study. (Good times!) But there is a citation for a different paper in which people were paid $0.24/minute, decreasing by $0.01 / minute every five minutes. If they used the same amounts here, then the maximum subjects could earn was $9.75. Anyway, here are the results: So… basically nothing? Because almost everyone waited the full 50 minutes? And they did this for only $9.75? Seems weird. I don’t really see this as evidence against semaglutide. Rather, I think this didn’t end up proving much in either direction. Outcome 3: Laboratory drinking So what’s with that initial figure? Well, after the delayed drinking experiment was over, the subjects were given 2 hours to drink as much as they wanted, up to some kind of safe limit. This is what led to the figure everyone is so excited about: When I first saw this, I too thought it looked good. I thought it looked so good that I started writing this post, eager to share the good news. But at some point I read the caption more carefully and my Spidey sense started tingling. There’s two issues here. First of all, subjects were free to skip this part of the experiment, and a lot did. Only 12 of the 24 subjects in the placebo group and 13 of 24 in the semaglutide group actually did it. This means the results are non-randomized. I mean, the people who declined to do this experiment would probably have drunk different amounts than those who agreed, right? So if semaglutide had any influence on people decision to participate (e.g. because it changed their relationship with alcohol, which is the hypothesis of this research) then the results would be biased. That bias could potentially go in either direction. But basically this means we’re sort of working with observational data. The second issue is that what’s being show in this plot is not data. I know it looks like data, but what’s shown are numbers derived from regression coefficients. In the appendix, you can find this table: Basically, they fit a regression to predict how much people drank in this experiment at the end of the study (“g-EtOH”) based on (a) how much they drank during the same experiment at the start of the study (“Baseline”) (b) their sex, and (c) if they got semaglutide or not (“Condition”). Those coefficients are in the B column. How exactly they got from these coefficients to the numbers in the figure isn’t entirely clear to me. But using a plot digitizer I found that the figure shows ~59.9 g for the placebo group and ~33.3 g for the semaglutide group, for a difference of 26.6 g. I believe that difference comes from the regression coefficient for “Condition” (-25.32) plus some adjustments for the fact that sex and baseline consumption vary a bit between the two groups. So… that’s not nothing! This is some evidence in favor of semaglutide being helpful. But it’s still basically just a regression coefficient computed on a non-randomized sample. Which is sad, since the point of RCTs is to avoid resorting to regression coefficients on non-randomized samples. Thus, I put much more faith in outcome #1. Discussion To summarize, the most reliable outcome of this paper was how much people reported drinking in daily life. No effect was observed there. The laboratory experiment suggests some effect, but the evidence is much weaker. When you combine the two, the results of this paper are quite bad, at least relative to my (high) hopes. Obviously, just because the results are disappointing does not mean the research was bad. The measure of science is the importance of the questions, not what the answers happen to be. It’s unfortunate that a non-randomized sample participated in the final drinking experiment, but what were they supposed to do, force them? This experiment involved giving a synthetic hormone and an addictive substance with people with a use disorder. If you have any doubts about the amount of work necessary to bring that to reality, I strongly encourage you to look at the FDA application. OK, fine, I admit that I do feel this paper “hides the bodies” slightly too effectively, in a way that could mislead people who aren’t experts or that don’t read the paper carefully. I think I’m on firm ground with that complaint, since in the discussions I’ve seen, 100% of people were in fact misled. But I’m sympathetic to the reality that most reviewers don’t share my enlightened views about judging science, and that a hypothetical paper written with my level of skepticism would never be published. (People think the problem with science is that it’s too woke. While I don’t really disagree, I still think the bigger problem is screwed up incentives that force everyone oversell everything, because that’s what you have to do to survive. But that’s a story for another time.) Anyway, despite these results, I’m still hopeful that GLP-1 drugs might help with addiction. This is a relatively small study, and it only lasted 9 weeks. I’m don’t think we can dismiss the huge number of anecdotes yet. And the laboratory experiment was at least a little promising. Given how destructive addictions can be, I vote for more research in this direction. Fortunately, given the billions of dollars to be made, that’s sure to happen. But given just how miraculous semaglutide is for obesity, and given the miraculous anecdotes, I don’t see how to spin this paper as anything but a letdown. It provides weak evidence for any effect and comes close to excluding the possibility of another miracle. If you’ve forgotten what miracles look like, here is the figure for body weight:

Some of my favorite internet people sometimes organize little community experiments. Like, let’s eat potatoes and see if we lose weight. Or, let’s try take some supplements and see if anxiety goes down. I’ve toyed with doing one myself, to see if theanine (a chemical in tea) really helps with stress. But sometimes, when everyone is having fun, some very mean very bad people show up and say, “HEY! YOU CAN’T DO THAT! THAT’S HUMAN SUBJECTS RESEARCH! YOU NEED TO GET APPROVAL FROM AN INSTITUTIONAL REVIEW BOARD!” So I wondered—is that right? Who exactly actually needs to get approval from an institutional review board (IRB)? More than a year later, I’m now convinced that: No single source on the internet actually answers that question. The answer is absurdly complex. The reason it’s so complex is that IRB rules are an illegible mishmash of things, some of which themselves have near-fractal complexity. If you stare at this long enough, it’s impossible not to question the degree to which we actually have “laws”. In this post, I’ll give the answer and then explain—in maddening detail—why I think that answer is right. But first I want to tell a little story. I used to live in an apartment with an extremely steep driveway. When I had visitors, I’d tell them, “At the bottom of the driveway you must slow to ⅒ of normal speed or your car will scrape the ground. I tell this to everyone and they only slow to ½ of normal speed and scrape their car on the ground. Don’t do that!” Then they’d only slow to ½ of normal speed and scrape their car on the ground. Why? I think because my visitors simply couldn’t believe my driveway was as stupid as it was. When I explained the rules of safe driveway usage, they mentally substituted the closest version of those rules that would be correct in a sane universe, one without driveways that form sudden 30° angles with the road. I suspect this is a general cognitive bias. So, who exactly needs IRB approval? Here are some myths: Myth: IRB approval is only needed for medical research. Myth: IRB approval is only needed for federally funded research. Myth: IRB approval is only needed for research at “institutions”. Myth: IRB approval is only needed if you want to publish in a journal. Wrong. But this is also a myth: Myth: IRB approval is needed for all research involving human subjects. Also wrong. As far as I can tell, after months of research, here is how it works: Now, I know what you’re thinking: “That can’t be right! The government can’t possibly claim to regulate what me and my roommates eat at home! That would be stupid!” Yes, it would be stupid. But who says the world makes sense? Now, would you actually be prosecuted for violating those rules? Unlikely. If a prosecutor went after you and you fought them in court, you might even be able to get some of these rules declared unconstitutional. But those are the rules as written. Caution: I estimate a 85% chance that this post contains at least one minor error and a 40% chance of a significant error. I’ve gone to insane lengths to try to get things right—I regret ever becoming interested in this topic—but this stuff is insanely convoluted. Past experience says that someone reading this knows much better than I do. If that’s you, let me know about any errors. Reminder of how law works in the US If you’re a sane person, and you want to know when IRB approval is needed, you might think something like, “I know! I’ll go find the IRB law and read it. Then I’ll know what’s legal and what isn’t. Yay!” Hahaha, no. This won’t work, for several reasons. For one, few federal rules actually come from laws. Usually politicians pass some broad law that says, “Human research is hereby regulated, details to come!” And then government employees write regulations that contain all the details, and those regulations have the power of law. And then the government employees issue “clarifications” that supposedly don’t change anything, but everyone treats like new laws. Also, the US constitution theoretically imposes severe limits on the power of the federal government. Article 1 section 8 says the federal government has the power to (1) tax and spend, (2) regulate commerce, (3) control citizenship, (4) create post offices, (5) protect IP, (6) make treaties and (7) do war. And the 10th amendment says that’s it: everything else is left to the states. The federal government doesn’t like those limits, so it evades them in various ways. The classic trick is to declare that everything is related to commerce and therefore almost anything can be considered “regulating commerce”. For example, it’s OK to make growing and eating wheat on your own property illegal, because growing and eating wheat is “commerce”. Another trick is to write rules that sort of say, “If your state/organization wants back any of the money it’s paying in taxes, then you must create and enforce the following rules: […]”. For example, in 1984, the US government decreed that all states must raise the legal age to buy alcohol to 21 or they would lose federal funding for highways. Finally, the limits of federal power are only clarified when someone fights them in court. Did you know that at some point, bureaucrats decided that your boss must guess your race and report it to the government? Is that an illegal overreach? Is it “regulating commerce” to make personal consumption of cannabis illegal in states that have legalized it? The way you find out is you break the rules, get prosecuted, and spend millions fighting in court. (For cannabis, the answer is yes, that is regulating commerce.) If you win, OK, no refunds on your legal fees. If you lose, you pay massive fines or go to prison. Also, courts have constantly changing judges and opinions. So what happens in practice is politicians write a vague law. Bureaucrats turn that law into very detailed (but often still vague) specific rules. Those rules might or might not be “legal”, but nobody want to risk fighting them in court. If the regulations are particularly ridiculous or likely to be overturned if challenged, prosecutors may quietly stop bringing cases. But the regulations still sit there on the books. And people still usually pay attention to them, because why risk it? OK! Where do IRB regulations come from? The Common Rule In 1932, the United States government and Tuskegee University began studying 600 poor black sharecroppers in Macon Country Alabama, ⅔ of whom had latent syphilis. The idea was to observe the progression of the disease if left untreated. By the late 1940s, syphilis was easily treatable with penicillin, but the men were never informed of their condition, never treated, and were even given fake treatments and diagnoses. In 1965, Peter Buxton joined the Public Health service and soon learned of the experiment. He was horrified and filed several official protests, but all were all rejected. Finally, in 1972, Buxton leaked the details to Jean Heller of the Associated Press which led to this front-page article on July 26, 1972. The led to widespread outrage and cancellation of the study. By this time, dozens of the men had died from syphilis and many of their wives and children had also become infected. It also led, in early 1973, a series of congressional hearings led by Ted Kennedy of Massachusetts. These hearings eventually produced the National Research Act. Several people voted for the first version of this bill and then against the second version, including one Joseph Biden of Delaware, though I can’t find any record of why. Anyway, it was signed into law by Richard Nixon on July 12, 1974. Formally speaking, the National Research Act did was to create a commission to develop guidelines for ethical research in human subjects, including how IRBs should work. But in reality, it also sent a message that federal agencies should use their existing authority to regulate research. Even before the Act was passed, the Department of Health and Human Services had started amending the federal code to regulate human research. In 1978, the commission issued a report on IRBs: The Commission believes that the rights of subjects should be protected by local review committees operating pursuant to federal regulations and located in institutions where research involving human subjects is conducted. […] The Commission further believes that institutions receiving federal support for the conduct of research involving human subjects should be governed by uniform federal regulations applicable to the review of all such research, whether it is supported by one federal department or another, or is not federally supported. Informed by this report, lots of different agencies issued different regulations. This continued until 1991, when 15 federal agencies harmonized on what is known as the Common Rule, because it is “common” to different federal agencies. This is now codified as Title 45 Part 46 of the federal code. Since 1991, this has been continuously updated and various other agencies have joined, notably the department of Labor. There are still a couple holdouts including the Nuclear Regulatory Commission, the National Endowment for the Humanities. The CIA is an odd case—they never issued any formal regulations, but supposedly apply the Common Rule because of a cryptic executive order from 1981. The FDA hasn’t joined because they have their own regulatory structure. So what does the Common Rule say? In principle, you can go here and read it. Good luck with that. It’s exactly as readable as “a series of regulations amended by many agencies over many decades” sounds like it would be. But basically (1) It requires informed consent. (2) It requires that risks are reasonable in relation to expected benefits, not including any long-range benefits from new knowledge. (3) It requires participants are selected fairly and equitably (e.g. not all poor black sharecroppers) Finally, (4) It requires that federally funded research with human subjects is reviewed by an IRB. Now, that might sound simple, but what exactly is “research”? This is defined by 45 CFR 46.102(l) to be a “systematic investigation designed to develop or contribute to generalizable knowledge.” The term “research” in this context, is defined by 45 CFR 46.102(l): It is a “systematic investigation designed to develop or contribute to generalizable knowledge.” It’s best to understand this with an example. Say you’re a teacher and you want to test if the cognitive bias of anchoring applies to your class. Is this research? If you’re doing it as part of normal “teaching stuff”, to demonstrate the effect to your students, then no. But if you’re doing it in hope of learning if the anchoring bias is real, then that is research. To some degree, the truth lies in your heart. (The definition of a “human subject” is also somewhat convoluted: What if you’re using data someone else gathered? What if you’re just using some human tissue samples or nonviable human embryos? But never mind.) Note that this applies to all research. The people who made these rules were almost all medical researchers thinking about serious medical risks. But they wrote the rules to apply to all research, including research where you might just ask people some questions. There are some narrow categories of research that are “exempt” from IRB review, e.g. research on standard educational methods or food taste evaluations, or certain benign behavioral interventions . Theoretically, an institution could allow investigators to decide for themselves if they are exempt, but this is not recommended, and in practice, essentially all institutions require you to apply for a determination of exemption. In reality, no research is really exempt, it’s just that certain categories of research have a somewhat more lightweight application process. To get a feel for how much friction this adds, here’s U. Michigan’s application for exemption. Up until the 1990s, IRBs at universities tended to be staffed with faculty “volunteers”. These were sympathetic to the needs of their colleagues, had little interest in reading (unreadable) federal regulations, and probably just wanted to get back to teaching and research. So IRBs were often pretty loose. But in 1996, Hoiyan Wan died after taking part of an experiment on the effects of smoking and air pollution the University of Rochester. And in 1999, Jesse Gelsinger died after taking an experimental gene therapy at the University Of Pennsylvania. Federal agencies cracked down and sent out hundreds of enforcement letters. Universities nervously responded with “hypercompliance”, leading to IRB process everyone loves today. Why is the Common Rule so common? The 1978 commission recommended that any institution accepting federal funding must apply IRB rules to all research, regardless of funding source. But the written regulations did not follow that recommendation. Theoretically, a university could accept federal funding for some research and still allow other research on human subjects without any IRB approval. But in practice, no university ever does that. Why? Well, before giving any federal funding to an institution, the government requires the institution to file a “Federalwise Assurance”. In this, the institution must promise to obey some “statement of ethical principles”. Supposedly this could be anything, but it appears that the only correct answer is “The Belmont Report”, an ethics statement issued by the same commission from 1978. Then there is a section where the institution may “optionally” promise to apply the common rule to all research, regardless of funding source. The government has a database of these “assurances”, but you can’t actually read them. Still, I found that a few prominent institutions that had published theirs. They all promised to abide by the Belmont report. Some (NYU, U. C. Irvine) promised to apply the Common Rule to all research, but most (Mayo Clinic, U. Florida, Emory, U. Michigan, MIT) do not. So if you’re a researcher at MIT, can you do non-federally funded research without an IRB? Nope. MIT rules still say you need an IRB for all research with human subjects. So do the Mayo Clinic’s. So do U. Florida’s rules. So do Emory’s rules. So do U. Michigan’s rules. What’s going on here? Throughout the 1970s, there was a debate about if IRB rules should apply to social science at all. The OHRP (which oversees the Common Rule) gradually asserted more and more broad IRB requirements. Ithiel de Sola Pool, a professor at MIT, protested this growing power. Apparently, a colleague had sought IRB approval to interview Boston anti-busing activists, and been rejected because this research could be used against them. Pool argued that much of social science should be exempt and got support from professional societies. But no matter. Here’s Charles McCarthy, director of the OHRP from 1977 to 1993, bragging about subverting the Carter and the Reagan administrations to add more regulations during the transition in 1980: But we were trying to decide when is the appropriate time to get the Secretary to sign off on the new regulations?  They had already been proposed, we had got comments from the public.  We’d incorporated the comments.  We wrote the preamble, it was ready to go.  And then when is the best time, ‘cause Harris could care less whether they every got published.  So we talked to the transition team, and they said Reagan will never allow new regulations to see the light of day, so if you’re going to get those out at all, you’d better get them out before he takes over. So we’re struggling with this problem, and the way we worked it out was that’s when we wrote, overnight one night–brilliant inspiration–we wrote–everybody stayed in the office, and we worked all night long, and we wrote the exemptions to the regulations, and we also wrote expedited review neither of which had ever been addressed by the Commission.  And that’s where they came from. Then we went to the transition team, and we said would the transition team endorse regulations that are less stringent than the previous regulations?  And, of course, they weren’t, but they looked like they were because we wrote some exceptions.  And so when we sent the package down to Harris, we said “Diminished Regulations for the Protection of Human Subjects.”  And that was the title.  And, of course, we knew nobody down there in the last weeks of the Harris administration getting ready to leave office would actually read it.  So they didn’t know what all that was about, but they could read the title. And so Secretary Harris signed that at her farewell party.  She put down her glass of champagne and signed the regulations on January 19, 1980, and went out of office on the 20th.  So we squeaked by.  Those were some of the adventures we had that were kind of harrowing for regulators, and that’s why some of the language in those exemptions is so convoluted because it was really a first draft.  But it survived and it’s still in there.  And it makes some sense, but if we could rewrite the regulations, we would run that through three or four more drafts and make them crystal clear.  They’re less than elegant writing, whereas if you read the rest of the regulation, I think you’ll find sub-part A is very well-written if you leave out the part about the exemptions. That’s a real quote that you can go read on the hhs.gov website today. The OHRP sent out “non-binding guidance” that suggested all research should be subject to IRB review. According to Hamburger (2007) in the 1990s, the Federalwise Assurance form implied that institutions had to apply the Common Rule to all research, despite the fact that no federal regulation to this effect existed. I can’t find the old form, but according to “persons who were in a position to know”, at one point all but about five institutions in the entire country had made this promise. By 2000-2005, the form made clear that promising to apply the Common Rule (and thus IRBs) to non federally-funded research was optional, and many institutions decline to do this. But they are only declining to make this assurance. Apparently, after years of applying the Common Rule to all research, it had risen to legally qualify as the “standard of care”, meaning that anyone that failed to apply it risked being legally “negligent “ under state law. So, having established the Common Rule as standard through questionably legal tactics, the government could now relax and rely on state tort laws. Now, say you’re affiliated with a university. Federal funded research without IRB approval is out. Any research at the university is out. But say you want to do research in your spare time, at home, without using any university resources. Is that OK? Policies here seem to vary. Some are clear that this is not OK. U.C. San Francisco says: UCSF faculty, staff, or students or researchers at UCSF-affiliated institutions conducting human subjects research require IRB approval before initiating the study. IRB approval is required regardless of the site of the study or the source of funding (if there is funding). Others are a bit ambiguous. MIT says: Any faculty member, employee or student at MIT who conducts human subjects research must apply [to the IRB] if the research involves any form of MIT involvement or support, including funding, personnel, facilities, academic credit or access to experimental subjects. But how could a MIT employee do research without involving MIT personnel, when they themselves are personnel? The closest I can find to anyone saying it’s OK to do independent research without an IRB is Columbia University which says an IRB is needed if: the research is conducted by or under the direction of any employee or agent (faculty/student/staff) of Columbia, in connection with his or her institutional responsibilities What exactly constitutes a “connection”? I guess you get to argue about the meaning of that term if/when you’re investigated for research misconduct. Journals OK! So if you’re doing any research at any federally funded institution, you need an IRB. If you work at such an institution, you could maybe do research at home, as long as it’s unrelated to your job and you’re willing to live dangerously. But suppose you’re not affiliated with any institution. Can you do research without an IRB? One additional barrier is that if you want to publish your results in a journal, almost all journals that publish human subjects research require IRB approval. For example, all Nature journals, Springer journals , and Taylor and Francis journals require you to identify the ethics committee and give a reference number. Science journals require IRB approval don’t seem to require details unless an editor requests them. (Those are publisher requirements. Specific journals can add additional restrictions.) Now, I’m not sure how carefully these journals actually check. My impression is that some do and some don’t. Maybe you could lie and get away with it, but papers do in fact retracted when a lack of IRB approval is revealed. State laws Another barrier is that some states have their own laws. In 1975, New York passed a law that required IRB approval for all human subjects research: Each person engaged in the conduct of human research or proposing to conduct human research shall affiliate himself with an institution or agency having a human research review committee, and such human research as he conducts or proposes to conduct shall be subject to review by such committee in the manner set forth in this section. Virginia law passed a very similar law in 1979. These go further than requiring IRB approval. They make independent human subjects research fully illegal! (Read that quote!) In most states, a random person could submit their research plan to an independent IRB. Not in New York or Virginia. In 2002, Maryland passed house bill 917. I find this text extremely confusing, but everyone seems to agree that that it means that all human subjects research requires IRB approval. I’ve seen many people claim that California also requires IRB approval for all human subjects research. While it’s true that California has some extra regulations, as far as I can tell none of these require IRB approval. FDA regulations Have you found all this to be too simple? Too straightforward and legible? The FDA is here to help. They have an independent set of IRB regulations. While the Common Rule IRB regulations are in Title 45 Part 46 (45 CFR 46), the FDA IRB regulations are in 21 CFR 56 and also 21 CFR Part 50. Whereas the Common Rule derives its constitutional authority from the idea that the government can decide how it wants to give out money, the FDA regulations seem derive their constitutional authority from the idea that it is regulating commerce. So FDA regulations apply regardless of where you work or how you’ve gotten funding. The FDA operates according to the 1938 Food, Drug, and Cosmetics Act and its many amendments, most notably the 1994 Dietary Supplement Health and Education Act. So what do the FDA regulations say? I found them to be much more painful to read than the (already extremely painful) Common Rule regulations. But after much thrashing and screaming, I think the core regulation is 21 CFR 56.103(a): […] any clinical investigation which must meet the requirements for prior submission (as required in parts 312, 812, and 813) […] shall not be initiated unless that investigation has been reviewed […] by an IRB meeting the requirements of this part. 21 CFR § 56.103 gives a two cases where you’re exempt from the above rule, but these aren’t much help. The first (21 CFR § 56.104) is that you’re exempt if you meet the conditions of which basically are that (1) you started your investigation in 1981 or earlier (2) it’s an emergency and you get IRB approval within 5 days, or (3) “taste and food quality evaluations and consumer acceptance studies if wholesome foods without additives are consumed or if a food is consumed that contains a food ingredient at or below the level and for a use found to be safe”. The second (21 CFR § 56.105) is that you’re exempt if you apply to the FDA to be exempt and they waive the requirements. Good luck with that. 21 CFR § 56.103 (b) says that if you apply to the FDA to get a drug approved, they may ignore your data if you should have gotten an IRB but you didn’t. But this an additional thing to worry about, not a waiver to do whatever you want if you don’t care about FDA approval. So what does this mean? Well, a “clinical investigation” is defined by 21 CFR 56.102(c) to be anything involving human subjects and a “test article”. And a “test article” which is defined in turn by 21 CFR 56.102(l) to be any “drug” for human use. So if you’re doing anything with a “drug” that requires “prior submission”, then you need an IRB. What do those terms mean? 21 CFR Part 312 makes clear that “prior submission” is referring to an investigational new drug (IND) application to the FDA. (Caution: An “IRB” and an “IND” are totally different things. An IRB is what this article is about. An IND is the kind of application that pharmaceutical companies make to the FDA before they can start clinical trials.) So, at first, this seems like good news: As long as you don’t need to do a (gigantic, extremely expensive) IND application to the FDA before starting your research, then FDA regulations don’t force you to use an IRB. Whew! Except… When are you supposed to submit an IND? Surely that’s not some incredibly broad set of circumstances that could apply to almost anyone doing almost anything, is it? Behold: This is a supposedly “nonbinding” document that that supposedly just explains old rules. Supposedly the FDA is just publishing this as a helpful service to all the mouth-breathers who can’t understand the federal regulations. But everyone in industry behaves as if it’s a new law. Which is understandable, since the written regulations are so confusing and ambiguous. If you read this guidance, you will discover that an IND is needed if: The research involves a “drug” The research is a “clinical investigation” The research is not exempt. If you want to avoid an IND, then 2. and 3. are little help. The term “clinical investigation” in is defined as anything where a “drug” is given to human subjects. And the exemptions are extremely narrow: They are only for marketed drugs where the drug is being used for the labeled purpose as directed. Testing something already approved for a disease? Not exempt. Testing if normal food helps with some new disease? Not exempt. You’re only exempt if you’re doing research on something the FDA has already accepted to be true. “Do I need an IND if I use a lawfully marketed drug for an unlabeled indication? If you are a health care provider and you prescribe a marketed drug to treat a patient for an unlabeled indication (also referred to as off-label use), an IND is not required because this use is considered to be within the scope of medical practice and not a clinical investigation. However, if you use the marketed drug for the same purpose in a clinical investigation intended to evaluate the drug’s ability to treat a disease or condition, an IND is required under part 312 unless the clinical investigation meets the criteria for an exemption for studies of lawfully marketed drugs (see 21 CFR 312.2(b) and section IV.A of this guidance.” “If a product containing substances generally recognized as safe (GRAS) for use in food is administered to subjects in a study intended to evaluate the effect of the substance on the pathogenesis of a human disease, is an IND required? Substances designated as GRAS for use in food are generally not approved as drug products. A clinical investigation of a GRAS substance that is intended to evaluate the product’s ability to diagnose, cure, mitigate, treat, or prevent disease requires an IND under part 312, unless the substance to be studied is also a lawfully marketed drug and the clinical investigation meets the criteria for exemption under 21 CFR 312.2(b).” If you want to avoid an IND, your best hope is that your research doesn’t involve a “drug”. Unfortunately, the term “drug” is defined extremely broadly, way beyond the colloquial meaning of “drug” in English. A drug is defined to include: articles intended for use in the diagnosis, cure, mitigation, treatment, or prevention of disease It also includes: articles (other than food) intended to affect the structure or any function of the body Get that? Here’s a little picture: Now, I know what you’re thinking. “That can’t be right! That would mean that if I wanted to study if some normal food reduced the odds of, say, diabetes, then I wouldn’t just need IRB approval, I would also need to submit a freaking IND! That would be stupid!” But the FDA is quite clear that this is right: As is the case for a dietary supplement, a food is considered to be a drug if it is “intended for use in the diagnosis, cure, mitigation, treatment, or prevention of disease,” except that a food may bear an authorized health claim about reducing the risk of a disease without becoming a drug (see section VI.D.3). Therefore, a clinical investigation intended to evaluate the effect of a food on a disease would require an IND under part 312. For example, a clinical investigation intended to evaluate the effect of a food on the signs and symptoms of Crohn’s disease would require an IND. FDA also says: Does a physician, in private practice, conducting research with an FDA regulated product, need to obtain IRB approval? Yes. The FDA regulations require IRB review and approval of regulated clinical investigations, whether or not the study involves institutionalized subjects. FDA has included non-institutionalized subjects because it is inappropriate to apply a double standard for the protection of research subjects based on whether or not they are institutionalized. An investigator should be able to obtain IRB review by submitting the research proposal to a community hospital, a university/medical school, an independent IRB, a local or state government health agency or other organizations. At first, I thought this couldn’t possibly be true. In 2017, it apparently cost $619,200 to submit an IND application to the FDA. However, in practice it appears that there are many exceptions and that there are two categories of INDs: Commercial and research. I assume the FDA doesn’t actually try to charge researchers $619,200 to review their research projects, but I don’t really know. The FDA’s IND application page is here. Have fun. I’m not sure how seriously people take these FDA regulations in practice. People do seem to get IND approval for research on actual (chemical) drugs. But does anyone actually seek IND approval if they want to study if diet influences obesity? Do they carefully avoid talking about disease, so they can fall under the looser “structure and function of the body”? And is private research on normal food really a form of inter-state commerce? I’m just not sure. What I’ve learned Mostly I’ve learned that the set of abstractions I was using to think about the world were wrong. I assumed that there were “laws” that someone had written, and that someone could read to understand what is legal and what isn’t. But in reality, politicians write vague laws, and then agencies write vague sprawling regulations, and then those agencies issue vague “clarifications”. Sometimes these might be unconstitutional or illegal, but no one knows unless they want to fight things in court. But courts are unpredictable, and even going to court is ruinously expensive. Trying to comply with the the written law is like trying to comply with a giant mound of potatoes. So people basically operate as if the law doesn’t exist. The “law” is what the people who enforce the law choose to enforce. If you need to live in this world, that’s what you pay attention to. The law is what you can get away with. What I would change All the IRB rules were designed by medical doctors to regulate the research of other medical doctors, thinking about incidents like the Tuskegee syphilis study. But here’s an opinion: Needing to submit an application before you can ask people to fill out some simple online survey is stupid. Needing to submit an application before you can interview a bunch of people is stupid. I like incremental change, so why don’t we start by making “exempt” research actually exempt? For example, 45 CFR 46.104(d)(3)(ii) exempts “benign behavioral interventions” from IRB review. But you still need to apply to prove you’re exempt. Why? As an analogy, driving a car is dangerous. Whenever I drive, I could easily kill someone. But the government doesn’t force me to submit a driving plan any time I want to go somewhere. Instead, if I misbehave, I am punished in retrospect. Why don’t we apply the same policy to research? Notes I found AI to be completely useless at parsing all these federal regulations. Maybe you need AI that’s specifically trained to be good at parsing legal text? Say I want to feed potatoes to my friends and see if that reduces diabetes. Maybe that’s dumb. Fine. But the FDA seems to be claiming that it is constitutionally permitted to regulate this because by doing this I would be engaging in “interstate commerce”. Really? Maybe the Chevron decision might have some implications for all this? As far as I can tell, no one has ever been prosecuted anyone for failure to comply with IRB requirements. It’s not even clear what the penalty would be. In practice, rules are enforced on institutions—if one failed to comply, then their “assurance” might be cancelled, meaning they couldn’t take any federal funding for research. Or, if they wanted to sell something, the FDA might refuse to approve it for sale. Are you chaotic good? Then you may find it interesting that it seems to be relatively easy to form a new IRB. (HHS rules / FDA rules). And that while all federally funded institutions require IRB approval for all research, only some require approval from their IRB. You can probably see where this is going. Further reading Philip Hamburger: Getting Permission (2007) IRB Licensing (2015) My Bookshelf Runneth Over: A history of US IRBs Ethical Imperialism: IRBs and the Social Sciences, 1965–2009 Many thanks to Willy Chertman.

What does “algorithmic ranking” bring to mind for you? Personally, I get visions of political ragebait and supplement hucksters and unnecessary cleavage. I see cratering attention spans and groups of friends on the subway all blankly swiping at glowing rectangles. I see overconfident charlatans and the hollow eyes eyes of someone reviewing 83 photo she just made her boyfriends take of her in front of a sunset. Most of all, I see dreams of creative expression perverted into a desperate scramble to do whatever it takes to please the Algorithm. Of course, lots of people like algorithmic ranking, too. I theorize that the skeptics are right and algorithmic ranking is in fact bad. But it’s not algorithmic ranking per se that’s bad—it’s just that the algorithms you’re used to don’t care about your goals. That might be an inevitable consequence of “enshittification”, but the solution isn’t to avoid all algorithms, but just to avoid algorithms you can’t control. This will become increasingly important in the future as algorithmic ranking becomes algorithmic everything. Why algorithmic ranking is bad for some people sometimes You’ve heard it all before. I think algorithmic ranking leads many people to spend time and emotional energy on things they’d rather not spend them on. I also think it leads lots of people to believe preposterous bullshit promoted by charismatic charlatans. I’m disturbed when I see how kids interact with addictive algorithms, but then I notice that adults are much the same. You know the story. A common defense of algorithmic ranking is that “your feed is your problem”. If you’re getting political ragebait and unnecessary cleavage, then that’s on you for engaging with that stuff. If you actually cared about the things you pretend you care about, everything would be fine. The problem is looking you in the mirror. I find this defense bewildering and almost hostile. I mean, where is it spelled out how you’re supposed to behave to get the content you want? Maybe there are little like buttons, but what do they do? How do they interact with all the other signals, like what you watch? It’s unclear. Now, I’m sure that many of the people who complain about ragebait and cleavage are in fact drawn to them in some way. Maybe they don’t swipe away fast enough when that stuff is inserted into their endless content trough. If they only had eyes for philosophy lectures and meditation videos, maybe that’s all they’d see. OK, but so what? Where’s the empathy? Everyone has some divergence between the urges they feel and the urges they wish they felt. We don’t judge alcoholics who throw away their booze. We don’t make fun of gambling addicts who avoid travel to Vegas or Monaco. So why is it wrong to not want lurid but unhealthy content dangled in front of you? One of the fundamental arts of being a human is using your “better self” to try to control your “lesser self”. You can put a giant calendar on your wall to track when you exercise. You can keep junk food out of your home. You can write your algorithmic ranking manifestos using an app that permanently deletes everything if you stop typing for 5 seconds. These tricks are good. We need more of them! Algorithmic ranking—as we know it—is the opposite. So what would I propose instead? How about… sliders? Why can’t I click a slider to say “more educational content” or “less political rage” or “no David Sinclair-esque supplement huckster bullshit”? Or how about an algorithm that just does what it says on the tin? Remember, YouTicBooX might let you “like” stuff, but the algorithm’s goal isn’t to show you stuff you’d like. The goal is to make money. Your likes are just another feature to be integrated with the rest of your behavioral profile for increasing engagement and targeting ads, thank you very much. The algorithm is running on you, not for you. But what about the market? Another common defense of current algorithms goes like this: If they’re so bad, then why did they win? If people wanted sliders, they would have picked services with sliders. But they didn’t. Go make SliderTube if you want. No one cares. There isn’t some pent-up demand for algorithms that give more control to our better selves. Sometimes people also gesture at the tyranny of the marginal user. The idea is that, sure, power users like control. But where companies really compete is in the fight for the “marginal user”, the person who is almost ready to quit. This person only vaguely understands what a phone is, has never heard of “algorithms”, likes flashing lights, and has the attention span of a pissed-off chimpanzee. They will not tolerate any complexity, so all sliders must go. Sorry, power users. There’s clearly something to these arguments. But still: If your portal to the world is designed for the marginal user, surely you’ve gone wrong somewhere, no? What happened to Netflix? In the long-long ago, Netflix had star ratings. You’d rate stuff between 1 and 5 stars and get a sorted list of predictions: Chunking Express 4.49 Days of Being Wild 4.51 Solaris 4.57 In the Mood for Love 4.62 Master and Commander: The Far Side of the World 4.98 Nowadays, you get a disorienting set of categories like DARK COMEDIES ABOUT ITALIAN FEUDALISM and LIFE IS SHORT—WATCH IT AGAIN and THINGS YOU’RE IN THE MIDDLE OF, HELPFULLY PLACED IN A INCONSISTENT LOCATION. Instead of star ratings, there are “match percentages”, but you have to interact to see them and they always seem to be 98%. What happened? Well, the story is in the public record. (See, for example this post and this post by Gibson Biddle.) In short, Netflix realized a bunch of things: That they needed to concentrate everything on increasing subscriber revenue. And that the main goal of recommendations should be subscriber retention, or making sure people don’t cancel. That the things people rate highly aren’t always the same as what they actually watch. It’s cool that you gave The Seventh Seal five stars. But after a long day at work and finally getting the kids to bed, are you really going to choose Andrei Rublev over The Great British Bachelorette and the Furious 7? That to retain people, you need to get them started watching new stuff. Lots of people want to watch Friends, so Netflix will pay $100 million/year for Friends. But if you just join, binge every episode of Friends, and then cancel, that’s bad. However, if the Friends button were to—say—randomly shift around in the interface, maybe while hunting for it you’ll get hooked on some other (hopefully cheaper) shows and stick around longer. That beyond your explicit ratings, there are lots of implicit signals like what watch, what you click on, what devices you use, and how long you stop scrolling when shown different kinds of thumbnails. These implicit signals are more useful than explicit rankings when predicting what to show you to keep you subscribed. That many people don’t want to rate stuff. And (I speculate) that this provides a convenient excuse to drop the whole star rating system and replace it with the “whatever the hell order we want” system that prevails today, where the match % means nothing and promises nothing. Netflix could have kept the star ratings as some kind of optional feature for the nerds, hidden away in some dusty submenu. But they didn’t. They 100% killed the star ratings for everyone. Why? I guess maintaining features takes work. But mostly I think they figured that most star rating diehard wouldn’t actually quit the platform. They’d grumble but accept the new system and thereby be Retained. And probably they were right. Now, I don’t mean to vilify Netflix. I mean, sure, it’s an amoral automaton doing whatever maximizes profits. But this is hardly the worst example of algorithmic ranking, and hey, this is capitalism! If Netflix tried to be “principled” and stuck with star ratings, maybe some other company would have displaced them? Don’t hate the player, hate the game? (Maybe don’t hate the game either?) What algorithmic ranking should be Let me summarize my argument so far: Algorithmic ranking as we know it is designed to maximize money for the companies doing the ranking, duh. That’s great if what you want is to be addicted or—more precisely—if your behavior happens to create incentives for rankings that are well-aligned with your goals in life. Otherwise, it’s not. Companies settled on these algorithms for a reason. It’s not because they’re evil, it’s because this is what’s profitable. If you accept all that, then what follows? One view is that this is further proof we must smash capitalism and end the malign power of the invisible hand. That’s intellectually coherent, but not my style. Another view is that, well, this is the outcome of the market. It’s pointless to fight the equilibrium, so we should just live with the algorithms. This is also not my style. A third view is that we must reject algorithmic ranking. Refuse to use any social media with algorithmic ranking. Subscribe to blogs using RSS. Install browser extensions that block YouTube’s recommendations. Chronological timelines only. Human curation only, forever. This third view is very much my style. One of the reasons I love blogging is that I can reach people without worrying about the damn algorithms. (Hi.) But I’ve come to believe it’s a dead end. After all, people like algorithmic ranking. Maybe with better interfaces or a bigger social movement, more people would shift towards human curation. But I suspect not that many, and the arc of history bends towards algorithms. And good algorithmic ranking—which you control—would be awesome. I mean, I appreciate that people subscribe to this blog. But I find it a little disturbing that if someone less well-known than me wrote the same thing, then many fewer people would read it. (I find it extremely disturbing that if someone more famous wrote it, then many more people would read it.) Yet that’s an inevitable consequence of relying on subscriptions instead of algorithms. This isn’t (only) an issue of vanity. I think it leads to an “invisible graveyard” of contributions that never happen. Say you’re a sane person who doesn’t want to spend hours every week writing for strangers on the internet. But maybe you’re—I don’t know—maybe you’re on the board of your local volunteer fire department. And maybe you have one sizzling banger to write about how local volunteer fire department boards should be organized. That information is important! Reality is fractally complex! But probably you won’t write it, because almost none of the people who’d benefit from it will ever see it. I think the solution is to embrace algorithmic ranking, but insist on “control”—to insist that the algorithm serves your goals and not someone else’s. How could that happen? In principle, we could all just refuse to use services without control. But I’m skeptical this would work, because of rug-pulls. The same forces that made TikTok into TikTok will still exist and history is filled with companies providing control early on, getting a dominant position, and then taking the control away. Theoretically everyone could leave at that point, but that rarely seems to happen in practice. Instead, I think the control needs to be somehow “baked in” from the beginning. There needs to be some kind of technological/legal/social structures in place that makes rug pulls impossible. What exactly should those structures be? And what exactly is “control”, after all? I don’t know! Those seem like difficult technical problems. But they don’t seem that hard, do they? I suspect the main reason they haven’t been solved is that we haven’t tried very hard. We should do that. And—I dare say—we should do it quickly. My guess is that algorithmic ranking will soon become a sort of all-encompassing algorithmic interface. More about that soon, but if all the information that enters your brain is being filtered by an algorithm, it seems important that you know the algorithm is on your side. Thanks: Steve Newman, Séb Krier

What is this? I am offering to act as a “mentor”, to you, in case that seems like something you’d find useful. How will it work? We will meet three times for 30 minutes. During those sessions, I’ll try to help you do whatever it is you’re trying to do. Then I’ll sit back and congratulate myself on everything you do for the rest of your life. Why are you doing this? It’s an experiment. I have a theory that there isn’t enough mentoring in the world, and that creating more mentoring might be an efficient step towards filling the universe with bliss-maximizing Dyson spheres. So I’ve decided to try this and see what happens. What topics are allowed? In principle, any topic. But I’ll probably be able to help you more if I have some kind of expertise or interest in whatever you’re trying to do. It could be anything related to statistics, science, AI, self-improvement, blogging, academia, writing, rationalism, (effective) altruism, air quality, the welfare of animals, or anything I’ve ever written about. Or it could be something else entirely. To encourage you to think broadly, I will try to pick at least one person with a topic not on the previous list. Who is eligible? Anyone. In particular, anyone at any age or career stage. But also the other kinds of anyone. What will it cost? Nothing. Is there an application? I have no idea how many people will be interested, but when scarce resources are given away, demand often exceeds supply. So there is an application, which is here: I basically just ask (1) what you want to do, and (2) how you hope I can help you. How many people will you pick? Three. If lots of people apply, I might try to recruit some other mentors. Or I might be lazy and disorganized and not do that. How will you pick? I will pick primarily based on how much I think I can help you and secondarily based on how much what you’re trying to do will advance the general welfare of the universe. I stress that the primary axis is primary! If you make a strong argument that I can help you become very rich or succeed in dating (unlikely) then I’ll pick you over some do-gooder that I can help slightly less. What makes you great that you can mentor other people? Nothing. Honestly, I could give you several reasons that I’m less than ideal as a mentor! But I suspect that mentoring is valuable enough (and undersupplied enough) that even less-than-ideal people like me can still be helpful. How will we meet? We will meet by video call using Signal. Will you share any details about me without my permission? I might write about the results of this experiment in general, reflecting on how it went, giving tips for other people who might try something similar, etc. The whole point of doing this is to experiment and understand if it works, after all. If you want me to share some details about you, I might. But I am very sensitive to privacy and I will not share anything (even broad details) without running it past you first and I’m extremely likely to agree to remove any information you ask me to remove, even if you’re being ridiculous. Is this a good opportunity to interview you or try to get you to go on a podcast, etc.? No. I will attack you.