This question will not go away. I am getting asked this questions at conferences now. The media also seems to think this is a compelling question, and they are fueled in part by the tech industry itself, some of whom indulge this question. I will admit the answer has become more complex and nuanced than a simple “no”, but it is still mostly no. Certainly the rise of neural networks, deep learning, transformer technology, and specifically large language models have ushered in the era of  – are we really sure current AI is not conscious? Our intuition tells us that they may be – LLMs are getting scary in how sophisticated they are and in their ability to simulate complex human conversations. Five years ago I was with the crowd who argued that LLMs are basically really good auto-complete apps, but this pithy sentiment no longer covers it. Even if they do not have a whisper of conscious awareness, they employ context-dependent logic. They are “intelligent” in their own way. And they are rapidly getting more impressive. So I get why people would be asking the question. Everyone I know who works with AI apps has their example of when they were blown away by an interaction with an LLM. For me, I was trouble-shooting a web-based app I was using, and ChatGPT was very effective at helping with the trouble-shooting and producing a differential problem list. At one point it asked for a screen capture so it could have more information, and it gave me a cogent analysis. But here’s the part I found surprising – it “noticed” another window open in the background and inferred from this that I was also using this other app in the process, and used this to generate further potential sources of the problem. As impressive as this is, I still think they are not conscious. I think the illusion of consciousness is generated by the fact that LLMs are simulating language, and we associate language with consciousness. No one thinks Midjourney, an art making program, is conscious, even though this is based on the same fundamental technology. Further, LLMs are still brittle in their own way, although the brittleness is being rapidly programmed out of them. What this means is that you can break them, or easily confuse them, in order to reveal that they are simulating language rather than truly thinking. This brittleness is revealed in several ways – sensitivity to exact prompt wording, accepting false premises in the prompt, and failure to recognize constraints. If changing the language but not the real meaning of a prompt gives different answers, then the LLM is responding statistically to the language, not genuinely reasoning about the information. Each iteration of LLMs is getting better at plugging these brittleness holes, but they are still there. Still, this doesn’t mean that there is no contextual reasoning happening. I think the best current summary is that LLMs are a combination of statistical language prediction (the auto-complete thing) and contextual reasoning. This doesn’t mean that they are conscious, however. What I believe it means is that humans find it surprising how much reasoning a machine can do without consciousness. Perhaps our own reasoning is more procedural than true deep thinking than we perceive or feel comfortable truly grappling with. We are also algorithmic machines, with different strengths and weaknesses than LLMs. But we are also conscious. So what is the difference? The difficulty in answering this question is that neuroscientists are still not certain what causes consciousness in the first place. This does not mean it’s a complete mystery or we know nothing – it means it is a complex multifactorial, and perhaps emergent, process that is not easily understood or explained. But we do have a lot of knowledge about it. There is no reason, in my opinion, to think that the different is biology vs silicon. I do believe that silicon can become conscious. Rather, it’s the process that matters, and we know some things about the process. Human consciousness requires that the brain be constantly self-activated. Turn off this activation (the brainstem activating system) and you will be in a coma – not conscious. We also know that the brain is constantly active and communicating with itself, in an endless loop of one network in the brain talking to other networks which talk to other networks and back to the original network, in a constant stream. It is this constant stream that is our consciousness. We also know there is a threshold, at least for wakeful consciousness. You need at least 45% of your cortex to be functional in order to generate wakeful consciousness. Are people who are not wakeful still conscious to some degree? Probably, but it’s difficult to understand what this means. If you are not making memories, for example, then we cannot interrogate what that consciousness was like. If people can still respond to their environment, does that imply consciousness, or could that simply be an unconscious reaction? For straightforward LLMs – prompt-response, I do not think they can be conscious, at least not human-level conscious. They are doing none of the things that are necessary for vertebrate consciousness. Things get a bit trickier for agentic AIs, that can iterate their own behavior, but still this is more like a self-iterative prompt than a true stream of consciousness. But admittedly, frontier agentic AI is starting to blur the lines a bit. But here’s the thing – even if I cannot say with absolute certainty that some of these frontier models have still not crossed over some fuzzy line into a type of consciousness, I can say that this consciousness must be fairly minimal compared to human-level consciousness. I can also say that such minimal consciousness is not what is speaking to you. It is not reasoning and generating the output of LLMs. We know what’s doing that, and it does not require consciousness or result from any process that can be reasonably inferred to be conscious (again, it may just be a prompt-response). Therefore, arguing that we cannot rule out some consciousness emerging from the processing of massive data centers does not mean you are speaking to that consciousness through your LLM. I may turn out to be wrong, but I don’t think we will back into AI consciousness through some unanticipated emergent property. I think we will have to design systems to do the kinds of things necessary for consciousness. It may be a deliberate attempt to make a conscious AI, or many of those elements may have some separate utility. Either way we will be able to look at the system and see that it is functioning in a way that can plausibly result in consciousness, and that consciousness will be proportional to the power and functionality of the system. LLMs are not it. They are not even on a path to become conscious. They are doing contextual reasoning, and it is getting steadily more impressive, but they are doing it without consciousness. The post Could AI Be Conscious? first appeared on NeuroLogica Blog.

Researchers have found that snowstorms are delivering vast numbers of tiny plastic particles to remote reaches of California's Sierra Nevada mountains. Snowmelt is a major source of fresh water for the state. Read more on E360 →

Economics often starts with an assumption that people are rational maximizers of their own self-interest; but it doesn't stop there. […]

Yesterday

A recent scroll through social media got me thinking about trust. About half my feed was filled with AI slop that astute users would call out as being machine-generated, often to the embarrassment of the rube who unknowingly spread it and sometimes to the delight of the poster who shared it deliberately because they found it amusing or useful for sowing confusion. Remember the good old days of GPT-3 when we could ask the chatbot anything and it would almost instantly and very confidently spit back a plausible but false answer? That’s when the word “hallucination” as applied to AI answers entered the lexicon. Six years and several versions of a few different chatbots later, I still can’t trust the answers I get, even though Claude or Gemini often do useful work for me. That’s the thing about current AI: Trust but verify. And verify you must. Apply that maxim at scale to maintain the integrity of the scientific record and you get an idea of the challenge facing publishers of peer-reviewed scholarship. In “I Used AI to Fake a Scientific Paper,” freelance writer and doctor of applied mathematics Benjamin Skuse tells the story of the human gatekeepers trying to beat back the rising tide of AI slop by attempting to get his own AI-generated papers past IEEE’s defenses. “He basically hit ‘run experiment’ and ‘write paper.’ And then he didn’t do a damn thing to it.” —Eliza Strickland As features and AI beat editor Eliza Strickland told me, Skuse was surprised at just how effortless it was to generate a paper that was more or less plausible and more or less science. For one of his experiments, “he basically hit ‘run experiment’ and ‘write paper,’” she said. “And then he didn’t do a damn thing to it. He just launched it into the maw of the publishing engine.” Skuse discovered that the combination of IEEE’s human-review processes and research-integrity checker programs are currently keeping pace in what amounts to a cat-and-mouse game against people using free tools. But the more advanced LLMs are no longer making the same dumb mistakes the free versions do. That leaves publishers locked in an arms race between increasingly plausible slop and the systems aimed at detecting it. Scholarly publishers aren’t the only ones trying to hold the line against slop. So is that global depository of human-generated knowledge, Wikipedia. For this month’s Five Questions, Strickland interviewed Ellia Lebleu, a founding member of WikiProject AI Cleanup, a volunteer effort to detect and delete AI-generated material from the platform. Lebleu’s team is confronting the same problem as commercial publishers, but with fewer people and fewer tools. “Wikipedia is known as a repository of human-generated information, and they want to hold on to that,” says Strickland. “Who knows how long they’ll be able to do that.” If we can’t collectively assume that the scientific record is trustworthy or that basic facts about the world are accurate, then our understanding of what is real unspools into uncertainty. Perhaps that delights quantum scientists and the grifters who thrive in an information environment where there is no shared reality. But that uncertainty ultimately undermines trust in the institutions that exist to curate the corpus of human knowledge and to promote the advancement of science and technology for the benefit of humanity. Or maybe in a world where AI is doing the science and writing the papers, it won’t matter whether or not humans trust the outputs. As Strickland put it, “Doesn’t it just feel crazy that we’ll have AI writing for other AIs at some point? We’ll just get left out of the loop.”