Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from matt.sh
agent-shell: stop spilling your outputs since the beginning of time, ai agents have wanted one thing at one thing only: redirecting stderr to stdout then piping the output to head or tail or grep or sed or awk. That’s all they really want. Except, they do it wrong the majority of the time (because “ai labs” are fundamentally incompetent at software engineering, as we’ve previously discussed before). Common ones I see routinely which are painful and require manual “reminders” into the agent to stop: uv run my-big-honking-test 2>&1 | grep -E "failed:|error:" | head -5 but then the test fails with a non-zero error code, the grep reports an error count, so then the agent RE-RUNS THE ENTIRE TEST with a different grep failure to “find the failing test case” except, the failing test case grep requires “guessing the future” regex, but the first attempt at “guessing the error regex” fails, then code agent continues re-running a 20 minute test suite with different grep filters until you remind it: “FILES EXIST! WE HAVE A FILE SYSTEM! WRITE OUTPUT TO FILES THEN READ THE FILES TO SEE THE ERRORS!” also, since current ai code agents have no feeling for “time” or “effort” they don’t care about running a 30 minute program-stderr-to-stdout-to-pipe-grep 10 times in a row even if each attempt takes 30 minutes. and a more toxic approach is where models decide to use live public URLs for reference material then curl | grep except, again, models try to “guess grep filters”, fail, then re-run the curl a dozen times with different “guess the regex” filters instead of saving the content locally and then searching through it (I often see claude get rate limited or blocked by upstream services because it “curls and greps” the same URLs in a loop, and you have to remind it every time “hey, files exist? you know? use files?”) I’ve run some bulk analysis about how often models “do the wrong thing” or “waste time guessing” (instead of following more proof-based data practices) over my claude session history (you do know all your claude session and all commands and tool calls are saved at ~/.claude right?) and here’s results: ways models try to read things Pattern Count Share of Bash commands Piped into a filter (grep/head/tail/awk/sed/etc.) 123,637 62.8% Bare narrowing command, no pipe 22,963 11.7% – Any pipe at all 135,413 68.8% Bare full-file read (cat) 1,573 0.8% Redirect-to-file, later read in full (“save then read”) 4,301 — common filter patterns guesses: Shape Count Likely context sed (bare, e.g. sed -n 'N,Mp' file) 15,123 line-range paging grep | head 9,497 search, then cap the results cd | head 5,998 cd dir && cmd \| head grep (bare) 4,523 direct search on a file uv | tail 3,864 tailing output of a uv run/uv pip command cd | tail 3,253 cd dir && cmd \| tail tail (bare) 2,764 log/output tailing ls | head 1,699 capped directory listing find | head 1,365 capped file search common slop guess counts: Metric Count Guess-then-give-up-and-read episodes (≥2 narrow attempts, then a full read) 541 Unresolved thrash episodes (≥3 narrow attempts, never followed by a full read) 2,701 Total narrow commands spent inside these episodes 5,006 Resolved episodes closed by the Read tool 447 (83%) Resolved episodes closed by a bare cat 94 (17%) multiple different cmd | filter guesses in a row here are full attempt-count distribution for episodes eventually resolving into a full read (example: running a command with a failed filter, then continuing to run cmd | filterA -> cmd | filterB over and over again until a “filter” matches instead of just saving output to a file and reading directly):
img,video{width:100%} claude-hooker The mystery of claude code over the past year and a half has been how does a world leading ‘AI productization lab’ continually generate such shitty amatuer globally distributed software and services over and over again? It’s almost like staffing a company entirely by 24 year olds seeing their net worth increase by $2 million per month for two years is a bad idea? Even the tech brain infosphere has noticed how claude code’s architecture is absolutely mornoic and amatuer and anthropic just continually ego-sociopath lies and nobody can stop them because hey, when you spend over $2 billion a month renting GPUs run by globally polluting gas turbines, they let you do it. don’t worry, bro, “it’s a javascript game engine in your console, ur not smaht enouf to understand our god-ai logic here!!!” as you may be able to tell, i’m deploying this post in annoyance/anger mode because I don’t understand why me, somebody who can’t afford a place to live and has 20 years of detailed technical experience, is giving free work to trillion dollar “ai” “startups” with good models but apparently abysmally amatuer software engineering practices. i guess it’s part of industry bubble syndrome. It’s not really worth thinking about what drives weird un-self-aware ego-sociopathic-autistic billionaires if you aren’t close enough to change anything in their minds though. The Bad i guess back to the point: given they have made claude so same-system dangerous with its cavalier attitude towards running destructive or system-resource-consuming actions you must install your own safeguards using claude code event hooks.
More in AI
Modern engineering projects increasingly combine human judgment with machine capability. An engineer can define a problem, then ask an artificial intelligence tool to search relevant research and technical standards, generate design options, test them in simulation, and compare the results. The engineer can then bring the results to specialists who understand the parts AI cannot judge on its own, such as whether a design can be built safely, whether it makes sense in a particular setting, or how it might affect its users. The final decision about whether and how to move forward remains a human responsibility. AI can help engineers do early project steps that once required several people with different technical skills. The engineer’s role is therefore expanding from performing each technical step to deciding how people, machines, and specialists’ knowledge should work together on the project. The problem, not the title Engineering has always been divided into disciplines, and for good reasons. Electrical, mechanical, civil, software, and other engineering fields require deep knowledge that takes years to develop. AI does not replace that need. It simply changes how easily engineers can explore the parts of a problem that sit outside their own specialty. Consider a software engineer working on a medical device. AI can help the engineer review relevant biomedical research, understand unfamiliar terms, analyze data, and build an early model. That does not make the engineer a physician or biomedical expert. It allows the engineer to ask better questions before working with the experts. The same is true in energy, transportation, manufacturing, and many other fields. A problem might begin as a software issue and then quickly involve hardware, safety measures, economics, human behavior, and regulations. Engineers do not need to master every field; they need to recognize which expertise the problem requires and bring the right people and tools together. The importance of systems thinking Systems thinking thus becomes especially important. It requires looking at the whole problem: understanding how its parts affect one another, identifying constraints and failure points, and making trade-offs. AI adds a new element to the process. Engineers must decide not only how the technical parts fit together but also what work to give to AI, what a person must check, when a specialist should make a decision, and when the problem needs to be redefined. An engineer’s value is not in knowing everything. It is in knowing how to frame the problem, how to ask the right questions, how to test the answers, and when another expert must lead the project. This aspect is crucial because technical challenges in engineering are seldom static. As new insights emerge, objectives might shift, unseen limitations can surface, and the initial premise might prove flawed. Rather than rigidly following the starting specifications, an effective engineer adapts and refines the scope as a deeper understanding of the system develops. Thus, engineering extends beyond mastering technical concepts; it is an adaptable, structured approach to navigating evolving challenges. New evidence can change the goal, reveal a hidden constraint, or show that the original hypothesis was wrong. Effective engineers go beyond merely acting on the initial problem statement. They update it as the system becomes better understood. In that sense, engineering is not only a body of technical knowledge, it is also a disciplined way of thinking about a project. Human judgment is important AI can produce convincing answers even when its assumptions are weak. It can generate code that works in one test but fails in another. It can summarize research without understanding which evidence is strongest, or optimize a design for the wrong goal. Such limits make human expertise more important, not less. Machines are useful where speed, scale, search, pattern recognition, and data generation matter. People are essential where context, values, responsibility, and judgment matter. The challenge is to decide where each belongs. An AI system might generate several designs for a medical sensor, for example, and predict how each will perform. An engineer can test the predictions and reject designs that violate technical requirements. A clinician can determine whether the sensor can be realistically used. Patients and practitioners might reveal problems that neither the engineer nor the developer considered. The final solution emerges from combining the perspectives. The engineer is still a builder and specialist but increasingly also a coordinator of knowledge. An engineer’s value is not in knowing everything. It is in knowing how to frame the problem, how to ask the right questions, how to test the answers, and when another expert must lead the project. IEEE’s vital role The broader form of engineering creates an important role for IEEE. Its societies already bring together expertise in various technologies, engineering education, technical standards, and other fields. That makes IEEE well placed to help engineers work across specialties without weakening the value of specialization. IEEE could create more communities organized around problems that require expertise from several fields. Examples include trustworthy clinical AI, resilient infrastructure, energy reliability, scientific discovery, and safe autonomous systems. An engineer could begin in a home discipline and then work with specialists from other IEEE communities as a project requires. IEEE also can help set practical standards for human-machine collaboration. Engineers will need clear guidance on when AI-generated work must be checked, how uncertainty should be documented, when a specialist must make a decision, and who is responsible when a system uses both human judgment and machine output. Those questions are becoming part of everyday engineering. They should not be left to each company or project manager to answer independently. A professional organization can provide common expectations that follow engineers as they move between jobs, industries, and technical fields. A larger definition of the engineer AI does not make engineering disciplines less important; it makes the connections between them easier to use. Deep expertise gives engineers the foundation to judge what is sound. Systems thinking helps them understand how different parts of a problem fit together. AI can extend engineers’ ability to explore, test, and communicate across fields. Human judgment gives the work direction and responsibility. An engineer is not defined only by a job title or a fixed set of tasks. An engineer is someone trained to understand complex systems, define and redefine problems, bring together the knowledge and tools needed to solve them, test whether solutions work, and take responsibility for the outcome. That is what it means to be an “unbound” engineer. The boundaries between specialties become easier to cross, and machines might take on more of the technical work, but the core of engineering remains human: deciding what problem is worth solving, what evidence is good enough, how the pieces fit together, and what to do when the answer is uncertain.
A lesson in learning, summarization, and explaining via LLMs
What is “the true cost of a token”?
I’ll be speaking at the Bayesian Interest Group November 2026 Virtual Meeting. The topic is using Bayesian models to interactively pre-estimate the value of applying an expensive inspection or procedure to a large population. For example: estimating the value of using an LLM security service on a great number of […]
I started this blog a few years ago when I began my journey studying immunology, to have a place to write about what I was learning and about applications of AI to the field. My studies were a success: I completed a Masters in Microbiology-Immunology, explored interesting applications, and met some awesome people along the way. At the end of last year, I decided to return to my origins and re-joined fast.ai, which had grown into Answer.AI while I was away. Why now? AI is an increasingly hated field, and for good reason. I wrote about my decision: There are tons of false and over-hyped claims about what AI can do, but underneath those, it is a genuinely useful technology. I want to use AI in ways that don’t degrade my own critical thinking skills and that don’t pretend that resources are unlimited. The biggest companies have made their vision for the technology seem like the only option. It isn’t. OpenAI, Anthropic, Google, and xAI don’t own AI. Their values and decisions aren’t the only version of what AI technology can be. I understand why many people are anti-AI. My new job is rooted in the same objections. I returned to the field because I want to figure out how to use AI in ways that protect human creativity, autonomy, and problem-solving. You can read my full essay here. I am excited to continue exploring how to design AI tools that better encourage human understanding and human autonomy, particularly in the area of education. I have written several essays in the last year, posted at fast.ai and answer.ai, about how AI intersects with education, coding, and science: Breaking the Spell of Vibe Coding Stop Saying Boredom is Good for Kids I Don’t Want a Learning Dashboard for My Child Risks and Limitations of AI in the Life Sciences My friends all hate AI; I just joined an AI startup Going forward, you can find me at fast.ai, answer.ai, on twitter, and on LinkedIn. I look forward to reading your responses. Create a free GitHub account to comment below.
