The agent principal-agent problem 2026-05-07 Code review is broken. The industry-established code review process, review-then-commit, was a straightforward mechanism that allowed a relatively low-trust group of engineers to collaborate. It appears to have been initially developed for the Apache server OSS project in the 90s, corporatized by Google in the early 2000s, and popularized throughout the industry by several means, most notable of which was the GitHub PR. It was very simple: This is not Michael Fagan's defect analysis work or the ticket-like processes used for critical systems changes in fields like aerospace. This will not catch your bugs. It will, however, communicate design changes to other engineers who maintain a mental model of the codebase, and reviewers can use the process to teach norms to contributors. It has advantages, and because there is a gate before the main branch changes, it does not require much trust. That makes it a great tool for scaling a company, because beyond ~10-12 engineers (the "two pizza" team, among other names), trust erodes rapidly. It is also great for scaling OSS. It puts work on reviewers, but there was work on the human making the change too. An imbalance existed but was often manageable. Agents broke this. If you insert an agent into the existing process, your best possible outcome is: This doubles the amount of review. But companies were already review limited. In a really well-functioning team, a code review cycle could take a day. (Between two engineers who get on well and intimately know each other's work, you could shrink this to an hour.) But across the industry the number was, optimistically, to get a review merged before agents.days Additionally, the whole reason engineers use agents is it improves productivity. More total changes are generated. So we doubled review, and increased the total changes. As you modify the old model, you run out of review bandwidth before you have extracted all the value you can from agents. (And anecdotally, you run out of bandwidth before you get even a fraction of the value of agents.) But things get worse, because no-one actually augments the old processes this way. What happens in reality are processes like this: This is an example of what economists call the : the reviewer is the principal, the contributor is the agent, and code review only worked because the reviewer could cheaply infer effort from reading the code. Agents collapse that signal. This is what is killing OSS, and it is commonly being referred to as "slop PRs". There is no incentive for the human driving the agent to actually read the code or spend time thinking about what the reviewer says.principal-agent problem The result is a radical imbalance. "Contributors" type a sentence or two, of the quality of a poor bug report, spend 5 minutes poking at the resulting program, and then generate serious review load for another engineer. You can do this with no understanding of the underlying project, its constraints, or the tools used to construct it. This is an unmanageable disaster. This does not even work in environments where the reviewer is paid to do the work, because they could be more productive by prompting the agent themselves. Small high-trust teams have an easy process they can adopt: There is still a human in the loop. There is still a reviewer who did not get deeply lost in the weeds of how a problem could be solved. Most importantly, there is no principal-agent problem, because the human driving the machine takes on the responsibility for its actions by owning the deployment. Anecdotal evidence suggests this works for small teams. With a team of nine at we have been able to make it work. We spend a lot more time writing integration tests, e2e tests, building agent-based workflows for analyzing commits for safety or performance or usability bugs to minimize risk. This is a lot of machinery teams traditionally do not develop until they are far larger and more mature, on the other hand it is much easier to develop thanks to agents. We also have had to be very selective about our colleagues and be intentional in our communication. But we ship this way.exe.dev This is not tenable in low-trust environments, i.e. large companies. You have to trust your co-workers to start a conversation about architectural changes before they do it. No-one at BigCo trusts their colleagues to make sweeping changes to a service they "own". And no-one at BigCo wants to be on the hook for a major outage without having coverage from a code review to smear the blame around. (Low trust environments are awful places.) I am sure there are small isolated teams at big companies that have broken with standard practices and are getting real value out of agents. I am also sure there are ICs who have work that lets them maximize the value of an agent without involving their colleagues. (E.g. if you work in quality, agents can help you write and execute endless large-scale experiments you never need get reviewed, just send out what works.) But the vast majority of big company engineers cannot make changes, especially cross-functional changes that agents do so well, without review eating all the productivity gains. As of writing this, I have not seen anyone describe a process that "scales" agent-driven development in a large company. There is, however, evidence from the past that it is possible. I would point to Microsoft in the 1990s, which did not have mandated review-before-commit practices. Some teams may have, but the company, while large, was organized as many independent teams constantly synchronized by QA processes. This is regarded as "old-fashioned" "cowboy" style development by proponents of the large-team processes that came before agents. But it did work. It created some of Microsoft's most long-lived successful products, like the win32 API. (And yes we could critique a 30 year old API endlessly, but it is still there and significantly better than some of its "replacements" that were built with code review processes.) Little appears to be written about this period of Microsoft history, if you were there I would love to hear or read about your experiences. Until someone develops robust processes for agent use in low-trust environments, small teams have a large force multiplier available to them that big teams do not. Ship while you can. A human makes a change. This change is packaged up, sent to another human for commentary. Rounds of commentary and adjustments continue until the reviewer approves (LGTMs) it. The change is committed. A human to make a change.instructs a machine The human reviews the code, iterates with comments until they approve it. This change is packaged up, sent to another human for commentary. Rounds of commentary and adjustments continue until the reviewer approves (LGTMs) it. The change is committed. A human instructs a machine to make a change. This change is lightly QA'd, packaged up, sent to another human for commentary. Rounds of commentary come back from the reviewer and until the reviewer approves (LGTMs) it.are sent wholesale to the machine for adjustments The change is committed. A human instructs a machine to make a change. The human reviews the code, iterates with comments until they approve it. They push the change to production and deploy. The crisis of code review The agent principal-agent problem Potential solutions Some hints in the history books

I am building a cloud 2026-04-22 Today is fundraising . As is the nature of writing for a larger audience, it is a formal, safe announcement. As it should be. Writing must necessarily become impersonal at scale. But I would like to write something personal about why I am doing this. What is the goal of building ? I am already the co-founder of that is doing very well, selling a product I love as much as when I first helped design and build it.announcement dayexe.devone startup What could possess me to go through all the pain of starting another company? Some fellow founders have looked at me with incredulity and shock that I would throw myself back into the frying pan. (Worse yet, experience tells me that most of the pain is still in my future.) It has been a genuinely hard question to answer because I start searching for a “big” reason, a principle or a social need, a reason or motivation beyond challenge. But I believe the truth is far simpler, and to some I am sure almost equally incredulous. I like computers. In some tech circles, that is an unusual statement. (“In this house, we curse computers!”) I get it, computers can be really frustrating. But I like computers. I always have. It is really fun getting computers to do things. Painful, sure, but the results are worth it. Small microcontrollers are fun, desktops are fun, phones are fun, and servers are fun, whether racked in your basement or in a data center across the world. I like them all. So it is no small thing for me when I admit: I do not like the cloud today. I want to. Computers are great, whether it is a BSD installed directly on a PC or a Linux VM. I can enjoy Windows, BeOS, Novell NetWare, I even installed OS/2 Warp back in the day and had a great time with it. Linux is particularly powerful today and a source of endless potential. And for all the pages of products, the cloud is just Linux VMs. Better, they are API driven Linux VMs. I should be in heaven. But every cloud product I try is wrong. Some are better than others, but I am constantly constrained by the choices cloud vendors make in ways that make it hard to get computers to do the things I want them to do. These issues go beyond UX or bad API design. Some of the fundamental building blocks of today’s clouds are the wrong shape. VMs are the wrong shape because they are tied to CPU/memory resources. I want to buy some CPUs, memory, and disk, and then run VMs on it. A Linux VM is a process running in another Linux’s cgroup, I should be able to run as many as I like on the computer I have. The only way to do that easily on today’s clouds is to take isolation into my own hands, with gVisor or nested virtualization on a single cloud VM, paying the nesting performance penalty, and then I am left with the job of running and managing, at a minimum, a reverse proxy onto my VMs. All because the cloud abstraction is the wrong shape. Clouds have tried to solve this with “PaaS” systems. Abstractions that are inherently less powerful than a computer, bespoke to a particular provider. Learn a new way to write software for each compute vendor, only to find half way into your project that something that is easy on a normal computer is nearly impossible because of some obscure limit of the platform system buried so deep you cannot find it until you are deeply committed to a project. Time and again I have said “this is the one” only to be betrayed by some half-assed, half-implemented, or half-thought-through abstraction. No thank you. Consider disk. Cloud providers want you to use remote block devices (or something even more limited and slow, like S3). When remote block devices were introduced they made sense, because computers used hard drives. Remote does not hurt sequential read/write performance, if the buffering implementation is good. Random seeks on a hard drive take 10ms, so 1ms RTT for the Ethernet connection to remote storage is a fine price to pay. It is a good product for hard drives and makes the cloud vendor’s life a lot easier because it removes an entire dimension from their standard instance types. But then we all switched to SSD. Seek time went from 10 milliseconds to 20 microseconds. Heroic efforts have cut the network RTT a bit for really good remote block systems, but the IOPS overhead of remote systems went from 10% with hard drives to more than 10x with SSDs. It is a lot of work to configure an EC2 VM to have 200k IOPS, and you will pay $10k/month for the privilege. My MacBook has 500k IOPS. Why are we hobbling our cloud infrastructure with slow disk? Finally networking. Hyperscalers have great networks. They charge you the earth for them and make it miserable to do deals with other vendors. The standard price for a GB of egress from a cloud provider is 10x what you pay racking a server in a normal data center. At moderate volume the multiplier is even worse. Sure, if you spend $XXm/month with a cloud the prices get much better, but most of my projects want to spend $XX/month, without the little m. The fundamental technology here is fine, but this is where limits are placed on you to make sure whatever you build cannot be affordable. Finally, clouds have painful APIs. This is where projects like K8S come in, papering over the pain so engineers suffer a bit less from using the cloud. But VMs are hard with Kubernetes because the cloud makes you do it all yourself with lumpy nested virtualization. Disk is hard because back when they were designing K8S Google didn’t really even do usable remote block devices, and even if you can find a common pattern among clouds today to paper over, it will be slow. Networking is hard because if it were easy you would private link in a few systems from a neighboring open DC and drop a zero from your cloud spend. It is tempting to dismiss Kubernetes as a scam, artificial make work designed to avoid doing real product work, but the truth is worse: it is a product attempting to solve an impossible problem: make clouds portable and usable. It cannot be done. You cannot solve the fundamental problems with cloud abstractions by building new abstractions on top. Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. We have been muddying along with these miserable clouds for 15 years now. We make do, in the way we do with all the unpleasant parts of our software stack, holding our nose whenever we have to deal with and trying to minimize how often that happens. This however, is the moment to fix it. This is the moment because something has changed: we have agents now. (Indeed my co-founder Josh and I started tinkering because we wanted to use LLMs in programming. It turns out what needs building for LLMs are better traditional abstractions.) Agents, by making it easiest to write code, means there will be a lot more software. Economists would call this an instance of . Each of us will write more programs, for fun and for work. We need private places to run them, easy sharing with friends and colleagues, minimal overhead.Jevons paradox With more total software in our lives the cloud, which was an annoying pain, becomes a much bigger pain. We need a lot more compute, we need it to be easier to manage. Agents help to some degree. If you trust them with your credentials they will do a great job driving the AWS API for you (though occasionally it will delete your production DB). But agents struggle with the fundamental limits of the abstractions as much as we do. You need more tokens than you should and you get a worse result than you should. Every percent of context window the agent spends thinking about how to contort classic clouds into working is context window is not using to solve your problem. So we are going to fix it. What we have launched on exe.dev today addresses the VM resource isolation problem: instead of provisioning individual VMs, you get CPU and memory and run the VMs you want. We took care of a TLS proxy and an authentication proxy, because I do not actually want my fresh VMs dumped directly on the internet. Your disk is local NVMe with blocks replicated off machine asynchronously. We have regions around the world for your machines, because you want your machines close. Your machines are behind an anycast network to give all your global users a low latency entrypoint to your product (and so we can build some new exciting things soon). There is a lot more to build here, from obvious things like static IPs to UX challenges like how to give you access to our automatic historical disk snapshots. Those will get built. And at the same time we are going right back to the beginning, racking computers in data centers, thinking through every layer of the software stack, exploring all the options for how we wire up networks. So, I am building a cloud. One I actually want to use. I hope it is useful to you.

Eight more months of agents 2026-02-08 I wrote up my experiences programming with LLMs , and updated it for the world of agents . A lot has changed since then, so here is an update.a bit over a year agoeight months ago We were prototyping our first agent, Sketch, when Claude Code was released 12 months ago. So I, by good fortune, got to be there and be excited right at the beginning. They could be helpful for some things some of the time! Agent harnesses have not improved much since then. There are things Sketch could do well six months ago that the most popular agents cannot do today. The agent harness is critical, there is plenty of innovation to be done there, but it is as interesting a space right now as compiler optimizations were during the megahertz explosion of the 1990s. Right now, it is all about the model. And on the models: there are plenty of public benchmarks but they have all been gamed to death. Ignore them. Clearly the frontier model companies have good internal evals, because the models have qualitatively changed dramatically. In February last year, Claude Code could write a quarter of my code. In February this year, the latest Opus model can write nine tenths of my code. It all needs to be carefully read, and regularly adjusted, but now I can and do rely on the model to do the adjustments for me. There has been no obvious change in models. Nothing like when GPT2 started talking back. There has however, clearly been a huge incremental improvement in the ability of coding models to get to useful results. (All of this, admittedly qualitative, progress is the most positive economic signal I see today.) At a big company, my time was 80-20 reading code to writing code. At a startup, it used to be closer to 50-50. Now it is 95-5. The history of IDEs is so strange. On the one hand, the IDE is obviously correct. Of course I should have a development environment that provides as much information and assistance as I can effectively use. By far the greatest IDE I have ever used was Visual Studio C++ 6.0 on Windows 2000. I have never felt like a toolchain was so complete and consistent with its environment as there. Since those glorious moments in 1999, I have spent more of my programming life outside of IDEs than in them. The truth of programming environments is they are a hot mess. Unix was great, the Howl's Moving Castle we have bolted onto an over-taxed set of Unix concepts, not so much. The same thing happened to that win32 API I used to use in VS6.0, still there, with a giant mess atop and around it and entirely unignorable. Then co-pilot came out and it seemed the IDE was inevitable. It did not matter how miserable it was trying to fit your IDE into your environment, you had to do the work because LLM-assisted auto-complete and edit were too powerful to ignore. They made my typing go 50% further and a large amount of the programming I do is typing limited, so the effect was enormous. In 2021, the IDE had won. In 2026, I don't use an IDE any more. The degree of certainty I felt about a copilot future, and the astonishing whiplash as agents gave me a better tool not four years later still surprises me. The only IDE-like feature I use today is go-to-def, which neovim is capable of with little configuration. So here I am, 2026, and I am back on Vi. Vi is turning 50 this year. A huge part of working with agents is discovering their limits. The limits keep moving right now, which means constant re-learning. But if you try some penny-saving cheap model like Sonnet, or a second rate local model, you do worse than waste your time, .you learn the wrong lessons I want local models to succeed more than anyone. I found LLMs entirely uninteresting until the day mixtral came out and I was able to get it kinda-sorta working locally on a very expensive machine. The moment I held one of these I finally appreciated it. And I know local models will win. At some point frontier models will face diminishing returns, local models will catch up, and we will be done being beholden to frontier models. That will be a wonderful day, but until then, you will not know what models will be capable of unless you use the best. Pay through the nose for Opus or GPT-7.9-xhigh-with-cheese. Don't worry, it's only for a few years. The constant stream of "may I run ?" from Claude Code and "I tried but cannot in my very-sophisticated sandbox" from Codex is a nightmare. You have to turn off the sandbox, which means you have to provide your own sandbox. I have tried just about everything and I highly recommend: use a fresh VM.cat foo.txtgo build This is why I am building . I need a VM, with an unconstrained agent, that I can trivially start up and type the one liner I would have otherwise put into an Apple Note named TODO and forgotten about. A good portion of the time Shelley turns a one-liner into a useful program.exe.dev I am having more fun programming than I ever have, because so many more of the programs I wish I could find the time to write actually exist. I wish I could share this joy with the people who are fearful about the changes agents are bringing. The fear itself I understand, I have fear more broadly about what the end-game is for intelligence on tap in our society. But in the limited domain of writing computer programs these tools have brought so much exploration and joy to my work. New technology brings a lot of challenges and reasonable concerns. I spend my days trying to push the limits of agents, so I see them fail catastrophically several times a week. Significant change also changes labor markets which has many effects, good and bad. In 1900, 33% of Americans lived on a farm, and 40% worked in agriculture. In 2000, less than one percent lived on farms and 1% of workers are in agriculture. That was a net benefit to the world, that we all don't have to work to eat. (The numbers are even more dramatic if you go back another century.) But a lot of pain and heartbreak can and did happen along the way. It is right to be concerned. But far more than measured analyses of the reality of the changes that are happening, I see hard anti-LLM takes that a year ago I disagreed with, and now I just cannot understand. It sounds like someone saying power tools should be outlawed in carpentry. I deeply appreciate hand-tool carpentry and mastery of the art, but people need houses and framing teams should obviously have skillsaws. To me that statement is as obvious as "water is wet". Most software is the wrong shape now. Most of the ways we try to solve problems are the wrong shape. To give you an example, consider Stripe Sigma. This product is a nice new SQL query system for your Stripe DB. It has a little LLM built into it to help you write queries. The LLM is not very good. I want Claude Code or Codex writing my queries. But Stripe launched a fancy Sigma UI with an integrated helper their API. There is a private alpha for the SQL REST endpoint that I do not have access to yet. So instead I had my agent do ETL-from-scratch: it used the standard Stripe APIs to query everything about my account, build a local SQLite DB, and now my agent queries against that far better than Sigma can.before I implemented that entire Stripe product (as it relates to me) by typing three sentences. It solves my problem better than their product. That's the world we are in today. By far the worst product I had to use every day in this new world were clouds, so that's what I'm building over at . It's a lot harder than it looks, but the entire point of the product is you should never feel that your agent should rewrite part of it for you.exe.dev Along the way I have developed a programming philosophy I now apply to everything: . The practical nature of writing software for customers has traditionally pushed us away from that philosophy. Product Managers have long had to find gentle ways to tell engineers: you are not the customer. Well, that has all been turned on its head. Every customer has an agent that will write code against your product for them. Build what programmers love and everyone will follow.the best software for an agent is whatever is best for a programmer Hopefully that philosophy will survive the next year of changes wrought by LLMs. Agents have improved dramatically in a year IDEs are clearly waning Using anything other than the frontier models is actively harmful Built-in agent sandboxes do not work I have far more programs and services than I used to I am extremely out of touch with anti-LLM arguments A lot has to change

How I program with Agents 2025-06-08 This is the second part of my ongoing self-education in how to adapt my programming experience to a world with computers that talk. The first part, , covered ways LLMs can be adapted into our existing tools (basically, autocomplete) and how careful prompting can replace traditional web search. Now I want to talk about the harder, and more rewarding act of using agents to program.How I program with LLMs It is worthwhile starting with a definition of the word “agent” in the context of LLMs. The “AI” hype cycle has been throwing this very generic word around longer than agents have actually been useful constructs. As a result there is a bit of smoke and mirrors marketing and general mysticism to dig through to find any value in the word. For someone with an engineering background there is now a straightforward definition: . That is, an agent is a for loop which contains an LLM call. The LLM can execute commands and see their output without a human in the loop.an agent is 9 lines of code @font-face { font-family: Excalifont; src: url(data:font/woff2;base64,d09GMgABAAAAABCIAA4AAAAAG/QAABAyAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGhYbhQQcegZgAIEEEQgKqDCeAws2AAE2AiQDaAQgBYMYByAbnBWjoqwRzpL9VUI2ZPLXsM7G0TiRFU22DbHd2awNUXefA9LgQvxZj/T9DU9z+3e1WxRLShkrNnolo7eGHlUSMqIcQzAKtDGx0IYPViBWYPKjsLAi+Oefk93/tjyIIIHA8iQImnDGnQiyjke0/CdU6X8A//7zr1OTVumOPxhaDpcAHRemvL3DLq/S+kFgCEiWbYUutQMFoPvg01/rr8ZZtx11jL8AsiPa8L/GWv4nlixeZDYVvHRCop3Yu+Wk/sU8iVlIoiEyTHdvRNVGJHRyIxaG1FmGm/CqYU0fxfc0CqPqAggAVLAGgEFQbqCKmyELpoNeLAnpwP0+1lkD3B9OezVw34pH1QF3KADAa6PO0+6sA8TuAFYlPC8REiKCaIVziYsFes84QMEAP72hoPjXZc9xtNPnhDgngdChMDgiEjIqWoGZ2BLJvYp1Ya6oMI0IPjoQIWGaixG+MwJFREFGQgPBOtogwFMJEiOMcEnsLkkT+WAYfo0tPDMFcYQghGwINF02ASKIBMi3UcFEA5vZYDygGAPqlypRzq2yvJgghnFMBwRUQngoWZ/pWxDdZ+WjQBIG2X1LhxbCEGstGwAQjwuBbkDpK0gP2dZ0tSaBPiDQuDT9bIX5daAmAQDrGsnXMDHBAyKR60eIB20oB7tiFkAk7CTlzyxRqgxlKiRVjTsqtQlUbK3h/P72v+sGHXfQXr022ahbVxCoA6CdmNad1B0FAOED4D95DCWORUCCYP8qyHpm3tP56CpvCC8dma7iSPmaCLr8lGaUJBiJyG3oyJ8a3f14a8bZYqKtnKW299O4CEndeHzBWC5yfSR35szShBodu5X6lnRjtpbbve89L87pgQECTwmm7zg0saVZpzMbLiBryTwgPvGpX+qYlXqEeSNqYB5xxMMIZ7YHKfWhL1aJbtd3QdxSExEQj7BCudyui+ayullfy9MU97C3lH7G/P0She/ZAyml0FgSI/YMKBCxzAGildr2Rq0XlJ16cCUX9JFF612GWIDNfRJjv/Y2ejjV9iBpAAjoij7y5k2VdNGYq4D4qE3z1drJbdoNXaxUGGrgJdvLi+44v0O/BuiexPYG9ZfyhCaQJpQaYHnfA9JEzdlJFKoCU4UKCi1/XoDW7kejbHYB7RQQILCjSxho7gyL2Smu5waLkfVJbQ8m7TylEJYcyDV+WGZRhr0EQtqGVMLAyUvSl5C2Hrjo0cmOGvmmsBhGmO95usd7v7R/MFsY+KRFy71NazQwLpBKWFvlP0HdwuSoVljmWYrxQjH63N+Gb2ce3ENt2eA9zoClvt8iyyDuCmBshonshxB6hnxz3qn5gIy5e9Xl2WhdFysa8yjCPIV0Jkfmq1VAhN+Zn3nan9ge5ktpinnGMxw5Ln6mqFCpm+GhpWE6A8iEk3PEZjLFuV+AlGI1wljIwgpu6OfhHD+3vRJNKIAJpDvdcgZCQkC8lxE+HkLhqMjkDX6+vh5UGigWffGXaf/WlvbYGympQTnSGznnmNve6qrICWb73idzLvTkqmC+NSSkJ9GrgablXiYEEupQ+K3h20pr111E6jFbbrw+9lzvSVfaw0GQ84EbM1RHbBF0f2x2qgEMLiRysEzbRvP+bskdYShURXOwZE/m8dry2vapqIEjba4apCswWH/bue2LnOGmCGYkDACx3IFnoy+m4vdG1ae2h7VJ01Iv4Ul9ShtvhG/z0YQ4HVvuxYHlRqc817s0uPFcmjeO7O4TGyHlxKY08Tw7fV6M/Lqty62hXImdufeMVLQMqUGY9fMeHq2XG7eyQ3wJl3qQ0gA6gQ/8AdZBzd1ujBbFbTGCPdn/cUVbfcRyFVq03D7DFayWeYfEY+7ArCqoefyldQFM7w+Ps5WZlbDCcUNPXipMqY0bGQdFqc13l3DkOu3U9oafyvWy6yj0jiUXsdl6OGlO6bWTekNUc3KHwhI1OoQIK2RtTZ4CUuoWWGias2O3Yd5f/3qQFR73J1uMrbo4N9gb/q/kU2rCIEdi1DSa1dmkGVqkBQUAHAa4GwFxEzHTJC8DAmRstAqkxqzOVMdy83++yPkDR81VGlcKzKxk0MsHRnPVaJ+bwzzLcLaPJzM0L0NGc81YAUhcevc/NLumzCr4cfmx7ckTci1AuvG7sqf1gIUh6yAVt4IYkBZYzhG/ZiydgcldQhhjSKmiKoONweLU/5dS2+uX/ImqONtVR61mYVxgkGvd97Zpdd6dXr4dHFjZk9x/eOj5uff7ttEMWpbbRNaeoaFpQKbBeJ4GxCc3S8GzwmhdXHIxGuQJpNRoLpXbI/z2THu95L09Z5BHqKmQEuuEqH7omraXeBQuENaG3DdkTxwD4sMVOSxD8+03xWFxbHGELdSLahKtCVEM3C66p87MGvjQ0jkMbd3TzlMJ+9P3qWZPb1XrKaDc3UDtmzhbaDLIAZh+yqif03x+HEy3vpwn3TG32UQsZBcpLpCcqsOeoXAy1Ga7aS7byDPMe/aB7gamjY4tNKG2t0l77brHdH+b0wS2Vzmg6sjVtSFSvgljQK1d1ty1g+kelW6cDE/0xQOWKv9jPb4aejLtX/LRq/b87rOuC5p9sRWw7JDfA9IPMOc0OuQvE2u2fBsquxwsLu0+FDy5Mn+sHbU/E7yz+N0OUN6m/bTLNmDQ4SpCkqyg5br926jP/PPFX4nKtg+ExCwSs03XkF4hiRjxBt6yil6AWFyv/hvz1+kBDaFnn5m3XTaHFAks3z8KiYgswi0/bGoCpvBEKWwfrZeMdrohkovIOA4mu+E3KD+kYpoOMEEHlfMr41I3YV4knkM0EvGcjELVXUxnkpL2Ma0D3Ceb9iw+uWOGe4dk2LL25f9i0SH9vykkPSNRC7/byNvzw46XO8rk/shwTxNjF3lXn/HUP+XDrqZOQgcb+ckeklTRgZYY9EJxvQ72shGYpY0ELAlJZ1x4HDVu48L5jIGiP3K6mmtTwGcmaQa9x05si0EDX8QGi3M1Y7hxCr/h/kRCIfSkPYDC7qLFbXie/V5e4Mkm5dQ9FXtDuYSsjtQ5qROBi0fZJBJSHQj+jBm491ec7ZN2C7t4RQCeOQrgcqTZxDcX1jVkC/XDgsIgEiPTwxbRKbSAQ95/IJeURdyWZWM7k+JFaUSuKd/IXnzsHgW/aJ11Fp4glkTdR5xbvLLy2gnZPAcPoM62Tma3eT9JB4GeKXh1yDoeilnDwQSIeFFz/WFFhq0hwfKV7X3mZ7v3OS1X8P9PWqdkS9BE3+v+aj2L/SYsm0qqQ9fDWLxGIGBz3yFbi8mNxJ2w9TBkEnIYDRn6V6i8WLW3LB6ziWxYiyg3IMOqueLJ2Nb75eksJeWqAWshZM+IslkjTStYECQoa8aYcrMdBBI8Tya5eh5n5xYRWmsRiUOKyIepWl/1fWxScA2fb5Bwss0rboak7slfcuHjxdRhdWPKym7KF3SJJsbx4WRftgvchyyXo/Wz5W70jMAif9mCxnkfymfTjbub+HvBScI9vat0XtXl5AyuL+RrPuhfKmyLYSTjLpUrtX/n583nJ/GFea7z6MnoN4D5lWyynBKrN/P0Psb6dEOByyA0wR6DjMHVcahqcXavA610B2RKqVUeF0jKh9ImEtI9uC9p3KrrLu1kz9cYY6YnYy5iaxY/Hhqz0ivQJtQ6ZrcnqBU5mcz+cb/3p7tY3z9d6y9OJh233Mz4yyUX5NP63FPejmBe+0qw1C2Ywiv0eazr9y4biJb09yHnPoSeW/qk7Y1MfRg2hHe/U35LhHZXY0417jqmAGxgKf64uuMLISUSJyVPHZdEvRMni0JwZCxOCOaVJrGYw+GIA69TPdrVwCahVou9EsW64qAsTw9d3NBnGpxT+9Ktro5hjT3Vr+527PXmf8ryj1zK7YStpHS7CvhknhaXVUWlJLi+HdFXnxItRVNyEB5XVXPzD/rGkoSp7Jb+a0anRMi5O8PGS0lGDXKTKPfpaKzhg548BE67r5pfz+X8pSzvBD3kts8zp8jHcVnG5nEbc7iTRV7MhWG9UDXPLaXVVd3bsNhpg8dvmLrlEwurUfTcflV8WtyRBjUFh2mctCRizLmzsRPYn863R/mBd1wPI7dDyndrImJEredkCn49EbzhEqUX/Jx8hvHxYXm1OraCRr1E4c8SAgSlBamC4XgWkgfz6N/lbExraCvaBYFf/2Luk9F30XixJdeRquLxvGM4XtnrPLN9oRCWuR60sPn9N6GJrjScTPndHab86z+W5+kFT8YQK5kWQDuIL42FELNgZxdtXD8YB7qLxGvhd2E8alteoWb1t3vZ8gMfOCEN7Mi0O5d3LonrI+ZAWceHrJJSR59M/1W3rTVgAjyCpEgzjXFZk7xGlfBiMXw9OovSACvAsARV4RzQC5PdWGXlLH2+dEcMX0ggeutDNKnswKT6TtMRr1jK0iFYIkJiGl8LTlX50UMg4+6LosyRoqs6766qhdW7h1ozWCnICcj9qaACXEd9K4/FfkMrq3wrd+hly6PjZkX3qyzbo2snen6wO4IF8bJMXqzWQtF3lEPC9EB/rzOHXNLbTEQfPwpErn29uxqsdjocVzOPuvhYyRuJoMwdSs0nmSa//oNtpuGLG4cGx/ygTnpZFyW8mqpeg5LgGd54VVjpohXDlh2L8ODGxljSd0g+UuIZ6ybSx4EFosV0EwfrEVL5KiEN6urgr5+bN6WQg5Hc58VwqJusE2eT2ZJW4OTR5oaGr78zEElHYhEebAJ6M/uiRKt8NqS7R0fx/HOxpilgbaIhw12vsZSc8yDUPsoXp2GwN/JG0WtqiuGmTsdSXu9p/DZ4YfqnV70Zb7JDDMPv4i713pMmAY52IE9ybp+EMl7Kick0nrxMGtJTcu22MZSdTe3Tlh+o4HUH+iCOMoniao9+YP1W0FL8ycLs4RdYGFpy8k73n3Z18Dtbr36aeXTlBCvayNnRN3AjRVc0PiTRVi+YBK5SCXiYfvbjyXbXVDhq/FgvQffuGRrVpjZXyEzJoBECP7KFihYrIgiExZKwoNT/SmSnSEOOxJV5xsm0/2zHEbSoi+4d5CfVopE069vvrBxw9IaiR70fo6+YF9rJYbyGPW1QzSM035kxdxpwMe3ROfu8nVMIZrDQebbURkh7pf2/h52y8Hbve3cAWZBzg3Wv/60pZER8IFKQx9sBAO6MzQQAAHd/ezbeH48+XUMBQAR38P3hxw8siWnfe//Tfn/uSwkwz0IA/goaCEg7RZkXWj/jdFtcVwWmA9L8aOZzggqAPDce88aXQmjcAeraRUQHoWWDYP9sIbOaIi+y5KDEFLNs/zlpgrtBQABsARICAB2gcLn/7x1WEgCgxrlEQDiOCRjdLoHwNUOgxEoEJooYBPOyATAbY4RiNSqVGanOKAFS2ZVrVKOYUyY7pwaVsroCCoGCBYezyivHcqjgrkpSQYmQWaScPIR88EQEPQeraZLGKon32amn2bcURmSHsRJUkmTFRJARIEdWCl5AveUSY1c9unIpJ+fRAqtYr0YMF3SXDRnYkexoMlFKUggAGf36CQMAAAA=); } User LLM prompt bash, patch,etc tool call tool result Response &End of turn That’s it. Like all simple things your instinct, quite reasonably, could be to say “so what?” It is a for loop. But the results are an astonishing improvement on the power of a raw large language model. Consider yourself in front of a whiteboard, writing a function in C with a marker designed to test if a UTF-8 string is valid. (This has actually happened to me, it is a standard interview technique. The stakes were high, this interview changed the course of my career!) How well you do at this task depends on your experience as a programmer, and your ability to fudge your inability to use external resources. You need to remember the encoding of UTF-8. You need to not confuse your C grammar with all the other C-like programming languages you have used over your career (is it name-then-type or type-then-name?). In daily life you get feedback from a compiler if you make a mistake, you can look up a specification of UTF-8, and best of all you can write your program and sprinkle some printfs in it to see what you got wrong. Asking an agentless LLM to write code is equivalent to asking you to write code on a whiteboard. It is an exercise in dredging up half-forgotten memories, executing a parser on an extraordinarily inefficient substrate, attempting to avoid hallucinating an actually-useful programming interface. The fact LLMs can invent programs out of whole cloth is an astonishing technical achievement that I consider myself blessed to have lived to see, but it is also not a huge shock that attaching a GPU to a virtual whiteboard is not going to do a significant amount of useful programming. But what happens if we give an LLM more than a virtual whiteboard? What if it could call the compiler, see the compiler errors, and get a chance to fix them before we see the result? What if it could use `grep` and `cat` to read existing files in a project? What if it could patch multiple existing files, including the unit tests and repeatedly run the tests? Agents are feedback-driven LLMs. Just as humans thrive in environments with feedback, LLMs go from nice demos to useful programmers when given a surprisingly small core set of tools that are very familiar to programmers: Agents are very good at navigating code bases using the bash tool: find, cat, grep -R, just the way those of us who predate IDEs tend to navigate code bases. Upfront it is instructed to commit changes to git, which it does by using the bash tool to run git add, git commit, etc. The result, compared to an LLM generating code without these tools available is significantly different. Of note: The downside of agents is time. A single sentence request that would have required a 200 token response can now generate tens of thousands of intermediate tokens driving tools, some web searching, and many runs of a project test suite. This takes several minutes and there are only so many cups of coffee we can healthily brew in one day. Today it may also appear costly (my last significant agent-driven commit cost me $1.15 in API credits!) but the cost will rapidly disappear as the chips driving the models continue to improve. GPUs are not nearly as restricted in their future progress as CPUs which are far more constrained in how they can physically improve (compiler-driven only goes so far). The fact that we still call LLM chips “graphics” chips shows the economic machine underneath software has a lot of restructuring to do before it is fully focused on LLM performance-per-watt.ILP Ultimately, agents spend CPU and GPU cycles doing intermediary work so that humans do not need to. Any time I can mechanise my labor I get more done, so agents are a huge step forward for me. I only end up writing a tiny fraction of the programs I want to due to a shortage of hours in the day, and with agent support I get a little further into the list of programs I wish I could write. May we all be so lucky as I have been this last year. As such, I am thoroughly convinced that agents are worth significant engineering investment addressing their limitations. It is relatively easy to see an example of an agent producing useful work. Drop one into your project, break off a small task and type it in, see what it does. Let me give you two examples. Let me talk you through an example of where I used an agent to do a significant amount of work on a project. I implemented the first pass of Github App auth for the hosted sketch.dev using sketch. It did the whole thing with 3-4 pieces of feedback as I clicked around the interface finding errors. That is an astonishing achievement. It is easy for the haughty to disregard the work of gluing well-known APIs together as “not real” programming, but it has been the practical experience of my career that for every hour of truly interesting programming I can do, I have had to do 10 or more hours of dreary work with APIs, libraries, dysfunctional compilers, build systems, or obtuse package managers to actually make that hour useful. Having a tool that lets me do 30 minutes of “not real” programming by writing a few carefully constructed sentences, and lets me vacuum the kids' room while it works, .preserves momentum But now comes the work. It implemented the GitHub app auth flow I wanted. It even met the stringent requirement I put on it: I asked it if it could avoid saving a token per user, and instead use the global private credentials of the app to drive everything to keep the database simple. It did it! In the process of doing it, it wrote some very bad code. First, is the huge security vulnerability it created. Because it let anyone who had authorized the app work with any repository authorized with the app, even if they did not have access to it. Disaster. Fortunately it is so bad that if we had got to testing with others on the team we would have spotted it quickly when our own private repos started appearing on each other’s repository lists. (And was such an obvious problem I caught it even before that.) A quick explanation of the problem to sketch.dev got it to fix it, it implemented authorization checks for users and got it right. Another amazing achievement, I wrote one sentence and got a reworked functioning commit on a branch. The next problem was performance. While the new code worked, it would have been unworkably slow as soon as it had several users. It generated a list of repositories the user had access to by: This meant that every time I wanted to display to the user a list of repositories they had authorized, I had to, for each GitHub org that had allowed the use of Sketch, do an API call to fetch their repository list, and then for every single repository do another API call to check if this person is a user. This means the number of API calls grows with the total number of product users, which is not going to work. It turns out, the problem was my original naive requirement to avoid storing per-user tokens from GitHub. It turns out they do not have any efficient app-auth-level API calls to determine what a user can access, the only way to do it efficiently is to ask what access the auth token has, and use an auth token for the user. Once I realized this I told sketch to go back and remove my original requirement: save per-user auth tokens and use them for everything. It quickly came up with efficient API calls. Telling this story took more words than I typed in total into Sketch to generate the GitHub auth code I wanted, and writing it took more effort than the code reviews that caught all the issues. I need to emphasize that because it is so easy to take a part of an anecdote like this and use the limits and mistakes of these tools to declare them “useless” or “dangerous”, while my experience is not that at all. What we have today can clearly not replace me as a programmer, but it did let me get done in a day a dreary task that I would have struggled to traditionally complete in a week. And I vacuumed the kid’s room while I was at it. Here is an example of something my agent needs to do regularly that it struggled with until I found a way to help it (and I believe captures a typical limitation people run into when first trying to work with LLMs). I learned an odd way of using SQL at Tailscale (from Brad and Maisem): make every table a JSON object. In particular, have only one “real” column and the rest generated from the JSON. So the typical table looks like: This has many pros and many cons. It acts as a poor man’s ORM as each table has an “obvious” data type matching each record. It makes adding to the schema trivial. You can choose to ADD COLUMN if you like but you do not have to. The SQL column constraints act as good dynamic checks on the quality of your JSON. It greatly increases the amount of stored data per row. You have to structure all your INSERT and UPDATEs in terms of JSON. Half a foot in the document database world, but I can still write an old fashioned JOIN. Etc. Pros and cons aside (that can be a fun blog post for the future), our agent often tripped over this style. When creating new tables and columns, it would sometimes, but not always, follow the generated column pattern. The very first time we added a table that did not use this all-generated-column style it got further confused, and would choose almost at random it seemed between styles. It turned out to be really easy to fix the agent’s behavior. At the top of the SQL schema file I tried adding a three sentence description of this. The key line seems to be “each table has a single concrete Data JSON column, all other columns are generated from it” and then a comment on the tables that did not follow this pattern explaining they were an exception to the norm, and the behavior improved dramatically. This is slightly counter-intuitive. My lived experience of instructions like this is that engineers heavily discount them. It may be ad(-like) blindness, it may be the challenge of keeping comments up to date, or it may be that most comments are not worth much attention.. Instead the industry norm for communicating this kind of knowledge is an unpleasant routine of engineers who do not know writing a PR that does it the wrong way, then receiving comments on it and telling them to do more work. LLMs seem to give the comments more work, hopefully for the better. One of the arguments against LLM as a code generation tool is that generating code is only a small fraction of the overall cost of code. The ongoing work of dealing with existing code is the vast majority of the cost, goes the argument. There are code bases where this statement is clearly true. In heavily used products with a growing user base adding new ways of using programs, engineers do spend most of their time navigating the unwritten misunderstood interdependencies of existing code. To someone whose job is that, a computer you can talk to that produces a usable result for “implement bubble sort in fortran” sits somewhere between a toy and nuisance. Sometimes attempts to compare this to financial concepts such as “assets” and “debts” are used, but I will skip them as they never seem to quite fit. Whether this understanding of engineering, which is correct for some projects, is correct for engineering as a whole is questionable. Very few programs ever reach the point that they are heavily used and long-lived. Almost everything has few users, or is short-lived, or both. Let’s not extrapolate from the experiences of engineers who only take jobs maintaining large existing products to the entire industry. Fortunately we do not have to answer the question of what the entirety of programming looks like to say if agents are valuable, because agents are potentially useful even in the maintenance of existing products. An agent is not just code generation. It is an LLM with a series of tools that reads code, and changes code by editing files. Agents are as happy removing code as adding it. The result is change. Yes, the change is “more work” because the person driving an agent has to understand the change being made. Yes, the agent may not have sufficient understanding to change large products yet. But change is the ultimate goal of the engineer driving the tool, and agents are demonstrating the ability to edit moderately large projects with some care. That makes them potentially useful tools everywhere in the programming industry. If agents are not good enough yet (and that’s a big if, it is well worth testing), they are on the right track and now have all the fundamentals to get there. A related, but tricker topic is one of the quieter arguments passed around for harder-to-use programming tools (for example, programming languages like C with few amenities and convoluted build systems) is that these tools act as gatekeepers on a project, stopping low-quality mediocre development. You cannot have sprawling dependencies on a project if no-one can figure out how to add a dependency. If you believe in an argument like this, then anything that makes it easier to write code: type safety, garbage collection, package management, and LLM-driven agents make things worse. If your goal is to decelerate and avoid change then an agent is not useful. Unlike somewhat mysterious ideas like the transformer architecture that underpins LLMs, mechanical feedback into an LLM seems “obvious.” It is clear to those of us who think about developer tools: I have been working on it for over a year now, but our first revision of sketch.dev in January, despite having go tooling wired into an LLM, hardly counts as an agent by the standards of what we can use today. (The difference in utility between the first revision of sketch and the current open source project is astonishing.) The utility of feedback is also clear to everyone who works in the ML field, as has been one of the core tenants of the field for 50 years.sketchreinforcement learning The answer is a critical chunk of the work for making agents useful is in the training process of the underlying models. Models have to robustly call the tools they are given and make good use of them. We are only now starting to see frontier models that are good at this. And while our goal is to eventually work entirely with open models, the open models are trailing the frontier models in our tool calling evals. We are confident the story will change in six months, but for now, useful repeated tool calling is a new feature for the underlying models.The LLMs of 2023 could not drive agents, the LLMs of 2025 are optimized for it. It is challenging to think about what is next in a fast-moving field where most engineers today are not even using these tools yet. But those of us building these tools need to think about it. Most use of agents today is either in the IDE or in a checked out repository on a development machine. This is an easy way to get started, it’s easy to install a vscode fork or a command line tool and run it. But it has two significant limits. The first major limit is that agents need a lot of safeguards built into them to avoid them running amok. One of my machines has production credentials squirreled away on it I can use to do deployments. As part of running commands is the agent going to grab those credentials and run my deploy script for its uncommitted changes? Avoiding this, if your agent runs directly on your real computer, involves a lot of asking the programmer to babysit the tool calls and give the agent permission to run commands. And even then, there are dangers. I can say “yes to all” to running curl, only to forget a web server I’m developing that’s running on localhost doesn’t have any auth yet and can read arbitrary files off disk. Oops, my production credentials are exposed again. The second major limit is requiring developers to use their bespoke manually-configured development environments to run an agent means we are effectively serializing execution of an agent. As mentioned earlier, one of the major weaknesses of agents is they take several minutes per turn to produce good results (and probably will for the foreseeable future). One way to make better use of our time is for an engineer to be driving several agents simultaneously, but this sort of agent deployment makes that impractical. We are exploring solving both of these issues in using containers. By default sketch creates a little development environment in a container with a copy of the source code and the runner has the ability to extract git commits from the container. This lets you run many simultaneously. (Other agents are also exploring this space, agents in general are a very busy space!)sketch.dev To give you an example of how this parallelism can work in practice, while I was working on the GitHub auth mentioned in the above example, I was about to complain in a group chat about how ugly a form I made is. Instead, I opened a second copy of Sketch and pasted in a screenshot of the form and wrote “this is ugly, please make it less ugly.” I went back to thinking about auth, and when I remembered I had typed that a half hour later I looked at the results and decided yes, it is an improvement. So I asked it to rebase, it resolved the merge conflict (one of my least favorite programming tasks!), and I pushed the update. It was not nearly the quality a real designer can produce, but it was certainly better than the awful unstyled form I had created during testing. In a past life I would have created an issue in the issue tracker, by which I mean I would have made a note in my personal to-do.txt file to create an issue, because issues are visible to other programmers and require typing something more coherent and constructive than pasting a screenshot and saying “this is ugly, please fix.” One of the great things about talking to an agent is if you only have a tiny bit of mental energy left you have a good chance of getting something of value out of 30 seconds work. The likelihood I would have ever turned the to-do.txt entry into a real issue is, honestly, pretty low. So our takeaway from exploring the UX of agents for the past six months is that we may finally have a good use for “development” containers. An open question that we are spending a lot of time exploring is: what does the IDE become in this environment? Let us say we start work by talking to an agent. The execution container can be completely derived from GitHub, the changes shown as a diff and pushed as a branch (or PR). Is that the actual workflow? In practice, many of the commits generated by sketch, or any other agent we have tried, require some human cleanup afterwards. (Our experience is, when programmers first use an agent most commits need manual intervention, but practice writing prompts reduces the number of necessary intervention.) It could be as simple as editing a comment or changing a variable name. It can be more significant. How do we make that work with a containerized world? So far we have several workflows that we quite like that other agents have not yet explored. One is making the diff view editable. You can type in the right-hand-side of Sketch’s diff view and it ends up in the commit and pushed for you. It is great for one-line edits. For the sorts of fixups where you want to run sed, or grep through the changes, or run the tests in an interesting way, we have had great success with giving the user ssh access to the container. Not only can you shell in (and we have a little web terminal in the UI too), it is easy to turn this into a vscode:// URL that can be opened directly in a traditional IDE, which is sometimes exactly what we want. Finally, we let you write “code review” style comments on the diff view in sketch.dev and send them back to the agent as feedback. Commenting directly on a line of a diff can greatly reduce the amount we have to type (and is very familiar from our long practice in the code review process). Overall, we are convinced that containers can be useful and warranted for programming. The idea has been around for a long time but I have never personally wanted to start programming in a container. But cleaning up a diff that an agent wrote for me in a container is far more interesting. The process of learning and experimenting with LLM-derived technology has been an exercise in humility. In general I love learning new things when the art of programming changes: dealing with the switch to multi-core programming, rethinking software design when SSDs replaced HDDs and dropped seek latencies, when everything was suddenly reachable on the same inter-network, these sorts of shifts in the industry are a joy to deal with. (Not to be confused with useless makework like the latest JavaScript framework, the latest cloud provider service, or the latest cluster orchestration software.) These sorts of challenges affected how my programs work: the choice of algorithms, languages, libraries, etc. But LLMs, and more specifically Agents, affect the process of writing programs in a new and confusing way. Absolutely every fundamental assumption about how I work has to be questioned, and it ripples through all the experience I have accumulated. There are days when it feels like I would be better off if I did not know anything about programming and started from scratch. And it is still changing. The way this all works today is very different than six months ago, and I do not believe we are at a stable point yet. I believe a lot of norms around team interactions will also be changing. For example, . It needs to be reinvented. The “IDE”, which has never been nearly as integrated as it has claimed to be, needs to be torn up and repurposed. The industry now seems aware of this, but has not taken an agent-first approach yet. There is a lot to do, and I suspect in six months things will be very different again. Curiosity and humility will get us through it, but even more than usual I would suggest turning away from internet forums where people talk in circles about this technology. That is a job for an agent.the mostly-broken process of half-hearted code review that has been adopted across the industry no longer solves the problems it barely solved before This post also appears on the .sketch blog Thanks to Sean McCullough, Philip Zeyliger, and Thomas Ptacek for reviewing this post. Define Agent Whiteboard programming Agents are LLMs with environmental feedback Example: Github App auth Example: SQL conventions around JSON “Asset” and “debt” models of code Why are we seeing agents now? What is next What does the IDE become? A final note bash(cmd) patch(hunks) todo(tasks) web_nav(url), web_eval(script), web_logs(), web_screenshot(), etc keyword_search(keywords) codereview() API use is greatly improved because the agent can web search for documentation and `curl` the docs into its context window. Compiler feedback reduces syntax errors and hallucinated interfaces. The compiler in a full development environment also improves dependency management, by helping the LLM understand features of the particular version of a dependency that a project uses. (This is an ongoing weakness of LLMs though, they will use documentation from a newer API version or make assumptions that apply only to older versions of a dependency. We have plans to address with this sketch.dev) Test failures help find errors in generated code, and have a positive reinforcing effect of getting the LLM to write tests for new code. LLMs can handle larger code bases than fit in the context window because they selectively choose which pieces of the codebase to read. Agents can try the end product themselves: run the code, take a screenshot of a page from the browser, feed it back into the model and keep tweaking the CSS based on the end-to-end rendering. When things go really wrong, read the server logs, find the panic, fix it and add a test. for install := range allAppInstallations { for r := range install.Repositories() { if r.IsCollaborator(user) { // add to available repositories } } } CREATE TABLE IF NOT EXISTS Cookie ( Cookie TEXT NOT NULL AS (Data->>'cookie') STORED UNIQUE, -- PK UserID INTEGER NOT NULL AS (Data->>'user_id') STORED REFERENCES User (UserID), Created INTEGER NOT NULL AS (unixepoch(Data->>'created')) STORED, LastUsed INTEGER AS (unixepoch(Data->>'last_used')) CHECK (LastUsed>0), Data JSONB NOT NULL );

How I program with LLMs 2025-01-06 This document is a summary of my personal experiences using generative models while programming over the past year. It has not been a passive process. I have intentionally sought ways to use LLMs while programming to learn about them. The result has been that I now regularly use LLMs while working and I consider their benefits net-positive on my productivity. (My attempts to go back to programming without them are unpleasant.) Along the way I have found oft-repeated steps that can be automated, and a few of us are working on building those into a tool specifically for Go programming: . It’s very early but so far the experience has been positive.sketch.dev I am typically curious about new technology. It took very little experimentation with LLMs for me to want to see if I could extract practical value. There is an allure to a technology that can (at least some of the time) craft sophisticated responses to challenging questions. It is even more exciting to watch a computer attempt to write a piece of a program as requested, and make solid progress. The only technological shift I have experienced that feels similar to me happened in 1995, when we first configured our LAN with a usable default route. We replaced the shared computer in the other room running Trumpet Winsock with a machine that could route a dialup connection, and all at once I had The Internet on tap. Having the internet all the time was astonishing, and felt like the future. Probably far more to me in that moment than to many who had been on the internet longer at universities, because I was immediately dropped into high internet technology: web browsers, JPEGs, and millions of people. Access to a powerful LLM feels like that. So I followed this curiosity, to see if a tool that can generate something mostly not wrong most of the time could be a net benefit in my daily work. The answer appears to be yes, generative models are useful for me when I program. It has not been easy to get to this point. My underlying fascination with the new technology is the only way I have managed to figure it out, so I am sympathetic when other engineers claim LLMs are “useless.” But as I have been asked more than once how I can possibly use them effectively, this post is my attempt to describe what I have found so far. There are three ways I use LLMs in my day-to-day programming: As this is about the of programming, this has been a fundamentally qualitative process that is hard to write about with quantitative rigor. The closest I will get to data is to say: it appears from my records that for every two hours of programming I do now, I accept more than 10 autocomplete suggestions, use LLM for a search-like task once, and program in a chat session once.practice The rest of this is about extracting value from chat-driven programming. Let me try to motivate this for the skeptical. A lot of the value I personally get out of chat-driven programming is I reach a point in the day when I know what needs to be written, I can describe it, but I don’t have the energy to create a new file, start typing, then start looking up the libraries I need. (I’m an early-morning person, so this is usually any time after 11am for me, though it can also be any time I context-switch into a different language/framework/etc.) LLMs perform that service for me in programming. They give me a first draft, with some good ideas, with several of the dependencies I need, and often some mistakes. Often, .I find fixing those mistakes is a lot easier than starting from scratch This means chat-based programming may not be for you. I am doing a particular kind of programming, product development, which could be roughly described as trying to bring programs to a user through a robust interface. That means I am building a lot, throwing away a lot, and bouncing around between environments. Some days I mostly write typescript, some days mostly Go. I spent a week in a C++ codebase last month exploring an idea, and just had an opportunity to learn the HTTP server-side events format. I am all over the place, constantly forgetting and relearning. If you spend more time proving your optimization of a cryptographic algorithm is not vulnerable to timing attacks than you do writing the code, I don’t think any of my observations here are going to be useful to you. Give an LLM a specific objective and all the background material it needs so it can craft a well-contained code review packet and expect it to adjust as you question it. There are two major elements to this: The ideal task for an LLM is one where it needs to use a lot of common libraries (more than a human can remember, so it is doing a lot of small-scale research for you), working to an interface you designed or produces a small interface you can verify as sensible quickly, and it can write readable tests. Sometimes this means choosing the library for it, if you want something obscure (though with open source code LLMs are quite good at this). You always need to pass an LLM’s code through a compiler and run the tests before spending time reading it. They all produce code that doesn’t compile sometimes. (Always making errors I find surprisingly human, every time I see one I think, there but for the grace of God go I.) The better LLMs are very good at recovering from their mistakes, often all they need is for you to paste the compiler error or test failure into the chat and they fix the code. There are vague tradeoffs we make every day around the cost of writing, the cost of reading, and the cost of refactoring code. Let’s take Go package boundaries as an example. The standard library has a package “net/http” that contains some fundamental types for dealing with wire format encoding, MIME types, etc. It contains an HTTP client, and an HTTP server. Should it be one package, or several? Reasonable people can disagree! So much so, I do not know if there is a correct answer today. What we have works, after 15 years of use it is still not clear to me that some other package arrangement would work better. Advantages of a larger package include: centralized documentation for callers, easier initial writing, easier refactoring, easier sharing of helper code without devising robust interfaces for them (which often involves pulling the fundamental types of a package out into yet another leaf package filled with types). The disadvantages include the package being harder to read because many different things are going on (try reading the net/http client implementation without tripping up and finding yourself in the server code for a few minutes), or it being harder to use because there is too much going on in it. For example I have a codebase that uses a C library in some fundamental types, but parts of the codebase need to be in a binary widely distributed to many platforms that does not technically need the C library, so have more packages than you might expect in the codebase isolating the use of the C library to avoid cgo in the multi-platform binary. There are no right answers here, instead we are trading off different types of work that an engineer will have to do (upfront and ongoing). LLMs influence those tradeoffs: Let me work an example to combine a few of the discussed ideas: Write a reservoir sampler for the quartiles of floats. First off, package structure. Were I doing this before LLMs, I would have chosen to have some sort of streamstat package that contained several algorithms, maybe one per file. This does not seem to be a unique opinion, here is an open source package following that model. Now, I want just this one algorithm in its own package. Other variants or related algorithms can have their own package.quantile Next up, what do we get from an LLM. The first pass is not bad. That prompt, with some details about wanting it in Go got me quartile_sampler.go: The core interface is good too: Great! There are also tests. An aside: this may be the place to stop. Sometimes I use LLM codegen as a form of specialized search. E.g. I’m curious about reservoir sampling, but want to see how the algorithm would be applied under some surprising constraint, for example time-windowed sampling. Instead of doing a literature search I might amend my prompt for an implementation that tracks freshness. (I could also ask it to include references to the literature in the comments, which I could manually check to see if it’s making things up or if there’s some solid research to work from.) Often I spend 60 seconds reading some generated code, see an obvious trick I hadn’t thought of, then throw it away and start over. Now I know the trick is possible. This is why it is so hard to attribute value generated by LLMs. Yes sometimes it makes bad code, gets stuck in a rut, makes up something impossible (it hallucinated a part of the monaco API I wish existed the other day) and wastes my time. It can also save me hours by pointing out something relevant I don’t know. Back to the code. Fascinatingly, the initial code produced didn’t compile. In the middle of the Quartiles implementation there was the line: Which is a fine line, sorted is a slice defined a few lines earlier. But the value is never used so gopls (and the Go compiler if you run go build) immediately says: This is a very easy fix. If I paste the error back into the LLM it will correct it. Though in this case, as I’m reading the code, it’s quite clear to me that I can just delete the line myself, so I do. Now the tests. I got what I expected. In quartile_sampler_test.go: Exactly the sort of thing I would write! I would run some cases through another implementation to generate expected outputs and copy them into a test like this. But there are two issues with this. The first is the LLM did run these numbers through another implementation. (To the best of my knowledge. When using a sophisticated LLM service, it is hard to say for sure what is happening behind the scenes.) It made them up, and LLMs have a reputation for being weak at arithmetic. So this sort of test, while reasonable for a human to write because we base it on the output of another tool, or if we are particularly old-school do some arithmetic ourselves, is not great from an LLM.not The second issue with this is we can do better. I am happy we now live in a time when programmers write their own tests, but we do not hold ourselves to the same standards with tests as we do with production code. That is a reasonable tradeoff, there are only so many hours in the day. But what LLMs lack in arithmetical prowess, they make up for in enthusiasm. Let’s ask for an even better test. This got us some new test code: The original test from above has been reworked to to use checkQuartiles and we have something new: This is fun, because it's wrong. My running tool immediately says:gopls Pasting that error back into the LLM gets it to regenerate the fuzz test such that it is built around a function that uses to extract floats from the data slice. Interactions like this point us towards automating the feedback from tools: all it needed was the obvious error message to make solid progress towards something useful. I was not needed.func(t *testing.T, data []byte)math.Float64frombits Doing a quick survey of the last few weeks of my LLM chat history shows (which as I mentioned earlier, is not a proper quantitative analysis by any measure) that more than 80% of the time there is a tooling error, the LLM can make useful progress without me adding any insight. About half the time it can completely resolve the issue without me saying anything of note, I am just acting as the messenger. There was a programming movement some 25 years ago focused around the principle “don’t repeat yourself.” As is so often the case with short snappy principles taught to undergrads, it got taken too far. There is a lot of cost associated with abstracting out a piece of code so it can be reused, it requires creating intermediate abstractions that must be learned, and it requires adding features to the factored out code to make it maximally useful to the maximum number of people, which means we depend on libraries filled with useless distracting features. The past 10-15 years has seen a far more tempered approach to writing code, with many programmers understanding it is better to reimplement a concept if the cost of sharing the implementation is higher than the cost of implementing and maintaining separate code. It is far less common for me to write on a code review “this isn’t worth it, separate the implementations.” (Which is fortunate, because people really don’t want to hear things like that after they have done all the work.) Programmers are getting better at tradeoffs. What we have now is a world where the tradeoffs have shifted. It is now easier to write more comprehensive tests. You can have the LLM write the fuzz test implementation you want but didn’t have the hours to build properly. You can spend a lot more time writing tests to be readable, because the LLM is not sitting there constantly thinking “it would be better for the company if I went and picked another bug off the issue tracker than doing this.” So the tradeoff shifts in favor of having more specialized implementations. The place where I expect this to be most visible is language-specific . Every major company API comes with dozens of these, usually low quality, wrappers written by people who aren’t actually using their implementations for a specific goal, instead are trying to capture every nook and cranny of an API in a large and complex interface. Even when it is done well, I have found it easier to go to the REST documentation (usually a set of curl commands), and implement a language wrapper for the 1% of the API I actually care about. It cuts down the amount of the API I need to learn upfront, and it cuts down how much future programmers (myself) reading the code need to understand.REST API wrappers For example, as part of my recent work on sketch.dev I implemented a Gemini API wrapper in Go. Even though the in Go has been carefully handcrafted by people who know the language well and clearly care, there is a lot to read to understand it:official wrapper My simplistic initial wrapper was 200 lines of code total, one method, three types. Reading the entire implementation is 20% of the work of reading the documentation of the official package, and if you decide to try digging into its implementation you will discover that it is a wrapper around another largely code-generated implementation with protos and grpc and the works. All I want is to cURL and parse a JSON object. There obviously comes a point in a project, where Gemini is the foundation of the entire app, where nearly every feature is used, where building on gRPC aligns well with the telemetry system elsewhere in your organization, where you should use the large official wrapper. But most of the time it is so much more time consuming, both upfront and ongoing, to do so given we almost always want only some wafer-thin sliver of whatever API we need to use today, that custom clients, largely written by a GPU, are far more effective for getting work done. So I foresee a world with far more specialized code, with fewer generalized packages, and more readable tests. Reusable code will continue to thrive around small robust interfaces and otherwise will be pulled apart into specialized code. Depending how well this is done, it will lead to either better software or worse software. I would expect both, with a long-term trend towards better software by the metrics that matter. As a programmer my instinct is to make computers do work for me. It is a lot of work getting value out of LLMs, how can a computer do it? I believe the key to solving a problem is not to overgeneralize. Solve a particular problem and then expand slowly. So instead of building a general-purpose UI for chat programming that is just as good at COBOL as it is for Haskell, we want to focus on one particular environment. The bulk of my programming is in Go, and so what I want is easy to imagine for a Go programmer: A few of us have built an early prototype of this: .sketch.dev The goal is not a “Web IDE” but rather to challenge the notion that chat-based programming even belongs in what is traditionally called an IDE. IDEs are collections of tools arranged for people. It is a delicate environment where I know what is going on. While an LLM is ultimately a developer tool, it is one that needs its own IDE to get the feedback it needs to operate effectively.I do not want an LLM spewing its first draft all over my current branch. Put another way: we didn’t embed goimports into sketch for it to be used by humans, but to get Go code closer to compiling using automatic signals, so that the compiler can provide better error feedback to the LLM driving it. It might be better to think of sketch.dev as a “Go IDE for LLMs”. This is all very recent work with a lot left to do, e.g. git integration so we can load existing packages for editing and drop the results on a branch. Better test feedback. More console control. (If the answer is to run sed, run sed. Be you the human or the LLM.) We are still exploring, but are convinced that focusing an environment for a particular kind of programming will give us better results than the generalized tool. Background Overview Why use chat at all? Chat-based LLMs do best with exam-style questions Extra code structure is much cheaper An example Where are we going? Better tests, maybe even less DRY Automating these observations: sketch.dev . This makes me more productive by doing a lot of the more-obvious typing for me. It turns out the current state of the art can be improved on here, but that’s a conversation for another day. Even the standard products you can get off the shelf are better for me than nothing. I convinced myself of that by trying to give them up. I could not go a week without getting frustrated by how much mundane typing I had to do before having a FIM model. This is the place to experiment first. Autocomplete . If I have a question about a complex environment, say “how do I make a button transparent in CSS” I will get a far better answer asking any consumer-based LLM, o1, sonnet 3.5, etc, than I do using an old fashioned web search engine and trying to parse the details out of whatever page I land on. (Sometimes the LLM is wrong. So are people. The other day I put my shoe on my head and asked my two year old what she thought of my hat. She dealt with it and gave me a proper scolding. I can deal with LLMs being wrong sometimes too.) Search . This is the hardest of the three. This is where I get the most value of LLMs, but also the one that bothers me the most. It involves learning a lot and adjusting how you program, and on principle I don’t like that. It requires at least as much messing about to get value out of LLM chat as it does to learn to use a slide rule, with the added annoyance that it is a non-deterministic service that is regularly changing its behavior and user interface. Indeed, the long-term goal in my work is to replace the need for chat-driven programming, to bring the power of these models to a developer in a way that is not so off-putting. But as of now I am dedicated to approaching the problem incrementally, which means figuring out how to do best with what we have and improve it.Chat-driven programming Avoid creating a situation with so much complexity and ambiguity that the LLM gets confused and produces bad results. This is why I have had little success with chat inside my IDE. My workspace is often messy, the repository I am working on is by default too large, it is filled with distractions. One thing humans appear to be much better than LLMs at (as of January 2025) is not getting distracted. That is why I still use an LLM via a web browser, because I want a blank slate on which to craft a well-contained request. Ask for work that is easy to verify. Your job as a programmer using an LLM is to read the code it produces, think about it, and decide if the work is good. You can ask an LLM to do things you would never ask a human to do. “Rewrite all of your new tests introducing an <intermediate concept designed to make the tests easier to read>” is an appalling thing to ask a human, you’re going to have days of tense back-and-forth about whether the cost of the work is worth the benefit. An LLM will do it in 60 seconds and not make you fight to get it done. Take advantage of the fact that .redoing work is extremely cheap As LLMs do better with exam-style questions, more and smaller packages make it easier to give a complete and yet isolated context for a piece of work. This is true for humans too, which is why we use packages at all, but we trade off package size against the extra typing/plumbing/filing to make more readable code. With an LLM both doing and benefiting from a big chunk of that extra work, the tradeoff shifts. (As a bonus, we humans get more readable code!) Smaller and more numerous packages can be compiled and tested independently of unrelated code. This is helpful for the LLM dev cycle as it means changes can be compiled and tested before other dependent packages are refactored, and the package has a simpler test environment. something like the Go playground, built around editing a package and tests with a chat interface onto editable code a little UNIX env where we can run go get and go test goimports integration gopls integration automatic model feedback: on model edit run go get, go build, go test, feedback missing packages, compiler errors, test failures to the model to try and get them fixed automatically // QuartileSampler maintains a reservoir sample to estimate quartiles of a stream // of float64 values. It uses reservoir sampling to maintain a representative // sample when processing large streams of data. type QuartileSampler struct { reservoir []float64 seen int maxSize int rng *rand.Rand } // NewQuartileSampler creates a new sampler that maintains up to maxSize samples. // A larger maxSize provides more accurate estimates but uses more memory. func NewQuartileSampler(maxSize int) *QuartileSampler { … } // Add adds a new value to the sampler. func (qs *QuartileSampler) Add(value float64) { … } // Quartiles returns the estimated 25th, 50th (median), and 75th percentiles // of the data seen so far. Returns (0,0,0) if no data has been added. func (qs *QuartileSampler) Quartiles() (q1, median, q3 float64) { if len(qs.reservoir) == 0 { return 0, 0, 0 } // Make a copy to avoid modifying the reservoir sorted := make([]float64, len(qs.reservoir)) copy(sorted, qs.reservoir) slices.Sort(sorted) n := len(sorted) q1 = percentile(sorted, 0.25) median = percentile(sorted, 0.50) q3 = percentile(sorted, 0.75) return q1, median, q3 } n := len(sorted) declared and not used: n func TestQuartileSampler(t *testing.T) { tests := []struct { name string data []float64 wantQ1 float64 wantMed float64 wantQ3 float64 epsilon float64 // allowed error for floating point comparison }{ { name: "basic sequence", data: []float64{1, 2, 3, 4, 5, 6, 7}, wantQ1: 2, wantMed: 4, wantQ3: 6, epsilon: 0.1, }, … } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { qs := NewQuartileSampler(100) // Use large reservoir for accurate testing for _, v := range tt.data { qs.Add(v) } q1, med, q3 := qs.Quartiles() if math.Abs(q1-tt.wantQ1) > tt.epsilon { t.Errorf("Q1 = %v, want %v (±%v)", q1, tt.wantQ1, tt.epsilon) } if math.Abs(med-tt.wantMed) > tt.epsilon { t.Errorf("Median = %v, want %v (±%v)", med, tt.wantMed, tt.epsilon) } if math.Abs(q3-tt.wantQ3) > tt.epsilon { t.Errorf("Q3 = %v, want %v (±%v)", q3, tt.wantQ3, tt.epsilon) } }) } } // referenceQuartiles calculates the exact quartiles for a slice of float64 values // using linear interpolation, matching the behavior expected from the sampler. func referenceQuartiles(data []float64) (q1, median, q3 float64) { … } // compareQuartiles checks if two sets of quartiles are within epsilon of each other. // Returns true if they match within the tolerance, false otherwise. func compareQuartiles(q1a, meda, q3a, q1b, medb, q3b, epsilon float64) bool { … } // checkQuartiles is a test helper that compares sampler output against the reference // implementation and reports any differences. func checkQuartiles(t *testing.T, data []float64, epsilon float64) { t.Helper() // Get reference values wantQ1, wantMed, wantQ3 := referenceQuartiles(data) // Get sampler values using a large reservoir for accuracy qs := NewQuartileSampler(1000) for _, v := range data { qs.Add(v) } gotQ1, gotMed, gotQ3 := qs.Quartiles() if !compareQuartiles(gotQ1, gotMed, gotQ3, wantQ1, wantMed, wantQ3, epsilon) { t.Errorf("Quartiles mismatch:\ngot (q1=%v, med=%v, q3=%v)\nwant (q1=%v, med=%v, q3=%v)\nepsilon=%v", gotQ1, gotMed, gotQ3, wantQ1, wantMed, wantQ3, epsilon) } } func FuzzQuartileSampler(f *testing.F) { // Add some seed corpus f.Add([]float64{1, 2, 3, 4, 5}) f.Fuzz(func(t *testing.T, data []float64) { // Use a larger epsilon for fuzzing since we might get more extreme values checkQuartiles(t, data, 0.2) }) } fuzzing arguments can only have the following types: string, bool, float32, float64, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, []byte $ go doc -all genai | wc -l 1155 In the tests, implement the simplest, most readable version of the standard code for quartiles over a fixed set of known values in a slice. Then pass the test cases through the standard code and the reservoir sampler and confirm they are within an epsilon of each other. Structure the comparison code such that it can be used in a fuzz test too.