More from Hixie's Natural Log
Over the past few months I've been writing notes for how I would write an LLM-based development tool. Specifically, giving these tools the power to run programs (especially compilers and tests) without having to either check every command for unexpected side-effects or trusting an LLM to do this checking for you, so as to avoid <a href="https://embracethered.com/blog/posts/2025/the-normalization-of-deviance-in-ai/">a disaster. This week, I dumped the <a href="https://github.com/Hixie/llmdevsilo/blob/main/docs/DESIGN.md">entire notes document into Anthropic's <a href="https://www.anthropic.com/news/claude-fable-5-mythos-5">Fable with a 1M context window and "Ultracode" mode, and gave it the following prompt: Implement this design in ~/dev/llmdevsilo/ Continue uninterrupted until it is complete. Two hours and about 2 million tokens later (approx $10 cost based on the plan I'm using), it had implemented the entire thing. There were cosmetic issues, to be sure. The native UI overflowed on small window sizes, the web client needed a tweak to work around browsers not liking the self-signed cert on the local websocket, that kind of thing. But overall it had implemented the entire thing. One shot, first attempt. (I'm not counting the two false starts caused by bugs in the Claude desktop app; those attempts were canceled within the first few minutes and didn't influence the prompt of the actual attempt.) Fable is... well, <a href="https://simonwillison.net/2026/Jun/11/fable-is-relentlessly-proactive/">relentlessly proactive is pretty accurate. So, I can now present <a href="https://github.com/Hixie/llmdevsilo/tree/main">llmdevsilo, or just "Silo" as Claude decided to call it. The details are in the design doc (and the additional documentation Claude generated, some as a result of additional prompting later), but at a high level: You launch a desktop app, and tell it to start a session ("harness"), pointing it at a directory and an LLM model. Currently it supports the Anthropic REST API, the Open AI REST and WebSocket APIs, and local models. (I've no idea how the local model stuff works; the design notes just said to support it and something was implemented for it, but I haven't tested it.) A harness process is instantiated. The directory you specified gets locked behind a disk image and mounted inside a sandbox. The only sandbox I've tested so far is macOS sandbox-exec, which seems to be the state of the art on Mac (despite being deprecated). On Linux it supports gVisor, but I haven't tested that yet. You can now talk to the LLM, and the LLM can run programs in the sandbox (it is given a set of tools similar to what the Claude interfaces implement, like Read/Write, Bash, etc). It cannot read your local files, only what was in the directory you gave it and a select set of binaries (like those in /usr/bin). The sandbox is also provided with a scratch directory. By default, any code running in the sandbox (and thus the LLM) can't access the network. If configured, the harness can provide the sandbox with network access via an HTTP proxy. The proxy supports TLS, there's a whole mechanism whereby a freshly minted temporary root CA is injected into the sandbox and the proxy generates certificates on the fly. Under gVisor, DNS is also proxied; only allow-listed names can be resolved. What all this means is you can use an LLM with untrusted third party dependencies, give it access to the web, give it a real dev environment with compilers, and be safe in the knowledge that none of your secrets will be <a href="https://medium.com/@nem841425bal/openais-codex-leaks-secrets-28f0bd749cc3">exfiltrated, your <a href="https://www.reddit.com/r/vibecoding/comments/1r96647/gpt_53_codex_wiped_my_entire_f_drive_with_a/">hard disk won't be wiped, and you never need to see a permissions prompt again, so no risk of prompt fatigue. This isn't the inherently imperfect "model watches the model" security, it's enforced by real sandbox boundaries. A core part of the design that I like is that the UI is separate from the harness. You can run the harness on one computer and connect to it from another, or from your phone. There is a Flutter-based app UI that works as a native app on desktop and mobile (so far only tested on macOS) and also works on web. There is a Rust-based terminal client. In principle there could be any number of other apps too. The UIs have a secure mechanism to connect (by design; as with everything else, I haven't studied the implementation). To add another client, you get a pairing code from an existing client and connect; they then mint asymmetric key pairs for reconnecting securely later. There are some limitations. The biggest is that I haven't audited any of this code yet. It could be full of holes. The entire codebase is itself written by an LLM. The security-sensitive parts are mainly in Rust and I am not fluent in Rust, nor in the security technologies it attempts to leverage and implement. Another big one is that if you're not able to use local models, you will need to use one of the APIs, which are a lot more expensive than the plans. For example, eyeballing the price tables, I think the original one-shot to create this would have cost about $100-$150 dollars in about two hours. Open AI and Claude only seem to allow you to use their plans with their own software, custom software is billed by the token. The final limitation worth being explicit about is that while everything outside the sandbox is supposed to be safe from exfiltration, if you allow any network egress then the contents of the sandbox are not. This matters if what you're developing isn't going to be open source, for example. I am happy to accept patches, but I may be slow to adopt them. Feel free to fork and run with this if you are interested. I have not finished tweaking it, I decided to post this in its current state because I ran out of tokens, and it seemed interesting enough to publish as is. The <a href="https://github.com/Hixie/llmdevsilo/commit/a252a24407a64aab2c7a08624ea13165787c5ea2">first commit in the repo is what the original prompt generated; subsequent commits are from additional discussions with the agent.
Over the past few months, I've been putting together some notes for future creators of UI frameworks, based on my experiences of the past 25-30 years. The work was originally commissioned by a customer for their use, but it seems like it might be of wider interest so I've decided to publish it. The primary copy is a Google doc, available at https://software.hixie.ch/ui-frameworks, on which people can leave comments. There is also a canonical PDF copy of this first edition of the document if that is a more convenient format for anyone. As always, I welcome feedback. The most helpful way to leave comments is on the Google doc itself.
When you build something, you have to pick some design goals and priorities. Ideally you do so explicitly, but even if you don't, you're still implicitly doing so based on your design choices. These choices are trade-offs. If you want to write a quiet song, it won't be loud. If you are writing a software tool and you want to prioritize speed over simplicity, then it won't be as simple as if you'd prioritized simplicity over speed. There are two main signs that you've succeeded at your goals. The first, and more pleasant, is that you get compliments about how your thing is like you wanted it to be. "I love that song, it's so quiet!" "Your tool is so fast!" Why thank you, that's exactly what I was going for. The second sign, though, is that you will get complaints. Specifically, people will complain that your thing does not achieve the things you didn't set out to achieve. "I wish this song was louder", "this tool is so hard to use". That you are receiving complaints at all means that people are aware of your creation; that they are complaining about what you specifically set out to make a non-goal is a side-effect of the fact that you made that trade-off. The worst thing to do, when you receive such complaints, is take them to heart and try to fix them. This is because by definition you wanted these complaints. They are a sign that the thing you built is built as you wanted to build it. The people complaining want something different, they don't want your thing. It's just that your thing is so good that it's the thing they're compelled towards even though it doesn't prioritize the things they care about most. If you try to fix these complaints, you will, again by definition, be compromising on your goals. If you make the song have a loud part, then it's no longer a quiet song. You wanted a quiet song. Now it's a song that's quiet in parts and loud in parts. It probably still doesn't satisfy the needs of the people who want a loud song, and now it also doesn't satisfy the needs of the people who wanted your original quiet song. If you make your tool easier to use by compromising on the speed, then now you have a tool that's both not as fast as it could be and not as usable as it could have been if you'd started with that as a goal. It's important, therefore, to separate out complaints into those that are complaints you expect based on your design goals (which you should acknowledge but not fix), vs complaints that are either orthogonal to your goals (which you can fix without compromising your goals), or that are in line with your goals (which you should prioritize, since that's what you said to yourself was most important in the first place).
My involvement in web standards started with the CSS working group. One of the things that we struggled with as a working group was that we would specify how the technology should work, but the browser vendors' implementations weren't exactly what we intended, and web authors would then write web pages that worked with those browsers, even though that meant the web pages themselves were also not doing things like the specifications said they should. The folks I worked with at the W3C (especially the academics and people working for organizations that did not themselves implement browsers) would frequently bemoan this state of affairs, expressing surprise at how they, the people in charge of the standards, were not being respected by the people implementing the standards. One of the key insights I had very early on in my work, before working on HTML5, which really influenced the WHATWG and its work, is the realization that the power dynamics at work were not at all the power dynamics that the folks at the W3C described. The reality of the situation was that the power lay entirely in the hands of the users. The users chose browsers. A browser vendor that ignored what the users wanted would lose market share. Market share is everything in this space. Browser vendors want users because they can convert users into dollars (in various ways, but they typically boil down to someone showing them ads and paying the browser vendors for the privilege). In turn, the browser vendors had more power than the specifications. What they implement is, by definition, what the technology is. The specification can say in absolute clarity that the keyword "marigold" should look yellow, but if a browser vendor makes it look red, then no web author is going to use it to mean yellow, and many will use it to mean red. There is a feedback loop here: if one browser implements "marigold" to mean red, and some important web site (or many unimportant web sites) rely on it, and say something like "best viewed in ThisOrThat browser!" because that's the one they use and in that browser it looks red and red is what looks best, then the other browser vendors are incentivised to make sure that the web page looks good in their browser too. Regardless of what the specification says, therefore, they are going to make "marigold" look red and not yellow. When I realized this, I also realized a corollary: if you have two competing specifications that both claim to define the same technology, but one matches what the browsers already do while the other one does not, the browser vendors are going to find it more useful to follow the one that matches what they do. This is because they can trust that implementing that specification will get them more market share. It means they won't have to stop and think at every step, "will following this specification cause me to lose users?". It is easier for them to use a specification that takes into account their needs in this way. We actually tried to explain this to the W3C membership. There was a big meeting in 2004 at Adobe in San Jose, the "W3C Workshop on Web Applications and Compound Documents". We tried to convey the above (I didn't quite understand it in the stark "power dynamic" terms yet, or at least, I didn't really express it in those terms, but if you read our position paper you can see this insight starting to crystalize). At this meeting, we made a pitch for the W3C to continue to maintain HTML and to care about what the browser vendors wanted. Representatives from Microsoft and Sun (in many ways arch enemies at the time) supported us. I seem to recall Apple being more quiet about it at the meeting but also essentially supporting the principles. The W3C membership resoundly rejected this whole concept. One of the W3C staff even explicitly said something along the lines of "if you want to do this you should do it elsewhere". That's what led to the WHATWG being founded a few weeks later. The WHATWG was founded on this core principle — the specifications need to actually specify reality. When the browsers disagree with the spec, the spec is by definition incorrect and needs to change, regardless of how much technically superior the design in the spec is. Naturally, when you provide browser vendors with something that valuable, they will follow. You end up with a weird inverted power dynamic. The spec writer (when they follow this principle) has all the power, but only within the space that the browser vendors are themselves willing to play; and the browser vendors have all the power, but only within the space that the users are willing to put up with. It's very easy to appear to be in control when you tell people to do the thing they were going to do anyway (or at least, one of the things they were willing to do if they were to think about it). There is a (probably apocryphal) quote supposedly by Alexandre Auguste Ledru-Rollin that is often cited in mockery of bad leadership, but that perfectly matches the power dynamic here: "There go my people; I must find out where they are going so I can lead them". (Thanks to Leonard Damhorst for prompting me to write this post.)
More in programming
My now-traditional “summary” of the conference along with my own commentary and thoughts.
Every year, browser vendors collaborate via the Interop project to synchronize their efforts on new HTML/CSS/JS features. This year, there’s a ton of super exciting features being considered! In this blog post, we’ll explore the CSS of the future, to get a glimpse of what we’ll soon be able to do.
Imprint’s average engineer is merging 4x more PRs than 2 years ago. Yay. Aggregated behind the average is something even more interesting: we have two teams that are merging 10x their PRs over that period. One engineer is merging 25x the PRs per week of our average engineer two years ago, and a number are merging 10x. That’s pretty neat, and the slope of the line suggests our average in a year is going to look much closer to today’s p90 than today’s p50. Even more importantly, we aren’t just shipping more, our quality has improved: incidents are down 42% half-over-half, and critical incidents are down 78% half-over-half. I think we’ve always known the quality/velocity tradeoff wasn’t a particularly useful construct, but that’s undeniable today. As we’ve walked this path, we–like everyone else progressing down it–have run into the problem of pull request review. The p50 pull request review took about 33 minutes to complete 2 years ago, but now takes us about 77 minutes. The slope here correlates tightly with the number of pull requests getting merged. Tightly enough that addressing it became one of my priorities for the week. We solved it! Just kidding, we’ve absolutely not solved it, but I think our current approach is an interesting one, which we call the small-change. The work discussed here was shaped and done by many folks at Imprint including Dan, Joy, Daniel, Lenny, Taylor, Ted, and Annie, among many others. Why review pull requests? There are four reasons we review pull requests: Knowledge sharing between author and reviewer – software engineering is generally learned via apprenticeship, and reviews are a primary mechanism of experienced engineers teaching others what good software looks like. We believe the current volume means that this knowledge sharing will have to move towards operating on clusters of pull requests, rather than individual pull requests Preventing errors – review has always been an expensive time to catch errors, and we believe that effective CICD along with harness verification techniques like poteto’s pstack are more effective mechanisms for preventing errors than review Preventing malicious behavior – my experience is that even careful code review will generally miss competent malicious behavior, although fortunately most malicious behavior is incompetent. We believe both sorts of behavior are best identified using a combination of fixed rules (modifying these specific objects/classes/files requires review), anti-abuse mechanisms that we’ve refined in our LLM-based member support experiences, auditing, and audit log inspection Compliance obligations in our documented controls – these can be changed, but today they require that an independent human reviews changes in a subset of our repositories In the modern development paradigm, we think there is a better solution for each of these problems. But, we also want to make these changes very cautiously. Small steps with small-change We believe we move fast by finding narrow, contained ways to validate new practices, so to iterate our way into an effective solution, we’ve developed the concept of a small-change. Here’s how it works: When a pull request is marked ready, CICD determines whether it is eligible to be a small-change. This is represented as a passing Github check. There is also a label for visibility, but it is the check itself, managed by the build and not human modifiable, that determines eligibility. (The label is used for initial filtering, but the check is still independently verified.) Something is a small-change if it meets all of these criteria: It is less than 1,000 lines of code There are no p1 or p2 findings from our ai-code-review automated review bot The author has “experience” in the service or application, defined as 10+ prior merged pull requests into that service or application’s directories. For changes working across multiple services or applications, the author must have experience in all of them to qualify Impacts fewer than N distinct services or applications. N is 1 on our frontend monorepo, and N is 3 in our backend monorepo, where typical changes will modify both a gRPC service and a common HTTP gateway Shared libraries and utilities files are not eligible for small-change Each repository has a .small_change file which allows per-path customization while we continue to tune the behaviors. For example, a service dealing with money movement can be excluded Our Slack automation tags every reference to a pull request with reacji labeling the state of that pull request, for example if it has failing tests, if it has been approved but tests are still running, or if it is ready to merge. We added a small-change reacji that indicates which of those pull requests meet the small-change criteria In our internal skills repository, we created two skills that have been distributed across harnesses: review-small-change which takes a list of repositories and reviews all open, non-draft, small-changes, including providing approval if they meet the review’s criteria (all checks pass including small-change check, and not touching on sensitive topics such as payments and security) review-pr-cluster which reviews batches of pull requests to identify common themes that came up, and suggest opportunities for discussion and adding verifications We set the social expectation that any error caused by a small-change is a failure in our system, and has to be solved at the system level: improvements to ai-code-review, to our verification mechanisms, to our CICD, to our harness, etc. This took about two days of infrequent attention to implement and rollout, although I imagine we will be tuning the knobs here for a long time. Maybe forever. fleet: centrally orchestrated harness I’ll briefly mention our other approach to reducing code review volume, which anchors on our centrally-orchestrated harness, fleet. (Which is similar to Stripe’s Minions although not following the one-shot approach in particular). We rely on fleet to implement and verify software in a highly controlled harness running in the cloud, with independent evaluation mechanisms to identify the correct number of independent reviewers for any given change. That number can be as low as one (e.g. fleet writes, one human reviews) for changes evaluated as safe. Results / next steps / reception I’ll be honest, it’s too early to give a useful result here. From early results, about 10% of changes qualify as a small-change. Those are now eligible for very fast approval, and generally get it unless they trip a flag, generally due to touching a security or money adjacent topic. While 10% is a small percentage, the core concept here is working, and ultimately will succeed or fail based on our ability to expand that percentage over the next few months in tandem with expanding our verification mechanisms to be increasingly comprehensive. This is ultimately going to fail or succeed depending on our ability to improve our verifications, which does require more difficult sorts of techniques to accomplish in low scale-complexity, high domain-complexity software than in the high scale-complexity, low domain-complexity products where many techniques like incremental rollouts are exceptionally effective. (Don’t get me wrong, they still work, they just provide less coverage when most users have different configuration and behavior.) Folks have not been uniformly excited about this, it’s a substantial change from how we’ve learned to be good engineers, and it’s uncomfortable to reevaluate whether a long-held best practice has become merely dogma. The most energizing part of the current cycle is that we get to reevaluate these practices we grew up on every week; the most painful part of the current cycle is that we’re likely to fall behind if we ever stop reevaluating them.
The park near my house has lots of chestnuts and my kids have been collecting them like crazy (it’s actually their favorite post-school activity). We accumulated quite a hoard. I thought it would be fun to make a launcher so we had something to do with our new chestnut collection. I wanted to go with a very simple crossbow-style design (like the famous D2 item) using latex tubing and a leather pouch for holding the chestnuts. This is what I came up with. It launches chestnuts about 220 feet, and it’s simple enough to build in an afternoon. The frame is a 42 inch 2x2 and a spare IKEA bed slat for the crossbeam, glued and screwed together. A 3/8 inch eye bolt goes through each end of the crossbeam, and the latex tubing is tied to those and to the pouch from a leather shepherd’s sling. Tying the tubing was the fiddliest part. It has to be stretched while you wrap it with butcher’s twine, so I clamped a couple of scrap blocks to my workbench to hold it. I like this design because the tubing can be wrapped around the eye bolts to easily add more tension and power. The Trigger My favorite part of the design was that I decided to use an archery release like this one for the trigger. It provides a safe and reliable way to hold the tubing back, and it’s made to let go cleanly. I 3D printed a little housing that mounts the release securely to the back of the stock, and the release clips onto a loop of archery D-loop cord on the back of the pouch. Then I added a dowel for a rear grip and an old handle I had previously removed from my shop vac. How Well Does It Work? We tried it at the park and it works surprisingly well. Aimed at about 35 to 40 degrees, it sends our chestnuts (on the small side, about 10 to 15 grams) about 220 feet. Thanks to the archery release, we haven’t had a single misfire. I got curious and did some rough math. To go that far with air drag, the chestnuts are leaving at somewhere around 85 to 100 mph, which works out to about the same energy as a paintball. So safety glasses are important and you never point it at anyone. You can find the 3D print on Printables and a full build guide on Instructables. Make sure to always use it with adult supervision :)