More from Daniel Marino
Beware! This post includes spoilers! I recently built an escape room game called CSScape Room. This isn’t my first JavaScript-free web game, but HTML and CSS have evolved significantly since my previous attempts, with newer additions allowing for more complex selectors and native interactions. Rather than saving this idea for a game jam, I built it purely for fun, which freed me from theme constraints and time pressure. I’ve enjoyed escape room games since childhood, and it was nostalgic to recreate that experience myself. This project pushed my artistic limits while challenging me to design puzzles and translate them into complex HTML and CSS. The learning process was fun, challenging, and sometimes tedious—mostly through trial and error. Process My creative process isn’t linear—it’s a blend of designing, puzzle creation, and coding that constantly influences each other. I frequently had to redesign or recode elements as the project evolved. There was also that time I accidentally deleted half my CSS because I wasn’t backing up to GitHub... lesson learned! 😬 This might sound chaotic, and honestly, it was. If you’re wondering where to start with a project like this, I began by prototyping the room navigation system. I figured that was the minimum viable feature—if I couldn’t make that work, I’d abandon the project. The solution I eventually found seems simple in retrospect, but I went through several iterations to discover it. This flexible approach makes sense for my creative projects. As I build something, both the in-progress work and my growing skills inevitably influences the entire project. I’m comfortable with this non-linear process—it also suits my ADHD brain, where I tend to lose interest if I work on the same thing for too long. Artwork I’d wanted to design a pixel art-styled game for some time but never felt confident enough to attempt it during a game jam because of the learning curve. I watched tutorials from Adam Yunis and Mort to get a crash course in pixel art best practices. Initially, progress was slow. I had to figure out 2D perspective with vanishing points, determine a color palette, practice shading techniques, and decide how much detail to include. While I tried to adhere to pixel art “rules,” I definitely broke some along the way. One challenge I set for myself was using only 32 colors to capture the feeling of an older gaming console. Once I got comfortable with shading and dithering, working within this constraint became easier. An added benefit to using 32 colors was it resulted in smaller image sizes—the game’s 79 images account for only about 25% of the total payload. I attempted to design sprites using dimensions in multiples of eight, but I’ll admit I became less strict about this as the project progressed. At a certain point, I was struggling enough with the color and styling limitations that this guideline became more of a starting point than a rule. I considered creating my own font, but after exhausting myself with all the artwork, I opted for Google’s PixelifySans instead. Almost all animation frames were individually drawn (except for the “one” TV animation). This was tedious, but I was determined to stay true to old-school techniques! I did use CSS to streamline some animations—for instance, I used animation-direction: alternate on the poster page curl to create a palindrome effect, halving the number of required sprites. Mechanics Like my previous game Heiro, this project primarily uses checkbox and radio button mechanics. However, the addition of the :has() pseudo-selector opened up many more possibilities. I also utilized the popover API to create more detailed interactions. Checkbox and Radio Selection Triggering interactions by toggling checkboxes and radio buttons isn’t new, but the :has() selector is a game-changer! Before this existed, you had to structure your markup so interactive elements were siblings. The :has() selector makes this far more flexible because you no longer need to rely on a specific HTML structure. #element { display: none; } :has(#checkbox:checked) #element { display: block; } Using this pattern, :has() looks for #checkbox anywhere on the page, meaning you don’t have to rely on #checkbox, its corresponding <label>, or #element being siblings. The markup structure is no longer a constraint. Most of this game functions on toggling checkboxes and radios to unlock, collect, and use items. Navigation I almost gave up on the current implementation, and used basic compass notation to avoid visual transitions between directions. After several failed attempts, I found a solution. The tricky part was determining how to transition into a direction from either left or right, depending on which arrow was clicked. My solution is conceptually simple but difficult to explain! First, I used radio buttons to determine which direction you’re facing (since you can only face one direction at a time). Second, I needed a way to determine if you’re entering a direction from west or east. This required eight radio buttons—two for each direction. For example, if you’re facing east (having come from facing north), you have two possible directions to go: west (returning to face north) or east (to face south). I needed to make the radio buttons visible that would take you north from east, and south from west. The CSS looks something like this: :has(#east-from-west:checked) :is( [for="south-from-west"], [for="north-from-east"]) { display: block; } This pattern was implemented for each direction, along with animations to ensure each room view slid in and out correctly. Zooming In I initially focused so much on checkbox mechanics that I assumed I’d need the same approach for zooming in on specific areas. Then I had a "Duh!" moment and realized the popover API would be perfect. Here’s the basic markup for looking at an individual book: <button popovertarget="book">Zoom in</button> <div id="book" popover> <!-- Book content goes here --> <button popovertarget="book" popovertargetaction="hide">Close</button> </div> Turning the Lights Off I procrastinated on implementing this feature because I thought I’d need to create darkened variations of all artwork. I don’t recall what inspired me to try blend modes, but I’m glad I did—the solution was surprisingly simple. When the light switch checkbox is toggled, a <div> becomes visible with a dark background color and mix-blend-mode: multiply. This multiplies the colors of the blending and base layers, resulting in a darker appearance. Playing the Crossword This required surprisingly complex CSS. Each square has three letters plus a blank tile, meaning four radio buttons. The :checked letter has a z-index of 3 to display above other letters, but also has pointer-events: none so clicks pass through to the next letter underneath (with z-index: 2). The remaining tiles have a z-index of 1. The CSS becomes even trickier when the last tile is :checked, requiring some creative selector gymnastics to target the first radio button in the stack again. Tools I created all artwork using Aseprite, which is specifically designed for pixel art. I probably only used a fraction of its features, and I’m not sure it actually made my life easier—it might have made things more difficult at times. I’m not giving up on it yet, though. I suspect I’ll occasionally discover features that make me think, “Oh, that’s way easier than what I was doing!” I started coding with basic HTML and CSS but eventually found navigation difficult with such a long HTML file. It also became tedious writing the same attributes for every <img /> element. I migrated the project to Eleventy to improve organization and create custom shortcodes for simplifying component creation. I used the html-minifier-terser npm package, which integrates well with Eleventy. I chose native CSS over Sass for several reasons: CSS now has native nesting for better organization and leaner code CSS has built-in variables HTTP/2 handles asset loading efficiently, eliminating the major benefit of bundling CSS files The game uses 12 CSS files with 12 <link rel="stylesheet" /> tags. The only Sass feature I missed was the ability to loop through style patterns for easier maintenance, but this wasn’t a significant issue. The game is hosted on GitHub Pages. During deployment, it runs an npm command to minify CSS using Lightning CSS. I mentioned accidentally deleting half my CSS earlier—this happened because I initially used Eleventy’s recommended approach with the clean-css npm package. I strongly advise against using this! This package doesn’t work with native CSS nesting. While losing code was frustrating, I rewrote much of it more efficiently, so there was a silver lining. Nice to Haves I initially wanted to make this game fully accessible, but the navigation system doesn’t translate well for screen reader users. I tried implementing a more compass-like navigation approach for keyboard users, but it proved unreliable and conflicted with the side-to-side approach. Adding text labels for interactive elements was challenging because you can’t track the :focus state of a <label> element. While you can track the :focus of the corresponding <input />, it wasn’t consistently reliable. The main keyboard accessibility issue is that the game exists as one long HTML page. When you navigate to face a different direction, keyboard focus remains elsewhere on the page, requiring extensive tabbing to reach navigation elements or item selection. I ultimately decided to make the game deliberately inaccessible by adding tabindex="-1" to all keyboard-accessible elements. I’d rather users recognize immediately that they can’t play with assistive technology than become frustrated with a partially broken experience. Sound would have been a nice addition, but I encountered the same issues as with my previous game Heiro. You can toggle the visibility of an <embed> element, but once it’s visible, you can’t hide it again—meaning there’s no way to toggle sound on and off. Conclusion CSScape Room was a fun but exhausting four-month project. It began as an experiment to see if creating a JavaScript-free escape room was possible—and the answer is definitely yes. I’ve only touched on some aspects here, so if you’re interested in the technical details, check out the source code on GitHub. Finally, I’d like to thank all my playtesters for their valuable feedback!
I’m a bit late to this, but back in summer 2024 I participated in the OST Composing Jam. The goal of this jam is to compose an original soundtrack (minimum of 3 minutes) of any style for an imaginary game. While I’ve composed a lot of video game music, I’ve never created an entire soundtrack around a single concept. Self Avoiding Walk by Daniel Marino To be honest, I wasn’t entirely sure where to start. I was torn between trying to come up with a story for a game to inspire the music, and just messing around with some synths and noodling on the keyboard. I did a little bit of both, but nothing really materialized. Synth + Metal ≈ Synthmetal Feeling a bit paralyzed, I fired up the ’ole RMG sequencer for inspiration. I saved a handful of randomized melodies and experimented with them in Reaper. After a day or two I landed on something I liked which was about the first 30 seconds or so of the second track: "Defrag." I love metal bands like Tesseract, Periphery, The Algorithm, Car Bomb, and Meshuggah. I tried experimenting with incorporating syncopated guttural guitar sounds with the synths. After several more days I finished "Defrag"—which also included "Kernel Panic" before splitting that into its own track. I didn’t have a clue what to do next, nor did I have a concept. Composing the rest of the music was a bit of a blur because I bounced around from song to song—iterating on the leitmotif over and over with different synths, envelopes, time signatures, rhythmic displacement, pitch shifting, and tweaking underlying chord structures. Production The guitars were recorded using DI with my Fender Squire and Behringer Interface. I’m primarily using the ML Sound Labs Amped Roots Free amp sim because the metal presets are fantastic and rarely need much fuss to get it sounding good. I also used Blue Cat Audio free amp sim for clean guitars. All the other instruments were MIDI tracks either programmed via piano roll or recorded with my Arturia MiniLab MKII. I used a variety of synth effects from my library of VSTs. I recorded this music before acquiring my Fender Squire Bass guitar, so bass was also programmed. Theme and Story At some point I had five songs that all sounded like they could be from the same game. The theme for this particular jam was "Inside my world." I had to figure out how I could write a story that corresponded with the theme and could align with the songs. I somehow landed on the idea of the main actor realizing his addiction to AI, embarking on a journey to "unplug." The music reflects his path to recovery, capturing the emotional and psychological evolution as he seeks to overcome his dependency. After figuring this out, I thought it would be cool to name all the songs using computer terms that could be metaphors for the different stages of recovery. Track listing Worm – In this dark and haunting opening track, the actor grapples with his addiction to AI, realizing he can no longer think independently. Defrag – This energetic track captures the physical and emotional struggles of the early stages of recovery. Kernel Panic – Menacing and eerie, this track portrays the actor’s anxiety and panic attacks as he teeters on the brink during the initial phases of recovery. Dæmons – With initial healing achieved, the real challenge begins. The ominous and chaotic melodies reflect the emotional turbulence the character endures. Time to Live – The actor, having come to terms with himself, experiences emotional growth. The heroic climax symbolizes the realization that recovery is a lifelong journey. Album art At the time I was messing around with Self-avoiding walks in generative artwork explorations. I felt like the whole concept of avoiding the self within the context of addiction and recovery metaphorically worked. So I tweaked some algorithms and generated the self-avoiding walk using JavaScript and the P5.js library. I then layered the self-avoiding walk over a photo I found visually interesting on Unsplash using a CSS blend mode. Jam results I placed around the top 50% out of over 600 entries. I would have liked to have placed higher, but despite my ranking, I thoroughly enjoyed composing the music! I’m very happy with the music, its production quality, and I also learned a lot. I would certainly participate in this style of composition jam again!
Over the past couple of years I’ve gotten into journaling. Recently I’ve been using a method where you’re given a single inspirational word as a prompt, and go from there. Unfortunately, the process of finding, saving, and accessing inspirational words was a bit of a chore: Google a list of “366 inspirational words”. Get taken to a blog bloated with ads and useless content all in an effort to generate SEO cred. Copy/paste the words into Notion. Fix how the words get formatted becasue Notion is weird, and I have OCD about formatting text. While this gets the job done, I felt like there was room to make this a more pleasant experience. All I really wanted was a small website that serves a single inspirational word on a daily basis without cruft or ads. This would allow me to get the content I want without having to scroll through a long list. I also don't want to manage or store the list of words. Once I've curated a list of words, I want to be done with it. Creating a microsite I love a good microsite, and so I decided to create one that takes all the chore out of obtaining a daily inspirational word. The website is built with all vanilla tech, and doesn’t use any frameworks! It’s nice and lean, and it’s footprint is only 6.5kb. Inspirational words While I’m not a huge fan of AI, I did leverage ChatGPT on obtaining 366 inspirational words. The benefit to ChatGPT was being able to get it to return the words as an array—cutting down on the tedium of having to convert the words I already had into an array. The words are stored in it’s own JSON file, and I use an async/await function to pull in the words, and then process the data upon return. Worth the effort I find these little projects fun and exciting because the scope is super tight, and makes for a great opportunity to learn new things. It’s definitely an overengineered solution to my problem, but it is a much more pleasant experience. And perhaps it will serve other people as well.
Over the past couple of years I’ve gotten into journaling. Recently I’ve been using a method where you’re given a single inspirational word as a prompt, and go from there. Unfortunately, the process of finding, saving, and accessing inspirational words was a bit of a chore: 1. Google a list of “366 inspirational words”. 2. Get taken to a blog bloated with ads and useless content all in an effort to generate SEO cred. 3. Copy/paste the words into Notion. 4. Fix how the words get formatted becasue Notion is weird, and I have OCD about formatting text. While this gets the job done, I felt like there was room to make this a more pleasant experience. All I really wanted was a small website that serves a single inspirational word on a daily basis without cruft or ads. This would allow me to get the content I want without having to scroll through a long list. I also don't want to manage or store the list of words. Once I've curated a list of words, I want to be done with it. ## Creating a microsite I love a good microsite, and so I decided to create one that takes all the chore out of obtaining a [daily inspirational word](https://starzonmyarmz.github.io/daily-inspirational-word/).  The website is built with all vanilla tech, and doesn’t use any frameworks! It’s nice and lean, and it’s footprint is only 6.5kb. ### Inspirational words While I’m not a huge fan of AI, I did leverage ChatGPT on obtaining 366 inspirational words. The benefit to ChatGPT was being able to get it to return the words as an array—cutting down on the tedium of having to convert the words I already had into an array. The words are stored in it’s own JSON file, and I use an async/await function to pull in the words, and then process the data upon return. ## Worth the effort I find these little projects fun and exciting because the scope is super tight, and makes for a great opportunity to learn new things. It’s definitely an overengineered solution to my problem, but it is a much more pleasant experience. And perhaps it will serve other people as well.
More in programming
<![CDATA[It has been a year since I set up my System76 Merkaat with Linux Mint. In July of 2024 I migrated from ChromeOS and the Merkaat has been my daily driver on the desktop. A year later I have nothing major to report, which is the point. Despite the occasional unplanned reinstallation I have been enjoying the stability of Linux and just using the PC. This stability finally enabled me to burn bridges with mainstream operating systems and fully embrace Linux and open systems. I'm ready to handle the worst and get back to work. Just a few years ago the frustration of troubleshooting a broken system would have made me seriously consider the switch to a proprietary solution. But a year of regular use, with an ordinary mix of quiet moments and glitches, gave me the confidence to stop worrying and learn to love Linux. linux a href="https://remark.as/p/journal.paoloamoroso.com/my-first-year-since-coming-back-to-linux"Discuss.../a Email | Reply @amoroso@oldbytes.space !--emailsub--]]>
The mystery In the previous article, I briefly mentioned a slight difference between the ESP-Prog and the reproduced circuit, when it comes to EN: Focusing on EN, it looks like the voltage level goes back to 3.3V much faster on the ESP-Prog than on the breadboard circuit. The grid is horizontally spaced at 2ms, so … Continue reading Overanalyzing a minor quirk of Espressif’s reset circuit → The post Overanalyzing a minor quirk of Espressif’s reset circuit appeared first on Quentin Santos.
There’s a lot of excitement about what AI (specifically the latest wave of LLM-anchored AI) can do, and how AI-first companies are different from the prior generations of companies. There are a lot of important and real opportunities at hand, but I find that many of these conversations occur at such an abstract altitude that they’re a bit too abstract. Sort of like saying that your company could be much better if you merely adopted software. That’s certainly true, but it’s not a particularly helpful claim. This post is an attempt to concisely summarize how AI agents work, apply that summary to a handful of real-world use cases for AI, and make the case that the potential of AI agents is equivalent to the potential of this generation of AI. By the end of this writeup, my hope is that you’ll be well-armed to have a concrete discussion about how LLMs and agents could change the shape of your company. How do agents work? At its core, using an LLM is an API call that includes a prompt. For example, you might call Anthropic’s /v1/message with a prompt: How should I adopt LLMs in my company? That prompt is used to fill the LLM’s context window, which conditions the model to generate certain kinds of responses. This is the first important thing that agents can do: use an LLM to evaluate a context window and get a result. Prompt engineering, or context engineering as it’s being called now, is deciding what to put into the context window to best generate the responses you’re looking for. For example, In-Context Learning (ICL) is one form of context engineering, where you supply a bunch of similar examples before asking a question. If I want to determine if a transaction is fraudulent, then I might supply a bunch of prior transactions and whether they were, or were not, fraudulent as ICL examples. Those examples make generating the correct answer more likely. However, composing the perfect context window is very time intensive, benefiting from techniques like metaprompting to improve your context. Indeed, the human (or automation) creating the initial context might not know enough to do a good job of providing relevant context. For example, if you prompt, Who is going to become the next mayor of New York City?, then you are unsuited to include the answer to that question in your prompt. To do that, you would need to already know the answer, which is why you’re asking the question to begin with! This is where we see model chat experiences from OpenAI and Anthropic use web search to pull in context that you likely don’t have. If you ask a question about the new mayor of New York, they use a tool to retrieve web search results, then add the content of those searches to your context window. This is the second important thing that agents can do: use an LLM to suggest tools relevant to the context window, then enrich the context window with the tool’s response. However, it’s important to clarify how “tool usage” actually works. An LLM does not actually call a tool. (You can skim OpenAI’s function calling documentation if you want to see a specific real-world example of this.) Instead there is a five-step process to calling tools that can be a bit counter-intuitive: The program designer that calls the LLM API must also define a set of tools that the LLM is allowed to suggest using. Every API call to the LLM includes that defined set of tools as options that the LLM is allowed to recommend The response from the API call with defined functions is either: Generated text as any other call to an LLM might provide A recommendation to call a specific tool with a specific set of parameters, e.g. an LLM that knows about a get_weather tool, when prompted about the weather in Paris, might return this response: [{ "type": "function_call", "name": "get_weather", "arguments": "{\"location\":\"Paris, France\"}" }] The program that calls the LLM API then decides whether and how to honor that requested tool use. The program might decide to reject the requested tool because it’s been used too frequently recently (e.g. rate limiting), it might check if the associated user has permission to use the tool (e.g. maybe it’s a premium only tool), it might check if the parameters match the user’s role-based permissions as well (e.g. the user can check weather, but only admin users are allowed to check weather in France). If the program does decide to call the tool, it invokes the tool, then calls the LLM API with the output of the tool appended to the prior call’s context window. The important thing about this loop is that the LLM itself can still only do one interesting thing: taking a context window and returning generated text. It is the broader program, which we can start to call an agent at this point, that calls tools and sends the tools’ output to the LLM to generate more context. What’s magical is that LLMs plus tools start to really improve how you can generate context windows. Instead of having to have a very well-defined initial context window, you can use tools to inject relevant context to improve the initial context. This brings us to the third important thing that agents can do: they manage flow control for tool usage. Let’s think about three different scenarios: Flow control via rules has concrete rules about how tools can be used. Some examples: it might only allow a given tool to be used once in a given workflow (or a usage limit of a tool for each user, etc) it might require that a human-in-the-loop approves parameters over a certain value (e.g. refunds more than $100 require human approval) it might run a generated Python program and return the output to analyze a dataset (or provide error messages if it fails) apply a permission system to tool use, restricting who can use which tools and which parameters a given user is able to use (e.g. you can only retrieve your own personal data) a tool to escalate to a human representative can only be called after five back and forths with the LLM agent Flow control via statistics can use statistics to identify and act on abnormal behavior: if the size of a refund is higher than 99% of other refunds for the order size, you might want to escalate to a human if a user has used a tool more than 99% of other users, then you might want to reject usage for the rest of the day it might escalate to a human representative if tool parameters are more similar to prior parameters that required escalation to a human agent LLMs themselves absolutely cannot be trusted. Anytime you rely on an LLM to enforce something important, you will fail. Using agents to manage flow control is the mechanism that makes it possible to build safe, reliable systems with LLMs. Whenever you find yourself dealing with an unreliable LLM-based system, you can always find a way to shift the complexity to a tool to avoid that issue. As an example, if you want to do algebra with an LLM, the solution is not asking the LLM to directly perform algebra, but instead providing a tool capable of algebra to the LLM, and then relying on the LLM to call that tool with the proper parameters. At this point, there is one final important thing that agents do: they are software programs. This means they can do anything software can do to build better context windows to pass on to LLMs for generation. This is an infinite category of tasks, but generally these include: Building general context to add to context window, sometimes thought of as maintaining memory Initiating a workflow based on an incoming ticket in a ticket tracker, customer support system, etc Periodically initiating workflows at a certain time, such as hourly review of incoming tickets Alright, we’ve now summarized what AI agents can do down to four general capabilities. Recapping a bit, those capabilities are: Use an LLM to evaluate a context window and get a result Use an LLM to suggest tools relevant to the context window, then enrich the context window with the tool’s response Manage flow control for tool usage via rules or statistical analysis Agents are software programs, and can do anything other software programs do Armed with these four capabilities, we’ll be able to think about the ways we can, and cannot, apply AI agents to a number of opportunities. Use Case 1: Customer Support Agent One of the first scenarios that people often talk about deploying AI agents is customer support, so let’s start there. A typical customer support process will have multiple tiers of agents who handle increasingly complex customer problems. So let’s set a goal of taking over the easiest tier first, with the goal of moving up tiers over time as we show impact. Our approach might be: Allow tickets (or support chats) to flow into an AI agent Provide a variety of tools to the agent to support: Retrieving information about the user: recent customer support tickets, account history, account state, and so on Escalating to next tier of customer support Refund a purchase (almost certainly implemented as “refund purchase” referencing a specific purchase by the user, rather than “refund amount” to prevent scenarios where the agent can be fooled into refunding too much) Closing the user account on request Include customer support guidelines in the context window, describe customer problems, map those problems to specific tools that should be used to solve the problems Flow control rules that ensure all calls escalate to a human if not resolved within a certain time period, number of back-and-forth exchanges, if they run into an error in the agent, and so on. These rules should be both rules-based and statistics-based, ensuring that gaps in your rules are neither exploitable nor create a terrible customer experience Review agent-customer interactions for quality control, making improvements to the support guidelines provided to AI agents. Initially you would want to review every interaction, then move to interactions that lead to unusual outcomes (e.g. escalations to human) and some degree of random sampling Review hourly, then daily, and then weekly metrics of agent performance Based on your learnings from the metric reviews, you should set baselines for alerts which require more immediate response. For example, if a new topic comes up frequently, it probably means a serious regression in your product or process, and it requires immediate review rather than periodical review. Note that even when you’ve moved “Customer Support to AI agents”, you still have: a tier of human agents dealing with the most complex calls humans reviewing the periodic performance statistics humans performing quality control on AI agent-customer interactions You absolutely can replace each of those downstream steps (reviewing performance statistics, etc) with its own AI agent, but doing that requires going through the development of an AI product for each of those flows. There is a recursive process here, where over time you can eliminate many human components of your business, in exchange for increased fragility as you have more tiers of complexity. The most interesting part of complex systems isn’t how they work, it’s how they fail, and agent-driven systems will fail occasionally, as all systems do, very much including human-driven ones. Applied with care, the above series of actions will work successfully. However, it’s important to recognize that this is building an entire software pipeline, and then learning to operate that software pipeline in production. These are both very doable things, but they are meaningful work, turning customer support leadership into product managers and requiring an engineering team building and operating the customer support agent. Use Case 2: Triaging incoming bug reports When an incident is raised within your company, or when you receive a bug report, the first problem of the day is determining how severe the issue might be. If it’s potentially quite severe, then you want on-call engineers immediately investigating; if it’s certainly not severe, then you want to triage it in a less urgent process of some sort. It’s interesting to think about how an AI agent might support this triaging workflow. The process might work as follows: Pipe all created incidents and all created tickets to this agent for review. Expose these tools to the agent: Open an incident Retrieve current incidents Retrieve recently created tickets Retrieve production metrics Retrieve deployment logs Retrieve feature flag change logs Toggle known-safe feature flags Propose merging an incident with another for human approval Propose merging a ticket with another ticket for human approval Redundant LLM providers for critical workflows. If the LLM provider’s API is unavailable, retry three times over ten seconds, then resort to using a second model provider (e.g. Anthropic first, if unavailable try OpenAI), and then finally create an incident that the triaging mechanism is unavailable. For critical workflows, we can’t simply assume the APIs will be available, because in practice all major providers seem to have monthly availability issues. Merge duplicates. When a ticket comes in, first check ongoing incidents and recently created tickets for potential duplicates. If there is a probable duplicate, suggest merging the ticket or incident with the existing issue and exit the workflow. Assess impact. If production statistics are severely impacted, or if there is a new kind of error in production, then this is likely an issue that merits quick human review. If it’s high priority, open an incident. If it’s low priority, create a ticket. Propose cause. Now that the incident has been sized, switch to analyzing the potential causes of the incident. Look at the code commits in recent deploys and suggest potential issues that might have caused the current error. In some cases this will be obvious (e.g. spiking errors with a traceback of a line of code that changed recently), and in other cases it will only be proximity in time. Apply known-safe feature flags. Establish an allow list of known safe feature flags that the system is allowed to activate itself. For example, if there are expensive features that are safe to disable, it could be allowed to disable them, e.g. restricting paginating through deeper search results when under load might be a reasonable tradeoff between stability and user experience. Defer to humans. At this point, rely on humans to drive incident, or ticket, remediation to completion. Draft initial incident report. If an incident was opened, the agent should draft an initial incident report including the timeline, related changes, and the human activities taken over the course of the incident. This report should then be finalized by the human involved in the incident. Run incident review. Your existing incident review process should take the incident review and determine how to modify your systems, including the triaging agent, to increase reliability over time. Safeguard to reenable feature flags. Since we now have an agent disabling feature flags, we also need to add a periodic check (agent-driven or otherwise) to reenable the “known safe” feature flags if there isn’t an ongoing incident to avoid accidentally disabling them for long periods of time. This is another AI agent that will absolutely work as long as you treat it as a software product. In this case, engineering is likely the product owner, but it will still require thoughtful iteration to improve its behavior over time. Some of the ongoing validation to make this flow work includes: The role of humans in incident response and review will remain significant, merely aided by this agent. This is especially true in the review process, where an agent cannot solve the review process because it’s about actively learning what to change based on the incident. You can make a reasonable argument that an agent could decide what to change and then hand that specification off to another agent to implement it. Even today, you can easily imagine low risk changes (e.g. a copy change) being automatically added to a ticket for human approval. Doing this for more complex, or riskier changes, is possible but requires an extraordinary degree of care and nuance: it is the polar opposite of the idea of “just add agents and things get easy.” Instead, enabling that sort of automation will require immense care in constraining changes to systems that cannot expose unsafe behavior. For example, one startup I know has represented their domain logic in a domain-specific language (DSL) that can be safely generated by an LLM, and are able to represent many customer-specific features solely through that DSL. Expanding the list of known-safe feature flags to make incidents remediable. To do this widely will require enforcing very specific requirements for how software is developed. Even doing this narrowly will require changes to ensure the known-safe feature flags remain safe as software is developed. Periodically reviewing incident statistics over time to ensure mean-time-to-resolution (MTTR) is decreasing. If the agent is truly working, this should decrease. If the agent isn’t driving a reduction in MTTR, then something is rotten in the details of the implementation. Even a very effective agent doesn’t relieve the responsibility of careful system design. Rather, agents are a multiplier on the quality of your system design: done well, agents can make you significantly more effective. Done poorly, they’ll only amplify your problems even more widely. Do AI Agents Represent Entirety of this Generation of AI? If you accept my definition that AI agents are any combination of LLMs and software, then I think it’s true that there’s not much this generation of AI can express that doesn’t fit this definition. I’d readily accept the argument that LLM is too narrow a term, and that perhaps foundational model would be a better term. My sense is that this is a place where frontier definitions and colloquial usage have deviated a bit. Closing thoughts LLMs and agents are powerful mechanisms. I think they will truly change how products are designed and how products work. An entire generation of software makers, and company executives, are in the midst of learning how these tools work. Software isn’t magic, it’s very logical, but what it can accomplish is magical. The same goes for agents and LLMs. The more we can accelerate that learning curve, the better for our industry.
This is not going to be a cakewalk like self driving cars. Most of comma’s competition is now out of business, taking billions and billions of dollars with it. Re: Tesla and FSD, we always expected Tesla to have the lead, but it’s not a winner take all market, it will look more like iOS vs Android. comma has been around for 10 years, is profitable, and is now growing rapidly. In self driving, most of the competition wasn’t even playing the right game. This isn’t how it is for ML frameworks. tinygrad’s competition is playing the right game, open source, and run by some quite smart people. But this is my second startup, so hopefully taking a bit more risk is appropriate. For comma to win, all it would take is people in 2016 being wrong about LIDAR, mapping, end to end, and hand coding, which hopefully we all agree now that they were. For tinygrad to win, it requires something much deeper to be wrong about software development in general. As it stands now, tinygrad is 14556 lines. Line count is not a perfect proxy for complexity, but when you have differences of multiple orders of magnitude, it might mean something. I asked ChatGPT to estimate the lines of code in PyTorch, JAX, and MLIR. JAX = 400k MLIR = 950k PyTorch = 3300k They range from one to two orders of magnitude off. And this isn’t even including all the libraries and drivers the other frameworks rely on, CUDA, cuBLAS, Triton, nccl, LLVM, etc…. tinygrad includes every single piece of code needed to drive an AMD RDNA3 GPU except for LLVM, and we plan to remove LLVM in a year or two as well. But so what? What does line count matter? One hypothesis is that tinygrad is only smaller because it’s not speed or feature competitive, and that if and when it becomes competitive, it will also be that many lines. But I just don’t think that’s true. tinygrad is already feature competitive, and for speed, I think the bitter lesson also applies to software. When you look at the machine learning ecosystem, you realize it’s just the same problems over and over again. The problem of multi machine, multi GPU, multi SM, multi ALU, cross machine memory scheduling, DRAM scheduling, SRAM scheduling, register scheduling, it’s all the same underlying problem at different scales. And yet, in all the current ecosystems, there are completely different codebases and libraries at each scale. I don’t think this stands. I suspect there is a simple formulation of the problem underlying all of the scheduling. Of course, this problem will be in NP and hard to optimize, but I’m betting the bitter lesson wins here. The goal of the tinygrad project is to abstract away everything except the absolute core problem in the cleanest way possible. This is why we need to replace everything. A model for the hardware is simple compared to a model for CUDA. If we succeed, tinygrad will not only be the fastest NN framework, but it will be under 25k lines all in, GPT-5 scale training job to MMIO on the PCIe bus! Here are the steps to get there: Expose the underlying search problem spanning several orders of magnitude. Due to the execution of neural networks not being data dependent, this problem is very amenable to search. Make sure your formulation is simple and complete. Fully capture all dimensions of the search space. The optimization goal is simple, run faster. Apply the state of the art in search. Burn compute. Use LLMs to guide. Use SAT solvers. Reinforcement learning. It doesn’t matter, there’s no way to cheat this goal. Just see if it runs faster. If this works, not only do we win with tinygrad, but hopefully people begin to rethink software in general. Of course, it’s a big if, this isn’t like comma where it was hard to lose. But if it wins… The main thing to watch is development speed. Our bet has to be that tinygrad’s development speed is outpacing the others. We have the AMD contract to train LLaMA 405B as fast as NVIDIA due in a year, let’s see if we succeed.
There’s a video on YouTube from “Technology Connections” — who I’ve never heard of or watched until now — called Algorithms are breaking how we think. I learned of this video from Gedeon Maheux of The Iconfactory fame. Speaking in the context of why they made Tapestry, he said the ideas in this video would be their manifesto. So I gave it a watch. Generally speaking, the video asks: Does anyone care to have a self-directed experience online, or with a computer more generally? I'm not sure how infrequently we’re actually deciding for ourselves these days [how we decide what we want to see, watch, and do on the internet] Ironically we spend more time than ever on computing devices, but less time than ever curating our own experiences with them. Which — again ironically — is the inverse of many things in our lives. Generally speaking, the more time we spend with something, the more we invest in making it our own — customizing it to our own idiosyncrasies. But how much time do you spend curating, customizing, and personalizing your digital experience? (If you’re reading this in an RSS reader, high five!) I’m not talking about “I liked that post, or saved that video, so the algorithm is personalizing things for me”. Do you know what to get yourself more of? Do you know where to find it? Do you even ask yourself these questions? “That sounds like too much work” you might say. And you’re right, it is work. As the guy in the video says: I'm one of those weirdos who think the most rewarding things in life take effort Me too. Email · Mastodon · Bluesky