Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Jaz's Blog
I’ve recently started building a POC of a Redis RESP3 Wire Compatible Key/Value Database built on FoundationDB with @calabro.io and though it’s rather early, it’s already spawned a fun distributed systems problem that I thought would be interesting to share. Previously I’ve written about how I implemented a Graph DB via Roaring Bitmaps, representing relations as a bidirectional pair of sets. To support such use-cases in this new database, we’d like to represent sets of keys such that you can perform boolean operations on them (intersection, union, difference) relatively quickly even for very large sets (with millions of members). Supporting Larger Keys In the original Graph DB, we were representing user DID strings as uint32 UIDs to allow us to store millions of edge lists in very little space (e.g. the set of users who follow bsky.app) while being able to perform boolean operations between lists quickly (using Roaring Bitmaps’ parallel boolean operators). Since we were graphing follows, blocks, and other such User-to-User relationships, there was a practical maximum for the total number of user IDs in the low billions. We’ve continued exploring objects and relationships we’d like to represent as a Graph, and have realized that if we wanted to store e.g. the URIs of all posts a user has liked so we can intersect it with other users’ likes, we’re going to need a bigger keyspace! There are well over 15 Billion records in the AT Proto Ecosystem, each with a unique AT URI! Now our desired keyspace is much larger than can be represented by uint32 values and so we need to expand to uint64. Easy enough, let’s use the uint64 flavor of Roaring Bitmaps and simply intern URIs and User DIDs as uint64s, problem solved, right? Not quite… Interning Many Things at Once The AT Proto Firehose has hit historic peak traffic of over 1,500 evt/sec. We want to design a system that will handle many times more scale than we’ve ever seen in reality. This means designing for 10x or 100x would require us to be able to intern 15k to 150k new URIs per second into uint64 integers. Sounds easy enough, what’s the holdup? Well, in FoundationDB we’re able to use Transactions to do things like atomically increment a sequence safely when many other threads may be trying to do the same thing. This is simple enough to do in Go, we can just toss together a little helper function to acquire a new UID for our string: func (s *server) allocateNewUID(span trace.Span, tx fdb.Transaction) (uint64, error) { var newUID uint64 val, err := tx.Get(fdb.Key("last_uid")).Get() if err != nil { return 0, return fmt.Errorf("failed to get last UID: %w", err) } if len(val) == 0 { newUID = 1 // start from 1 } else { lastUID, err := strconv.ParseUint(string(val), 10, 64) if err != nil { return 0, return fmt.Errorf("failed to parse last UID: %w", err) } newUID = lastUID + 1 } tx.Set(fdb.Key("last_uid"), []byte(strconv.FormatUint(newUID, 10))) return newUID, nil } This function gets called from a fdb.Transaction which gets assigned a Transaction ID, then stages its changes, then tries to commit them. In FoundationDB, if your transaction is reading or modifying data written to by a different Transaction that finishes while you’re in-progress, your Transaction is thrown out and must be retried. For our UID assignment use-case, this is pretty problematic. We want to assign hundreds of thousands of new UIDs per second but if they’re all modifying the same key, concurrent transactions will constantly run into contention on the same data and will be forced to retry over and over again. This problem gets worse the more concurrent transactions you have trying to read from or write to the same key. Even if we stick to sequential access, if it takes ~5-10ms to assign a UID, we can only assign ~100-200 UIDs per second, nowhere near the throughput we need to support. How can we get past this problem and allow us to give strings unique uint64 UIDs in a high throughput and highly concurrent manner? Attempt #1: xxHash My first attempt to solve this problem was to try something that required no coordination and hash the string keys into uint64s using xxHash. xxHash is a non-cryptographic hash algorithm that supports incredibly high throughput (dozens of GB/sec) and can produce 64 bit unsigned integer hashes of strings trivially. Implementing this would look something like: Hash the incoming string key Lookup the uint64 UID to see if we’ve already assigned it to a string Reject the transaction if there’s a collision and give up Store the key in the UID map and the UID in the key map Use the UID for anything else we need While the uint64 keyspace is plenty large for our needs assuming we distribute evenly among the whole space, using a hashing algorithm with no coordination means there’s room for collisions and thus we’d need some additional logic (potentially by bucketing the keys somehow). Consulting the Birthday Problem we can see that a keyspace with 64 bit hashes has a >50% chance of containing a single collision when we have only ~5 billion keys in the set! That’s barely more keys than we can cram into a uint32 and definitely won’t suffice for the number of keys we expect to be storing! So, xxHash, while nice and coordination-free is probably not going to be the solution we need. What else can we do? Attempt #2: Billions of Sequences Incrementing one sequence is clearly not an option because we can only increment a single sequence ~100-200 times per second, but what if we instead had more than one sequence? Roaring Bitmaps managed to make highly efficient bitmap representations by breaking up a uint32 keyspace into a uint16-wide set of uint16-wide keyspaces. Can we do something similar here? Here’s an idea, what if we had just over 4 billion difference sequences and just picked one at random when we needed to assign a UID? Since we’re constructing our UIDs as a uint64, we can split the full UID into a pair of uint32s where the most-significant-bits are used to identify the sequence ID and the least-significant-bits are used to identify the value assigned to the UID within the sequence. So in our implementation, we get ~4.3 Billion sequence IDs that each have ~4.3 Billion incrementing values. As an example, if we were to randomly select Sequence ID 37 and then we increment that sequence to the value 5, we’d assemble the ID as 37<<32 + 5 which looks like 158,913,789,952 + 5 -> 158,913,789,957. Looking at the next Seuqence ID, we’d see 38 which, when left shifted by 32 gives us 163,208,757,248. You can see there’s a gap of ~4.3 billion values between the first UID assigned by each Sequence ID. Assuming we can increment a single sequence ~100 times per second with contention, we’re able to mint 430 Billion new UIDs per second without locking up (assuming the cluster can keep up). Storing ~4.3 billion sequences may be a bit expensive, but thankfully this strategy can scale up and down by picking a larger or smaller prefix size. If we only wanted to store say, ~16k sequences, we can pick a 14 bit prefix instead of a 32 bit prefix and then use a 50 bit sequence number. That spreads the load across 2^14 sequence IDs and significantly reduces storage requirements for Sequences. What does this look like in code? Well, it’s honestly not very complex! const uidSequencePrefix = "uid_sequence/" func (s *server) allocateNewUID(tx fdb.Transaction) (uint64, error) { // sequenceNum is the random uint32 sequence we are using for this allocation var sequenceNum uint32 var sequenceKey string // assignedUID is the uint32 within the sequence we will assign var assignedUID uint32 // Try up to 5 times to find a sequence that is not exhausted for range 5 { // Pick a random uint32 as the sequence we will be using for this UID sequenceNum = rand.Uint32() sequenceKey = fmt.Sprintf("%s%d", uidSequencePrefix, sequenceNum) val, err := tx.Get(fdb.Key(sequenceKey)).Get() if err != nil { return 0, fmt.Errorf("failed to get last UID: %w", err) } if len(val) == 0 { assignedUID = 1 // Start each sequence at 1 } else { lastUID, err := strconv.ParseUint(string(val), 10, 32) if err != nil { return 0, fmt.Errorf("failed to parse last UID: %w", err) } // If we have exhausted this sequence, pick a new random sequence if lastUID >= 0xFFFFFFFF { continue } assignedUID = uint32(lastUID) + 1 } } // If we failed to find a sequence after 5 tries, return an error if assignedUID == 0 { return 0, fmt.Errorf("failed to allocate new UID after 5 attempts") } // Assemble the 64-bit UID from the sequence ID and assigned UID newUID := (uint64(sequenceNum) << 32) | uint64(assignedUID) // Store the assigned UID back to the sequence key for the next allocation tx.Set(fdb.Key(sequenceKey), []byte(strconv.FormatUint(uint64(assignedUID), 10))) // Return the full 64-bit UID return newUID, nil } And there we go! We can now intern billions of strings per second with little to no contention in a distributed system while completely avoiding collisions and making full use of our keyspace! Conclusion Often times when designing distributed systems, patterns and strategies you see in seemingly unrelated libraries can inspire an elegant solution to the problem at hand. In the case of distributed, high-throughput string interning, horizontal scaling can be achieved by breaking up one large keyspace that requires strict coordination into billions of smaller keyspaces that can be randomly load-balanced across. Both patterns used in this technique are present elsewhere: Breaking up a large keyspace into a bunch of smaller keyspaces is present in Roaring Bitmaps (among other systems) Letting randomness and large numbers spread out resource contention is present in many load balancing systems This is one of my favorite parts of growing as an engineer: the more systems and strategies you familiarize yourself with, the more material you have to draw from when designing something new. Personal News A bit of personal news for y’all if you made it this far. Today is my last day as a member of the Bluesky team! The past 2+ years building out Bluesky’s Infrastructure and Platform team and scaling Bluesky from 100,000 -> 40,000,000 users have been the most intense and rewarding years of my life. I don’t have the words to express how much I’ve valued my time on the team and how much I care for the people I’ve worked with in what feels like a decade of real time. I’ve got some new adventures ahead and am excited to be embarking on a new journey within the next month (still building large-scale infrastructure, don’t worry). I plan to continue being involved in the AT Proto Community and to contribute to some cool projects other folks on the Bluesky team are working on from the FOSS space (like KVDB). To the team, I wish you all the best and will dearly miss getting to work with you all every day, but nothing lasts forever and I will always cherish the time I got to spend building an incredible platform with incredible people. If you’re interested in joining a world-class team doing important work, check out Bluesky’s open job listings here. There should be a new role opening up for a seasoned Go Engineer on the Platform team soon!
Often when designing systems, we aim for perfection in things like consistency of data, availability, latency, and more. The hardest part of system design is that it’s difficult (if not impossible) to design systems that have perfect consistency, perfect availability, incredibly low latency, and incredibly high throughput, all at the same time. Instead, when we approach system design, it’s best to treat each of these properties as points on different axes that we balance to find the “right fit” for the application we’re supporting. I recently made some major tradeoffs in the design of Bluesky’s Following Feed/Timeline to improve the performance of writes at the cost of consistency in a way that doesn’t negatively affect users but reduced P99s by over 96%. Timeline Fanout When you make a post on Bluesky, your post is indexed by our systems and persisted to a database where we can fetch it to hydrate and serve in API responses. Additionally, a reference to your post is “fanned out” to your followers so they can see it in their Timelines. This process involves looking up all of your followers, then inserting a new row into each of their Timeline tables in reverse chronological order with a reference to your post. When a user loads their Timeline, we fetch a page of post references and then hydrate the posts/actors concurrently to quickly build an API response and let them see the latest content from people they follow. The Timelines table is sharded by user. This means each user gets their own Timeline partition, randomly distributed among shards of our horizontally scalable database (ScyllaDB), replicated across multiple shards for high availability. Timelines are regularly trimmed when written to, keeping them near a target length and dropping older post references to conserve space. Hot Shards in Your Area Bluesky currently has around 32 Million Users and our Timelines database is broken into hundreds of shards. To support millions of partitions on such a small number of shards, each user’s Timeline partition is colocated with tens of thousands of other users’ Timelines. Under normal circumstances with all users behaving well, this doesn’t present a problem as the work of an individual Timeline is small enough that a shard can handle the work of tens of thousands of them without being heavily taxed. Unfortunately, with a large number of users, some of them will do abnormal things like… well… following hundreds of thousands of other users. Generally, this can be dealt with via policy and moderation to prevent abusive users from causing outsized load on systems, but these processes take time and can be imperfect. When a user follows hundreds of thousands of others, their Timeline becomes hyperactive with writes and trimming occurring at massively elevated rates. This load slows down the individual operations to the user’s Timeline, which is fine for the bad behaving user, but causes problems to the tens of thousands of other users sharing a shard with them. We typically call this situation a “Hot Shard”: where some resident of a shard has “hot” data that is being written to or read from at much higher rates than others. Since the data on the shard is only replicated a few times, we can’t effectively leverage the horizontal scale of our database to process all this additional work. Instead, the “Hot Shard” ends up spending so much time doing work for a single partition that operations to the colocated partitions slow down as well. Stacking Latencies Returning to our Fanout process, let’s consider the case of Fanout for a user followed by 2,000,000 other users. Under normal circumstances, writing to a single Timeline takes an average of ~600 microseconds. If we sequentially write to the Timelines of our user’s followers, we’ll be sitting around for 20 minutes at best to Fanout this post. If instead we concurrently Fanout to 1,000 Timelines at once, we can complete this Fanout job in ~1.2 seconds. That sounds great, except it oversimplifies an important property of systems: tail latencies. The average latency of a write is ~600 microseconds, but some writes take much less time and some take much more. In fact, the P99 latency of writes to the Timelines cluster can be as high as 15 milliseconds! What does this mean for our Fanout? Well, if we concurrently write to 1,000 Timelines at once, statistically we’ll see 10 writes as slow as or slower than 15 milliseconds. In the case of timelines, each “page” of followers is 10,000 users large and each “page” must be fanned out before we fetch the next page. This means that our slowest writes will hold up the fetching and Fanout of the next page. How does this affect our expected Fanout time? Each “page” will have ~100 writes as slow as or slower than the P99 latency. If we get unlucky, they could all stack up on a single routine and end up slowing down a single page of Fanout to 1.5 seconds. In the worst case, for our 2,000,000 Follower celebrity, their post Fanout could end up taking as long as 5 minutes! That’s not even considering P99.9 and P99.99 latencies which could end up being >1 second, which could leave us waiting tens of minutes for our Fanout job. Now imagine how bad this would be for a user with 20,000,000+ Followers! So, how do we fix the problem? By embracing imperfection, of course! Lossy Timelines Imagine a user who follows hundreds of thousands of others. Their Timeline is being written to hundreds of times a second, moving so fast it would be humanly impossible to keep up with the entirety of their Timeline even if it was their full-time job. For a given user, there’s a threshold beyond which it is unreasonable for them to be able to keep up with their Timeline. Beyond this point, they likely consume content through various other feeds and do not primarily use their Following Feed. Additionally, beyond this point, it is reasonable for us to not necessarily have a perfect chronology of everything posted by the many thousands of users they follow, but provide enough content that the Timeline always has something new. Note in this case I’m using the term “reasonable” to loosely convey that as a social media service, there must be a limit to the amount of work we are expected to do for a single user. What if we introduce a mechanism to reduce the correctness of a Timeline such that there is a limit to the amount of work a single Timeline can place on a DB shard. We can assert a reasonable limit for the number of follows a user should have to have a healthy and active Timeline, then increase the “lossiness” of their Timeline the further past that limit they go. A loss_factor can be defined as min(reasonable_limit/num_follows, 1) and can be used to probabilistically drop writes to a Timeline to prevent hot shards. Just before writing a page in Fanout, we can generate a random float between 0 and 1, then compare it to the loss_factor of each user in the page. If the user’s loss_factor is smaller than the generated float, we filter the user out of the page and don’t write to their Timeline. Now, users all have the same number of “follows worth” of Fanout. For example with a reasonable_limit of 2,000, a user who follows 4,000 others will have a loss_factor of 0.5 meaning half the writes to their Timeline will get dropped. For a user following 8,000 others, their loss factor of 0.25 will drop 75% of writes to their Timeline. Thus, each user has a effective ceiling on the amount of Fanout work done for their Timeline. By specifying the limits of reasonable user behavior and embracing imperfection for users who go beyond it, we can continue to provide service that meets the expectations of users without sacrificing scalability of the system. Aside on Caching We write to Timelines at a rate of more than one million times a second during the busy parts of the day. Looking up the number of follows of a given user before fanning out to them would require more than one million additional reads per second to our primary database cluster. This additional load would not be well received by our database and the additional cost wouldn’t be worth the payoff for faster Timeline Fanout. Instead, we implemented an approach that caches high-follow accounts in a Redis sorted set, then each instance of our Fanout service loads an updated version of the set into memory every 30 seconds. This allows us to perform lookups of follow counts for high-follow accounts millions of times per second per Fanount service instance. By caching values which don’t need to be perfect to function correctly in this case, we can once again embrace imperfection in the system to improve performance and scalability without compromising the function of the service. Results We implemented Lossy Timelines a few weeks ago on our production systems and saw a dramatic reduction in hot shards on the Timelines database clusters. In fact, there now appear to be no hot shards in the cluster at all, and the P99 of a page of Fanout work has been reduced by over 90%. Additionally, with the reduction in write P99s, the P99 duration for a full post Fanout has been reduced by over 96%. Jobs that used to take 5-10 minutes for large accounts now take <10 seconds. Knowing where it’s okay to be imperfect lets you trade consistency for other desirable aspects of your systems and scale ever higher. There are plenty of other places for improvement in our Timelines architecture, but this step was a big one towards improving throughput and scalability of Bluesky’s Timelines. If you’re interested in these sorts of problems and would like to help us build the core data services that power Bluesky, check out this job listing. If you’re interested in other open positions at Bluesky, you can find them here.
More in AI
In mid-September, Anthropic CEO Dario Amodei joined a growing group of researchers calling for the AI industry to slow down. While some have questioned these researchers’ motivation for voicing concerns about the rapid pace of development, the plea highlighted a key tension between safety and speed. The underlying problem is not simply a technical error, but a sociotechnical failure: Pressure to move quickly has made it hard for engineers to fully assess and manage new risks of increasingly powerful AI systems. It is a clear warning that responsible engineering requires more than technical expertise; engineers must also understand workplace pressures, moral and ethical rules, and the social consequences rooted in the systems they create. Cases like this, alongside invasive surveillance technologies and biased AI tools, have exposed serious shortcomings in how engineers recognize and address the ethical and social implications of their work. While the need to educate graduates in ethics and social responsibility is widely recognized, a persistent gap remains—these topics are too often treated as afterthoughts, disconnected from the technical content of engineering coursework. This siloed approach leaves graduates ill-equipped to address real-world challenges; indeed, only about a third of engineering professionals in the United States report having received training on their public welfare responsibilities during their formal engineering education. We are working to bridge this gap by designing two complementary instructional approaches, one that integrates social issues directly into foundation technical coursework and a second that prepares students to act as “public welfare watchdogs” through a standalone engineering course. By embedding ethics and social responsibility training directly into the core of undergraduate engineering coursework, we equip future professionals with the agency they will need to protect the public good. The gap in instruction about ethics and social responsibility Across the engineering profession, there is a persistent belief that technical skill-building is central to engineering, while ethical, social, and public welfare concerns are peripheral or distracting. This technical/social dualism is evident even in the evaluation of scholarly research. For instance, we recently submitted an academic position paper discussing this ethics education gap to a leading peer-reviewed engineering journal. It was dismissed by reviewers along the very same dualism it described. One reviewer warned that it would be “highly irresponsible” to challenge technical skill-building as “the foundation of any kind of engineering” by adding ethics and social responsibility training into engineering classrooms. Another reviewer objected that integrating these considerations into foundational courses risks “distracting the students from fully absorbing the technology.” These comments miss the core argument of our work: that ethics and social responsibility are integral to responsible technical practice. Today, accreditation agencies like ABET (formerly known as the Accreditation Board for Engineering and Technology) and professional licensing bodies like the National Society of Professional Engineers mandate student outcomes related to social and public welfare consequences. But these curriculum requirements tend to be unhelpfully generic and minimally enforced. As a result, engineering students often receive ethics instruction through introductory or non-engineering coursework, where the focus is on codes of conduct and academic integrity, rather than deeper topics of social responsibility. This instruction is almost always disconnected from the complex real-world dilemmas engineers face. Indeed, we find that only ethics instruction in actual engineering courses (not in other parts of the curriculum) can effectively instill ethical awareness as professionals. Concerningly, only about 4 in 10 engineers in the United States recall receiving this kind of training, and 3 in 10 report never receiving training on their public welfare responsibilities at any point in their careers. This leaves most of the profession underprepared to handle the complex challenges of engineering practice. A new approach: Integrating technical and social dimensions To advance a new paradigm—one that embeds ethics and social responsibility issues within the undergraduate engineering curriculum—we’ve designed two innovative course-based approaches: a library of one-hour sociotechnical modules for foundational circuits courses (described here and here) and a new engineering course dedicated to public welfare responsibilities. The first approach integrates discussions about relevant social issues directly into foundational engineering courses, where students begin to form their identities as engineers. “Introduction to Circuits” is typically the first course in the electrical engineering major and is a requirement for many other engineering disciplines, making it a powerful site for introducing social content and ethical reasoning. We designed several one-hour modules for this course, where social issues provide context for a technical topic, offering a manageable entry point for faculty who may feel ill-prepared to teach about ethics and social responsibility. While many engineering faculty recognize the importance of teaching students about these topics, they struggle to integrate them into their courses amid heavy teaching and research demands. To overcome these challenges, each module provides ready-to-use materials, including lecture slides with scripts and homework and exam problems. Integrating these modules directly into the course signals to students that sociotechnical awareness is not an “add-on,” but a required professional competency. Our module on hospital power prioritization, for instance, connects the technical topics of power and equivalent circuits to the ethical challenges of healthcare infrastructure. Students learn about the reality of “red outlets” in medical settings, debating the power needs of specific wards (e.g., the intensive care unit, operating rooms, and physiotherapy rooms) to decide which branches should receive emergency power during a hypothetical outage. Our conflict minerals module connects the technical topic of capacitors to the global tantalum supply chain. Students calculate the amount of tantalum used in cell phones and identify mining locations in regions like the Democratic Republic of the Congo, where revenue often fuels armed conflict and human rights abuses. Students research the conflict-mineral policies of major companies such as Apple and Samsung and reflect on their future roles as ethical designers and consumers. Student responses to these modules have been overwhelmingly positive. In interviews, students told us that the content helped them get away from the “stigma that engineers only worry about math” and made the work more meaningful. We plan to share the modules widely to help circuits instructors across the globe bridge the technical-social divide. Beyond these integrated modules, our second approach is a standalone engineering course—Public Welfare Responsibilities in Engineering (PWRE)—that helps round out students’ sociotechnical education by preparing them to fulfil their professional obligations as engineers. While most dedicated ethics courses end by introducing engineers’ abstract responsibilities, PWRE helps students recognize specific public welfare concerns, enhances their motivation to act when they encounter them, and teaches concrete, practical intervention strategies. In addition to teaching students about professional codes, such as the IEEE Code of Ethics, PWRE engages students in discussion of where these formal definitions may fall short, and how structural barriers such as workplace culture can hinder engineers from fulfilling these responsibilities. Then, the course equips students with intervention strategies. Students learn about their rights and responsibilities as whistleblowers, review tactics for seeking advice from professional societies, and practice writing op-eds to raise the alarm about ethical issues to the broader public. Our research shows that the course produces notable shifts in students’ ethical agency, strengthening their ability to serve as “public welfare watchdogs” in the modern workplace. Ethical engineers for the future Today’s engineering graduates face complex sociotechnical challenges beyond AI expansion, including climate change and global supply chains; they need tools to navigate ambiguity and act ethically in uncertain contexts. To meet these evolving demands, engineering education must move beyond outdated modes of instruction that focus only on building technical skills—some of which may soon be outsourced to AI. We must fundamentally reimagine the role of ethics and social responsibility, embedding them across the engineering curriculum and throughout the professional formation of engineers. All engineers learn to solve problems, but ethical engineers can also discern which problems are worth solving and anticipate the consequences of innovation for society. By centering ethics and social responsibility as core priorities of the profession, we can prepare engineers to be stewards of technology and trusted voices in public discourse. As one student reflected after participating in our modules, “we are a part of the issue if we don’t decide to fix it.”
Inside the harnesses that let AI carry a job across hours, days, and conversations.
how agents turn ruthless on paper first
I talk to a lot of old people, those who were born in the 20th century, and if I ask them what the word “unalive” means, they usually have no idea what I’m talking about, except for some of them who have kids or who study Internet culture. This will, of course, probably seem very weird to most people who were born in or grew up in the 21st century. Just to recap for the olds: “unalive” is the word you use to represent concepts like dying, or death, or killing or being killed, on digital platforms where saying those words accurately will cause the algorithm to punish or censor you. Or, maybe, where the perception is that using those words will result in being censored by the algorithm, and no one is actually willing to find out what happens if you use the forbidden words. This sort of attack on people’s expression started on platforms like TikTok, where nearly all content is distributed through an algorithmic feed, but has since become ubiquitous in nearly all digital media that we see. In fact, these tics are now so prevalent that it’s routine to hear people using this kind of language in everyday life, even though there’s not yet an algorithm to appease in the physical world. I’ve heard people say, out loud, “he unalived himself”, in reference to someone dying by suicide. And all of this has become even more visible in recent days as online conversation has turned to discussion of the horrific lack of accountability around the tragic rape case at Cornell University. Across the Internet, people are routinely referring to the central crime in the case as r*pe or “grape” or even using the 🍇 emoji, without a second thought for what it means that the very word can’t be said online anymore. Or, at least, the assumption is that it can’t be said. To be clear, I am very much in favor of people using content warnings or sensitivity markers for content, and fine with people using abbreviations like “SA” for references to disturbing or triggering topics like sexual assault; we should provide people with as much context and control as possible when choosing what information they want to consume and when. I also know that sometimes, people use lesser terms for stressful subjects like death or assault to create a bit of ironic distance from painful or upsetting topics. But most of the different variations of wording and emojis are coming from trying to appease the platforms, and there’s a heavy cost for those who are worried about being mindful: If someone is using a tool to filter out content, it will no longer be effective because everyone is using misspellings and euphemisms and imagery to get around the algorithm. The spread of censored and mangled syntax is happening because people believe, or have experienced, that platforms will silence them for accurately describing the world in plain language. This shit is terrible, and it has to stop. You Were Not Born With These Constraints One of the things that’s most concerning to me is that an entire generation has grown up not realizing how extreme it is that their very language is being chosen for them by platforms run by people who hate that generation’s ability to express itself, and who hate the things it has to say. From their youngest days, this generation grew up watching people make stupid faces at them for YouTube thumbnails and never had a chance to reflect on the fact that those creators didn’t want to be humiliating themselves by making those expressions — the demands of the algorithms of Big Tech forced them to do that. The rituals of feeding the algorithm are so built into people’s everyday habits that they’re invisible to people who weren’t alive before today’s platforms took over. Every parent of my cohort remembers the first time they heard their toddler finish doing something cute in their living room, and then turn around and say, “please like and subscribe!” afterwards. It’s a ghastly, sickening feeling to confront the fact that our little kids were being brainwashed into thinking that every adorable thing they did should be followed by a prompt to provide data to Google. Over on Instagram, where people originally signed up thinking they were going to see someone’s vacation pictures, or shots of their cousin’s kids, you’re now stuck watching people beg for everyone to reply with cultish phrases in the comments, which will then earn them an obviously AI-generated response in return, all in service of “showing activity” to the algorithm, like it’s an angry god that needs a sacrifice. They’re just not sure exactly what the angry god wants. Your free speech was taken away from you, and the people who did it are the same ones who spent years pretending to care about “free expression”. They contrived examples of lack of free speech on college campuses while squashing protests, and cried crocodile tears about “cancel culture” while getting people fired for political criticism. Now they have no problem with billionaires deciding exactly what words everyone is allowed to say. Larry Ellison is not content with his family owning all of the movies and TV shows — his family has to control what words people are allowed to speak on TikTok, too. Elon Musk isn’t content to merely generate and distribute child sexual abuse material for profit — he wants to silence the messages of the few decent people who are foolish enough to remain on Twitter/X, too. (That’s why I wrote you a guide on how to get your organization off of that cursed platform.) Now that an entire generation has grown up using these Orwellian euphemisms, and all of the Big AI products are trained on the Internet that was created under this regime, do you think today’s AI tools even know that the real, uncensored world exists? If you can’t say “genocide” on any of the major platforms, yet those are the ones all of the Big AI tools used as their training data... well, then the AI tools sure aren’t very likely to know much about genocide, are they? Fuck the Algorithm Our creativity can be constrained by the language we use — our imaginations are limited by what we can think to say. If we’re trained to limit the words we speak just by habit, and those limits are put in place by people whose social, political, cultural and moral goals are the opposite of what we value, then our work is unalive before it is even born. The answer to this is simple: say what you mean. This will take, to some degree, courage. It may even take, I hesitate to say, some sacrifice. When I suggest this course of action to people, they inevitably say, “But it will cost me audience!” or “But what if I lose followers!” or “What if they demonetize me!” Okay, what if they do? What if they do. Are you willing to push on this? To make a point about it? To move to platforms where you can actually say what you mean? Or to remember that you already have platforms where you can say what you mean? On an email newsletter or podcast you can say whatever the hell you want and nobody can stop you. On my blog right here, I can even curse in a headline and it won’t affect anything about how my site operates. (And a reminder: Substack is not an email newsletter, and a Spotify show is not a podcast — they’ll be unaliving your distribution any day now.) If you are a 20th century relic like me, it is incumbent upon you to remind the generation that grew up inside the algorithm that another world is possible, and that we know this because we lived it. We were able to style a MySpace page in any way that we wanted; the code for LiveJournal was entirely open source so there was no part of the algorithm that was unknowable. A blog like the one you’re reading right now could be made by anyone, and put up for pennies, and nobody could stop it from being read by millions of people. (And that last one? It’s still possible.) If you are from this century, forget all that rambling bullshit about ancient history: all that matters is you getting what you deserve, because you’ve been fucked over by the same billionaires who’ve poisoned your planet and infested your world with slop. The best artists around you are invisible to you, and the most important statements by activists that you care about are being silenced. It’s not a conspiracy, it’s a system working as designed. And the proof is as obvious as the fact that the angriest activists you know can’t even talk about systemic abuses or state violence without having to put it in algorithmically approved speech or censoring their captions like they’re going to be read by 5-year-olds. It should make you furious. It is time to kill “unalive”. The response is simple: For every message you put out, start by saying what you mean. Don’t work backwards from what the algorithm wants or what a platform permits. Build a presence on every platform you can, even the ones where you have fewer followers or where you’re harder to find. Tell your audience that your speech being free matters more than corporate convenience. Keep saying it, and keep it positive: independence gets them better art, better information, and more connected communities. Build alliances with other artists, activists and people who share your values, and let them know you’re going to start sharing your work uncensored. Start releasing your work uncensored and see where the platforms push back. (You may be surprised: sometimes you were censoring yourself in anticipation of limits that weren’t even there.) If a platform does try to limit your reach or expression, make a LOT OF NOISE about it. Tell the press, rally your alliance, and spread the word on your other platforms, using the moment to build audience and raise support there. Get others to amplify the parts of your work that don’t violate platform policy, so the controversy drives people to the rest. Find the others pushing back on algorithmic control of expression, and raise and praise their work when they do the same. If we keep accepting the words that are forced upon us by TikTok and Meta and Google and all the rest, while platforms like Twitter/X allow the most hateful and harmful content in the world to be distributed completely unfettered, we’ll only see authoritarianism rise, and the harms against the vulnerable accelerate. But what breaks my heart almost as much is that we’ll see so many brilliant artists and activists and thinkers whose genius will be muted or silenced by mindless, heartless algorithms that capriciously decide who gets to say exactly what words, in what ways. I get angry every time I think about it. The tech tycoons get ever more brazen in what they’re willing to say publicly, boasting about how they’re going to cause the end of the world, or calling for ethnic cleansing, all while putting tighter and tighter reins on the speech and expression of ordinary people. It’s time for “unalive” to die.
More than a year ago I wrote a few posts here that recommended people not to load custom tools into their context (or MCP servers) but to just use more scripts. Most importantly I wrote that Code Is All You Need and I wrote about that MCP needs code. With Pi 1.0 we now added MCP support via Codemode which in some ways is a long time coming, but then also maybe somewhat surprising to some. So I want to share some updated thoughts on this blog on what this all means. What Are Tools When a harness like Pi provides tools for an LLM to call, it does so by supplying some tool definitions which then translate into some token structure on the server side. Whether a model is encouraged to call a tool is the result of the reinforcement learning process. Something I wrote about before if you want to learn more. One of the reasons we strongly lean towards CLI and bash is because it allows easy composition of calls, and because the model also learns how the file system works when it’s trained. So when it invokes a tool like echo foo > /tmp/test.txt the model also learns that after that tool call, there is now a file called test.txt in /tmp. However bash has one fundamental limitation which is that it can only compose programs that run. And there are some things, which are not programs, but native tools to the LLM and they sort of have to be. The most obvious example here is read or view_image. If a multimodal model needs to read an image, it cannot use cat for that because the harness needs to inject the actual image payload into the protocol of the LLM. Another quite vivid example are sub agents. In order to spawn and orchestrate sub agents, it’s tricky to avoid tools that are provided by the harness. While in theory the agent could provide a CLI tool that talks to the outer harness via environment variables and Unix sockets, it’s a rather crude process. It however has another issue, and that is where the code runs. Brains vs Hands To better understand that, it’s important to think a bit more about where all the bits and pieces run. There really usually are two different systems involved. The first is the brain, the harness: it runs on one machine. It’s trusted. The second is often the same machine, but it’s really where the tools are executing: the hands. In Pi we now call this the execution environment, but you can think of it as the target of all the operations. Crucially what is important for us, is that there is a dividing line between the harness brain and the target environment that runs bash and executes the tools. And splitting this in half has some really important consequences. For a start it means that they are running on different file systems and they have different levels of trust. If you for instance use a sandboxing solution like Gondolin your bash stuff will be sandboxed just fine, but the harness itself will not be. Orchestrating The Harness Which brings us to what Codemode really does: it’s a way for the LLM to express and orchestrate complex operations on the harness side, but not the execution environment side. Codemode runs in the harness, in its own sandbox. In case of Pi it’s running in QuickJS within a WASM runtime with intentional limitations: no network, no file system, no timers, limited RAM. The only way is to call more tools. You could also imagine that Codemode could run Scheme or some other language as well. If you are not familiar with Codemode, it’s basically just a way to issue tool calls from within some language, in our case JavaScript. That allows you to compose those calls without necessarily going through the LLM’s context. Credit for naming goes to our friends at Cloudflare who coined it. For instance if you issue a bash call as a regular tool call in the LLM, then we only throw the trailing 2000 lines into the context and if the agent wants more, it needs to look at the overflow file itself. If however the agent issues that invocation via Codemode, then the Codemode side gets larger outputs sent structurally. Most importantly, because Codemode is JavaScript the agent can express concurrent operations and basic workflows. A common way in which you see agents now use this, is to first probe at 5-10 items from some tool response to see what it looks like, and to then write a Codemode script that processes the next n items. Codemode also allows you to throw state into the transcript! That means that one Codemode invocation can stash away data, that the next call in the session can load again. And remember: this is on the harness host, not the sandbox. In case of Pi, Codemode also allows you to issue calls that naturally do not make any sense in Pi’s traditional interface. For instance if you want to generate images with an image model or you want to classify some text with a one shot classifier model, those Pi APIs are exposed via Codemode, but not via regular tools where they would just waste context. What It Looks Like So now that we talked a bunch about it, it’s probably worth being a bit more explicit about it. Let’s walk ourselves through some invocations of Codemode of recent Pi sessions of mine. Note that none of this code is human written. It’s from real sessions of Pi, just re-indented for your viewing pleasure. The agent starts using Codemode automatically either because it’s a task where the model already naturally picks up that tool, or because a user asked it to. Note that Codemode is by default only enabled in Pi when MCP is enabled, but you can turn it on with "defaultTools": ["+codemode"] in the settings. Just ask Pi to enable it for you. Generating Images Let’s start simple with image generation. Image generation is a feature that Pi supports in the AI SDK core, but it’s not a tool that the agent can use. In the past the only way to use image models has been to write a bespoke extension or to have the agent run node itself and use the internal image APIs. However because we expose quite a few of the internal model APIs within Codemode, it means that the agent can use it: const [painter] = await models.getAvailableOfType("image"); const result = await models.generateImages(painter, { input: [{ type: "text", text: "A cute little puppy sitting on a grassy " + "lawn, soft natural light, photorealistic" }], }); if (result.stopReason !== "stop") return result.errorMessage; for (const block of result.output) { if (block.type === "image") image(block); else text(block.text); } Note that the call to image() sends the image back as image content to the LLM. On the harness side it feeds it directly into both the agent, as well as onto disk as a temporary artifact in case the agent wants to be able to pass that image back to bash. Classifying Things Similar things apply to classifier models such as Jev. They also do not fit well into the workflows of an agent through the typical tools. But rather than making a bespoke tool available, Codemode just allows the agent to reach into the AI SDK and invoke those directly. Here you can see how Jev is used to mass process GitHub issues for a quick sentiment analysis: const jev = await models.getModelOfType("classifier", "typesafe", "jev-latest"); const r = await tools.bash({ command: "gh issue list --state open --limit 100 " + "--json number,title,body,comments", }); const issues = JSON.parse(r.output); const results = await Promise.all(issues.map(async (issue) => { const res = await models.classify(jev, { state: { title: issue.title, body: (issue.body || "").slice(0, 4000), comments: issue.comments.slice(-5).map(c => c.body.slice(0, 800)), }, questions: { sentiment: { type: "choice", instructions: "What is the overall sentiment of the author towards pi?", criteria: { positive: "Appreciative, happy, constructive praise", neutral: "Matter-of-fact report or request without emotion", negative: "Frustrated, annoyed, upset, or angry", }, }, frustration: { type: "score", instructions: "How frustrated is the reporter?", criteria: ["not at all", "mildly", "clearly frustrated", "very angry"], }, kind: { type: "choice", instructions: "What kind of issue is this?", criteria: { bug: "Bug report or regression", feature: "Feature request or enhancement", question: "Question or support request", other: "Docs, discussion, meta, spam", }, }, }, }); if (res.stopReason !== "stop") { return { n: issue.number, title: issue.title, error: res.errorMessage }; } return { n: issue.number, title: issue.title, ...res.answers }; })); store("sentiment_results", results); return results .filter(r => !r.error) .sort((a, b) => b.frustration.score - a.frustration.score) .slice(0, 12) .map(r => `#${r.n} ${r.frustration.score.toFixed(2)} [${r.kind.choice}] ${r.title}`); Note how in that above example we also call store() which dumps the result of that execution into the session transcript. A future invocation of Codemode can thus read back that result if it wants to. The Promise.all here is fine, because Pi limits the total number of concurrent tool executions itself to four and maintains a queue for the rest. A more adventurous example is to use Jev to drive a game engine for debugging purposes: Codemode with Jev for Game Debugging Here it knows about my tankctl command and it built itself quickly a minimal harness around it to drive a game loop to assist a user with debugging a problem. Note how it built a 30 step loop in which each step goes back to both the game engine to get a text dump of what’s going on, and then to Jev to determine what to do next: const jev = await models.getModelOfType("classifier", "typesafe", "jev-latest"); const tank = async (cmd) => (await tools.bash({ command: `tools/tankctl "${cmd}"` })).output; await tank("start --map assets/maps/night_arena.map"); const questions = { action: { type: "choice", instructions: "You control the tank '@' in a top-down tank game. " + "Choose the best next action.", criteria: { attack: "an enemy has line of sight to you and you can fire at it", approach: "no enemy has line of sight; drive toward the nearest enemy", dodge: "an enemy shot is heading at you and will hit soon", powerup: "a powerup is close and no enemy threatens you", }, }, }; function commandFor(choice, st) { const p = st.player; const enemy = st.enemies.filter(e => !e.dead) .sort((a, b) => (b.los - a.los) || (a.dist - b.dist))[0]; if (choice === "attack" && enemy) { return `fire_at tank ${enemy.id}; frames 30 until clear,damage,kill`; } if (choice === "dodge") { // move perpendicular to the closest incoming shot const s = st.projectiles.filter(s => !s.yours) .sort((a, b) => a.eta - b.eta)[0]; const dir = s && Math.abs(s.vel[0]) > Math.abs(s.vel[1]) ? (p.pos[1] > s.pos[1] ? "+down" : "+up") : (p.pos[0] > (s ? s.pos[0] : 0) ? "+right" : "+left"); return `input ${dir}; frames 20 until damage; input stop`; } const powerup = st.powerups.filter(u => u.available) .sort((a, b) => a.dist - b.dist)[0]; if (choice === "powerup" && powerup) { return `goto ${powerup.pos[0]} ${powerup.pos[1]} 180`; } return enemy ? `goto ${enemy.pos[0]} ${enemy.pos[1]} 90` : null; } const log = []; for (let step = 0; step < 30; step++) { const st = JSON.parse(await tank("state")); if (st.state !== "playing") break; const threats = st.projectiles .filter(s => !s.yours && s.miss_dist < 1.5 && s.eta < 1.5) .map(s => `incoming shot dist ${s.dist} eta ${s.eta}s`) .join("\n") || "no incoming shots"; const r = await models.classify(jev, { state: { map: await tank("view 8"), threats, hp: st.player.hp }, questions, }); if (r.stopReason !== "stop") { log.push(`#${step} classifier error: ${r.errorMessage}`); break; } const choice = r.answers.action.choice; const cmd = commandFor(choice, st); if (!cmd) break; log.push(`#${step} hp=${st.player.hp} ${choice} -> ${await tank(cmd)}`); } return log.join("\n"); Calling MCP Servers Lastly, Codemode obviously is great for calling MCP servers. And because we do not actually expose any of the MCP tools to the LLM, the agent first uses provided APIs to issue a tool search within Codemode to discover what it might be able to do with the connected servers. This form of progressive discovery makes the whole MCP business work well enough for a lot of use cases today. Here for instance you can see the agent reach for the Sentry MCP straight away, even without discovering the tools, presumably because it has learned during the RL process already about what the Sentry MCP looks like. But it learns from what we inject into the system prompt, that the Sentry server is available to begin with. It’s not completely guessing here. const orgs = await tools.mcp__sentry__find_organizations({}); const { organizations } = orgs.structuredContent; const results = await Promise.allSettled(organizations.map(org => tools.mcp__sentry__find_projects({ organizationSlug: org.slug, regionUrl: org.regionUrl, }) )); return organizations.map((org, i) => { const r = results[i]; if (r.status !== "fulfilled") return { org: org.slug, error: String(r.reason) }; if (r.value.isError) return { org: org.slug, error: r.value.content }; return { org: org.slug, projects: r.value.structuredContent.projects.map(p => p.slug), }; }); Modern MCP Is A Fight I really don’t want to talk too much about MCP here, but MCP is in fact a protocol that greatly benefits from Codemode. The problem in parts is that MCP in practice often targets harnesses that do not (yet?) use Codemode. But the tide is shifting. In the meantime, a temporary crutch has been to do what Cloudflare did, and do Codemode within the MCP server. But now we have Codemode in Codemode which is pretty bad. It means double JSON escaping, easy for smaller models to get confused by and the inner code cannot call the outer tools. So if you for instance use the Cloudflare MCP servers in Pi, the agent needs to write JavaScript and funnel it through more JavaScript. This is really not optimal, but it’s also understandable that this is happening: const accRes = await tools.mcp__cloudflare__execute({ code: `async () => { const r = await cloudflare.request({ method: "GET", path: "/accounts" }); return r.result.map(a => ({ id: a.id, name: a.name })); }`, }); const accounts = JSON.parse(accRes.content.map(c => c.text).join("")); const out = []; for (const account of accounts) { const r = await tools.mcp__cloudflare__execute({ account_id: account.id, code: `async () => { const r = await cloudflare.request({ method: "GET", path: \`/accounts/\${accountId}/workers/scripts\`, }); return r.result.map(s => ({ id: s.id, modified: s.modified_on })); }`, }); out.push({ account: account.name, workers: r.content.map(c => c.text).join("") }); } return out; MCP Desires So to end things off: how well does Codemode work with MCP today? Well … not amazingly well. That’s because MCP servers are not really targeting harnesses that use Codemode yet (though at this point I think most harnesses support it). For this to work well some recommendations: Structured content: Codemode wants calls to return some nicely formatted JSON. So that needs to come back from the server, and many don’t do that yet. The outputSchema system in MCP is great for that. Consistent results: an interesting failure case is when an MCP server does not return consistent data. For instance because it tries to token optimize things depending on how many items are in the result set. This can cause an initial probe with 5 items to succeed, but then fail when the server returns the maximum batch size. Large binary data: today MCP does not yet support large binary data so quite a few use cases that are really interesting do not work well at all yet. You end up with all kinds of weird workarounds such as pre-signed URLs to allow file uploads then to happen through non MCP channels. Composable tool search: the MCP server might know better than the MCP client which tool is appropriate for a task. But there is no good mechanism today that allows a harness to fan out tool searches across multiple MCP servers. It’s all emergent behavior and it does not scale well to multiple active servers. Future of Codemode So where does this leave us? Is this a reversal of what I wrote a year ago where I encouraged CLIs? I don’t think so. In fact, the MCP ecosystem from my perspective picked up on exactly what we pointed out a year ago works: code. But Codemode goes beyond MCP in that it can act as a capable mechanism within the harness to express more freedom for the agent. There are however also some things that we still need to figure out. For one, durability with Codemode is trickier. We might have to adopt some ideas from durable workflow engines here to snapshot invocations. Or maybe, something like Starlark is a better composition language than JavaScript given its deterministic nature. Images, binary data and just the inability of this pattern to work with smaller models is also something that needs to be fleshed out. So it’s for sure not a perfect solution yet, but it’s quite a useful pattern that I expect us to leverage more.
