Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
63
Often when designing systems, we aim for perfection in things like consistency of data, availability, latency, and more. The hardest part of system design is that it’s difficult (if not impossible) to design systems that have perfect consistency, perfect availability, incredibly low latency, and incredibly high throughput, all at the same time. Instead, when we approach system design, it’s best to treat each of these properties as points on different axes that we balance to find the “right fit” for the application we’re supporting. I recently made some major tradeoffs in the design of Bluesky’s Following Feed/Timeline to improve the performance of writes at the cost of consistency in a way that doesn’t negatively affect users but reduced P99s by over 96%. Timeline Fanout When you make a post on Bluesky, your post is indexed by our systems and persisted to a database where we can fetch it to hydrate and serve in API responses. Additionally, a reference to your post is “fanned out” to your followers so they can see it in their Timelines. This process involves looking up all of your followers, then inserting a new row into each of their Timeline tables in reverse chronological order with a reference to your post. When a user loads their Timeline, we fetch a page of post references and then hydrate the posts/actors concurrently to quickly build an API response and let them see the latest content from people they follow. The Timelines table is sharded by user. This means each user gets their own Timeline partition, randomly distributed among shards of our horizontally scalable database (ScyllaDB), replicated across multiple shards for high availability. Timelines are regularly trimmed when written to, keeping them near a target length and dropping older post references to conserve space. Hot Shards in Your Area Bluesky currently has around 32 Million Users and our Timelines database is broken into hundreds of shards. To support millions of partitions on such a small...
19th Feb 2025

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Jaz's Blog

Reverse Twins
2nd Oct 2025 • 24 votes
Turning Billions of Strings into Integers Every Second Without Collisions

I’ve recently started building a POC of a Redis RESP3 Wire Compatible Key/Value Database built on FoundationDB with @calabro.io and though it’s rather early, it’s already spawned a fun distributed systems problem that I thought would be interesting to share. Previously I’ve written about how I implemented a Graph DB via Roaring Bitmaps, representing relations as a bidirectional pair of sets. To support such use-cases in this new database, we’d like to represent sets of keys such that you can perform boolean operations on them (intersection, union, difference) relatively quickly even for very large sets (with millions of members). Supporting Larger Keys In the original Graph DB, we were representing user DID strings as uint32 UIDs to allow us to store millions of edge lists in very little space (e.g. the set of users who follow bsky.app) while being able to perform boolean operations between lists quickly (using Roaring Bitmaps’ parallel boolean operators). Since we were graphing follows, blocks, and other such User-to-User relationships, there was a practical maximum for the total number of user IDs in the low billions. We’ve continued exploring objects and relationships we’d like to represent as a Graph, and have realized that if we wanted to store e.g. the URIs of all posts a user has liked so we can intersect it with other users’ likes, we’re going to need a bigger keyspace! There are well over 15 Billion records in the AT Proto Ecosystem, each with a unique AT URI! Now our desired keyspace is much larger than can be represented by uint32 values and so we need to expand to uint64. Easy enough, let’s use the uint64 flavor of Roaring Bitmaps and simply intern URIs and User DIDs as uint64s, problem solved, right? Not quite… Interning Many Things at Once The AT Proto Firehose has hit historic peak traffic of over 1,500 evt/sec. We want to design a system that will handle many times more scale than we’ve ever seen in reality. This means designing for 10x or 100x would require us to be able to intern 15k to 150k new URIs per second into uint64 integers. Sounds easy enough, what’s the holdup? Well, in FoundationDB we’re able to use Transactions to do things like atomically increment a sequence safely when many other threads may be trying to do the same thing. This is simple enough to do in Go, we can just toss together a little helper function to acquire a new UID for our string: func (s *server) allocateNewUID(span trace.Span, tx fdb.Transaction) (uint64, error) { var newUID uint64 val, err := tx.Get(fdb.Key("last_uid")).Get() if err != nil { return 0, return fmt.Errorf("failed to get last UID: %w", err) } if len(val) == 0 { newUID = 1 // start from 1 } else { lastUID, err := strconv.ParseUint(string(val), 10, 64) if err != nil { return 0, return fmt.Errorf("failed to parse last UID: %w", err) } newUID = lastUID + 1 } tx.Set(fdb.Key("last_uid"), []byte(strconv.FormatUint(newUID, 10))) return newUID, nil } This function gets called from a fdb.Transaction which gets assigned a Transaction ID, then stages its changes, then tries to commit them. In FoundationDB, if your transaction is reading or modifying data written to by a different Transaction that finishes while you’re in-progress, your Transaction is thrown out and must be retried. For our UID assignment use-case, this is pretty problematic. We want to assign hundreds of thousands of new UIDs per second but if they’re all modifying the same key, concurrent transactions will constantly run into contention on the same data and will be forced to retry over and over again. This problem gets worse the more concurrent transactions you have trying to read from or write to the same key. Even if we stick to sequential access, if it takes ~5-10ms to assign a UID, we can only assign ~100-200 UIDs per second, nowhere near the throughput we need to support. How can we get past this problem and allow us to give strings unique uint64 UIDs in a high throughput and highly concurrent manner? Attempt #1: xxHash My first attempt to solve this problem was to try something that required no coordination and hash the string keys into uint64s using xxHash. xxHash is a non-cryptographic hash algorithm that supports incredibly high throughput (dozens of GB/sec) and can produce 64 bit unsigned integer hashes of strings trivially. Implementing this would look something like: Hash the incoming string key Lookup the uint64 UID to see if we’ve already assigned it to a string Reject the transaction if there’s a collision and give up Store the key in the UID map and the UID in the key map Use the UID for anything else we need While the uint64 keyspace is plenty large for our needs assuming we distribute evenly among the whole space, using a hashing algorithm with no coordination means there’s room for collisions and thus we’d need some additional logic (potentially by bucketing the keys somehow). Consulting the Birthday Problem we can see that a keyspace with 64 bit hashes has a >50% chance of containing a single collision when we have only ~5 billion keys in the set! That’s barely more keys than we can cram into a uint32 and definitely won’t suffice for the number of keys we expect to be storing! So, xxHash, while nice and coordination-free is probably not going to be the solution we need. What else can we do? Attempt #2: Billions of Sequences Incrementing one sequence is clearly not an option because we can only increment a single sequence ~100-200 times per second, but what if we instead had more than one sequence? Roaring Bitmaps managed to make highly efficient bitmap representations by breaking up a uint32 keyspace into a uint16-wide set of uint16-wide keyspaces. Can we do something similar here? Here’s an idea, what if we had just over 4 billion difference sequences and just picked one at random when we needed to assign a UID? Since we’re constructing our UIDs as a uint64, we can split the full UID into a pair of uint32s where the most-significant-bits are used to identify the sequence ID and the least-significant-bits are used to identify the value assigned to the UID within the sequence. So in our implementation, we get ~4.3 Billion sequence IDs that each have ~4.3 Billion incrementing values. As an example, if we were to randomly select Sequence ID 37 and then we increment that sequence to the value 5, we’d assemble the ID as 37<<32 + 5 which looks like 158,913,789,952 + 5 -> 158,913,789,957. Looking at the next Seuqence ID, we’d see 38 which, when left shifted by 32 gives us 163,208,757,248. You can see there’s a gap of ~4.3 billion values between the first UID assigned by each Sequence ID. Assuming we can increment a single sequence ~100 times per second with contention, we’re able to mint 430 Billion new UIDs per second without locking up (assuming the cluster can keep up). Storing ~4.3 billion sequences may be a bit expensive, but thankfully this strategy can scale up and down by picking a larger or smaller prefix size. If we only wanted to store say, ~16k sequences, we can pick a 14 bit prefix instead of a 32 bit prefix and then use a 50 bit sequence number. That spreads the load across 2^14 sequence IDs and significantly reduces storage requirements for Sequences. What does this look like in code? Well, it’s honestly not very complex! const uidSequencePrefix = "uid_sequence/" func (s *server) allocateNewUID(tx fdb.Transaction) (uint64, error) { // sequenceNum is the random uint32 sequence we are using for this allocation var sequenceNum uint32 var sequenceKey string // assignedUID is the uint32 within the sequence we will assign var assignedUID uint32 // Try up to 5 times to find a sequence that is not exhausted for range 5 { // Pick a random uint32 as the sequence we will be using for this UID sequenceNum = rand.Uint32() sequenceKey = fmt.Sprintf("%s%d", uidSequencePrefix, sequenceNum) val, err := tx.Get(fdb.Key(sequenceKey)).Get() if err != nil { return 0, fmt.Errorf("failed to get last UID: %w", err) } if len(val) == 0 { assignedUID = 1 // Start each sequence at 1 } else { lastUID, err := strconv.ParseUint(string(val), 10, 32) if err != nil { return 0, fmt.Errorf("failed to parse last UID: %w", err) } // If we have exhausted this sequence, pick a new random sequence if lastUID >= 0xFFFFFFFF { continue } assignedUID = uint32(lastUID) + 1 } } // If we failed to find a sequence after 5 tries, return an error if assignedUID == 0 { return 0, fmt.Errorf("failed to allocate new UID after 5 attempts") } // Assemble the 64-bit UID from the sequence ID and assigned UID newUID := (uint64(sequenceNum) << 32) | uint64(assignedUID) // Store the assigned UID back to the sequence key for the next allocation tx.Set(fdb.Key(sequenceKey), []byte(strconv.FormatUint(uint64(assignedUID), 10))) // Return the full 64-bit UID return newUID, nil } And there we go! We can now intern billions of strings per second with little to no contention in a distributed system while completely avoiding collisions and making full use of our keyspace! Conclusion Often times when designing distributed systems, patterns and strategies you see in seemingly unrelated libraries can inspire an elegant solution to the problem at hand. In the case of distributed, high-throughput string interning, horizontal scaling can be achieved by breaking up one large keyspace that requires strict coordination into billions of smaller keyspaces that can be randomly load-balanced across. Both patterns used in this technique are present elsewhere: Breaking up a large keyspace into a bunch of smaller keyspaces is present in Roaring Bitmaps (among other systems) Letting randomness and large numbers spread out resource contention is present in many load balancing systems This is one of my favorite parts of growing as an engineer: the more systems and strategies you familiarize yourself with, the more material you have to draw from when designing something new. Personal News A bit of personal news for y’all if you made it this far. Today is my last day as a member of the Bluesky team! The past 2+ years building out Bluesky’s Infrastructure and Platform team and scaling Bluesky from 100,000 -> 40,000,000 users have been the most intense and rewarding years of my life. I don’t have the words to express how much I’ve valued my time on the team and how much I care for the people I’ve worked with in what feels like a decade of real time. I’ve got some new adventures ahead and am excited to be embarking on a new journey within the next month (still building large-scale infrastructure, don’t worry). I plan to continue being involved in the AT Proto Community and to contribute to some cool projects other folks on the Bluesky team are working on from the FOSS space (like KVDB). To the team, I wish you all the best and will dearly miss getting to work with you all every day, but nothing lasts forever and I will always cherish the time I got to spend building an incredible platform with incredible people. If you’re interested in joining a world-class team doing important work, check out Bluesky’s open job listings here. There should be a new role opening up for a seasoned Go Engineer on the Platform team soon!

26th Sep 2025 • 1 votes
Emoji Griddle
30th Oct 2024 • 35 votes
Jetstream: Shrinking the AT Proto Firehose by >99%

Bluesky recently saw a massive spike in activity in response to Brazil’s ban of Twitter. As a result, the AT Proto event firehose provided by Bluesky’s Relay at bsky.network has increased in volume by a huge amount. The average event rate during this surge increased by ~1,300%. Before this new surge in activity, the firehose would produce around 24 GB/day of traffic. After the surge, this volume jumped to over 232 GB/day! Keeping up with the full, verified firehose quickly became less practical on cheap cloud infrastructure with metered bandwidth. To help reduce the burden of operating bots, feed generators, labelers, and other non-verifying AT Proto services, I built Jetstream as an alternative, lightweight, filterable JSON firehose for AT Proto. How the Firehose Works The AT Proto firehose is a mechanism used to keep verified, fully synced copies of the repos of all users. Since repos are represented as Merkle Search Trees, each firehose event contains an update to the user’s MST which includes all the changed blocks (nodes in the path from the root to the modified leaf). The root of this path is signed by the repo owner, and a consumer can keep their copy of the repo’s MST up-to-date by applying the diff in the event. For a more in-depth explanation of how Merkle Trees are constructed, check out this explainer. Practically, this means that for every small JSON record added to a repo, we also send along some number of MST blocks (which are content-addressed hashes and thus very information-dense) that are mostly useful for consumers attempting to keep a fully synced, verified copy of the repo. You can think of this as the difference between cloning a git repo v.s. just grabbing the latest version of the files without the .git folder. In this case, the firehose effectively streams the diffs for the repository with commits, signatures, and metadata, which is inherently heavier than a point-in-time checkout of the repo. Because firehose events with repo updates are signed by the repo owner, they allow a consumer to process events from any operator without having to trust the messenger. This is the “Authenticated” part of the Authenticated Transfer (AT) Protocol and is crucial to the correct functioning of the network. That being said, of the hundreds of consumers of Bluesky’s production Relay, >90% of them are building feeds, bots, and other tools that don’t keep full copies of the entire network and don’t verify MST operations at all. For these consumers, all they actually process is the JSON records created, updated, and deleted in each event. If consumers already trust the provider to do validation on their end, they could get by with a much more lightweight data stream. How Jetstream Works Jetstream is a streaming service that consumes an AT Proto com.atproto.sync.subscribeRepos stream and converts it into lightweight, friendly JSON. If you want to try it out yourself, you can connect to my public Jetstream instance and view all posts on Bluesky in realtime: $ websocat "wss://jetstream2.us-east.bsky.network/subscribe?wantedCollections=app.bsky.feed.post" Note: the above instance is operated by Bluesky PBC and is free to use, more instances are listed in the official repo Readme Jetstream converts the CBOR-encoded MST blocks produced by the AT Proto firehose and translates them into JSON objects that are easier to interface with using standard tooling available in programming languages. Since Repo MSTs only contain records in their leaf nodes, this means Jetstream can drop all of the blocks in an event except for those of the leaf nodes, typically leaving only one block per event. In reality, this means that Jetstream’s JSON firehose is nearly 1/10 the size of the full protocol firehose for the same events, but lacks the verifiability and signatures included in the protocol-level firehose. Jetstream events end up looking something like: { "did": "did:plc:eygmaihciaxprqvxpfvl6flk", "time_us": 1725911162329308, "type": "com", "commit": { "rev": "3l3qo2vutsw2b", "type": "c", "collection": "app.bsky.feed.like", "rkey": "3l3qo2vuowo2b", "record": { "$type": "app.bsky.feed.like", "createdAt": "2024-09-09T19:46:02.102Z", "subject": { "cid": "bafyreidc6sydkkbchcyg62v77wbhzvb2mvytlmsychqgwf2xojjtirmzj4", "uri": "at://did:plc:wa7b35aakoll7hugkrjtf3xf/app.bsky.feed.post/3l3pte3p2e325" } }, "cid": "bafyreidwaivazkwu67xztlmuobx35hs2lnfh3kolmgfmucldvhd3sgzcqi" } } Each event lets you know the DID of the repo it applies to, when it was seen by Jetstream (a time-based cursor), and up to one updated repo record as serialized JSON. Check out this 10 second CPU profile of Jetstream serving 200k evt/sec to a local consumer: By dropping the MST and verification overhead by consuming from relay we trust, we’ve reduced the size of a firehose of all events on the network from 232 GB/day to ~41GB/day, but we can do better. Jetstream and zstd I recently read a great engineering blog from Discord about their use of zstd to compress websocket traffic to/from their Gateway service and client applications. Since Jetstream emits marshalled JSON through the websocket for developer-friendliness, I figured it might be a neat idea to see if we could get further bandwidth reduction by employing zstd to compress events we send to consumers. zstd has two basic operating modes, “simple” mode and “streaming” mode. Streaming Compression At first glance, streaming mode seems like it’d be a great fit. We’ve got a websocket connection with a consumer and streaming mode allows the compression to get more efficient over the lifetime of the connection. I went and implemented a streaming compression version of Jetstream where a consumer can request compression when connecting and will get zstd compressed JSON sent as binary messages over the socket instead of plaintext. Unfortunately, this had a massive impact on Jetstream’s server-side CPU utilization. We were effectively compressing every message once per consumer as part of their streaming session. This was not a scalable approach to offering compression on Jetstream. Additionally, Jetstream stores a buffer of the past 24 hours (configurable) of events on disk in PebbleDB to allow consumers to replay events before getting transitioned into live-tailing mode. Jetstream stores serialized JSON in the DB, so playback is just shuffling the bytes into the websocket without having to round-trip the data into a Go struct. When we layer in streaming compression, playback becomes significantly more expensive because we have to compress outgoing events on-the-fly for a consumer that’s catching up. In real numbers, this increased CPU usage of Jetstream by 23% while lowering the throughput of playback from ~200k evt/sec to ~28k evt/sec for a single local consumer. When in streaming mode, we can’t leverage the bytes we compress for one consumer and reuse them for another consumer because zstd’s streaming context window may not be in sync between the two consumers. They haven’t received exactly the same data in the session so the clients on the other end don’t have their state machines in the same state. Since streaming mode’s primary advantage is giving us eventually better efficiency as the encoder learns about the data, what if we just taught the encoder about the data at the start and compress each message statelessly? Dictionary Mode zstd offers a mechanism for initializing an encoder/decoder with pre-optimized settings by providing a dictionary trained on a sample of the data you’ll be encoding/decoding. Using this dictionary, zstd essentially uses it’s smallest encoded representations for the most frequently seen patterns in the sample data. In our case, where we’re compressing serialized JSON with a common event shape and lots of common property names, training a dictionary on a large number of real events should allow us to represent the common elements among messages in the smallest number of bytes. For take two of Jetstream with zstd, let’s to use a single encoder for the whole service that utilizes a custom dictionary trained on 100,000 real events. We can use this encoder to compress every event as we see it, before persisting and emitting it to consumers. Now we end up with two copies of every event, one that’s just serialized JSON, and one that’s statelessly compressed to zstd using our dictionary. Any consumers that want compression can have a copy of the dictionary on their end to initialize a decoder, then when we broadcast the shared compressed event, all consumers can read it without any state or context issues. This requires the consumers and server to have a pre-shared dictionary, which is a major drawback of this implementation but good enough for our purposes. That leaves the problem of event playback for compression-enabled clients. An easy solution here is to just store the compressed events as well! Since we’re only sticking the JSON records into our PebbleDB, the actual size of the 24 hour playback window is <8GB with sstable compression. If we store a copy of the JSON serialized event and a copy of the zstd compressed event, this will, at most, double our storage requirements. Then during playback, if the consumer requests compression, we can just shuffle bytes out of the compressed version of the DB into their socket instead of having to move it through a zstd encoder. Savings Running with a custom dictionary, I was able to get the average Jetstream event down from 482 bytes to just 211 bytes (~0.44 compression ratio). Jetstream allows us to live tail all posts on Bluesky as they’re posted for as little as ~850 MB/day, and we could keep up with all events moving through the firehose during the Brazil Twitter Exodus weekend for 18GB/day (down from 232GB/day). With this scheme, Jetstream is required to compress each event only once before persisting it to disk and emitting it to connected consumers. The CPU impact of these changes is significant in proportion to Jetstream’s incredibly light load but it’s a flat cost we pay once no matter how many consumers we have. (CPU profile from a 30 second pprof sample with 12 consumers live-tailing Jetstream) Additionally, with Jetstream’s shared buffer broadcast architecture, we keep memory allocations incredibly low and the cost per consumer on CPU and RAM is trivial. In the allocation profile below, more than 80% of the allocations are used to consume the full protocol firehose. The total resident memory of Jetstream sits below 16MB, 25% of which is actually consumed by the new zstd dictionary. To bring it all home, here’s a screenshot from the dashboard of my public Jetstream instance serving 12 consumers all with various filters and compression settings, running on a $5/mo OVH VPS. At our new baseline firehose activity, a consumer of the protocol-level firehose would require downloading ~3.16TB/mo to keep up. A Jetstream consumer getting all created, updated, and deleted records without compression enabled would require downloading ~400GB/mo to keep up. A Jetstream consumer that only cares about posts and has zstd compression enabled can get by on as little as ~25.5GB/mo, <99% of the full weight firehose. Feel free to join the conversation about Jetstream and zstd on Bluesky.

24th Sep 2024 • 41 votes

More in AI

We Need to Embed Ethics Into Our Engineering Courses

In mid-September, Anthropic CEO Dario Amodei joined a growing group of researchers calling for the AI industry to slow down. While some have questioned these researchers’ motivation for voicing concerns about the rapid pace of development, the plea highlighted a key tension between safety and speed. The underlying problem is not simply a technical error, but a sociotechnical failure: Pressure to move quickly has made it hard for engineers to fully assess and manage new risks of increasingly powerful AI systems. It is a clear warning that responsible engineering requires more than technical expertise; engineers must also understand workplace pressures, moral and ethical rules, and the social consequences rooted in the systems they create. Cases like this, alongside invasive surveillance technologies and biased AI tools, have exposed serious shortcomings in how engineers recognize and address the ethical and social implications of their work. While the need to educate graduates in ethics and social responsibility is widely recognized, a persistent gap remains—these topics are too often treated as afterthoughts, disconnected from the technical content of engineering coursework. This siloed approach leaves graduates ill-equipped to address real-world challenges; indeed, only about a third of engineering professionals in the United States report having received training on their public welfare responsibilities during their formal engineering education. We are working to bridge this gap by designing two complementary instructional approaches, one that integrates social issues directly into foundation technical coursework and a second that prepares students to act as “public welfare watchdogs” through a standalone engineering course. By embedding ethics and social responsibility training directly into the core of undergraduate engineering coursework, we equip future professionals with the agency they will need to protect the public good. The gap in instruction about ethics and social responsibility Across the engineering profession, there is a persistent belief that technical skill-building is central to engineering, while ethical, social, and public welfare concerns are peripheral or distracting. This technical/social dualism is evident even in the evaluation of scholarly research. For instance, we recently submitted an academic position paper discussing this ethics education gap to a leading peer-reviewed engineering journal. It was dismissed by reviewers along the very same dualism it described. One reviewer warned that it would be “highly irresponsible” to challenge technical skill-building as “the foundation of any kind of engineering” by adding ethics and social responsibility training into engineering classrooms. Another reviewer objected that integrating these considerations into foundational courses risks “distracting the students from fully absorbing the technology.” These comments miss the core argument of our work: that ethics and social responsibility are integral to responsible technical practice. Today, accreditation agencies like ABET (formerly known as the Accreditation Board for Engineering and Technology) and professional licensing bodies like the National Society of Professional Engineers mandate student outcomes related to social and public welfare consequences. But these curriculum requirements tend to be unhelpfully generic and minimally enforced. As a result, engineering students often receive ethics instruction through introductory or non-engineering coursework, where the focus is on codes of conduct and academic integrity, rather than deeper topics of social responsibility. This instruction is almost always disconnected from the complex real-world dilemmas engineers face. Indeed, we find that only ethics instruction in actual engineering courses (not in other parts of the curriculum) can effectively instill ethical awareness as professionals. Concerningly, only about 4 in 10 engineers in the United States recall receiving this kind of training, and 3 in 10 report never receiving training on their public welfare responsibilities at any point in their careers. This leaves most of the profession underprepared to handle the complex challenges of engineering practice. A new approach: Integrating technical and social dimensions To advance a new paradigm—one that embeds ethics and social responsibility issues within the undergraduate engineering curriculum—we’ve designed two innovative course-based approaches: a library of one-hour sociotechnical modules for foundational circuits courses (described here and here) and a new engineering course dedicated to public welfare responsibilities. The first approach integrates discussions about relevant social issues directly into foundational engineering courses, where students begin to form their identities as engineers. “Introduction to Circuits” is typically the first course in the electrical engineering major and is a requirement for many other engineering disciplines, making it a powerful site for introducing social content and ethical reasoning. We designed several one-hour modules for this course, where social issues provide context for a technical topic, offering a manageable entry point for faculty who may feel ill-prepared to teach about ethics and social responsibility. While many engineering faculty recognize the importance of teaching students about these topics, they struggle to integrate them into their courses amid heavy teaching and research demands. To overcome these challenges, each module provides ready-to-use materials, including lecture slides with scripts and homework and exam problems. Integrating these modules directly into the course signals to students that sociotechnical awareness is not an “add-on,” but a required professional competency. Our module on hospital power prioritization, for instance, connects the technical topics of power and equivalent circuits to the ethical challenges of healthcare infrastructure. Students learn about the reality of “red outlets” in medical settings, debating the power needs of specific wards (e.g., the intensive care unit, operating rooms, and physiotherapy rooms) to decide which branches should receive emergency power during a hypothetical outage. Our conflict minerals module connects the technical topic of capacitors to the global tantalum supply chain. Students calculate the amount of tantalum used in cell phones and identify mining locations in regions like the Democratic Republic of the Congo, where revenue often fuels armed conflict and human rights abuses. Students research the conflict-mineral policies of major companies such as Apple and Samsung and reflect on their future roles as ethical designers and consumers. Student responses to these modules have been overwhelmingly positive. In interviews, students told us that the content helped them get away from the “stigma that engineers only worry about math” and made the work more meaningful. We plan to share the modules widely to help circuits instructors across the globe bridge the technical-social divide. Beyond these integrated modules, our second approach is a standalone engineering course—Public Welfare Responsibilities in Engineering (PWRE)—that helps round out students’ sociotechnical education by preparing them to fulfil their professional obligations as engineers. While most dedicated ethics courses end by introducing engineers’ abstract responsibilities, PWRE helps students recognize specific public welfare concerns, enhances their motivation to act when they encounter them, and teaches concrete, practical intervention strategies. In addition to teaching students about professional codes, such as the IEEE Code of Ethics, PWRE engages students in discussion of where these formal definitions may fall short, and how structural barriers such as workplace culture can hinder engineers from fulfilling these responsibilities. Then, the course equips students with intervention strategies. Students learn about their rights and responsibilities as whistleblowers, review tactics for seeking advice from professional societies, and practice writing op-eds to raise the alarm about ethical issues to the broader public. Our research shows that the course produces notable shifts in students’ ethical agency, strengthening their ability to serve as “public welfare watchdogs” in the modern workplace. Ethical engineers for the future Today’s engineering graduates face complex sociotechnical challenges beyond AI expansion, including climate change and global supply chains; they need tools to navigate ambiguity and act ethically in uncertain contexts. To meet these evolving demands, engineering education must move beyond outdated modes of instruction that focus only on building technical skills—some of which may soon be outsourced to AI. We must fundamentally reimagine the role of ethics and social responsibility, embedding them across the engineering curriculum and throughout the professional formation of engineers. All engineers learn to solve problems, but ethical engineers can also discern which problems are worth solving and anticipate the consequences of innovation for society. By centering ethics and social responsibility as core priorities of the profession, we can prepare engineers to be stewards of technology and trusted voices in public discourse. As one student reflected after participating in our modules, “we are a part of the issue if we don’t decide to fix it.”

54 minutes ago • 1 votes
Agentic Primitives 101

Inside the harnesses that let AI carry a job across hours, days, and conversations.

19 hours ago • 1 votes
Agent governance looks less like moral education and more like institutional economics

how agents turn ruthless on paper first

3 days ago • 1 votes
We are going to kill “unalive”

I talk to a lot of old people, those who were born in the 20th century, and if I ask them what the word “unalive” means, they usually have no idea what I’m talking about, except for some of them who have kids or who study Internet culture. This will, of course, probably seem very weird to most people who were born in or grew up in the 21st century. Just to recap for the olds: “unalive” is the word you use to represent concepts like dying, or death, or killing or being killed, on digital platforms where saying those words accurately will cause the algorithm to punish or censor you. Or, maybe, where the perception is that using those words will result in being censored by the algorithm, and no one is actually willing to find out what happens if you use the forbidden words. This sort of attack on people’s expression started on platforms like TikTok, where nearly all content is distributed through an algorithmic feed, but has since become ubiquitous in nearly all digital media that we see. In fact, these tics are now so prevalent that it’s routine to hear people using this kind of language in everyday life, even though there’s not yet an algorithm to appease in the physical world. I’ve heard people say, out loud, “he unalived himself”, in reference to someone dying by suicide. And all of this has become even more visible in recent days as online conversation has turned to discussion of the horrific lack of accountability around the tragic rape case at Cornell University. Across the Internet, people are routinely referring to the central crime in the case as r*pe or “grape” or even using the 🍇 emoji, without a second thought for what it means that the very word can’t be said online anymore. Or, at least, the assumption is that it can’t be said. To be clear, I am very much in favor of people using content warnings or sensitivity markers for content, and fine with people using abbreviations like “SA” for references to disturbing or triggering topics like sexual assault; we should provide people with as much context and control as possible when choosing what information they want to consume and when. I also know that sometimes, people use lesser terms for stressful subjects like death or assault to create a bit of ironic distance from painful or upsetting topics. But most of the different variations of wording and emojis are coming from trying to appease the platforms, and there’s a heavy cost for those who are worried about being mindful: If someone is using a tool to filter out content, it will no longer be effective because everyone is using misspellings and euphemisms and imagery to get around the algorithm. The spread of censored and mangled syntax is happening because people believe, or have experienced, that platforms will silence them for accurately describing the world in plain language. This shit is terrible, and it has to stop. You Were Not Born With These Constraints One of the things that’s most concerning to me is that an entire generation has grown up not realizing how extreme it is that their very language is being chosen for them by platforms run by people who hate that generation’s ability to express itself, and who hate the things it has to say. From their youngest days, this generation grew up watching people make stupid faces at them for YouTube thumbnails and never had a chance to reflect on the fact that those creators didn’t want to be humiliating themselves by making those expressions — the demands of the algorithms of Big Tech forced them to do that. The rituals of feeding the algorithm are so built into people’s everyday habits that they’re invisible to people who weren’t alive before today’s platforms took over. Every parent of my cohort remembers the first time they heard their toddler finish doing something cute in their living room, and then turn around and say, “please like and subscribe!” afterwards. It’s a ghastly, sickening feeling to confront the fact that our little kids were being brainwashed into thinking that every adorable thing they did should be followed by a prompt to provide data to Google. Over on Instagram, where people originally signed up thinking they were going to see someone’s vacation pictures, or shots of their cousin’s kids, you’re now stuck watching people beg for everyone to reply with cultish phrases in the comments, which will then earn them an obviously AI-generated response in return, all in service of “showing activity” to the algorithm, like it’s an angry god that needs a sacrifice. They’re just not sure exactly what the angry god wants. Your free speech was taken away from you, and the people who did it are the same ones who spent years pretending to care about “free expression”. They contrived examples of lack of free speech on college campuses while squashing protests, and cried crocodile tears about “cancel culture” while getting people fired for political criticism. Now they have no problem with billionaires deciding exactly what words everyone is allowed to say. Larry Ellison is not content with his family owning all of the movies and TV shows — his family has to control what words people are allowed to speak on TikTok, too. Elon Musk isn’t content to merely generate and distribute child sexual abuse material for profit — he wants to silence the messages of the few decent people who are foolish enough to remain on Twitter/X, too. (That’s why I wrote you a guide on how to get your organization off of that cursed platform.) Now that an entire generation has grown up using these Orwellian euphemisms, and all of the Big AI products are trained on the Internet that was created under this regime, do you think today’s AI tools even know that the real, uncensored world exists? If you can’t say “genocide” on any of the major platforms, yet those are the ones all of the Big AI tools used as their training data... well, then the AI tools sure aren’t very likely to know much about genocide, are they? Fuck the Algorithm Our creativity can be constrained by the language we use — our imaginations are limited by what we can think to say. If we’re trained to limit the words we speak just by habit, and those limits are put in place by people whose social, political, cultural and moral goals are the opposite of what we value, then our work is unalive before it is even born. The answer to this is simple: say what you mean. This will take, to some degree, courage. It may even take, I hesitate to say, some sacrifice. When I suggest this course of action to people, they inevitably say, “But it will cost me audience!” or “But what if I lose followers!” or “What if they demonetize me!” Okay, what if they do? What if they do. Are you willing to push on this? To make a point about it? To move to platforms where you can actually say what you mean? Or to remember that you already have platforms where you can say what you mean? On an email newsletter or podcast you can say whatever the hell you want and nobody can stop you. On my blog right here, I can even curse in a headline and it won’t affect anything about how my site operates. (And a reminder: Substack is not an email newsletter, and a Spotify show is not a podcast — they’ll be unaliving your distribution any day now.) If you are a 20th century relic like me, it is incumbent upon you to remind the generation that grew up inside the algorithm that another world is possible, and that we know this because we lived it. We were able to style a MySpace page in any way that we wanted; the code for LiveJournal was entirely open source so there was no part of the algorithm that was unknowable. A blog like the one you’re reading right now could be made by anyone, and put up for pennies, and nobody could stop it from being read by millions of people. (And that last one? It’s still possible.) If you are from this century, forget all that rambling bullshit about ancient history: all that matters is you getting what you deserve, because you’ve been fucked over by the same billionaires who’ve poisoned your planet and infested your world with slop. The best artists around you are invisible to you, and the most important statements by activists that you care about are being silenced. It’s not a conspiracy, it’s a system working as designed. And the proof is as obvious as the fact that the angriest activists you know can’t even talk about systemic abuses or state violence without having to put it in algorithmically approved speech or censoring their captions like they’re going to be read by 5-year-olds. It should make you furious. It is time to kill “unalive”. The response is simple: For every message you put out, start by saying what you mean. Don’t work backwards from what the algorithm wants or what a platform permits. Build a presence on every platform you can, even the ones where you have fewer followers or where you’re harder to find. Tell your audience that your speech being free matters more than corporate convenience. Keep saying it, and keep it positive: independence gets them better art, better information, and more connected communities. Build alliances with other artists, activists and people who share your values, and let them know you’re going to start sharing your work uncensored. Start releasing your work uncensored and see where the platforms push back. (You may be surprised: sometimes you were censoring yourself in anticipation of limits that weren’t even there.) If a platform does try to limit your reach or expression, make a LOT OF NOISE about it. Tell the press, rally your alliance, and spread the word on your other platforms, using the moment to build audience and raise support there. Get others to amplify the parts of your work that don’t violate platform policy, so the controversy drives people to the rest. Find the others pushing back on algorithmic control of expression, and raise and praise their work when they do the same. If we keep accepting the words that are forced upon us by TikTok and Meta and Google and all the rest, while platforms like Twitter/X allow the most hateful and harmful content in the world to be distributed completely unfettered, we’ll only see authoritarianism rise, and the harms against the vulnerable accelerate. But what breaks my heart almost as much is that we’ll see so many brilliant artists and activists and thinkers whose genius will be muted or silenced by mindless, heartless algorithms that capriciously decide who gets to say exactly what words, in what ways. I get angry every time I think about it. The tech tycoons get ever more brazen in what they’re willing to say publicly, boasting about how they’re going to cause the end of the world, or calling for ethnic cleansing, all while putting tighter and tighter reins on the speech and expression of ordinary people. It’s time for “unalive” to die.

3 days ago • 1 votes
What is Codemode

More than a year ago I wrote a few posts here that recommended people not to load custom tools into their context (or MCP servers) but to just use more scripts. Most importantly I wrote that Code Is All You Need and I wrote about that MCP needs code. With Pi 1.0 we now added MCP support via Codemode which in some ways is a long time coming, but then also maybe somewhat surprising to some. So I want to share some updated thoughts on this blog on what this all means. What Are Tools When a harness like Pi provides tools for an LLM to call, it does so by supplying some tool definitions which then translate into some token structure on the server side. Whether a model is encouraged to call a tool is the result of the reinforcement learning process. Something I wrote about before if you want to learn more. One of the reasons we strongly lean towards CLI and bash is because it allows easy composition of calls, and because the model also learns how the file system works when it’s trained. So when it invokes a tool like echo foo > /tmp/test.txt the model also learns that after that tool call, there is now a file called test.txt in /tmp. However bash has one fundamental limitation which is that it can only compose programs that run. And there are some things, which are not programs, but native tools to the LLM and they sort of have to be. The most obvious example here is read or view_image. If a multimodal model needs to read an image, it cannot use cat for that because the harness needs to inject the actual image payload into the protocol of the LLM. Another quite vivid example are sub agents. In order to spawn and orchestrate sub agents, it’s tricky to avoid tools that are provided by the harness. While in theory the agent could provide a CLI tool that talks to the outer harness via environment variables and Unix sockets, it’s a rather crude process. It however has another issue, and that is where the code runs. Brains vs Hands To better understand that, it’s important to think a bit more about where all the bits and pieces run. There really usually are two different systems involved. The first is the brain, the harness: it runs on one machine. It’s trusted. The second is often the same machine, but it’s really where the tools are executing: the hands. In Pi we now call this the execution environment, but you can think of it as the target of all the operations. Crucially what is important for us, is that there is a dividing line between the harness brain and the target environment that runs bash and executes the tools. And splitting this in half has some really important consequences. For a start it means that they are running on different file systems and they have different levels of trust. If you for instance use a sandboxing solution like Gondolin your bash stuff will be sandboxed just fine, but the harness itself will not be. Orchestrating The Harness Which brings us to what Codemode really does: it’s a way for the LLM to express and orchestrate complex operations on the harness side, but not the execution environment side. Codemode runs in the harness, in its own sandbox. In case of Pi it’s running in QuickJS within a WASM runtime with intentional limitations: no network, no file system, no timers, limited RAM. The only way is to call more tools. You could also imagine that Codemode could run Scheme or some other language as well. If you are not familiar with Codemode, it’s basically just a way to issue tool calls from within some language, in our case JavaScript. That allows you to compose those calls without necessarily going through the LLM’s context. Credit for naming goes to our friends at Cloudflare who coined it. For instance if you issue a bash call as a regular tool call in the LLM, then we only throw the trailing 2000 lines into the context and if the agent wants more, it needs to look at the overflow file itself. If however the agent issues that invocation via Codemode, then the Codemode side gets larger outputs sent structurally. Most importantly, because Codemode is JavaScript the agent can express concurrent operations and basic workflows. A common way in which you see agents now use this, is to first probe at 5-10 items from some tool response to see what it looks like, and to then write a Codemode script that processes the next n items. Codemode also allows you to throw state into the transcript! That means that one Codemode invocation can stash away data, that the next call in the session can load again. And remember: this is on the harness host, not the sandbox. In case of Pi, Codemode also allows you to issue calls that naturally do not make any sense in Pi’s traditional interface. For instance if you want to generate images with an image model or you want to classify some text with a one shot classifier model, those Pi APIs are exposed via Codemode, but not via regular tools where they would just waste context. What It Looks Like So now that we talked a bunch about it, it’s probably worth being a bit more explicit about it. Let’s walk ourselves through some invocations of Codemode of recent Pi sessions of mine. Note that none of this code is human written. It’s from real sessions of Pi, just re-indented for your viewing pleasure. The agent starts using Codemode automatically either because it’s a task where the model already naturally picks up that tool, or because a user asked it to. Note that Codemode is by default only enabled in Pi when MCP is enabled, but you can turn it on with "defaultTools": ["+codemode"] in the settings. Just ask Pi to enable it for you. Generating Images Let’s start simple with image generation. Image generation is a feature that Pi supports in the AI SDK core, but it’s not a tool that the agent can use. In the past the only way to use image models has been to write a bespoke extension or to have the agent run node itself and use the internal image APIs. However because we expose quite a few of the internal model APIs within Codemode, it means that the agent can use it: const [painter] = await models.getAvailableOfType("image"); const result = await models.generateImages(painter, { input: [{ type: "text", text: "A cute little puppy sitting on a grassy " + "lawn, soft natural light, photorealistic" }], }); if (result.stopReason !== "stop") return result.errorMessage; for (const block of result.output) { if (block.type === "image") image(block); else text(block.text); } Note that the call to image() sends the image back as image content to the LLM. On the harness side it feeds it directly into both the agent, as well as onto disk as a temporary artifact in case the agent wants to be able to pass that image back to bash. Classifying Things Similar things apply to classifier models such as Jev. They also do not fit well into the workflows of an agent through the typical tools. But rather than making a bespoke tool available, Codemode just allows the agent to reach into the AI SDK and invoke those directly. Here you can see how Jev is used to mass process GitHub issues for a quick sentiment analysis: const jev = await models.getModelOfType("classifier", "typesafe", "jev-latest"); const r = await tools.bash({ command: "gh issue list --state open --limit 100 " + "--json number,title,body,comments", }); const issues = JSON.parse(r.output); const results = await Promise.all(issues.map(async (issue) => { const res = await models.classify(jev, { state: { title: issue.title, body: (issue.body || "").slice(0, 4000), comments: issue.comments.slice(-5).map(c => c.body.slice(0, 800)), }, questions: { sentiment: { type: "choice", instructions: "What is the overall sentiment of the author towards pi?", criteria: { positive: "Appreciative, happy, constructive praise", neutral: "Matter-of-fact report or request without emotion", negative: "Frustrated, annoyed, upset, or angry", }, }, frustration: { type: "score", instructions: "How frustrated is the reporter?", criteria: ["not at all", "mildly", "clearly frustrated", "very angry"], }, kind: { type: "choice", instructions: "What kind of issue is this?", criteria: { bug: "Bug report or regression", feature: "Feature request or enhancement", question: "Question or support request", other: "Docs, discussion, meta, spam", }, }, }, }); if (res.stopReason !== "stop") { return { n: issue.number, title: issue.title, error: res.errorMessage }; } return { n: issue.number, title: issue.title, ...res.answers }; })); store("sentiment_results", results); return results .filter(r => !r.error) .sort((a, b) => b.frustration.score - a.frustration.score) .slice(0, 12) .map(r => `#${r.n} ${r.frustration.score.toFixed(2)} [${r.kind.choice}] ${r.title}`); Note how in that above example we also call store() which dumps the result of that execution into the session transcript. A future invocation of Codemode can thus read back that result if it wants to. The Promise.all here is fine, because Pi limits the total number of concurrent tool executions itself to four and maintains a queue for the rest. A more adventurous example is to use Jev to drive a game engine for debugging purposes: Codemode with Jev for Game Debugging Here it knows about my tankctl command and it built itself quickly a minimal harness around it to drive a game loop to assist a user with debugging a problem. Note how it built a 30 step loop in which each step goes back to both the game engine to get a text dump of what’s going on, and then to Jev to determine what to do next: const jev = await models.getModelOfType("classifier", "typesafe", "jev-latest"); const tank = async (cmd) => (await tools.bash({ command: `tools/tankctl "${cmd}"` })).output; await tank("start --map assets/maps/night_arena.map"); const questions = { action: { type: "choice", instructions: "You control the tank '@' in a top-down tank game. " + "Choose the best next action.", criteria: { attack: "an enemy has line of sight to you and you can fire at it", approach: "no enemy has line of sight; drive toward the nearest enemy", dodge: "an enemy shot is heading at you and will hit soon", powerup: "a powerup is close and no enemy threatens you", }, }, }; function commandFor(choice, st) { const p = st.player; const enemy = st.enemies.filter(e => !e.dead) .sort((a, b) => (b.los - a.los) || (a.dist - b.dist))[0]; if (choice === "attack" && enemy) { return `fire_at tank ${enemy.id}; frames 30 until clear,damage,kill`; } if (choice === "dodge") { // move perpendicular to the closest incoming shot const s = st.projectiles.filter(s => !s.yours) .sort((a, b) => a.eta - b.eta)[0]; const dir = s && Math.abs(s.vel[0]) > Math.abs(s.vel[1]) ? (p.pos[1] > s.pos[1] ? "+down" : "+up") : (p.pos[0] > (s ? s.pos[0] : 0) ? "+right" : "+left"); return `input ${dir}; frames 20 until damage; input stop`; } const powerup = st.powerups.filter(u => u.available) .sort((a, b) => a.dist - b.dist)[0]; if (choice === "powerup" && powerup) { return `goto ${powerup.pos[0]} ${powerup.pos[1]} 180`; } return enemy ? `goto ${enemy.pos[0]} ${enemy.pos[1]} 90` : null; } const log = []; for (let step = 0; step < 30; step++) { const st = JSON.parse(await tank("state")); if (st.state !== "playing") break; const threats = st.projectiles .filter(s => !s.yours && s.miss_dist < 1.5 && s.eta < 1.5) .map(s => `incoming shot dist ${s.dist} eta ${s.eta}s`) .join("\n") || "no incoming shots"; const r = await models.classify(jev, { state: { map: await tank("view 8"), threats, hp: st.player.hp }, questions, }); if (r.stopReason !== "stop") { log.push(`#${step} classifier error: ${r.errorMessage}`); break; } const choice = r.answers.action.choice; const cmd = commandFor(choice, st); if (!cmd) break; log.push(`#${step} hp=${st.player.hp} ${choice} -> ${await tank(cmd)}`); } return log.join("\n"); Calling MCP Servers Lastly, Codemode obviously is great for calling MCP servers. And because we do not actually expose any of the MCP tools to the LLM, the agent first uses provided APIs to issue a tool search within Codemode to discover what it might be able to do with the connected servers. This form of progressive discovery makes the whole MCP business work well enough for a lot of use cases today. Here for instance you can see the agent reach for the Sentry MCP straight away, even without discovering the tools, presumably because it has learned during the RL process already about what the Sentry MCP looks like. But it learns from what we inject into the system prompt, that the Sentry server is available to begin with. It’s not completely guessing here. const orgs = await tools.mcp__sentry__find_organizations({}); const { organizations } = orgs.structuredContent; const results = await Promise.allSettled(organizations.map(org => tools.mcp__sentry__find_projects({ organizationSlug: org.slug, regionUrl: org.regionUrl, }) )); return organizations.map((org, i) => { const r = results[i]; if (r.status !== "fulfilled") return { org: org.slug, error: String(r.reason) }; if (r.value.isError) return { org: org.slug, error: r.value.content }; return { org: org.slug, projects: r.value.structuredContent.projects.map(p => p.slug), }; }); Modern MCP Is A Fight I really don’t want to talk too much about MCP here, but MCP is in fact a protocol that greatly benefits from Codemode. The problem in parts is that MCP in practice often targets harnesses that do not (yet?) use Codemode. But the tide is shifting. In the meantime, a temporary crutch has been to do what Cloudflare did, and do Codemode within the MCP server. But now we have Codemode in Codemode which is pretty bad. It means double JSON escaping, easy for smaller models to get confused by and the inner code cannot call the outer tools. So if you for instance use the Cloudflare MCP servers in Pi, the agent needs to write JavaScript and funnel it through more JavaScript. This is really not optimal, but it’s also understandable that this is happening: const accRes = await tools.mcp__cloudflare__execute({ code: `async () => { const r = await cloudflare.request({ method: "GET", path: "/accounts" }); return r.result.map(a => ({ id: a.id, name: a.name })); }`, }); const accounts = JSON.parse(accRes.content.map(c => c.text).join("")); const out = []; for (const account of accounts) { const r = await tools.mcp__cloudflare__execute({ account_id: account.id, code: `async () => { const r = await cloudflare.request({ method: "GET", path: \`/accounts/\${accountId}/workers/scripts\`, }); return r.result.map(s => ({ id: s.id, modified: s.modified_on })); }`, }); out.push({ account: account.name, workers: r.content.map(c => c.text).join("") }); } return out; MCP Desires So to end things off: how well does Codemode work with MCP today? Well … not amazingly well. That’s because MCP servers are not really targeting harnesses that use Codemode yet (though at this point I think most harnesses support it). For this to work well some recommendations: Structured content: Codemode wants calls to return some nicely formatted JSON. So that needs to come back from the server, and many don’t do that yet. The outputSchema system in MCP is great for that. Consistent results: an interesting failure case is when an MCP server does not return consistent data. For instance because it tries to token optimize things depending on how many items are in the result set. This can cause an initial probe with 5 items to succeed, but then fail when the server returns the maximum batch size. Large binary data: today MCP does not yet support large binary data so quite a few use cases that are really interesting do not work well at all yet. You end up with all kinds of weird workarounds such as pre-signed URLs to allow file uploads then to happen through non MCP channels. Composable tool search: the MCP server might know better than the MCP client which tool is appropriate for a task. But there is no good mechanism today that allows a harness to fan out tool searches across multiple MCP servers. It’s all emergent behavior and it does not scale well to multiple active servers. Future of Codemode So where does this leave us? Is this a reversal of what I wrote a year ago where I encouraged CLIs? I don’t think so. In fact, the MCP ecosystem from my perspective picked up on exactly what we pointed out a year ago works: code. But Codemode goes beyond MCP in that it can act as a capable mechanism within the harness to express more freedom for the agent. There are however also some things that we still need to figure out. For one, durability with Codemode is trickier. We might have to adopt some ideas from durable workflow engines here to snapshot invocations. Or maybe, something like Starlark is a better composition language than JavaScript given its deterministic nature. Images, binary data and just the inability of this pattern to work with smaller models is also something that needs to be fleshed out. So it’s for sure not a perfect solution yet, but it’s quite a useful pattern that I expect us to leverage more.

3 days ago • 1 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in