Full Width 
FOCUS MODE
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from exist
Often when designing systems, we aim for perfection in things like consistency of data, availability, latency, and more. The hardest part of system design is that it’s difficult (if not impossible) to design systems that have perfect consistency, perfect availability, incredibly low latency, and incredibly high throughput, all at the same time. Instead, when we approach system design, it’s best to treat each of these properties as points on different axes that we balance to find the “right fit” for the application we’re supporting. I recently made some major tradeoffs in the design of Bluesky’s Following Feed/Timeline to improve the performance of writes at the cost of consistency in a way that doesn’t negatively affect users but reduced P99s by over 96%. Timeline Fanout When you make a post on Bluesky, your post is indexed by our systems and persisted to a database where we can fetch it to hydrate and serve in API responses. Additionally, a reference to your post is “fanned out” to your followers so they can see it in their Timelines. This process involves looking up all of your followers, then inserting a new row into each of their Timeline tables in reverse chronological order with a reference to your post. When a user loads their Timeline, we fetch a page of post references and then hydrate the posts/actors concurrently to quickly build an API response and let them see the latest content from people they follow. The Timelines table is sharded by user. This means each user gets their own Timeline partition, randomly distributed among shards of our horizontally scalable database (ScyllaDB), replicated across multiple shards for high availability. Timelines are regularly trimmed when written to, keeping them near a target length and dropping older post references to conserve space. Hot Shards in Your Area Bluesky currently has around 32 Million Users and our Timelines database is broken into hundreds of shards. To support millions of partitions on such a small number of shards, each user’s Timeline partition is colocated with tens of thousands of other users’ Timelines. Under normal circumstances with all users behaving well, this doesn’t present a problem as the work of an individual Timeline is small enough that a shard can handle the work of tens of thousands of them without being heavily taxed. Unfortunately, with a large number of users, some of them will do abnormal things like… well… following hundreds of thousands of other users. Generally, this can be dealt with via policy and moderation to prevent abusive users from causing outsized load on systems, but these processes take time and can be imperfect. When a user follows hundreds of thousands of others, their Timeline becomes hyperactive with writes and trimming occurring at massively elevated rates. This load slows down the individual operations to the user’s Timeline, which is fine for the bad behaving user, but causes problems to the tens of thousands of other users sharing a shard with them. We typically call this situation a “Hot Shard”: where some resident of a shard has “hot” data that is being written to or read from at much higher rates than others. Since the data on the shard is only replicated a few times, we can’t effectively leverage the horizontal scale of our database to process all this additional work. Instead, the “Hot Shard” ends up spending so much time doing work for a single partition that operations to the colocated partitions slow down as well. Stacking Latencies Returning to our Fanout process, let’s consider the case of Fanout for a user followed by 2,000,000 other users. Under normal circumstances, writing to a single Timeline takes an average of ~600 microseconds. If we sequentially write to the Timelines of our user’s followers, we’ll be sitting around for 20 minutes at best to Fanout this post. If instead we concurrently Fanout to 1,000 Timelines at once, we can complete this Fanout job in ~1.2 seconds. That sounds great, except it oversimplifies an important property of systems: tail latencies. The average latency of a write is ~600 microseconds, but some writes take much less time and some take much more. In fact, the P99 latency of writes to the Timelines cluster can be as high as 15 milliseconds! What does this mean for our Fanout? Well, if we concurrently write to 1,000 Timelines at once, statistically we’ll see 10 writes as slow as or slower than 15 milliseconds. In the case of timelines, each “page” of followers is 10,000 users large and each “page” must be fanned out before we fetch the next page. This means that our slowest writes will hold up the fetching and Fanout of the next page. How does this affect our expected Fanout time? Each “page” will have ~100 writes as slow as or slower than the P99 latency. If we get unlucky, they could all stack up on a single routine and end up slowing down a single page of Fanout to 1.5 seconds. In the worst case, for our 2,000,000 Follower celebrity, their post Fanout could end up taking as long as 5 minutes! That’s not even considering P99.9 and P99.99 latencies which could end up being >1 second, which could leave us waiting tens of minutes for our Fanout job. Now imagine how bad this would be for a user with 20,000,000+ Followers! So, how do we fix the problem? By embracing imperfection, of course! Lossy Timelines Imagine a user who follows hundreds of thousands of others. Their Timeline is being written to hundreds of times a second, moving so fast it would be humanly impossible to keep up with the entirety of their Timeline even if it was their full-time job. For a given user, there’s a threshold beyond which it is unreasonable for them to be able to keep up with their Timeline. Beyond this point, they likely consume content through various other feeds and do not primarily use their Following Feed. Additionally, beyond this point, it is reasonable for us to not necessarily have a perfect chronology of everything posted by the many thousands of users they follow, but provide enough content that the Timeline always has something new. Note in this case I’m using the term “reasonable” to loosely convey that as a social media service, there must be a limit to the amount of work we are expected to do for a single user. What if we introduce a mechanism to reduce the correctness of a Timeline such that there is a limit to the amount of work a single Timeline can place on a DB shard. We can assert a reasonable limit for the number of follows a user should have to have a healthy and active Timeline, then increase the “lossiness” of their Timeline the further past that limit they go. A loss_factor can be defined as min(reasonable_limit/num_follows, 1) and can be used to probabilistically drop writes to a Timeline to prevent hot shards. Just before writing a page in Fanout, we can generate a random float between 0 and 1, then compare it to the loss_factor of each user in the page. If the user’s loss_factor is smaller than the generated float, we filter the user out of the page and don’t write to their Timeline. Now, users all have the same number of “follows worth” of Fanout. For example with a reasonable_limit of 2,000, a user who follows 4,000 others will have a loss_factor of 0.5 meaning half the writes to their Timeline will get dropped. For a user following 8,000 others, their loss factor of 0.25 will drop 75% of writes to their Timeline. Thus, each user has a effective ceiling on the amount of Fanout work done for their Timeline. By specifying the limits of reasonable user behavior and embracing imperfection for users who go beyond it, we can continue to provide service that meets the expectations of users without sacrificing scalability of the system. Aside on Caching We write to Timelines at a rate of more than one million times a second during the busy parts of the day. Looking up the number of follows of a given user before fanning out to them would require more than one million additional reads per second to our primary database cluster. This additional load would not be well received by our database and the additional cost wouldn’t be worth the payoff for faster Timeline Fanout. Instead, we implemented an approach that caches high-follow accounts in a Redis sorted set, then each instance of our Fanout service loads an updated version of the set into memory every 30 seconds. This allows us to perform lookups of follow counts for high-follow accounts millions of times per second per Fanount service instance. By caching values which don’t need to be perfect to function correctly in this case, we can once again embrace imperfection in the system to improve performance and scalability without compromising the function of the service. Results We implemented Lossy Timelines a few weeks ago on our production systems and saw a dramatic reduction in hot shards on the Timelines database clusters. In fact, there now appear to be no hot shards in the cluster at all, and the P99 of a page of Fanout work has been reduced by over 90%. Additionally, with the reduction in write P99s, the P99 duration for a full post Fanout has been reduced by over 96%. Jobs that used to take 5-10 minutes for large accounts now take <10 seconds. Knowing where it’s okay to be imperfect lets you trade consistency for other desirable aspects of your systems and scale ever higher. There are plenty of other places for improvement in our Timelines architecture, but this step was a big one towards improving throughput and scalability of Bluesky’s Timelines. If you’re interested in these sorts of problems and would like to help us build the core data services that power Bluesky, check out this job listing. If you’re interested in other open positions at Bluesky, you can find them here.
Bluesky recently saw a massive spike in activity in response to Brazil’s ban of Twitter. As a result, the AT Proto event firehose provided by Bluesky’s Relay at bsky.network has increased in volume by a huge amount. The average event rate during this surge increased by ~1,300%. Before this new surge in activity, the firehose would produce around 24 GB/day of traffic. After the surge, this volume jumped to over 232 GB/day! Keeping up with the full, verified firehose quickly became less practical on cheap cloud infrastructure with metered bandwidth. To help reduce the burden of operating bots, feed generators, labelers, and other non-verifying AT Proto services, I built Jetstream as an alternative, lightweight, filterable JSON firehose for AT Proto. How the Firehose Works The AT Proto firehose is a mechanism used to keep verified, fully synced copies of the repos of all users. Since repos are represented as Merkle Search Trees, each firehose event contains an update to the user’s MST which includes all the changed blocks (nodes in the path from the root to the modified leaf). The root of this path is signed by the repo owner, and a consumer can keep their copy of the repo’s MST up-to-date by applying the diff in the event. For a more in-depth explanation of how Merkle Trees are constructed, check out this explainer. Practically, this means that for every small JSON record added to a repo, we also send along some number of MST blocks (which are content-addressed hashes and thus very information-dense) that are mostly useful for consumers attempting to keep a fully synced, verified copy of the repo. You can think of this as the difference between cloning a git repo v.s. just grabbing the latest version of the files without the .git folder. In this case, the firehose effectively streams the diffs for the repository with commits, signatures, and metadata, which is inherently heavier than a point-in-time checkout of the repo. Because firehose events with repo updates are signed by the repo owner, they allow a consumer to process events from any operator without having to trust the messenger. This is the “Authenticated” part of the Authenticated Transfer (AT) Protocol and is crucial to the correct functioning of the network. That being said, of the hundreds of consumers of Bluesky’s production Relay, >90% of them are building feeds, bots, and other tools that don’t keep full copies of the entire network and don’t verify MST operations at all. For these consumers, all they actually process is the JSON records created, updated, and deleted in each event. If consumers already trust the provider to do validation on their end, they could get by with a much more lightweight data stream. How Jetstream Works Jetstream is a streaming service that consumes an AT Proto com.atproto.sync.subscribeRepos stream and converts it into lightweight, friendly JSON. If you want to try it out yourself, you can connect to my public Jetstream instance and view all posts on Bluesky in realtime: $ websocat "wss://jetstream2.us-east.bsky.network/subscribe?wantedCollections=app.bsky.feed.post" Note: the above instance is operated by Bluesky PBC and is free to use, more instances are listed in the official repo Readme Jetstream converts the CBOR-encoded MST blocks produced by the AT Proto firehose and translates them into JSON objects that are easier to interface with using standard tooling available in programming languages. Since Repo MSTs only contain records in their leaf nodes, this means Jetstream can drop all of the blocks in an event except for those of the leaf nodes, typically leaving only one block per event. In reality, this means that Jetstream’s JSON firehose is nearly 1/10 the size of the full protocol firehose for the same events, but lacks the verifiability and signatures included in the protocol-level firehose. Jetstream events end up looking something like: { "did": "did:plc:eygmaihciaxprqvxpfvl6flk", "time_us": 1725911162329308, "type": "com", "commit": { "rev": "3l3qo2vutsw2b", "type": "c", "collection": "app.bsky.feed.like", "rkey": "3l3qo2vuowo2b", "record": { "$type": "app.bsky.feed.like", "createdAt": "2024-09-09T19:46:02.102Z", "subject": { "cid": "bafyreidc6sydkkbchcyg62v77wbhzvb2mvytlmsychqgwf2xojjtirmzj4", "uri": "at://did:plc:wa7b35aakoll7hugkrjtf3xf/app.bsky.feed.post/3l3pte3p2e325" } }, "cid": "bafyreidwaivazkwu67xztlmuobx35hs2lnfh3kolmgfmucldvhd3sgzcqi" } } Each event lets you know the DID of the repo it applies to, when it was seen by Jetstream (a time-based cursor), and up to one updated repo record as serialized JSON. Check out this 10 second CPU profile of Jetstream serving 200k evt/sec to a local consumer: By dropping the MST and verification overhead by consuming from relay we trust, we’ve reduced the size of a firehose of all events on the network from 232 GB/day to ~41GB/day, but we can do better. Jetstream and zstd I recently read a great engineering blog from Discord about their use of zstd to compress websocket traffic to/from their Gateway service and client applications. Since Jetstream emits marshalled JSON through the websocket for developer-friendliness, I figured it might be a neat idea to see if we could get further bandwidth reduction by employing zstd to compress events we send to consumers. zstd has two basic operating modes, “simple” mode and “streaming” mode. Streaming Compression At first glance, streaming mode seems like it’d be a great fit. We’ve got a websocket connection with a consumer and streaming mode allows the compression to get more efficient over the lifetime of the connection. I went and implemented a streaming compression version of Jetstream where a consumer can request compression when connecting and will get zstd compressed JSON sent as binary messages over the socket instead of plaintext. Unfortunately, this had a massive impact on Jetstream’s server-side CPU utilization. We were effectively compressing every message once per consumer as part of their streaming session. This was not a scalable approach to offering compression on Jetstream. Additionally, Jetstream stores a buffer of the past 24 hours (configurable) of events on disk in PebbleDB to allow consumers to replay events before getting transitioned into live-tailing mode. Jetstream stores serialized JSON in the DB, so playback is just shuffling the bytes into the websocket without having to round-trip the data into a Go struct. When we layer in streaming compression, playback becomes significantly more expensive because we have to compress outgoing events on-the-fly for a consumer that’s catching up. In real numbers, this increased CPU usage of Jetstream by 23% while lowering the throughput of playback from ~200k evt/sec to ~28k evt/sec for a single local consumer. When in streaming mode, we can’t leverage the bytes we compress for one consumer and reuse them for another consumer because zstd’s streaming context window may not be in sync between the two consumers. They haven’t received exactly the same data in the session so the clients on the other end don’t have their state machines in the same state. Since streaming mode’s primary advantage is giving us eventually better efficiency as the encoder learns about the data, what if we just taught the encoder about the data at the start and compress each message statelessly? Dictionary Mode zstd offers a mechanism for initializing an encoder/decoder with pre-optimized settings by providing a dictionary trained on a sample of the data you’ll be encoding/decoding. Using this dictionary, zstd essentially uses it’s smallest encoded representations for the most frequently seen patterns in the sample data. In our case, where we’re compressing serialized JSON with a common event shape and lots of common property names, training a dictionary on a large number of real events should allow us to represent the common elements among messages in the smallest number of bytes. For take two of Jetstream with zstd, let’s to use a single encoder for the whole service that utilizes a custom dictionary trained on 100,000 real events. We can use this encoder to compress every event as we see it, before persisting and emitting it to consumers. Now we end up with two copies of every event, one that’s just serialized JSON, and one that’s statelessly compressed to zstd using our dictionary. Any consumers that want compression can have a copy of the dictionary on their end to initialize a decoder, then when we broadcast the shared compressed event, all consumers can read it without any state or context issues. This requires the consumers and server to have a pre-shared dictionary, which is a major drawback of this implementation but good enough for our purposes. That leaves the problem of event playback for compression-enabled clients. An easy solution here is to just store the compressed events as well! Since we’re only sticking the JSON records into our PebbleDB, the actual size of the 24 hour playback window is <8GB with sstable compression. If we store a copy of the JSON serialized event and a copy of the zstd compressed event, this will, at most, double our storage requirements. Then during playback, if the consumer requests compression, we can just shuffle bytes out of the compressed version of the DB into their socket instead of having to move it through a zstd encoder. Savings Running with a custom dictionary, I was able to get the average Jetstream event down from 482 bytes to just 211 bytes (~0.44 compression ratio). Jetstream allows us to live tail all posts on Bluesky as they’re posted for as little as ~850 MB/day, and we could keep up with all events moving through the firehose during the Brazil Twitter Exodus weekend for 18GB/day (down from 232GB/day). With this scheme, Jetstream is required to compress each event only once before persisting it to disk and emitting it to connected consumers. The CPU impact of these changes is significant in proportion to Jetstream’s incredibly light load but it’s a flat cost we pay once no matter how many consumers we have. (CPU profile from a 30 second pprof sample with 12 consumers live-tailing Jetstream) Additionally, with Jetstream’s shared buffer broadcast architecture, we keep memory allocations incredibly low and the cost per consumer on CPU and RAM is trivial. In the allocation profile below, more than 80% of the allocations are used to consume the full protocol firehose. The total resident memory of Jetstream sits below 16MB, 25% of which is actually consumed by the new zstd dictionary. To bring it all home, here’s a screenshot from the dashboard of my public Jetstream instance serving 12 consumers all with various filters and compression settings, running on a $5/mo OVH VPS. At our new baseline firehose activity, a consumer of the protocol-level firehose would require downloading ~3.16TB/mo to keep up. A Jetstream consumer getting all created, updated, and deleted records without compression enabled would require downloading ~400GB/mo to keep up. A Jetstream consumer that only cares about posts and has zstd compression enabled can get by on as little as ~25.5GB/mo, <99% of the full weight firehose. Feel free to join the conversation about Jetstream and zstd on Bluesky.
Over the past few weeks, I’ve been building out server-side short video support for Bluesky. The major aim of this feature is to support short (90 second max) video streaming at a quality that doesn’t cost an arm and a leg for us to provide for free. In order to stay within these constraints, we’re considering making use of a video CDN that can bear the brunt of the bandwidth required to support Video-on-Demand streaming. While the CDN is a pretty fully-featured product, we want to avoid too much vendor lock-in and provide some enhancements to our streaming platform that requires extending their offering and getting creative with video streaming protocols. Some of the things we’d like to be able to do that don’t work out-of-the-box are: Track view counts, viewer sessions, and duration viewed to provide better feedback for video performance. Provide dynamic closed-caption support with the flexibility to automate them in the future. Store a transcoded version of source files somewhere durable to provide a “source of truth” for videos when needed. Append a “trailer” to the end of video streams for some branding in a TikTok-esque 3-second snippet. In this post I’ll be focusing on the HLS-related features above, namely view/duration accounting, closed captions, and trailers. HLS is Just a Bunch of Text files HTTP Live Streaming (HLS) is a standard established by Apple in 2009 that allows for adaptive-bitrate live and Video-on-Demand (VOD) streaming. For the purposes of this blog post, I’ll restrict my explanations to how HLS VOD streaming works. A player that implements the HLS protocol is capable of dynamically adjusting the quality of a streamed video based on network conditions. Additionally, a server that implements the HLS protocol should provide one or more variants of a media stream which accommodate varying network qualities to allow for graceful degradation of stream quality without stopping playback. HLS implements this by producing a series of plaintext (.m3u8) “playlist” files that tell the player what bitrates and resolutions the server provides so that the player can decide which variant it should stream. HLS differentiates between two kinds of “playlist” files: Master Playlists, and Media Playlists. Master Playlists A Master Playlist is the first file fetched by your video player. It contains a series of variants which point to child Media Playlists. It also describes the approximate bitrate of the variant sources and the codecs and resolutions used by those sources. $ curl https://my.video.host.com/video_15/playlist.m3u8 #EXTM3U #EXT-X-VERSION:3 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=688540,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=640x360 360p/video.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=1921217,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=1280x720 720p/video.m3u8 In the above file, the key things to notice are the RESOLUTION parameters and the {res}/video.m3u8 links. Your media player will generally start with the lowest resolution version before jumping up to higher resolutions once the network speed between you and the server is dialed in. The links in this file are pointers to Media Playlists, generally as relative paths from the Master Playlist such that, if we wanted to grab the 720p Media Playlist, we’d navigate to: https://my.video.host.com/video_15/720p/video.m3u8. A Master Playlist can also contain multi-track audio directives and directives for closed-captions but for now let’s move onto the Media Playlist. Media Playlists A Media Playlist is yet another plaintext file that provides your video player with two key bits of data: a list of media Segments (encoded as .ts video files) and headers for each Segment that tell the player the runtime of the media. $ curl https://my.video.host.com/video_15/720p/video.m3u8 #EXTM3U #EXT-X-VERSION:3 #EXT-X-PLAYLIST-TYPE:VOD #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:4 #EXTINF:4.000, video0.ts #EXTINF:4.000, video1.ts #EXTINF:4.000, video2.ts #EXTINF:4.000, video3.ts #EXTINF:4.000, video4.ts #EXTINF:2.800, video5.ts This Media Playlist describes a video that’s 22.8 seconds long (5 x 4-second Segments + 1 x 2.8-second Segment). The playlist describes a VOD piece of media, meaning we know this playlist contains the entirety of the media the player needs. The TARGETDURATION tells us the maximum length of each Segment so the player knows how many Segments to buffer ahead of time. During live streaming, that also lets the player know how frequently to refresh the playlist file to discover new Segments. Finally the EXTINF headers for each Segment indicate the duration of the following .ts Segment file and the relative paths of the video#.ts tell the player where to load the actual media files from. Where’s the Actual Media? At this point, the video player has loaded two .m3u8 playlist files and got lots of metadata about how to play the video but it hasn’t actually loaded any media files. The .ts files referenced in the Media Playlist are where the real media is, so if we wanted to control the playlists but let the CDN handle serving actual media, we can just redirect those video#.ts requests to our CDN. .ts files are Transport Stream MPEG-2 encoded short media files that can contain video or audio and video. Tracking Views To track views of our HLS streams, we can leverage the fact that every video player must first load the Master Playlist. When a user requests the Master Playlist, we can modify the results dynamically to provide a SessionID to each response and allow us to track the user session without cookies or headers: #EXTM3U #EXT-X-VERSION:3 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=688540,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=640x360 360p/video.m3u8?session_id=12345 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=1921217,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=1280x720 720p/video.m3u8?session_id=12345 Now when their video player fetches the Media Playlists, it’ll include a query-string that we can use to identify the streaming session, ensuring we don’t double-count views on the video and can track which Segments of video were loaded in the session. #EXTM3U #EXT-X-VERSION:3 #EXT-X-PLAYLIST-TYPE:VOD #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:4 #EXTINF:4.000, video0.ts?session_id=12345&duration=4 #EXTINF:4.000, video1.ts?session_id=12345&duration=4 #EXTINF:4.000, video2.ts?session_id=12345&duration=4 #EXTINF:4.000, video3.ts?session_id=12345&duration=4 #EXTINF:4.000, video4.ts?session_id=12345&duration=4 #EXTINF:2.800, video5.ts?session_id=12345&duration=2.8 Finally when the video player fetches the media Segment files, we can measure the Segment view before we redirect to our CDN with a 302, allowing us to know the amount of video-seconds loaded in the session and which Segments were loaded. This method has limitations, namely that a media player loading a segment doesn’t necessarily mean it showed that segment to the viewer, but it’s the best we can do without an instrumented media player. Adding Subtitles Subtitles are included in the Master Playlist as a variant and then are referenced in each of the video variants to let the player know where to load subs from. #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="en_subtitle",DEFAULT=NO,AUTOSELECT=yes,LANGUAGE="en",FORCED="no",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog",URI="subtitles/en.m3u8" #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=688540,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=640x360,SUBTITLES="subs" 360p/video.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=1921217,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=1280x720,SUBTITLES="subs" 720p/video.m3u8 Just like with the video Media Playlists, we need a Media Playlist file for the subtitle track as well so that the player knows where to load the source files from and what duration of the stream they cover. $ curl https://my.video.host.com/video_15/subtitles/en.m3u8 #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:22.8 #EXTINF:22.800, en.vtt In this case, since we’re only serving a short video, we can just provide a single Segment that points at a WebVTT subtitle file encompassing the entire duration of the video. If you crack open the en.vtt file you’ll see something like: $ curl https://my.video.host.com/video_15/subtitles/en.vtt WEBVTT 00:00.000 --> 00:02.000 According to all known laws of aviation, 00:02.000 --> 00:04.000 there is no way a bee should be able to fly. 00:04.000 --> 00:06.000 Its wings are too small to get its fat little body off the ground. ... The media player is capable of reading WebVTT and presenting the subtitles at the right time to the viewer. For longer videos you may want to break up your VTT files into more Segments and update the subtitle Media Playlist accordingly. To provide multiple languages and versions of subtitles, just add more EXT-X-MEDIA:TYPE=SUBTITLES lines to the Master Playlist and tweak the NAME, LANGUAGE (if different), and URI of the additional subtitle variant definitions. #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="en_subtitle",DEFAULT=NO,AUTOSELECT=yes,LANGUAGE="en",FORCED="no",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog",URI="subtitles/en.m3u8" #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="fr_subtitle",DEFAULT=NO,AUTOSELECT=yes,LANGUAGE="fr",FORCED="no",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog",URI="subtitles/fr.m3u8" #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="ja_subtitle",DEFAULT=NO,AUTOSELECT=yes,LANGUAGE="ja",FORCED="no",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog",URI="subtitles/ja.m3u8" Appending a Trailer For branding purposes (and in other applications, for advertising purposes), it can be helpful to insert Segments of video into a playlist to change the content of the video without requiring the content be appended to and re-encoded with the source file. Thankfully, HLS allows us to easily insert Segments into the Media Playlist using this one neat trick: #EXTM3U #EXT-X-VERSION:3 #EXT-X-PLAYLIST-TYPE:VOD #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:4 #EXTINF:4.000, video0.ts #EXTINF:4.000, video1.ts #EXTINF:4.000, video2.ts #EXTINF:4.000, video3.ts #EXTINF:4.000, video4.ts #EXTINF:2.800, video5.ts #EXT-X-DISCONTINUITY #EXTINF:3.337, trailer0.ts #EXTINF:1.201, trailer1.ts #EXTINF:1.301, trailer2.ts #EXT-X-ENDLIST In this Media Playlist we use HLS’s EXT-X-DISCONTINUITY header to let the video player know that the following Segments may be in a different bitrate, resolution, and aspect-ratio than the preceding content. Once we’ve provided the discontinuity header, we can add more Segments just like normal that point at a different media source broken up into .ts files. Remember, HLS allows us to use relative or absolute paths here, so we could provide a full URL for these trailer#.ts files, or virtually route them so they can retain the path context of the currently viewed video. Note that we don’t need to provide the discontinuity header here, and we could also name the trailer files something like video{6-8}.ts if we wanted to, but for clarity and proper player behavior, it’s best to use the discontinuity header if your trailer content doesn’t match the bitrate and resolution of the other video Segments. When the video player goes to play this media, it will continue from video5.ts to trailer0.ts without missing a beat, making it appear as if the trailer is part of the original video. This approach allows us to dynamically change the contents of the trailer for all videos, heavily cache the trailer .ts Segment files for performance, and avoid having to encode the trailer onto the end of every video source file. Conclusion At the end of the day, we’ve now got a video streaming service capable of tracking views and watch session durations, dynamic closed caption support, and branded trailers to help grow the platform. HLS is not a terribly complex protocol. The vast majority of it is human-readable plaintext files and is easy to inspect in the wild to how it’s used in production. When I started this project, I knew next to nothing about the protocol but was able to download some .m3u8 files and get digging to discover how the protocol worked, then build my own implementation of a HLS server to accommodate the video streaming needs of Bluesky. To learn more about HLS, you can check out the official RFC here which describes all the features discussed above and more. I hope this post encourages you to go explore other protocols you use every day by poking at them in the wild, downloading the files your browser interprets for you, and figuring out how simple some of these apparently “complex” systems are. If you’re interested in solving problems like these, take a look at our open Job Recs. If you have any questions about HLS, Bluesky, or other distributed, @scale social media infrastructure, you can find me on Bluesky here and you can discuss this post here.
More in AI
The debate about prioritizing speed or safety is over and reality has made the decision for us.
On shadow libraries, legal documents, and judicial skepticism.
Today's links Hate the player AND the game: But above all, hate the crooked ump. Hey look at this: Delights to delectate. Object permanence: Library Tor nodes vs the DHS; Egg-board psyops; Fury Road amputation cosplay; NYPD's dirtiest cop. Upcoming appearances: Where to find me. Recent appearances: Where I've been. Latest books: You keep readin' em, I'll keep writin' 'em. Upcoming books: Like I said, I'll keep writin' 'em. Colophon: All the rest. Hate the player AND the game (permalink) The epigram for my forthcoming book, Enshittification: Why Everything Suddenly Got Worse and What To Do About It is a quote from Ed Zitron: "I hate them for what they've done to the computer" (Ed even recorded a little cameo of this for the audiobook): https://www.kickstarter.com/projects/doctorow/enshittification-the-drm-free-audiobook/ Ed's a smart and passionate guy, and this was definitely the quote to sum up the rage I felt as I wrote the book. Ed's got a whole theory of who "they" are and "what they did to the computer," which he calls "the Rot Economy": https://www.wheresyoured.at/the-rot-economy/ The Rot Economy describes the ideology of bosses, starting with monsters like GE's Jack Welch, who financialized companies, optimizing them for making short term cash gains for investors, at the expense of their workers, their customers, their products and services, and, ultimately, their long-term health. For Ed, these bosses (especially tech bosses) are the sociopaths who destroyed "the computer" (a stand-in for tech more generally). I don't disagree at all. The there is a direct, undeniable line from the ideas and conduct of tech bosses and the tech hellscape we live in today. A good read on this subject is Anil Dash's scorching post from yesterday, "How Tim Cook sold out Steve Jobs": https://www.anildash.com/2025/09/09/how-tim-cook-sold-out-steve-jobs/ I find the Rot Economy hypothesis entirely compelling, but also, incomplete. Ed's explaining why we should hate the players and why we should hate the game, but the enshittification thesis goes even further and explains why we need to hate the umpires – the policymakers, enforcers, economists and legal theorists who created the enshittogenic environment in which the Rot Economy took hold. Some early reviews of Enshittification have expressed dissatisfaction with book's "solutions" section, complaining that all the solutions are policy oriented, and there's nothing suggested for us to do in our capacity as individual consumers: https://pluralistic.net/2025/07/31/unsatisfying-answers/#systemic-problems Those criticisms are correct: there is nothing we can do as individual consumers. Agonizing about your consumption choices will not fight enshittification any more than conscientiously sorting your recycling will end the climate emergency. Enshittification isn't caused by "lazy consumers" who choose "convenience" or are "too cheap to pay for online services": https://pluralistic.net/2024/04/12/give-me-convenience/#or-give-me-death The wellspring of enshittification isn't poor consumption choices, it's poor policy choices. The reason monsters are able to destroy our online lives isn't their personal moral failings, it's the system that rewards predatory, deceptive and unfair commercial practices and elevates their foremost practitioners to positions of power within firms: https://pluralistic.net/2023/07/28/microincentives-and-enshittification/ And here's the kicker: we know where those policy choices came from! The people who made these policy choices did so in living memory. They were warned at the time about the foreseeable consequences of their choices. They made those choices anyway. They faced zero consequences for doing so, even after every one of the prophesied horrors came to pass. Not only were they spared consequences for their actions, but they prospered as a result – they are revered as statesmen, lawyers, scholars and titans of economics. As Trashfuture showrunner Riley Quinn often says, the curse of being a leftist is that you have object permanence – you actually remember the stuff that happened and how it happened. You don't live in an eternal now that has no causal relationship to the past. It's not enough to hate the player, nor the game – we've got to remember the crooked umps who rigged the match. We have to say their names, because that's how we root out their terrible ideas and ensure that our policy interventions make real change. If Elon Musk OD'ed on ketamine tomorrow, there'd be ten Big Balls who'd tear each others' throats out in the ensuing succession fight, and the next guy would be just as stupid, racist, and authoritarian. Musk, Cook, Zuck, Pichai, Nadella, Larry Ellison – they're just filling the monster-shaped holes that policy-makers installed in our society. Start with Robert Bork, the jurist who championed the "consumer welfare" theory of antitrust, which promotes monopolies as efficient and counsels policymakers not to punish companies that take over markets, because the only way to really dominate a market is to be so good that everyone chooses your products and services. Wouldn't it just be perverse to use public funds to shut down the public's favorite companies? Bork was a virulent racist, a Nixonite criminal, and he was dead wrong about the law and the economics of monopoly: https://pluralistic.net/2022/02/20/we-should-not-endure-a-king/ Bork's legacy of pro-monopoly advocacy is, unsurprisingly, monopolies. Monopolies that make everything more expensive and worse: from athletic shoes to microchips, glass bottles to pharmaceuticals, pro wrestling to eyeglasses: https://www.openmarketsinstitute.org/learn/monopoly-by-the-numbers These monopolies did not arise because of the iron laws of economics. They are not the product of the great forces of history. They are the direct and undeniable consequence of Robert Bork convincing the world's governments to embrace his bullshit, pro-monopoly policies. Satan took Bork to hell in 2012, but you know who's still with us? Bruce Lehman. Bruce Lehman was Bill Clinton's copyright czar, the man who, in his own words, "did an end-run around Congress" by getting an UN treaty passed that obliged its signatories to ban reverse engineering: https://www.cbc.ca/listen/cbc-podcasts/1353-the-naked-emperor/episode/16145640-ctrl-ctrl-ctrl Lehman's used the treaty to get Congress to pass the Digital Millennium Copyright Act (DMCA) and section 1201 of the DMCA made it a felony to break DRM. Bruce Lehman is why farmers can't fix their own tractors, hospitals can't fix their own ventilators, and your mechanic can't fix your car. He's why, when the manufacturer of your artificial eyes bricks a computer that is permanently wired to your nervous system, no one else can revive it: https://pluralistic.net/2022/12/12/unsafe-at-any-speed/ Bruce Lehman is why you can't use the apps of your choosing on your phone or games console. He's why we can't preserve beloved old video games. He's why Apple and Google get to steal 30 cents out of every dollar you send to a performer, software author, or creator through an app: https://pluralistic.net/2025/05/01/its-not-the-crime/#its-the-coverup Yeah, Tim Cook is a venal billionaire who owes his wealth to the Chinese sweatshops of iPhone City, where they had to install suicide nets to catch the workers who'd rather end it all than work another day for Tim Apple, but Tim Cook's power over those workers is owed to Bruce Lehman and Robert Bork. Then there's the ISP sector, whose Net Neutrality violations and underinvestment mean that people who live in the country where the internet was invented have some of the slowest, most expensive internet in the world. Big ISP bosses are some of the worst people on Earth. Take Thomas Rutledge, who CEO of Charter/Spectrum when covid broke out. At the time, Rutledge was America's highest-paid CEO. He dictated that his back-office staff could not work from home (imagine a telco boss who doesn't believe in telework!), and those back-offices all turned into super-spreader sites. Rutledge's field workers – the people who came to our homes and upgraded our internet so we could work from home – did not get PPE or danger pay. Instead, they got vouchers exclusively redeemable at restaurants that had shut down during the pandemic: https://pluralistic.net/2020/04/22/filternet/#thomas-rutledge-murderer Fuck Thomas Rutledge and may his name be a curse forever. But the reason Thomas Rutledge – and all the other terrible telco bosses – were able to reap millions by supplying us with dogshit internet while literally murdering their employees was that Trump's FCC chairman, an ex-Verizon lawyer named Ajit Pai, let them get away with it: https://pluralistic.net/2021/02/12/ajit-pai/#pai Ajit Pai engaged in some of the most flagrant cheating ever seen in American regulation (prior to Jan 20, 2025, at least). When he decided to kill Net Neutrality, he accepted obviously fraudulent comments into the official record, including one million identical comments from @pornhub.com email addresses, as well as millions of comments whose return addresses were taken from darknet data-dumps, including the email addresses of dead people and of sitting US senators who supported Net Neutrality: https://pluralistic.net/2023/11/10/digital-redlining/#stop-confusing-the-issue-with-relevant-facts Pai – and his co-conspirators – are the umps who rigged the game. Hate Thomas Rutledge to be sure, but to prevent people like Rutledge from gaining power over your digital life in future, you must remember Ajit Pai with the special form of white-hot rage that keeps people like him from ever making policy decisions again. Then there's Canada's hall of shame, which is full of monsters. Two of my least favorite are James Moore and Tony Clement, who, as ministers under Stephen Harper, rammed through a Canadian version of the DMCA, 2012's Bill C-11, despite their own consultation, which found that Canadians overwhelmingly rejected the idea: https://pluralistic.net/2024/11/15/radical-extremists/#sex-pest Clement (now a disgraced sex-pest) and Moore (still accepted into polite society as a corporate lawyer) are the reason that Canada's Right to Repair and interop laws are dead on arrival. THey're also why Canada can't retaliate against Trump's tariffs by jailbreaking US products, making everything cheaper for Canadians and birthing new, global Canadian tech businesses: https://pluralistic.net/2025/01/15/beauty-eh/#its-the-only-war-the-yankees-lost-except-for-vietnam-and-also-the-alamo-and-the-bay-of-ham In Europe, there's Axel Voss, the man behind 2019's "filternet" proposal, which requires tech platforms to spend hundreds of millions of euros for copyright filters that use AI to process everything posted to the public internet in Europe and block anything the AI thinks is "copyrighted": https://memex.craphound.com/2019/03/26/article-13-will-wreck-the-internet-because-swedish-meps-accidentally-pushed-the-wrong-voting-button/ For years, Voss maintained that none of this was true, that there would be no filters, and dismissed his critics as hysterical fools: https://memex.craphound.com/2019/04/03/after-months-of-insisting-that-article13-doesnt-require-filters-top-eu-commissioner-says-article-13-requires-filters/ But then, after his law passed, he admitted he "didn't know what he was voting for": https://memex.craphound.com/2018/09/14/father-of-the-catastrophic-copyright-directive-reveals-he-didnt-know-what-he-was-voting-for/ Fuck the media lobbyists who spent hundreds of millions of euros to push this catastrophic law through: https://memex.craphound.com/2018/12/13/clash-of-the-corporate-titans-whos-spending-what-in-europes-copyright-directive-battle/ But especially and forever, fuck Axel Voss, the policymaker who helped turn those corporate bribes into policy. Ed Zitron is right to hate the people who implement the Rot Economy for what they did to the computer. But those people are only doing what policymakers let them do. Corporate monsters thrive in an enshittogenic environment. But political monsters are the ones create that enshittogenic environment. They're the ones who are terraforming our planet to sideline human life and replace it with the immortal colony organisms we call "limited liability corporations." Hey look at this (permalink) Dwayne Johnson Will Play the Chicken Man in ‘Lizard Music’ https://gizmodo.com/dwayne-johnson-to-next-play-the-chicken-man-in-lizard-music-2000655464 Qualifying Conditions https://www.jwz.org/blog/2025/09/qualifying-conditions/ Cindy Cohn Is Leaving the EFF, but Not the Fight for Digital Rights https://www.wired.com/story/eff-cindy-cohn-stepping-down/ Five technological achievements! (That we won’t see any time soon.) https://crookedtimber.org/2025/09/09/five-technological-achievements-that-we-wont-see-any-time-soon/ A notional design studio. https://ethanmarcotte.com/wrote/a-notional-design-studio/ Object permanence (permalink) #20yrsago Anti-trusted-computing video https://www.lafkon.net/tc/ #10yrsago Library offers Tor nodes; DHS tells them to stop https://www.propublica.org/article/library-support-anonymous-internet-browsing-effort-stops-after-dhs-email #10yrsago Ashley Madison’s passwords were badly encrypted, 15 million+ passwords headed for the Web https://arstechnica.com/information-technology/2015/09/ashley-madison-password-crack-could-spell-trouble-across-the-internet/ #10yrsago Heathrow security insists that ice is a liquid https://gizmodo.com/what-happens-if-you-take-frozen-liquids-through-airport-1729772148 #10yrago DoJ says it will consider jailing executives who order corporate crimes https://www.nytimes.com/2015/09/10/us/politics/new-justice-dept-rules-aimed-at-prosecuting-corporate-executives.html #10yrsago Government-run egg board waged high-price, secret PSYOPS war on vegan egg-replacement https://www.theguardian.com/business/2015/sep/06/usda-american-egg-board-paid-bloggers-hampton-creek #10yrago Using sandwiches to teach the Socratic method https://web.archive.org/web/20140810204054/https://medium.com/@kmikeym/is-this-a-sandwich-50b1317eb3f5 #10yrago Fury Road cosplay: amputated arm edition https://web.archive.org/web/20150911194228/http://www.tor.com/2015/09/09/afternoon-roundup-furiosa-real-prosthetic-arm-cosplay/ #5yrsago Kids' smart-watches unsafe at any speed https://pluralistic.net/2020/09/10/booksellers-vs-big-tech/#digital-parenting #5yrsago Georgia voter suppression, quantified https://pluralistic.net/2020/09/10/booksellers-vs-big-tech/#georgia-suppression #5yrsago The rise and rise of one of NYPD's dirtiest cops https://pluralistic.net/2020/09/10/booksellers-vs-big-tech/#50a #5yrago Inaudible https://pluralistic.net/2020/09/10/booksellers-vs-big-tech/#audible-exclusive Upcoming appearances (permalink) Ithaca: Enshittification at Buffalo Street Books, Sept 11 https://buffalostreetbooks.com/event/2025-09-11/cory-doctorow-tcpl-librarian-judd-karlman Ithaca: AD White keynote (Cornell), Sep 12 https://deanoffaculty.cornell.edu/events/keynote-cory-doctorow-professor-at-large/ Ithaca: Enshittification at Autumn Leaves Books, Sept 13 https://www.autumnleavesithaca.com/event-details/enshittification-why-everything-got-worse-and-what-to-do-about-it Ithaca: Radicalized Q&A (Cornell), Sept 16 https://events.cornell.edu/event/radicalized-qa-with-author-cory-doctorow Ithaca: The Counterfeiters (Dinner/Movie Night) (Cornell), Sept 17 https://adwhiteprofessors.cornell.edu/visits/cory-doctorow/ Ithaca: Communication Power, Policy, and Practice (Cornell), Sept 18 https://events.cornell.edu/event/policy-provocations-a-conversation-about-communication-power-policy-and-practice Ithaca: A Reverse-Centaur's Guide to Being a Better AI Critic (Cornell), Sept 18 https://events.cornell.edu/event/2025-nordlander-lecture-in-science-public-policy NYC: Enshittification and Renewal (Cornell Tech), Sept 19 https://www.eventbrite.com/e/enshittification-and-renewal-a-conversation-with-cory-doctorow-tickets-1563948454929 NYC: Brooklyn Book Fair, Sept 21 https://brooklynbookfestival.org/event/big-techs-big-heist-cory-doctorow-in-conversation-with-adam-becker/ DC: Enshittification with Rohit Chopra (Politics and Prose), Oct 8 https://politics-prose.com/cory-doctorow-10825 NYC: Enshittification with Lina Khan (Brooklyn Public Library), Oct 9 https://www.bklynlibrary.org/calendar/cory-doctorow-discusses-central-library-dweck-20251009-0700pm New Orleans: DeepSouthCon63, Oct 10-12 http://www.contraflowscifi.org/ Chicago: Enshittification with Anand Giridharadas (Chicago Humanities), Oct 15 https://www.oldtownschool.org/concerts/2025/10-15-2025-kara-swisher-and-cory-doctorow-on-enshittification/ San Francisco: Enshittification at Public Works (The Booksmith), Oct 20 https://app.gopassage.com/events/doctorow25 Madrid: Conferencia EUROPEA 4D (Virtual), Oct 28 https://4d.cat/es/conferencia/ Miami: Enshittification at Books & Books, Nov 5 https://www.eventbrite.com/e/an-evening-with-cory-doctorow-tickets-1504647263469 Recent appearances (permalink) Nerd Harder! (This Week in Tech) https://twit.tv/shows/this-week-in-tech/episodes/1047 Techtonic with Mark Hurst https://www.wfmu.org/playlists/shows/155658 Cory Doctorow DESTROYS Enshittification (QAA Podcast) https://soundcloud.com/qanonanonymous/cory-doctorow-destroys-enshitification-e338 Latest books (permalink) "Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels). "The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org). "The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). "The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245). "Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. "Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com Upcoming books (permalink) "Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025 "Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025 https://us.macmillan.com/books/9780374619329/enshittification/ "Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026 "Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026 "The Memex Method," Farrar, Straus, Giroux, 2026 "The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, 2026 Colophon (permalink) Today's top sources: Currently writing: "The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. FIRST DRAFT COMPLETE AND SUBMITTED. A Little Brother short story about DIY insulin PLANNING This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net. https://creativecommons.org/licenses/by/4.0/ Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution. How to get Pluralistic: Blog (no ads, tracking, or data-collection): Pluralistic.net Newsletter (no ads, tracking, or data-collection): https://pluralistic.net/plura-list Mastodon (no ads, tracking, or data-collection): https://mamot.fr/@pluralistic Medium (no ads, paywalled): https://doctorow.medium.com/ Twitter (mass-scale, unrestricted, third-party surveillance and advertising): https://twitter.com/doctorow Tumblr (mass-scale, unrestricted, third-party surveillance and advertising): https://mostlysignssomeportents.tumblr.com/tagged/pluralistic "When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ISSN: 3066-764X
And a big change for this newsletter
Today's links Fingerspitzengefühl: IP vs process knowledge. Hey look at this: Delights to delectate. Object permanence: Buddhist hell theme-park; ORG launches; Yahoo spies for Beijing; Secret plan for border laptop-searches; BBC Creative Archive launches; Penn and Teller BBS; Immortan Trump; IP. Upcoming appearances: Where to find me. Recent appearances: Where I've been. Latest books: You keep readin' em, I'll keep writin' 'em. Upcoming books: Like I said, I'll keep writin' 'em. Colophon: All the rest. Fingerspitzengefühl (permalink) This was the plan: America would stop making things and instead make recipes, the "IP" that could be sent to other countries to turn into actual stuff, in distant lands without the pesky environmental and labor rules that forced businesses accept reduced profits because they weren't allowed to maim their workers and poison the land, air and water. This was quite a switch! At the founding of the American republic, the US refused to extend patent protection to foreign inventors. The inventions of foreigners would be fair game for Americans, who could follow their recipes without paying a cent, and so improve the productivity of the new nation without paying rent to old empires over the sea. It was only once America found itself exporting as much as it imported that it saw fit to recognize the prerogatives of foreign inventors, as part of reciprocal agreements that required foreigners to seek permission and pay royalties to American patent-holders. But by the end of the 20th Century, America's ruling class was no longer interested in exporting things; they wanted to export ideas, and receive things in return. You can see why: America has a limited supply of things, but there's an infinite supply of ideas (in theory, anyway). There was one problem: why wouldn't the poor-but-striving nations abroad copy the American Method for successful industrialization? If ignoring Europeans' patents allowed America to become the richest and most powerful nation in the world, why wouldn't, say, China just copy all that American "IP"? If seizing foreigners' inventions without permission was good enough for Thomas Jefferson, why not Jiang Zemin? America solved this problem with the promise of "free trade." The World Trade Organization divided the world into two blocs: countries that could trade with one another without paying tariffs, and the rabble without who had to navigate a complex O(^2) problem of different tariff schedules between every pair of nations. To join the WTO club, countries had to sign up to a side-treaty called the Trade-Related Aspects of Intellectual Property Rights (TRIPS). Under the TRIPS, the Jeffersonian plan for industrialization (taking foreigners' ideas without permission) was declared a one-off, a scheme only the US got to try and no other country could benefit from. For China to join the WTO and gain tariff-free access to the world's markets, it would have to agree to respect foreign patents, copyrights, trademarks and other "IP." We know the story of what followed over the next quarter-century: China became the world's factory, and became so structurally important that even if it violated its obligations under the TRIPS, "stealing the IP" of rich nations, no one could afford to close their borders to Chinese imports, because every country except China had forgotten how to make things. But this isn't the whole story – it's not even the most important part of it. In his new book Breakneck, Dan Wang (a Chinese-born Canadian who has lived extensively in Silicon Valley and in China) devotes a key chapter to "process knowledge": https://danwang.co/breakneck/ What's "process knowledge"? It's all the intangible knowledge that workers acquire as they produce goods, combined with the knowledge that their managers acquire from overseeing that labor. The Germans call it "Fingerspitzengefühl" ("fingertip-feeling"), like the sense of having a ball balanced on your fingertips, and knowing exactly which way it will tip as you tilt your hand this way or that. Wang's book is big and complicated, and I haven't yet finished it. There's plenty I disagree with Wang about – I think he overstates the role of proceduralism in slowing down American progress and understates the role monopoly and oligarchy play in corrupting the rule of law. But the chapter on process knowledge is revelatory. Don't take my word for it: read Henry Farrell, who says that "[process knowledge] is the message of Dan Wang's new book": https://www.programmablemutter.com/p/process-knowledge-is-crucial-to-economic And Dan Davies, who uses the example of the UK's iconic Brompton bikes to explain the importance of process knowledge: https://backofmind.substack.com/p/the-brompton-ness-of-it-all Process knowledge is everything from "Here's how to decant feedstock into this gadget so it doesn't jam," to "here's how to adjust the flow of this precursor on humid days to account for the changes in viscosity" to "if you can't get the normal tech to show up and calibrate the part, here's the phone number of the guy who retired last year and will do it for time-and-a-half." It can also be decidedly high-tech. A couple years ago, the legendary hardware hacker Andrew "bunnie" Huang explained to me his skepticism about the CHIPS Act's goal of onshoring the most advanced (4-5nm) chips. Bunnie laid out the process by which these chips are etched: first you need to make the correct wavelength of light for the nanolithography machine. Stage one of that is spraying droplets of molten tin into an evacuated chamber, where each droplet is tracked by a computer vision system that targets them to be hit with a highly specialized laser that smashes each droplet into a precise coin shape. Then, a second kind of extremely esoteric laser evaporates each of these little tin coins to make a specific kind of tin vapor that can be used to generate the right wavelength of light. This light is then played over two wafers on reciprocating armatures; each wafer needs to be precisely (as in nanograms and nanometers) the same dimensions and weight, otherwise the moving platters they slide back and forth on will get out of balance and the wafers will be spoiled as they are mis-etched. This process is so esoteric, and has so many figurative and literal moving parts, that it needs to be closely overseen and continuously adjusted by someone with a PhD in electrical engineering. That overseer needs to wear a clean-room suit, and they have to work an eight-hour shift without a bathroom, food or water break (because getting out of the suit means going through an airlock means shutting down the system means long delays and wastage). That PhD EENG is making $50k/year. Bunnie's topline explanation for the likely failure of the CHIPS Act is that this is a process that could only be successfully executed in a country "with an amazing educational system and a terrible passport." For bunnie, the extensive educational subsidies that produced Taiwan's legion of skilled electrical engineers and the global system that denied them the opportunity to emigrate to higher-wage zones were the root of the country's global dominance in advanced chip manufacture. I have no doubt that this is true, but I think it's incomplete. What bunnie is describing isn't merely the expertise imparted by attaining a PhD in electrical engineering – it's the process knowledge built up by generations of chip experts who debugged generations of systems that preceded the current tin-vaporizing Rube Goldberg machines. Even if you described how these machines worked to a doctoral EENG who had never worked in this specific field, they couldn't oversee these machines. Sure, they'd have the technical background to be seriously impressed by how cool all this shit is, and you might be able to train them don a bunny suit and hold onto their bladders for 8 hours and make the machine go, but simply handing them the "IP" for this process will not get you a chip foundry. It's undeniable that there's been plenty of Chinese commercial espionage, some of it with state backing. But in reading Wang, it's clear that the country's leaders have cooled on the importance of "IP" – indeed, these days, they call it "imaginary property," and call the IP economy the "imaginary economy" (contrast with the "real economy" of making stuff). Wang evocatively describes how China built up its process knowledge over the WTO years, starting with simple assembly of complex components made abroad, then progressing to making those components, then progressing to coming up with novel ways to reconfiguring them ("a drone is a cellphone with propellers"). He explains how the vicious cycle of losing process knowledge accelerated the decline of manufacturing in the west: every time a factory goes to China, US manufacturers that had been in its supply chain lose process knowledge. You can no longer call up that former supplier and brainstorm solutions to tricky production snags, which means that other factories in the supply chain suffer, and they, too get offshored to China. America's vicious cycle was China's virtuous cycle. The process knowledge that drained out of America accumulated in China. Years of experience solving problems in earlier versions of new equipment and processes gives workers a conceptual framework to debug the current version – they know about the raw mechanisms subsumed in abstraction layers and sealed packages and can visualize what's going on inside those black boxes. Likewise in colonial America: taking foreigners' patents was just table-stakes. Real improvement came from the creation of informal communities built around manufacturing centers, and from the pollinators who spread innovations around among practitioners. Long before John Deere turned IP troll and locked farmers out of servicing their own tractors, they paid and army of roving engineers who would visit farmers to learn about the ways they'd improved their tractors, and integrate these improvements into new designs: https://securityledger.com/2019/03/opinion-my-grandfathers-john-deere-would-support-our-right-to-repair/ But here's the thing: while "IP" can be bought and sold by the capital classes, process knowledge is inseparably vested in the minds and muscle-memory of their workers. People who own the instructions are constitutionally prone to assuming that making the recipe is the important part, while following the recipe is donkey-work you can assign to any freestanding oaf who can take instruction. Think of John Philip Sousa, decrying the musicians who recorded and sold his compositions on early phonograms: These talking machines are going to ruin the artistic development of music in this country. When I was a boy…in front of every house in the summer evenings, you would find young people together singing the songs of the day or old songs. Today you hear these infernal machines going night and day. We will not have a vocal cord left. The vocal cord will be eliminated by a process of evolution, as was the tail of man when he came from the ape. For Sousa, musicians were just the trained monkeys who followed the instructions that talented composers set down on paper and handed off to other trained monkeys to print and distribute for sale. The exaltation of "IP" over process knowledge is part of the ancient practice of bosses denigrating their workers' contribution to the bottom line. It's key to the myth that workers can be replaced by AI: an AI can consume all the "IP" produced by workers, but it doesn't have their process knowledge. It can't, because process knowledge is embodied and enmeshed, it is relational and physical. It doesn't appear in training data. In other words, elevating "IP" over process knowledge is a form of class war. And now that the world's store of process knowledge has been sent to the global south, the class war has gone racial. Think of how Howard Dean – now a paid shill for the pharma lobby – peddled the racist lie that there was no point in dropping patent protections for the covid vaccines, because brown people in poor countries were too stupid to make advanced vaccines: https://pluralistic.net/2021/04/08/howard-dino/#the-scream The truth is that the world's largest vaccine factories are to be found in the global south, particularly India, and these factories sit at the center of a vast web of process knowledge, embedded in relationships and built up with hard-won problem-solving. Bosses would love it if process knowledge didn't matter, because then workers could finally be tamed by industry. We could just move the "IP" around to the highest bidders with the cheapest workforces. But Wang's book makes a forceful argument that it's easier to build up a powerful, resilient society based on process knowledge than it is to do so with IP. What good is a bunch of really cool recipes if no one can follow them? I think that bosses are, psychoanalytically speaking, haunted by the idea that their workers own the process knowledge that is at the heart of their profits. That's why bosses are so obsessed with noncompete "agreements." If you can't own your workers' expertise, then you must own your workers. Any time a debate breaks out over noncompetes, a boss will say something like, "My intellectual property walks out the door of my shop every day at 5PM." They're wrong: the intellectual property is safely stored on the company's hard drives – it's the process knowledge that walks out the door. You can see this in the prepper dreaming of the ruling class. Preppers are consumed by "disaster fantasies" in which the world ends in a way that they – and they alone – can put to rights. In Dancing at Armageddon: Survivalism and Chaos in Modern Times, the ethnographer Richard Mitchell describes a water chemist who is obsessed with terrorists poisoning the water supply: https://pluralistic.net/2020/03/22/preppers-are-larpers/#preppers-unprepared This chemist has stockpiled everything he would need to restore order after a mass water-supply poisoning. But when Mitchell presses him to explain why he thinks it's likely that his town's water supply would be poisoned by terrorists, the prepper is at a loss. Eventually, he basically confesses that it would just be really cool if the world ended in such a way that only he could save it. Which is a problem for a boss. The chemist has a lot of process knowledge, he knows how to do stuff. But the boss knows how to raise money from investors, how to ignore the company's essential qualitative traits (such as the relationships between workers) and reduce the firm to a set of optimizable spreadsheet cells that are legible to the financial markets. What kind of crisis recovery demands those skills? As I posit in my novella "The Masque of the Red Death," the perfect boss fantasy is one in which the boss hunkers down in a luxury bunker while the rabble rebuild civilization from the ashes: https://pluralistic.net/2020/03/14/masque-of-the-red-death/#masque And once that task is complete, the boss emerges from his hidey-hole with an army of mercenaries in bomb-collars, a vast cache of AR-15s, gemstone-quality emeralds, and thumbdrives full of bitcoin, and does what he does best – takes over the show and tells everyone else what to do, from the comfort his high-walled fortress, with its mountain of canned goods and its harem. The absurdity of this – as I try to show with my story – is that the process knowledge of wheedling, bullying and coercing other people to work for you is actually not very useful. The IP you can buy and sell an inert curiosity until it finds its way to people who can put it into process. Hey look at this (permalink) Statement on discourse about ActivityPub and AT Protocol https://github.com/swicg/general/blob/master/statements%2F2025-09-05-activitypub-and-atproto-discourse.md A message from Emily James, director of the upcoming documentary Enshittification: The Film. https://www.kickstarter.com/projects/doctorow/enshittification-the-drm-free-audiobook/posts/4478169 The story of how RSS beat Microsoft https://buttondown.com/blog/rss-vs-ice Ideas Have Consequences The Impact of Law and Economics on American Justice https://academic.oup.com/qje/advance-article/doi/10.1093/qje/qjaf042/8241352 Object permanence (permalink) #20yrsago BBC Creative Archive pilot launches http://news.bbc.co.uk/2/hi/entertainment/4225914.stm #20yrsago Gold Rush-era sailing ship ruin excavated in San Fran https://web.archive.org/web/20050910151416/https://www.sfgate.com/cgi-bin/article.cgi?f=/n/a/2005/09/06/state/n154446D61.DTL #20yrsago iTunes phone gratuitously crippled by DRM https://web.archive.org/web/20051001030643/http://playlistmag.com/weblogs/todayatplaylist/2005/09/hiddengoodies/index.php #20yrsago My photos from the Buddhist hells of the Singaporean Tiger Balm themepark https://memex.craphound.com/2005/09/07/corys-photos-from-the-buddhist-hells-of-the-singaporean-tiger-balm-themepark/ #20yrsago Online Rights Group UK launches https://web.archive.org/web/20051120005155/http://www.openrightsgroup.org/ #20yrsago Yahoo rats out Chinese reporter to Beijing, writer gets 10 years in jail http://news.bbc.co.uk/2/hi/asia-pacific/4221538.stm #15yrsago Secret copyright treaty: USA caves on border laptop/phone/MP3 player searches for copyright infringement https://www.michaelgeist.ca/2010/09/acta-enforcement-practice-chapter/ #15yrsago Login screens from Penn and Teller BBS, 1987 https://www.flickr.com/photos/davidkha/4969386169/ #10yrsago Antihoarding: When “decluttering” becomes a compulsion https://www.theatlantic.com/health/archive/2015/09/ocd-obsessive-compulsive-decluttering-hoarding/401591/ #10yrsago NZ bans award-winning YA novel after complaints from conservative Christian group https://www.theguardian.com/world/2015/sep/07/new-zealand-bans-into-the-river-teenage-novel-outcry-christian-group #10yrsago Immortan Trump https://imgur.com/gallery/relevant-donald-trump-cos-play-OQe2rU5 #5yrsago Antitrust trouble for cloud services https://pluralistic.net/2020/09/08/attack-surface-kickstarter/#reasonable-agreements #5yrsago FTC about to hammer Intuit https://pluralistic.net/2020/09/08/attack-surface-kickstarter/#tax-fraud #5yrsago IP https://pluralistic.net/2020/09/08/attack-surface-kickstarter/#control #5yrsago My first-ever Kickstarter https://pluralistic.net/2020/09/08/attack-surface-kickstarter/#asks #5yrsago David Graeber on Spectre TV https://pluralistic.net/2020/09/07/facebook-v-humanity/#spectre #5yrsago Facebook's foreseeable election consequences https://pluralistic.net/2020/09/07/facebook-v-humanity/#zuck-off Upcoming appearances (permalink) Ithaca: Enshittification at Buffalo Street Books, Sept 11 https://buffalostreetbooks.com/event/2025-09-11/cory-doctorow-tcpl-librarian-judd-karlman Ithaca: AD White keynote (Cornell), Sep 12 https://deanoffaculty.cornell.edu/events/keynote-cory-doctorow-professor-at-large/ Ithaca: Enshittification at Autumn Leaves Books, Sept 13 https://www.autumnleavesithaca.com/event-details/enshittification-why-everything-got-worse-and-what-to-do-about-it Ithaca: Radicalized Q&A (Cornell), Sept 16 https://events.cornell.edu/event/radicalized-qa-with-author-cory-doctorow Ithaca: The Counterfeiters (Dinner/Movie Night) (Cornell), Sept 17 https://adwhiteprofessors.cornell.edu/visits/cory-doctorow/ Ithaca: Communication Power, Policy, and Practice (Cornell), Sept 18 https://events.cornell.edu/event/policy-provocations-a-conversation-about-communication-power-policy-and-practice Ithaca: A Reverse-Centaur's Guide to Being a Better AI Critic (Cornell), Sept 18 https://events.cornell.edu/event/2025-nordlander-lecture-in-science-public-policy NYC: Enshittification and Renewal (Cornell Tech), Sept 19 https://www.eventbrite.com/e/enshittification-and-renewal-a-conversation-with-cory-doctorow-tickets-1563948454929 DC: Enshittification with Rohit Chopra (Politics and Prose), Oct 8 https://politics-prose.com/cory-doctorow-10825 NYC: Enshittification with Lina Khan (Brooklyn Public Library), Oct 9 https://www.bklynlibrary.org/calendar/cory-doctorow-discusses-central-library-dweck-20251009-0700pm New Orleans: DeepSouthCon63, Oct 10-12 http://www.contraflowscifi.org/ Chicago: Enshittification with Anand Giridharadas (Chicago Humanities), Oct 15 https://www.oldtownschool.org/concerts/2025/10-15-2025-kara-swisher-and-cory-doctorow-on-enshittification/ San Francisco: Enshittification at Public Works (The Booksmith), Oct 20 https://app.gopassage.com/events/doctorow25 Madrid: Conferencia EUROPEA 4D (Virtual), Oct 28 https://4d.cat/es/conferencia/ Miami: Enshittification at Books & Books, Nov 5 https://www.eventbrite.com/e/an-evening-with-cory-doctorow-tickets-1504647263469 Recent appearances (permalink) Nerd Harder! (This Week in Tech) https://twit.tv/shows/this-week-in-tech/episodes/1047 Techtonic with Mark Hurst https://www.wfmu.org/playlists/shows/155658 Cory Doctorow DESTROYS Enshittification (QAA Podcast) https://soundcloud.com/qanonanonymous/cory-doctorow-destroys-enshitification-e338 Latest books (permalink) "Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels). "The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org). "The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). "The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245). "Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. "Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com Upcoming books (permalink) "Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025 "Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025 https://us.macmillan.com/books/9780374619329/enshittification/ "Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026 "Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026 "The Memex Method," Farrar, Straus, Giroux, 2026 "The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, 2026 Colophon (permalink) Today's top sources: Currently writing: "The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. FIRST DRAFT COMPLETE AND SUBMITTED. A Little Brother short story about DIY insulin PLANNING This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net. https://creativecommons.org/licenses/by/4.0/ Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution. How to get Pluralistic: Blog (no ads, tracking, or data-collection): Pluralistic.net Newsletter (no ads, tracking, or data-collection): https://pluralistic.net/plura-list Mastodon (no ads, tracking, or data-collection): https://mamot.fr/@pluralistic Medium (no ads, paywalled): https://doctorow.medium.com/ Twitter (mass-scale, unrestricted, third-party surveillance and advertising): https://twitter.com/doctorow Tumblr (mass-scale, unrestricted, third-party surveillance and advertising): https://mostlysignssomeportents.tumblr.com/tagged/pluralistic "When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ISSN: 3066-764X
