Full Width [alt+shift+f] FOCUS MODE Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
17
I’ve been working on a growing series of services that archive, analyze, and represent data from a social network. Part of this process involves archiving every post on a social network, running Computer Vision models on every image posted to the network, and running sentiment analysis on the text of every post on the network. Exposition: Custom Feeds I’ve built different services and patterns for generating this analysis data, but have recently been trying to produce custom feeds of posts based on analytic subgroups. One such example is a “Cat Feed” which is a feed of all posts that contain pictures of cats (and sometimes not cats based on CV model accuracy). Another example is a “Cluster Feed” which uses social cluster assignments of users on the platform to produce curated feeds authored by the members of a given cluster. Producing these custom feeds requires building a View of the data in the database that conforms to numerous potential groupings: posts with cat pictures could be...
over a year ago

Comments

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from exist

When Imperfect Systems are Good, Actually: Bluesky's Lossy Timelines

Often when designing systems, we aim for perfection in things like consistency of data, availability, latency, and more. The hardest part of system design is that it’s difficult (if not impossible) to design systems that have perfect consistency, perfect availability, incredibly low latency, and incredibly high throughput, all at the same time. Instead, when we approach system design, it’s best to treat each of these properties as points on different axes that we balance to find the “right fit” for the application we’re supporting. I recently made some major tradeoffs in the design of Bluesky’s Following Feed/Timeline to improve the performance of writes at the cost of consistency in a way that doesn’t negatively affect users but reduced P99s by over 96%. Timeline Fanout When you make a post on Bluesky, your post is indexed by our systems and persisted to a database where we can fetch it to hydrate and serve in API responses. Additionally, a reference to your post is “fanned out” to your followers so they can see it in their Timelines. This process involves looking up all of your followers, then inserting a new row into each of their Timeline tables in reverse chronological order with a reference to your post. When a user loads their Timeline, we fetch a page of post references and then hydrate the posts/actors concurrently to quickly build an API response and let them see the latest content from people they follow. The Timelines table is sharded by user. This means each user gets their own Timeline partition, randomly distributed among shards of our horizontally scalable database (ScyllaDB), replicated across multiple shards for high availability. Timelines are regularly trimmed when written to, keeping them near a target length and dropping older post references to conserve space. Hot Shards in Your Area Bluesky currently has around 32 Million Users and our Timelines database is broken into hundreds of shards. To support millions of partitions on such a small number of shards, each user’s Timeline partition is colocated with tens of thousands of other users’ Timelines. Under normal circumstances with all users behaving well, this doesn’t present a problem as the work of an individual Timeline is small enough that a shard can handle the work of tens of thousands of them without being heavily taxed. Unfortunately, with a large number of users, some of them will do abnormal things like… well… following hundreds of thousands of other users. Generally, this can be dealt with via policy and moderation to prevent abusive users from causing outsized load on systems, but these processes take time and can be imperfect. When a user follows hundreds of thousands of others, their Timeline becomes hyperactive with writes and trimming occurring at massively elevated rates. This load slows down the individual operations to the user’s Timeline, which is fine for the bad behaving user, but causes problems to the tens of thousands of other users sharing a shard with them. We typically call this situation a “Hot Shard”: where some resident of a shard has “hot” data that is being written to or read from at much higher rates than others. Since the data on the shard is only replicated a few times, we can’t effectively leverage the horizontal scale of our database to process all this additional work. Instead, the “Hot Shard” ends up spending so much time doing work for a single partition that operations to the colocated partitions slow down as well. Stacking Latencies Returning to our Fanout process, let’s consider the case of Fanout for a user followed by 2,000,000 other users. Under normal circumstances, writing to a single Timeline takes an average of ~600 microseconds. If we sequentially write to the Timelines of our user’s followers, we’ll be sitting around for 20 minutes at best to Fanout this post. If instead we concurrently Fanout to 1,000 Timelines at once, we can complete this Fanout job in ~1.2 seconds. That sounds great, except it oversimplifies an important property of systems: tail latencies. The average latency of a write is ~600 microseconds, but some writes take much less time and some take much more. In fact, the P99 latency of writes to the Timelines cluster can be as high as 15 milliseconds! What does this mean for our Fanout? Well, if we concurrently write to 1,000 Timelines at once, statistically we’ll see 10 writes as slow as or slower than 15 milliseconds. In the case of timelines, each “page” of followers is 10,000 users large and each “page” must be fanned out before we fetch the next page. This means that our slowest writes will hold up the fetching and Fanout of the next page. How does this affect our expected Fanout time? Each “page” will have ~100 writes as slow as or slower than the P99 latency. If we get unlucky, they could all stack up on a single routine and end up slowing down a single page of Fanout to 1.5 seconds. In the worst case, for our 2,000,000 Follower celebrity, their post Fanout could end up taking as long as 5 minutes! That’s not even considering P99.9 and P99.99 latencies which could end up being >1 second, which could leave us waiting tens of minutes for our Fanout job. Now imagine how bad this would be for a user with 20,000,000+ Followers! So, how do we fix the problem? By embracing imperfection, of course! Lossy Timelines Imagine a user who follows hundreds of thousands of others. Their Timeline is being written to hundreds of times a second, moving so fast it would be humanly impossible to keep up with the entirety of their Timeline even if it was their full-time job. For a given user, there’s a threshold beyond which it is unreasonable for them to be able to keep up with their Timeline. Beyond this point, they likely consume content through various other feeds and do not primarily use their Following Feed. Additionally, beyond this point, it is reasonable for us to not necessarily have a perfect chronology of everything posted by the many thousands of users they follow, but provide enough content that the Timeline always has something new. Note in this case I’m using the term “reasonable” to loosely convey that as a social media service, there must be a limit to the amount of work we are expected to do for a single user. What if we introduce a mechanism to reduce the correctness of a Timeline such that there is a limit to the amount of work a single Timeline can place on a DB shard. We can assert a reasonable limit for the number of follows a user should have to have a healthy and active Timeline, then increase the “lossiness” of their Timeline the further past that limit they go. A loss_factor can be defined as min(reasonable_limit/num_follows, 1) and can be used to probabilistically drop writes to a Timeline to prevent hot shards. Just before writing a page in Fanout, we can generate a random float between 0 and 1, then compare it to the loss_factor of each user in the page. If the user’s loss_factor is smaller than the generated float, we filter the user out of the page and don’t write to their Timeline. Now, users all have the same number of “follows worth” of Fanout. For example with a reasonable_limit of 2,000, a user who follows 4,000 others will have a loss_factor of 0.5 meaning half the writes to their Timeline will get dropped. For a user following 8,000 others, their loss factor of 0.25 will drop 75% of writes to their Timeline. Thus, each user has a effective ceiling on the amount of Fanout work done for their Timeline. By specifying the limits of reasonable user behavior and embracing imperfection for users who go beyond it, we can continue to provide service that meets the expectations of users without sacrificing scalability of the system. Aside on Caching We write to Timelines at a rate of more than one million times a second during the busy parts of the day. Looking up the number of follows of a given user before fanning out to them would require more than one million additional reads per second to our primary database cluster. This additional load would not be well received by our database and the additional cost wouldn’t be worth the payoff for faster Timeline Fanout. Instead, we implemented an approach that caches high-follow accounts in a Redis sorted set, then each instance of our Fanout service loads an updated version of the set into memory every 30 seconds. This allows us to perform lookups of follow counts for high-follow accounts millions of times per second per Fanount service instance. By caching values which don’t need to be perfect to function correctly in this case, we can once again embrace imperfection in the system to improve performance and scalability without compromising the function of the service. Results We implemented Lossy Timelines a few weeks ago on our production systems and saw a dramatic reduction in hot shards on the Timelines database clusters. In fact, there now appear to be no hot shards in the cluster at all, and the P99 of a page of Fanout work has been reduced by over 90%. Additionally, with the reduction in write P99s, the P99 duration for a full post Fanout has been reduced by over 96%. Jobs that used to take 5-10 minutes for large accounts now take <10 seconds. Knowing where it’s okay to be imperfect lets you trade consistency for other desirable aspects of your systems and scale ever higher. There are plenty of other places for improvement in our Timelines architecture, but this step was a big one towards improving throughput and scalability of Bluesky’s Timelines. If you’re interested in these sorts of problems and would like to help us build the core data services that power Bluesky, check out this job listing. If you’re interested in other open positions at Bluesky, you can find them here.

6 months ago 52 votes
Emoji Griddle
10 months ago 28 votes
Jetstream: Shrinking the AT Proto Firehose by >99%

Bluesky recently saw a massive spike in activity in response to Brazil’s ban of Twitter. As a result, the AT Proto event firehose provided by Bluesky’s Relay at bsky.network has increased in volume by a huge amount. The average event rate during this surge increased by ~1,300%. Before this new surge in activity, the firehose would produce around 24 GB/day of traffic. After the surge, this volume jumped to over 232 GB/day! Keeping up with the full, verified firehose quickly became less practical on cheap cloud infrastructure with metered bandwidth. To help reduce the burden of operating bots, feed generators, labelers, and other non-verifying AT Proto services, I built Jetstream as an alternative, lightweight, filterable JSON firehose for AT Proto. How the Firehose Works The AT Proto firehose is a mechanism used to keep verified, fully synced copies of the repos of all users. Since repos are represented as Merkle Search Trees, each firehose event contains an update to the user’s MST which includes all the changed blocks (nodes in the path from the root to the modified leaf). The root of this path is signed by the repo owner, and a consumer can keep their copy of the repo’s MST up-to-date by applying the diff in the event. For a more in-depth explanation of how Merkle Trees are constructed, check out this explainer. Practically, this means that for every small JSON record added to a repo, we also send along some number of MST blocks (which are content-addressed hashes and thus very information-dense) that are mostly useful for consumers attempting to keep a fully synced, verified copy of the repo. You can think of this as the difference between cloning a git repo v.s. just grabbing the latest version of the files without the .git folder. In this case, the firehose effectively streams the diffs for the repository with commits, signatures, and metadata, which is inherently heavier than a point-in-time checkout of the repo. Because firehose events with repo updates are signed by the repo owner, they allow a consumer to process events from any operator without having to trust the messenger. This is the “Authenticated” part of the Authenticated Transfer (AT) Protocol and is crucial to the correct functioning of the network. That being said, of the hundreds of consumers of Bluesky’s production Relay, >90% of them are building feeds, bots, and other tools that don’t keep full copies of the entire network and don’t verify MST operations at all. For these consumers, all they actually process is the JSON records created, updated, and deleted in each event. If consumers already trust the provider to do validation on their end, they could get by with a much more lightweight data stream. How Jetstream Works Jetstream is a streaming service that consumes an AT Proto com.atproto.sync.subscribeRepos stream and converts it into lightweight, friendly JSON. If you want to try it out yourself, you can connect to my public Jetstream instance and view all posts on Bluesky in realtime: $ websocat "wss://jetstream2.us-east.bsky.network/subscribe?wantedCollections=app.bsky.feed.post" Note: the above instance is operated by Bluesky PBC and is free to use, more instances are listed in the official repo Readme Jetstream converts the CBOR-encoded MST blocks produced by the AT Proto firehose and translates them into JSON objects that are easier to interface with using standard tooling available in programming languages. Since Repo MSTs only contain records in their leaf nodes, this means Jetstream can drop all of the blocks in an event except for those of the leaf nodes, typically leaving only one block per event. In reality, this means that Jetstream’s JSON firehose is nearly 1/10 the size of the full protocol firehose for the same events, but lacks the verifiability and signatures included in the protocol-level firehose. Jetstream events end up looking something like: { "did": "did:plc:eygmaihciaxprqvxpfvl6flk", "time_us": 1725911162329308, "type": "com", "commit": { "rev": "3l3qo2vutsw2b", "type": "c", "collection": "app.bsky.feed.like", "rkey": "3l3qo2vuowo2b", "record": { "$type": "app.bsky.feed.like", "createdAt": "2024-09-09T19:46:02.102Z", "subject": { "cid": "bafyreidc6sydkkbchcyg62v77wbhzvb2mvytlmsychqgwf2xojjtirmzj4", "uri": "at://did:plc:wa7b35aakoll7hugkrjtf3xf/app.bsky.feed.post/3l3pte3p2e325" } }, "cid": "bafyreidwaivazkwu67xztlmuobx35hs2lnfh3kolmgfmucldvhd3sgzcqi" } } Each event lets you know the DID of the repo it applies to, when it was seen by Jetstream (a time-based cursor), and up to one updated repo record as serialized JSON. Check out this 10 second CPU profile of Jetstream serving 200k evt/sec to a local consumer: By dropping the MST and verification overhead by consuming from relay we trust, we’ve reduced the size of a firehose of all events on the network from 232 GB/day to ~41GB/day, but we can do better. Jetstream and zstd I recently read a great engineering blog from Discord about their use of zstd to compress websocket traffic to/from their Gateway service and client applications. Since Jetstream emits marshalled JSON through the websocket for developer-friendliness, I figured it might be a neat idea to see if we could get further bandwidth reduction by employing zstd to compress events we send to consumers. zstd has two basic operating modes, “simple” mode and “streaming” mode. Streaming Compression At first glance, streaming mode seems like it’d be a great fit. We’ve got a websocket connection with a consumer and streaming mode allows the compression to get more efficient over the lifetime of the connection. I went and implemented a streaming compression version of Jetstream where a consumer can request compression when connecting and will get zstd compressed JSON sent as binary messages over the socket instead of plaintext. Unfortunately, this had a massive impact on Jetstream’s server-side CPU utilization. We were effectively compressing every message once per consumer as part of their streaming session. This was not a scalable approach to offering compression on Jetstream. Additionally, Jetstream stores a buffer of the past 24 hours (configurable) of events on disk in PebbleDB to allow consumers to replay events before getting transitioned into live-tailing mode. Jetstream stores serialized JSON in the DB, so playback is just shuffling the bytes into the websocket without having to round-trip the data into a Go struct. When we layer in streaming compression, playback becomes significantly more expensive because we have to compress outgoing events on-the-fly for a consumer that’s catching up. In real numbers, this increased CPU usage of Jetstream by 23% while lowering the throughput of playback from ~200k evt/sec to ~28k evt/sec for a single local consumer. When in streaming mode, we can’t leverage the bytes we compress for one consumer and reuse them for another consumer because zstd’s streaming context window may not be in sync between the two consumers. They haven’t received exactly the same data in the session so the clients on the other end don’t have their state machines in the same state. Since streaming mode’s primary advantage is giving us eventually better efficiency as the encoder learns about the data, what if we just taught the encoder about the data at the start and compress each message statelessly? Dictionary Mode zstd offers a mechanism for initializing an encoder/decoder with pre-optimized settings by providing a dictionary trained on a sample of the data you’ll be encoding/decoding. Using this dictionary, zstd essentially uses it’s smallest encoded representations for the most frequently seen patterns in the sample data. In our case, where we’re compressing serialized JSON with a common event shape and lots of common property names, training a dictionary on a large number of real events should allow us to represent the common elements among messages in the smallest number of bytes. For take two of Jetstream with zstd, let’s to use a single encoder for the whole service that utilizes a custom dictionary trained on 100,000 real events. We can use this encoder to compress every event as we see it, before persisting and emitting it to consumers. Now we end up with two copies of every event, one that’s just serialized JSON, and one that’s statelessly compressed to zstd using our dictionary. Any consumers that want compression can have a copy of the dictionary on their end to initialize a decoder, then when we broadcast the shared compressed event, all consumers can read it without any state or context issues. This requires the consumers and server to have a pre-shared dictionary, which is a major drawback of this implementation but good enough for our purposes. That leaves the problem of event playback for compression-enabled clients. An easy solution here is to just store the compressed events as well! Since we’re only sticking the JSON records into our PebbleDB, the actual size of the 24 hour playback window is <8GB with sstable compression. If we store a copy of the JSON serialized event and a copy of the zstd compressed event, this will, at most, double our storage requirements. Then during playback, if the consumer requests compression, we can just shuffle bytes out of the compressed version of the DB into their socket instead of having to move it through a zstd encoder. Savings Running with a custom dictionary, I was able to get the average Jetstream event down from 482 bytes to just 211 bytes (~0.44 compression ratio). Jetstream allows us to live tail all posts on Bluesky as they’re posted for as little as ~850 MB/day, and we could keep up with all events moving through the firehose during the Brazil Twitter Exodus weekend for 18GB/day (down from 232GB/day). With this scheme, Jetstream is required to compress each event only once before persisting it to disk and emitting it to connected consumers. The CPU impact of these changes is significant in proportion to Jetstream’s incredibly light load but it’s a flat cost we pay once no matter how many consumers we have. (CPU profile from a 30 second pprof sample with 12 consumers live-tailing Jetstream) Additionally, with Jetstream’s shared buffer broadcast architecture, we keep memory allocations incredibly low and the cost per consumer on CPU and RAM is trivial. In the allocation profile below, more than 80% of the allocations are used to consume the full protocol firehose. The total resident memory of Jetstream sits below 16MB, 25% of which is actually consumed by the new zstd dictionary. To bring it all home, here’s a screenshot from the dashboard of my public Jetstream instance serving 12 consumers all with various filters and compression settings, running on a $5/mo OVH VPS. At our new baseline firehose activity, a consumer of the protocol-level firehose would require downloading ~3.16TB/mo to keep up. A Jetstream consumer getting all created, updated, and deleted records without compression enabled would require downloading ~400GB/mo to keep up. A Jetstream consumer that only cares about posts and has zstd compression enabled can get by on as little as ~25.5GB/mo, <99% of the full weight firehose. Feel free to join the conversation about Jetstream and zstd on Bluesky.

11 months ago 31 votes
How HLS Works

Over the past few weeks, I’ve been building out server-side short video support for Bluesky. The major aim of this feature is to support short (90 second max) video streaming at a quality that doesn’t cost an arm and a leg for us to provide for free. In order to stay within these constraints, we’re considering making use of a video CDN that can bear the brunt of the bandwidth required to support Video-on-Demand streaming. While the CDN is a pretty fully-featured product, we want to avoid too much vendor lock-in and provide some enhancements to our streaming platform that requires extending their offering and getting creative with video streaming protocols. Some of the things we’d like to be able to do that don’t work out-of-the-box are: Track view counts, viewer sessions, and duration viewed to provide better feedback for video performance. Provide dynamic closed-caption support with the flexibility to automate them in the future. Store a transcoded version of source files somewhere durable to provide a “source of truth” for videos when needed. Append a “trailer” to the end of video streams for some branding in a TikTok-esque 3-second snippet. In this post I’ll be focusing on the HLS-related features above, namely view/duration accounting, closed captions, and trailers. HLS is Just a Bunch of Text files HTTP Live Streaming (HLS) is a standard established by Apple in 2009 that allows for adaptive-bitrate live and Video-on-Demand (VOD) streaming. For the purposes of this blog post, I’ll restrict my explanations to how HLS VOD streaming works. A player that implements the HLS protocol is capable of dynamically adjusting the quality of a streamed video based on network conditions. Additionally, a server that implements the HLS protocol should provide one or more variants of a media stream which accommodate varying network qualities to allow for graceful degradation of stream quality without stopping playback. HLS implements this by producing a series of plaintext (.m3u8) “playlist” files that tell the player what bitrates and resolutions the server provides so that the player can decide which variant it should stream. HLS differentiates between two kinds of “playlist” files: Master Playlists, and Media Playlists. Master Playlists A Master Playlist is the first file fetched by your video player. It contains a series of variants which point to child Media Playlists. It also describes the approximate bitrate of the variant sources and the codecs and resolutions used by those sources. $ curl https://my.video.host.com/video_15/playlist.m3u8 #EXTM3U #EXT-X-VERSION:3 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=688540,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=640x360 360p/video.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=1921217,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=1280x720 720p/video.m3u8 In the above file, the key things to notice are the RESOLUTION parameters and the {res}/video.m3u8 links. Your media player will generally start with the lowest resolution version before jumping up to higher resolutions once the network speed between you and the server is dialed in. The links in this file are pointers to Media Playlists, generally as relative paths from the Master Playlist such that, if we wanted to grab the 720p Media Playlist, we’d navigate to: https://my.video.host.com/video_15/720p/video.m3u8. A Master Playlist can also contain multi-track audio directives and directives for closed-captions but for now let’s move onto the Media Playlist. Media Playlists A Media Playlist is yet another plaintext file that provides your video player with two key bits of data: a list of media Segments (encoded as .ts video files) and headers for each Segment that tell the player the runtime of the media. $ curl https://my.video.host.com/video_15/720p/video.m3u8 #EXTM3U #EXT-X-VERSION:3 #EXT-X-PLAYLIST-TYPE:VOD #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:4 #EXTINF:4.000, video0.ts #EXTINF:4.000, video1.ts #EXTINF:4.000, video2.ts #EXTINF:4.000, video3.ts #EXTINF:4.000, video4.ts #EXTINF:2.800, video5.ts This Media Playlist describes a video that’s 22.8 seconds long (5 x 4-second Segments + 1 x 2.8-second Segment). The playlist describes a VOD piece of media, meaning we know this playlist contains the entirety of the media the player needs. The TARGETDURATION tells us the maximum length of each Segment so the player knows how many Segments to buffer ahead of time. During live streaming, that also lets the player know how frequently to refresh the playlist file to discover new Segments. Finally the EXTINF headers for each Segment indicate the duration of the following .ts Segment file and the relative paths of the video#.ts tell the player where to load the actual media files from. Where’s the Actual Media? At this point, the video player has loaded two .m3u8 playlist files and got lots of metadata about how to play the video but it hasn’t actually loaded any media files. The .ts files referenced in the Media Playlist are where the real media is, so if we wanted to control the playlists but let the CDN handle serving actual media, we can just redirect those video#.ts requests to our CDN. .ts files are Transport Stream MPEG-2 encoded short media files that can contain video or audio and video. Tracking Views To track views of our HLS streams, we can leverage the fact that every video player must first load the Master Playlist. When a user requests the Master Playlist, we can modify the results dynamically to provide a SessionID to each response and allow us to track the user session without cookies or headers: #EXTM3U #EXT-X-VERSION:3 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=688540,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=640x360 360p/video.m3u8?session_id=12345 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=1921217,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=1280x720 720p/video.m3u8?session_id=12345 Now when their video player fetches the Media Playlists, it’ll include a query-string that we can use to identify the streaming session, ensuring we don’t double-count views on the video and can track which Segments of video were loaded in the session. #EXTM3U #EXT-X-VERSION:3 #EXT-X-PLAYLIST-TYPE:VOD #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:4 #EXTINF:4.000, video0.ts?session_id=12345&duration=4 #EXTINF:4.000, video1.ts?session_id=12345&duration=4 #EXTINF:4.000, video2.ts?session_id=12345&duration=4 #EXTINF:4.000, video3.ts?session_id=12345&duration=4 #EXTINF:4.000, video4.ts?session_id=12345&duration=4 #EXTINF:2.800, video5.ts?session_id=12345&duration=2.8 Finally when the video player fetches the media Segment files, we can measure the Segment view before we redirect to our CDN with a 302, allowing us to know the amount of video-seconds loaded in the session and which Segments were loaded. This method has limitations, namely that a media player loading a segment doesn’t necessarily mean it showed that segment to the viewer, but it’s the best we can do without an instrumented media player. Adding Subtitles Subtitles are included in the Master Playlist as a variant and then are referenced in each of the video variants to let the player know where to load subs from. #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="en_subtitle",DEFAULT=NO,AUTOSELECT=yes,LANGUAGE="en",FORCED="no",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog",URI="subtitles/en.m3u8" #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=688540,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=640x360,SUBTITLES="subs" 360p/video.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=0,BANDWIDTH=1921217,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=1280x720,SUBTITLES="subs" 720p/video.m3u8 Just like with the video Media Playlists, we need a Media Playlist file for the subtitle track as well so that the player knows where to load the source files from and what duration of the stream they cover. $ curl https://my.video.host.com/video_15/subtitles/en.m3u8 #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:22.8 #EXTINF:22.800, en.vtt In this case, since we’re only serving a short video, we can just provide a single Segment that points at a WebVTT subtitle file encompassing the entire duration of the video. If you crack open the en.vtt file you’ll see something like: $ curl https://my.video.host.com/video_15/subtitles/en.vtt WEBVTT 00:00.000 --> 00:02.000 According to all known laws of aviation, 00:02.000 --> 00:04.000 there is no way a bee should be able to fly. 00:04.000 --> 00:06.000 Its wings are too small to get its fat little body off the ground. ... The media player is capable of reading WebVTT and presenting the subtitles at the right time to the viewer. For longer videos you may want to break up your VTT files into more Segments and update the subtitle Media Playlist accordingly. To provide multiple languages and versions of subtitles, just add more EXT-X-MEDIA:TYPE=SUBTITLES lines to the Master Playlist and tweak the NAME, LANGUAGE (if different), and URI of the additional subtitle variant definitions. #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="en_subtitle",DEFAULT=NO,AUTOSELECT=yes,LANGUAGE="en",FORCED="no",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog",URI="subtitles/en.m3u8" #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="fr_subtitle",DEFAULT=NO,AUTOSELECT=yes,LANGUAGE="fr",FORCED="no",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog",URI="subtitles/fr.m3u8" #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="ja_subtitle",DEFAULT=NO,AUTOSELECT=yes,LANGUAGE="ja",FORCED="no",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog",URI="subtitles/ja.m3u8" Appending a Trailer For branding purposes (and in other applications, for advertising purposes), it can be helpful to insert Segments of video into a playlist to change the content of the video without requiring the content be appended to and re-encoded with the source file. Thankfully, HLS allows us to easily insert Segments into the Media Playlist using this one neat trick: #EXTM3U #EXT-X-VERSION:3 #EXT-X-PLAYLIST-TYPE:VOD #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:4 #EXTINF:4.000, video0.ts #EXTINF:4.000, video1.ts #EXTINF:4.000, video2.ts #EXTINF:4.000, video3.ts #EXTINF:4.000, video4.ts #EXTINF:2.800, video5.ts #EXT-X-DISCONTINUITY #EXTINF:3.337, trailer0.ts #EXTINF:1.201, trailer1.ts #EXTINF:1.301, trailer2.ts #EXT-X-ENDLIST In this Media Playlist we use HLS’s EXT-X-DISCONTINUITY header to let the video player know that the following Segments may be in a different bitrate, resolution, and aspect-ratio than the preceding content. Once we’ve provided the discontinuity header, we can add more Segments just like normal that point at a different media source broken up into .ts files. Remember, HLS allows us to use relative or absolute paths here, so we could provide a full URL for these trailer#.ts files, or virtually route them so they can retain the path context of the currently viewed video. Note that we don’t need to provide the discontinuity header here, and we could also name the trailer files something like video{6-8}.ts if we wanted to, but for clarity and proper player behavior, it’s best to use the discontinuity header if your trailer content doesn’t match the bitrate and resolution of the other video Segments. When the video player goes to play this media, it will continue from video5.ts to trailer0.ts without missing a beat, making it appear as if the trailer is part of the original video. This approach allows us to dynamically change the contents of the trailer for all videos, heavily cache the trailer .ts Segment files for performance, and avoid having to encode the trailer onto the end of every video source file. Conclusion At the end of the day, we’ve now got a video streaming service capable of tracking views and watch session durations, dynamic closed caption support, and branded trailers to help grow the platform. HLS is not a terribly complex protocol. The vast majority of it is human-readable plaintext files and is easy to inspect in the wild to how it’s used in production. When I started this project, I knew next to nothing about the protocol but was able to download some .m3u8 files and get digging to discover how the protocol worked, then build my own implementation of a HLS server to accommodate the video streaming needs of Bluesky. To learn more about HLS, you can check out the official RFC here which describes all the features discussed above and more. I hope this post encourages you to go explore other protocols you use every day by poking at them in the wild, downloading the files your browser interprets for you, and figuring out how simple some of these apparently “complex” systems are. If you’re interested in solving problems like these, take a look at our open Job Recs. If you have any questions about HLS, Bluesky, or other distributed, @scale social media infrastructure, you can find me on Bluesky here and you can discuss this post here.

a year ago 28 votes
An entire Social Network in 1.6GB (GraphD Part 2)

In Part 1 of this series, we tried to answer the question “who do you follow who also follows user B” in Bluesky, a social network with millions of users and hundreds of millions of follow relationships. At the conclusion of the post, we’d developed an in-memory graph store for the network that uses HashMaps and HashSets to keep track of the followers of every user and the set of users they follow, allowing bidirectional lookups, intersections, unions, and other set operations for combining social graph data. I received some helpful feedback after that post where several people pointed me towards Roaring Bitmaps as a potential improvement on my implementation. They were right, Roaring Bitmaps would be an excellent fit for my Graph service, GraphD, and could also provide me with a much needed way to quickly persist and load the Graph data to and from disk on startup, hopefully reducing the startup time of the service. What are Bitmaps? If you just want to dive into the Roaring Bitmap spec, you can read the paper here, but it might be easier to first talk about bitmaps in general. You can think of a bitmap as a vector of one-bit values (like booleans) that let you encode a set of integer values. For instance, say we have 10,000 users on our website and want to keep track of which users have validated their email addresses. We could do this by creating a list of the uint32 user IDs of each user, in which case if all 10,000 users have validated their emails we’re storing 10k * 32 bits = 40KB. Or, we could create a vector of single-bit values that’s 10,000 bits long (10k / 8 = 1.25KB), then if a user has confirmed their email we can set the value at the index of their UID to 1. If we want to create a list of all the UIDs of validated accounts, we can walk the vector and record the index of each non-zero bit. If we want to check if user n has validated their email, we can do a O(1) lookup in the bitmap by loading the bit at index n and checking if it’s set. When Bitmaps get Big and Sparse Now when talking about our social network problem, we’re dealing with a few more than 10,000 UIDs. We need to keep track of 5.5M users and whether or not the user follows or is followed by any of the other 5.5M users in the network. To keep a bitmap of “People who follow User A”, we’re going to need 5.5M bits which would require (5.5M / 8) ~687KB of space. If we wanted to keep bitmaps of “People who follow User A” and “People who User A follows”, we’d need ~1.37MB of space per user using a simple bitmap, meaning we’d need 5,500,000 * 1.37MB = ~7.5 Terabytes of space! Clearly this isn’t an improvement of our strategy from Part 1, so how can we make this more efficient? One strategy for compressing the bitmap is to take consecutive runs of 0’s or 1’s (i.e. 00001110000001) in the bitmap and turn them into a number. For instance if we had an account that followed only the last 100 accounts in our social network, the first 5,499,900 indices in our bitmap would be 0’s and so we could represent the bitmap by saying: 5,499,900 0's, then 100 1's which you notice I’ve written here in a lot fewer than 687KB and a computer could encode using two uint32 values plus two bits (one indicator bit for the state of each run) for a total of 66 bits. This strategy is called Run Length Encoding (RLE) and works pretty well but has a few drawbacks: mainly if your data is randomly and heavily populated, you may not have many consecutive runs (imagine a bitset where every odd bit is set and every even bit is unset). Also lookups and evaluation of the bitset requires walking the whole bitset to figure out where the index you care about lives in the compressed format. Thankfully there’s a more clever way to compress bitmaps using a strategy called Roaring Bitmaps. A brief description of the storage strategy for Roaring Bitmaps from the official paper is as follows: We partition the range of 32-bit indexes ([0, n)) into chunks of 2^16 integers sharing the same 16 most significant digits. We use specialized containers to store their 16 least significant bits. When a chunk contains no more than 4096 integers, we use a sorted array of packed 16-bit integers. When there are more than 4096 integers, we use a 2^16-bit bitmap. Thus, we have two types of containers: an array container for sparse chunks and a bitmap container for dense chunks. The 4096 threshold insures that at the level of the containers, each integer uses no more than 16 bits. These bitmaps are designed to support both densely and sparsely distributed data and can provide high performance binary set operations (and/or/etc.) operating on the containers within two or more bitsets in parallel. For more info on how Roaring Bitmaps work and some neat diagrams, check out this excellent primer on Roaring Bitmaps by Vikram Oberoi. So, how does this help us build a better graph? GraphD, Revisited with Roaring Bitmaps Let’s get back to our GraphD Service, this time in Go instead of Rust. For each user we can keep track of a struct with two bitmaps: type FollowMap struct { followingBM *roaring.Bitmap followingLk sync.RWMutex followersBM *roaring.Bitmap followersLk sync.RWMutex } Our FollowMap gives us a Roaring Bitmap for both the set of users we follow, and the set of users who follow us. Adding a Follow to the graph just requires we set the right bits in both user’s respective maps: // Note I've removed locking code and error checks for brevity func (g *Graph) addFollow(actorUID, targetUID uint32) { actorMap, _ := g.g.Load(actorUID) actorMap.followingBM.Add(targetUID) targetMap, _ := g.g.Load(targetUID) targetMap.followersBM.Add(actorUID) } Even better if we want to compute the intersections of two sets (i.e. the people User A follows who also follow User B) we can do so in parallel: // Note I've removed locking code and error checks for brevity func (g *Graph) IntersectFollowingAndFollowers(actorUID, targetUID uint32) ([]uint32, error) { actorMap, ok := g.g.Load(actorUID) targetMap, ok := g.g.Load(targetUID) intersectMap := roaring.ParAnd(4, actorMap.followingBM, targetMap.followersBM) return intersectMap.ToArray(), nil } Storing the entire graph as Roaring Bitmaps in-memory costs us around 6.5GB of RAM and allows us to perform set intersections between moderately large sets (with hundreds of thousands of set bits) in under 500 microseconds while serving over 70k req/sec! And the best part of all? We can use Roaring’s serialization format to write these bitmaps to disk or transfer them over the network. Storing 164M Follows in 1.6GB In the original version of GraphD, on startup the service would read a CSV file with an adjacency list of the (ActorDID, TargetDID) pairs of all follows on the network. This required creating a CSV dump of the follows table, pausing writes to the follows table, then bringing up the service and waiting 5 minutes for it to read the CSV file, intern the DIDs as uint32 UIDs, and construct the in-memory graph. This process is slow, pauses writes for 5 minutes, and every time our service restarts we have to do it all over again! With Roaring Bitmaps, we’re now given an easy way to effectively serialize a version of the in-memory graph that is many times smaller than the adjacency list CSV and many times faster to load. We can serialize the entire graph into a SQLite DB on the local machine where each row in a table contains: (uid, DID, followers_bitmap, following_bitmap) Loading the entire graph from this SQLite DB can be done in around ~20 seconds: // Note I've removed locking code and error checks for brevity rows, err := g.db.Query(`SELECT uid, did, following, followers FROM actors;`) for rows.Next() { var uid uint32 var did string var followingBytes []byte var followersBytes []byte rows.Scan(&uid, &did, &followingBytes, &followersBytes) followingBM := roaring.NewBitmap() followingBM.FromBuffer(followingBytes) followersBM := roaring.NewBitmap() followersBM.FromBuffer(followersBytes) followMap := &FollowMap{ followingBM: followingBM, followersBM: followersBM, followingLk: sync.RWMutex{}, followersLk: sync.RWMutex{}, } g.g.Store(uid, followMap) g.setUID(did, uid) g.setDID(uid, did) } While the service is running, we can also keep track of the UIDs of actors who have added or removed a follow since the last time we saved the DB, allowing us to periodically flush changes to the on-disk SQLite only for bitmaps that have updated. Syncing our data every 5 seconds while tailing the production firehose takes 2ms and writes an average of only ~5MB to disk per flush. The crazy part of this is, the on-disk representation of our entire follow network is only ~1.6GB! Because we’re making use of Roaring’s compressed serialized format, we can turn the ~6.5GB of in-memory maps into 1.6GB of on-disk data. Our largest bitmap, the followers of the bsky.app account with over 876k members, becomes ~500KB as a blob stored in SQLite. So, to wrap up our exploration of Roaring Bitmaps for first-degree graph databases, we saw: A ~20% reduction in resident memory size compared to HashSets and HashMaps A ~84% reduction in the on-disk size of the graph compared to an adjacency list A ~93% reduction in startup time compared to loading from an adjacency list A ~66% increase in throughput of worst-case requests under load A ~59% reduction in p99 latency of worst-case requests under low My next iteration on this problem will likely be to make use of DGraph’s in-memory Serialized Roaring Bitmap library that allows you to operate on fully-compressed bitmaps so there’s no need to serialize and deserialize them when reading from or writing to disk. It also probably results in significant memory savings as well! If you’re interested in solving problems like these, take a look at our open Backend Developer Job Rec. You can find me on Bluesky here, you can chat about this post here.

a year ago 31 votes

More in AI

Mass Intelligence

From GPT-5 to nano banana: everyone is getting access to powerful AI

12 hours ago 5 votes
Pluralistic: The capitalism of fools (28 Aug 2025)

Today's links The capitalism of fools: Trump's mirror-world New Deal. Hey look at this: Delights to delectate. Object permanence: IBM's fabric design; Nixon Cthulu; Surveillance capitalism is capitalism, with surveillance; Dismaland ad; Outdoor ed vs TB; Mathematicians' fave chalk. Upcoming appearances: Where to find me. Recent appearances: Where I've been. Latest books: You keep readin' em, I'll keep writin' 'em. Upcoming books: Like I said, I'll keep writin' 'em. Colophon: All the rest. The capitalism of fools (permalink) As Trump rails against free trade, demands public ownership stakes in corporations that receive government funds, and (selectively) enforces antitrust law, some (stupid) people are wondering, "Is Trump a communist?" In The American Prospect, David Dayen writes about the strange case of Trump's policies, which fly in the face of right wing economic orthodoxy and have the superficial trappings of a leftist economic program: https://prospect.org/economy/2025-08-28-judge-actually-existing-trump-economy/ The problem isn't that tariffs are always bad, nor is it that demanding state ownership stakes in structurally important companies that depend on public funds is bad policy. The problem is that Trump's version of these policies sucks, because everything Trump touches dies, and because he governs solely on vibes, half-remembered wisdom imparted by the last person who spoke to him, and the dying phantoms of old memories as they vanish beneath a thick bark of amyloid plaque. Take Trump's demand for a 10% stake in Intel (a course of action endorsed by no less than Bernie Sanders). Intel is a company in trouble, whose financialization has left it dependent on other companies (notably TMSC) to make its most advanced chips. The company has hollowed itself out, jettisoning both manufacturing capacity and cash reserves, pissing away the funds thus freed up on stock buybacks and dividends. Handing Trump a 10% "golden share" does nothing to improve Intel's serious structural problems. And if you take Trump at his word and accept that securing US access to advanced chips is a national security priority, Trump's Intel plan does nothing to advance that access. But it gets worse: Trump also says denying China access to these chips is a national security priority, but he greenlit Nvidia's plan to sell its top-of-the-range silicon to China in exchange for a gaudy statuette and a 15% export tax. It's possible to pursue chip manufacturing as a matter of national industrial policy, and it's even possible to achieve this goal by taking ownership stakes in key firms – because it's often easier to demand corporate change via a board seat than it is to win the court battles needed to successfully invoke the Defense Production Act. The problem is that Trumpland is uninterested in making any of that happen. They just want a smash and grab and some red meat for the base: "Look, we made Intel squeal!" Then there's the Trump tariffs. Writing in Vox EU, Lausanne prof of international business Richard Baldwin writes about the long and checkered history of using tariffs to incubate and nurture domestic production: https://www.nakedcapitalism.com/2025/08/trumpian-tariffs-rerun-the-failed-strategy-of-import-substitution-industrialization.html The theory of tariffs goes like this: if we make imports more expensive by imposing a tax on them (tariffs are taxes that are paid by consumers, after all), then domestic manufacturers will build factories and start manufacturing the foreign goods we've just raised prices on. This is called "import substitution," and it really has worked, but only in a few cases. What do those cases have in common? They were part of a comprehensive program of "export discipline, state-directed credit, and careful government–business coordination": https://academic.oup.com/book/10201 In other words, tariffs only work to reshore production where there is a lot of careful planning, diligent data-collection, and review. Governments have to provide credit to key firms to get them capitalized, provide incentives, and smack nonperformers around. Basically, this is the stuff that Biden did for renewables with the energy sector, and – to a lesser extent – for silicon with the CHIPS Act. Trump's not doing any of that. He's just winging it. There's zero follow-through. It's all about appearances, soundbites, and the libidinal satisfaction of watching corporate titans bend the knee to your cult leader. This is also how Trump approaches antitrust. When it comes to corporate power, both Trump and Biden's antitrust enforcers are able to strike terror into the hearts of corporate behemoths. The difference is that the Biden administration prioritized monopolists based on how harmful they were to the American people and the American economy, whereas Trump's trustbusters target companies based on whether Trump is mad at them: https://pluralistic.net/2024/11/12/the-enemy-of-your-enemy/#is-your-enemy What's more, any company willing to hand a million or two to a top Trump enforcer can just walk away from the charges: https://prospect.org/power/2025-08-19-doj-insider-blows-whistle-pay-to-play-antitrust-corruption/ In her 2023 book Doppelganger, Naomi Klein introduces the idea of a right-wing "mirror world" that offers a conspiratorial, unhinged version of actual problems that leftists wrestle with: https://pluralistic.net/2023/09/05/not-that-naomi/#if-the-naomi-be-klein-youre-doing-just-fine For example, the antivax movement claims that pharma companies operate on the basis of unchecked greed, without regard to the harm their defective products cause to everyday people. When they talk about this, they sound an awful like leftists who are angry that the Sacklers killed a million Americans with their opiods and then walked away with billions of dollars: https://pluralistic.net/2023/12/05/third-party-nonconsensual-releases/#au-recherche-du-pedos-perdue Then there are the conspiracy theories about voting machines. Progressives have been sounding the alarm about the security defects in voting machine since the Bush v Gore years, but that doesn't mean that Venezuelan hackers stole the 2020 election for Biden: https://pluralistic.net/2021/01/11/seeing-things/#ess When anti-15-minute-city weirdos warn that automated license-plate cameras are a gift to tyrants both petty and gross, they are repeating a warning that leftists have sounded since the Patriot Act: https://locusmag.com/2023/05/commentary-cory-doctorow-the-swivel-eyed-loons-have-a-point/ The mirror-world is a world where real problems (the rampant sexual abuse of children by powerful people and authortiy figures) are met with fake solutions (shooting up pizza parlors and transferring Ghislaine Maxwell to a country-club prison): https://www.bbc.com/news/articles/czd049y2qymo Most of the people stuck in the mirror world are poor and powerless, because desperation makes you an easy mark for grifters peddling conspiracy theories. But Trump's policies on corporate power are what happens in the mirror world inhabited by the rich and powerful. Trump is risking the economic future of every person in America (except a few cronies), but that's not the only risk here. There's also the risk that reasonable people will come to view industrial policy, government stakes in publicly supported companies, and antitrust as reckless showboating, a tactic exclusively belonging to right wing nutjobs and would-be dictators. Sociologists have a name for this: they call it "schismogenesis," when a group defines itself in opposition to its rivals. Schismogenesis is progressives insisting that voting machines and pharma companies are trustworthy and that James Comey is a resistance hero: https://pluralistic.net/2021/12/18/schizmogenesis/ After we get rid of Trump, America will be in tatters. We're going to need big, muscular state action to revive the nation and rebuild its economy. We can't afford to let Trump poison the well for the very idea of state intervention in corporate activity. Hey look at this (permalink) Thinking Ahead to the Full Military Takeover of Cities https://www.hamiltonnolan.com/p/thinking-ahead-to-the-full-military Framework is working on a giant haptic touchpad, Trackpoint nub, and eGPU for its laptops https://www.theverge.com/news/766161/framework-egpu-haptic-touchpad-trackpoint-nub National says "fuck you" on the right to repair https://norightturn.blogspot.com/2025/08/national-says-fuck-you-on-right-to.html?m=1 Tax the Rich. They’ll Stay https://www.rollingstone.com/politics/political-commentary/zohran-mamdani-tax-rich-new-york-city-1235414327/ Welcome to the Free Online Tax Preparation Feedback Survey https://irsresearch.gov1.qualtrics.com/jfe/form/SV_ewDJ6DeBj3ockGa Object permanence (permalink) #20yrsago Cops have to pay $41k for stopping man from videoing them https://web.archive.org/web/20050905015507/http://www.paed.uscourts.gov/documents/opinions/05D0847P.pdf #20yrsago Commercial music in podcasts: the end of free expression? https://memex.craphound.com/2005/08/26/commercial-music-in-podcasts-the-end-of-free-expression/ #10yrsago North Dakota cops can now use lobbyist-approved taser/pepper-spray drones https://www.thedailybeast.com/first-state-legalizes-taser-drones-for-cops-thanks-to-a-lobbyist/ #10yrsago Illinois mayor appoints failed censor to town library board https://ncac.org/news/blog/mayor-appoints-would-be-censor-to-library-board #10yrsago IBM’s lost, glorious fabric design https://collection.cooperhewitt.org/users/mepelman/visits/qtxg/87597377/ #10yrsago Former mayor of SLC suing NSA for warrantless Olympic surveillance https://www.techdirt.com/2015/08/26/prominent-salt-lake-city-residents-sue-nsa-over-mass-warrantless-surveillance-during-2002-olympics/ #10yrsago Health’s unkillable urban legend: “You must drink 8 glasses of water/day” https://www.nytimes.com/2015/08/25/upshot/no-you-do-not-have-to-drink-8-glasses-of-water-a-day.html?_r=0 #10yrsago Austin Grossman’s CROOKED: the awful, cthulhoid truth about Richard Nixon https://memex.craphound.com/2015/08/26/austin-grossmans-crooked-the-awful-cthulhoid-truth-about-richard-nixon/ #10yrsago After Katrina, FBI prioritized cellphone surveillance https://www.muckrock.com/news/archives/2015/aug/27/stingray-katrina/ #10yrsago Germany’s spy agency gave the NSA the private data of German citizens in exchange for Xkeyscore access https://www.zeit.de/digital/datenschutz/2015-08/xkeyscore-nsa-domestic-intelligence-agency #10yrsago Elaborate spear-phishing attempt against global Iranian and free speech activists, including an EFF staffer https://citizenlab.ca/2015/08/iran_two_factor_phishing/ #10yrsago Commercial for Banksy’s Dismaland https://www.youtube.com/watch?v=V2NG-MgHqEk #5yrsago Outdoor education beat TB in 1907 https://pluralistic.net/2020/08/27/cult-chalk/#tb #5yrsago Hagoromo, mathematicians' cult chalk https://pluralistic.net/2020/08/27/cult-chalk/#hagoromo #5yrsago Principles for platform regulation https://pluralistic.net/2020/08/27/cult-chalk/#eff-eu #5yrsago It's blursday https://pluralistic.net/2020/08/26/destroy-surveillance-capitalism/#blursday #5yrsago Surveillance Capitalism is just capitalism, plus surveillance https://pluralistic.net/2020/08/26/destroy-surveillance-capitalism/#surveillance-monopolism Upcoming appearances (permalink) Ithaca: AD White keynote (Cornell), Sep 12 https://deanoffaculty.cornell.edu/events/keynote-cory-doctorow-professor-at-large/ DC: Enshittification at Politics and Prose, Oct 8 https://politics-prose.com/cory-doctorow-10825 New Orleans: DeepSouthCon63, Oct 10-12 http://www.contraflowscifi.org/ Chicago: Enshittification with Kara Swisher (Chicago Humanities), Oct 15 https://www.oldtownschool.org/concerts/2025/10-15-2025-kara-swisher-and-cory-doctorow-on-enshittification/ San Francisco: Enshittification at Public Works (The Booksmith), Oct 20 https://app.gopassage.com/events/doctorow25 Miami: Enshittification at Books & Books, Nov 5 https://www.eventbrite.com/e/an-evening-with-cory-doctorow-tickets-1504647263469 Recent appearances (permalink) Divesting from Amazon’s Audible and the Fight for Digital Rights (Libro.fm) https://pocketcasts.com/podcasts/9349e8d0-a87f-013a-d8af-0acc26574db2/00e6cbcf-7f27-4589-a11e-93e4ab59c04b The Utopias Podcast https://www.buzzsprout.com/2272465/episodes/17650124 Tariffs vs IP Law (Firewalls Don't Stop Dragons) https://www.youtube.com/watch?v=LFABFe-5-uQ Latest books (permalink) "Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels). "The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org). "The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). "The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245). "Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. "Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com Upcoming books (permalink) "Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025 "Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025 https://us.macmillan.com/books/9780374619329/enshittification/ "Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026 "Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026 "The Memex Method," Farrar, Straus, Giroux, 2026 "The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, 2026 Colophon (permalink) Today's top sources: Currently writing: "The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. (1090 words yesterday, 45491 words total). A Little Brother short story about DIY insulin PLANNING This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net. https://creativecommons.org/licenses/by/4.0/ Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution. How to get Pluralistic: Blog (no ads, tracking, or data-collection): Pluralistic.net Newsletter (no ads, tracking, or data-collection): https://pluralistic.net/plura-list Mastodon (no ads, tracking, or data-collection): https://mamot.fr/@pluralistic Medium (no ads, paywalled): https://doctorow.medium.com/ Twitter (mass-scale, unrestricted, third-party surveillance and advertising): https://twitter.com/doctorow Tumblr (mass-scale, unrestricted, third-party surveillance and advertising): https://mostlysignssomeportents.tumblr.com/tagged/pluralistic "When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ISSN: 3066-764X

14 hours ago 3 votes
ML for SWEs 65: The AI bubble is popping and why that's a good thing

The future of the industry and how to get the most out of your AI coding assistant

2 days ago 4 votes
Pluralistic: By all means, tread on those people (26 Aug 2025)

Today's links By all means, tread on those people: We know you love freedom, we just wish you'd share. Hey look at this: Delights to delectate. Object permanence: The right to bear cameras; GOP wants slavery for undocumented migrants; Telepresence Nazi-punching. Upcoming appearances: Where to find me. Recent appearances: Where I've been. Latest books: You keep readin' em, I'll keep writin' 'em. Upcoming books: Like I said, I'll keep writin' 'em. Colophon: All the rest. By all means, tread on those people (permalink) Just as Martin Niemöller's "First They Came" has become our framework for understanding the rise of fascism in Nazi Germany, so, too is Wilhoit's Law the best way to understand America's decline into fascism: https://en.wikipedia.org/wiki/First_They_Came In case you're not familiar with Frank Wilhoit's amazing law, here it is: Conservatism consists of exactly one proposition, to wit: There must be in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect. https://crookedtimber.org/2018/03/21/liberals-against-progressives/#comment-729288 The thing that makes Wilhoit's Law so apt to this moment – and to our understanding of the recent history that produced this moment – is how it connects the petty with the terrifying, the trivial with the radical, the micro with the macro. It's a way to join the dots between fascists' business dealings, their interpersonal relationships, and their political views. It describes a continuum that ranges from minor commercial grifts to martial law, and shows how tolerance for the former creates the conditions for the latter. The gross ways in which Wilhoit's Law applies are easy to understand. The dollar value of corporate wage-theft far outstrips the total dollars lost to all other forms of property crime, and yet there is virtually no enforcement against bosses who steal their workers' paychecks, while petty property crimes can result in long prison sentences (depending on your skin color and/or bank balance): https://www.opportunityinstitute.org/blog/post/organized-retail-theft-wage-theft/ Elon Musk values "free speech" and insists on his right to brand innocent people as "pedos," but he also wants the courts to destroy organizations that publish their opinions about his shitty business practices: https://www.mediamatters.org/elon-musk Fascists turn crybaby when they're imprisoned for attempting a murderous coup, but buy merch celebrating the construction of domestic concentration camps where people are locked up without trial: https://officialalligatoralcatraz.com/shop That stuff is all easy to see, but I want to draw a line between these gross violations of Wilhoit's Law and pettier practices that have been creating the conditions for the present day Wilhoit Dystopia. Take terms of service. The Federalist Society – whose law library could save a lot of space by throwing away all its books and replacing them with a framed copy of Wilhoit's Law – has long held that merely glancing at a web-page or traversing the doorway of a shop is all it takes for you to enter into a "contract" by which you surrender all of your rights. Every major corporation – and many smaller ones – now routinely seek to bind both workers and customers to garbage-novellas of onerous, unreadable legal conditions. If we accept that this is how contracts work, then this should be perfectly valid, right? By reading these words, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. This indemnity will survive the termination of your relationship with your employer. I mean, why not? What principle – other than "in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect" – makes terms of service valid, and this invalid? Then there's binding arbitration. Corporations routinely bind their workers and customers to terms that force them to surrender their right to sue, no matter how badly they are injured through malice or gross negligence. This practice used to be illegal, until Antonin Scalia opened the hellmouth and unleashed binding arbitration on the world: https://brooklynworks.brooklaw.edu/cgi/viewcontent.cgi?article=1443&amp;&amp;context=blr There's a pretty clever hack around binding arbitration: mass arbitration, whereby lots of wronged people coordinate to file claims, which can cost a dirty corporation more than a plain old class-action suit: https://pluralistic.net/2021/06/02/arbitrary-arbitration/#petard Of course, Wilhoit's Law provides corporations with a way around this: they can reserve the right not to arbitrate and to force you into a class action suit if that's advantageous to them: https://pluralistic.net/2025/08/15/dogs-breakfast/#by-clicking-this-you-agree-on-behalf-of-your-employer-to-release-me-from-all-obligations-and-waivers-arising-from-any-and-all-NON-NEGOTIATED-agreements Heads they win, tails you lose. Or take the nature of property rights themselves. Conservatives say they revere property rights above all else, claiming that every other human right stems from the vigorous enforcement of property relations. What is private property? For that, we turn to the key grifter thinkfluencer Sir William Blackstone, and his 1768 "Commentaries on the Laws of England": That sole and despotic dominion which one man claims and exercises over the external things of the world, in total exclusion of the right of any other individual in the universe. https://oll.libertyfund.org/pages/blackstone-on-property-1753 Corporations love the idea of their property rights, but they're not so keen on your property rights. Think of the practice of locking down digital devices – from phones to cars to tractors – so that they can't be repaired by third parties, use generic ink or parts, or load third-party apps except via an "app store": https://memex.craphound.com/2012/01/10/lockdown-the-coming-war-on-general-purpose-computing/ A device you own, but can only use in ways that its manufacturer approves of, sure doesn't sound like "sole and despotic dominion" to me. Some corporations (and their weird apologists) like to claim that, by buying their product, you've agreed not to use it except in ways that benefit their shareholders, even when that is to your own detriment: https://pluralistic.net/2024/01/12/youre-holding-it-wrong/#if-dishwashers-were-iphones Apple will say, "We've been selling iPhones for nearly 20 years now. It can't possibly come as a surprise to you that you're not allowed to install apps that we haven't approved. If that's important to you, you shouldn't have bought an iPhone." But the obvious rejoinder to this is, "People have been given sole and despotic dominion over the things they purchased since time immemorial. If the thought of your customers using their property in ways that displease you causes you to become emotionally disregulated, perhaps you shouldn't have gotten into the manufacturing business." But as indefensibly wilhoitian as Apple's behavior might be, Google has just achieved new depths of wilhoitian depravity, with a rule that says that starting soon, you will no longer be able to install apps of your choosing on your Android device unless Google first approves of them: https://9to5google.com/2025/08/25/android-apps-developer-verification/ Like Apple, Google says that this is to prevent you from accidentally installing malicious software. Like Apple, Google does put a lot of effort into preventing its customers from being remotely attacked. And, like Apple, Google will not protect you from itself: https://pluralistic.net/2023/02/05/battery-vampire/#drained When it comes to vetoing your decisions about which programs your Android device can run, Google has an irreconcilable conflict of interest. Google, after all, is a thrice-convicted monopolist who have an interest in blocking you from installing programs that interfere with its profits, under the pretense of preventing you from coming to harm. And – like Apple – Google has a track record of selling its users out to oppressive governments. Apple blocked all working privacy tools for its Chinese users at the behest of the Chinese government, while Google secretly planned to release a version of its search engine that would enforce Chinese censorship edicts and help the Chinese government spy on its people: https://en.wikipedia.org/wiki/Dragonfly_(search_engine) Google's CEO Sundar Pichai, personally gave one million dollars to Donald Trump for a seat on the dais at this year's inauguration (so did Apple CEO Tim Cook). Both men are in a position to help the self-described dictator make good on his promise to spy on and arrest Americans who disagree with his totalitarian edicts. All of this makes Google's announcement extraordinarily reckless, but also very, very wilhoitian. After all, Google jealously guards its property rights from you, but insists that your property rights need to be subordinated to its corporate priorities: "in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect." We can see this at work in the way that Google treats open source software and free software. Google's software is "open source" – for us. We have the right to look at the code and do free work for Google to identify and fix bugs in the code. But only Google gets a say in how that code is deployed on its cloud servers. They have software freedom, while we merely have software transparency: https://pluralistic.net/2025/07/14/pole-star/#gnus-not-utilitarian Big companies love to both assert their own property rights while denying you yours. Take the music industry: they are required to pay different royalties to musicians depending on whether they're "selling" music, or "licensing" music. Sales pay a fraction of the royalties of a licensing deal, so it's far better for musicians when their label licenses their music than when they sell it. When you or I click the "buy" button in an online music store, we are confronted with a "licensing agreement," that limits what we may do with our digital purchase. Things that you get automatically when you buy music in physical form – on a CD, say – are withheld through these agreements. You can't re-sell your digital purchases as used goods. You can't give them away. You can't lend them out. You can't divide them up in a divorce. You can't leave them to your kids in your will. It's not a sale, so the file isn't your property. But when the label accounts for that licensing deal to a musician, the transaction is booked as a sale, which entitles the creative worker to a fraction of the royalties that they'd get from a license. Somehow, digital media exists in quantum superposition: it is a licensing deal when we click the buy button, but it is a sale when it shows up on a royalty statement. It's Schroedinger's download: https://pluralistic.net/2022/06/21/early-adopters/#heads-i-win Now, a class action suit against Amazon over this very issue has been given leave to progress to trial: https://www.hollywoodreporter.com/business/business-news/prime-video-lawsuit-movie-license-ownership-1236353127/ The plaintiffs insist that because Amazon showed them a button that said, "Buy this video" but then slapped it with licensing conditions that take away all kinds of rights (Amazon can even remotely delete your videos after you "buy" them) that they have been ripped off in a bait-and-switch. Amazon's defense is amazing. They've done what any ill-prepared fifth grader would do when called on the carpet; they quoted Webster's: Quoting Webster’s Dictionary, it said that the term means “rights to the use or services of payment” rather than perpetual ownership and that its disclosures properly warn people that they may lose access. People are increasingly pissed off with this bullshit, whereby things that you "buy" are not yours, and your access to them can be terminated at any time. The Stop Killing Games campaign is pushing for the rights of gamers to own the games they buy forever, even if the company decides to shut down its servers: https://www.stopkillinggames.com/ I've been pissed off about this bullshit since forever. It's one of the main reasons I convinced my publishers to let me sell my own ebooks and audiobooks, out of my own digital storefront. All of those books are sold, not licensed, and come without any terms or conditions: https://craphound.com/shop/ The ability to change the terms after the sale is a major source of enshittification. I call it the "Darth Vader MBA," as in "I am altering the deal. Pray I do not alter it any further": https://pluralistic.net/2023/10/26/hit-with-a-brick/#graceful-failure Naturally the ebooks and audiobooks in the Kickstarter for pre-sales of my next book, Enshittification are also sold without any terms and conditions: https://www.kickstarter.com/projects/doctorow/enshittification-the-drm-free-audiobook/ Look, I don't think that personal consumption choices can fix systemic problems. You're not going to fix enshittification – let alone tyranny – by shopping, even if you're very careful: https://pluralistic.net/2025/07/31/unsatisfying-answers/#systemic-problems But that doesn't mean that there isn't a connection between the unfair bullshit that monopolies cram down our throat and the rise of fascism. It's not just that the worst enshittifiers also the biggest Trump donors, it's that Wilhoit's Law powers enshittification. Wiloitism is shot through the Maga movement. The Flu Klux Klan wants to ban you from wearing a mask for health reasons, but they will defend to the death the right of ICE brownshirts to run around in gaiters and Oakleys as they kidnap our neighbors off the streets. Conservative bedwetters will donate six figures to a Givesendgo set up by some crybaby with a viral Rumble video about getting 86'ed from a restaurant for wearing a Maga hat, but they literally want to imprison trans people for wearing clothes that don't conform to their assigned-at-birth genders. They'll piss and moan about being "canceled" because of hecklers at the speeches they give for the campus chapter of the Hitler Youth, but they experience life-threatening priapism when students who object to the Israeli genocide of Palestinians are expelled, arrested and deported. Then there's their abortion policies, which hold that personhood begins at conception, but ends at birth, and can only be re-established by forming an LLC. It's "in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect" all the way down. I'm not saying that bullshit terms of service, wage theft, binding arbitration gotchas, or victim complexes about your kids going no-contact because you won't shut the fuck up about "the illegals" at Thanksgiving are the same as the actual fascist dictatorship being born around us right now or the genocide taking place in Gaza. But I am saying that they come from the same place. The ideology of "in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect" underpins the whole ugly mess. After we defeat these fucking fascists, after the next installment of the Nuremburg trials, after these eichmenn and eichwomenn get their turns in the dock, we're going to have to figure out how to keep them firmly stuck to the scrapheap of history. For this, I propose a form of broken windows policing; zero-tolerance for any activity or conduct that implies that there are "in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect." We should treat every attempt to pull any of these scams as an inch (or a yard, or a mile) down the road to fascist collapse. We shouldn't suffer practitioners of this ideology to be in our company, to run our institutions, or to work alongside of us. We should recognize them for the monsters they are. Hey look at this (permalink) Citizen Is Using AI to Generate Crime Alerts With No Human Review. It’s Making a Lot of Mistakes https://www.404media.co/citizen-is-using-ai-to-generate-crime-alerts-with-no-human-review-its-making-a-lot-of-mistakes/ How To Argue With An AI Booster https://www.wheresyoured.at/how-to-argue-with-an-ai-booster/ We must fight age verification with all we have https://www.usermag.co/p/we-must-fight-age-verification-with Sqinks: A Transreal Cyberpunk Love Story https://www.kickstarter.com/projects/rudyrucker/sqinks LibreOffice 25.8: a Strategic Asset for Governments and Enterprises Focused on Digital Sovereignty and Privacy https://blog.documentfoundation.org/blog/2025/08/25/libreoffice-25-8-backgrounder/ Object permanence (permalink) #20yrsago Oakland sheriffs detain people for carrying cameras https://thomashawk.com/2005/08/right-to-bear-cameras.html #10yrsago New Zealand gov’t promises secret courts for accused terrorists https://www.nzherald.co.nz/nz/attorney-general-says-law-society-got-it-wrong-over-secret-courts/E5JHYBTMVSIBZ62UNGEWB4DPEA/?c_id=1&amp;objectid=11503094 #10yrsago Platform Cooperativism: a worker-owned Uber for everything https://platformcoop.net/ #10yrsago GOP “kingmaker” proposes enslavement as an answer to undocumented migrants https://www.thedailybeast.com/iowa-gop-kingmaker-has-a-slavery-proposal-for-immigration/ #10yrsago Six years after unprovoked beating, Denver cop finally fired https://kdvr.com/news/video-evidence-determined-fate-of-denver-officer-in-excessive-force-dispute-fired-after-6-years/ #10yrsago Samsung fridges can leak your Gmail logins https://web.archive.org/web/20150825014450/https://www.pentestpartners.com/blog/hacking-defcon-23s-iot-village-samsung-fridge/ #10yrsago German student ditches apartment, buys an unlimited train pass https://www.washingtonpost.com/news/worldviews/wp/2015/08/22/how-one-german-millennial-chose-to-live-on-trains-rather-than-pay-rent/ #10yrsago Ashley Madison’s founding CTO claimed he hacked competing dating site https://www.wired.com/2015/08/ashley-madison-leak-reveals-ex-cto-hacked-competing-site/ #5yrsago Telepresence Nazi-punching https://pluralistic.net/2020/08/25/anxietypunk/#smartibots #5yrsago Ballistic Kiss https://pluralistic.net/2020/08/25/anxietypunk/#bk Upcoming appearances (permalink) Ithaca: AD White keynote (Cornell), Sep 12 https://deanoffaculty.cornell.edu/events/keynote-cory-doctorow-professor-at-large/ DC: Enshittification at Politics and Prose, Oct 8 https://politics-prose.com/cory-doctorow-10825 New Orleans: DeepSouthCon63, Oct 10-12 http://www.contraflowscifi.org/ Chicago: Enshittification with Kara Swisher (Chicago Humanities), Oct 15 https://www.oldtownschool.org/concerts/2025/10-15-2025-kara-swisher-and-cory-doctorow-on-enshittification/ San Francisco: Enshittification at Public Works (The Booksmith), Oct 20 https://app.gopassage.com/events/doctorow25 Miami: Enshittification at Books & Books, Nov 5 https://www.eventbrite.com/e/an-evening-with-cory-doctorow-tickets-1504647263469 Recent appearances (permalink) Divesting from Amazon’s Audible and the Fight for Digital Rights (Libro.fm) https://pocketcasts.com/podcasts/9349e8d0-a87f-013a-d8af-0acc26574db2/00e6cbcf-7f27-4589-a11e-93e4ab59c04b The Utopias Podcast https://www.buzzsprout.com/2272465/episodes/17650124 Tariffs vs IP Law (Firewalls Don't Stop Dragons) https://www.youtube.com/watch?v=LFABFe-5-uQ Latest books (permalink) "Picks and Shovels": a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books (US), Head of Zeus (UK), February 2025 (https://us.macmillan.com/books/9781250865908/picksandshovels). "The Bezzle": a sequel to "Red Team Blues," about prison-tech and other grifts, Tor Books (US), Head of Zeus (UK), February 2024 (the-bezzle.org). "The Lost Cause:" a solarpunk novel of hope in the climate emergency, Tor Books (US), Head of Zeus (UK), November 2023 (http://lost-cause.org). "The Internet Con": A nonfiction book about interoperability and Big Tech (Verso) September 2023 (http://seizethemeansofcomputation.org). Signed copies at Book Soup (https://www.booksoup.com/book/9781804291245). "Red Team Blues": "A grabby, compulsive thriller that will leave you knowing more about how the world works than you did before." Tor Books http://redteamblues.com. "Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin", on how to unrig the markets for creative labor, Beacon Press/Scribe 2022 https://chokepointcapitalism.com Upcoming books (permalink) "Canny Valley": A limited edition collection of the collages I create for Pluralistic, self-published, September 2025 "Enshittification: Why Everything Suddenly Got Worse and What to Do About It," Farrar, Straus, Giroux, October 7 2025 https://us.macmillan.com/books/9780374619329/enshittification/ "Unauthorized Bread": a middle-grades graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2026 "Enshittification, Why Everything Suddenly Got Worse and What to Do About It" (the graphic novel), Firstsecond, 2026 "The Memex Method," Farrar, Straus, Giroux, 2026 "The Reverse-Centaur's Guide to AI," a short book about being a better AI critic, Farrar, Straus and Giroux, 2026 Colophon (permalink) Today's top sources: Currently writing: "The Reverse Centaur's Guide to AI," a short book for Farrar, Straus and Giroux about being an effective AI critic. (1019 words yesterday, 42282 words total). A Little Brother short story about DIY insulin PLANNING This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net. https://creativecommons.org/licenses/by/4.0/ Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution. How to get Pluralistic: Blog (no ads, tracking, or data-collection): Pluralistic.net Newsletter (no ads, tracking, or data-collection): https://pluralistic.net/plura-list Mastodon (no ads, tracking, or data-collection): https://mamot.fr/@pluralistic Medium (no ads, paywalled): https://doctorow.medium.com/ Twitter (mass-scale, unrestricted, third-party surveillance and advertising): https://twitter.com/doctorow Tumblr (mass-scale, unrestricted, third-party surveillance and advertising): https://mostlysignssomeportents.tumblr.com/tagged/pluralistic "When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla READ CAREFULLY: By reading this, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ISSN: 3066-764X

3 days ago 5 votes