Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Marco.org
I’ve been taking a break from Mastodon as my social home on the web for about six weeks. It has been more difficult than I’d usually admit. I have an amazing local community in my town that I love and am deeply involved in. But nobody here is a programmer, or even a computer nerd, so it doesn’t serve my need for a tech community. I work alone, for myself, in my house. It’s a fortunate lifestyle with many upsides, but it means I don’t get that form of community at work, either. I’ve had this feeling before… as a teenager in the 1990s. It’s profoundly lonely. But when I got internet access, that started to change. I found my people. First in the Total Annihilation community (RIP, Annihilated), then the HardOCP Forum, then the Something Awful Forums, then Tumblr, then Twitter and our community of Apple bloggers, and then Mastodon. Communities form, change, and dissolve over time. None of those are, today, anything like what they were when I most enjoyed them. We change, too. I didn’t move on from most of those solely because they changed — it was usually because we both changed, and it was no longer a good fit. My break from Mastodon has been a relief in some ways, as I’ve felt increasing divergence between what I wanted and what I was getting from it, but I didn’t know where else to go. Instagram, TikTok, and YouTube are all more video-broadcast platforms than communities. All I’ve found on Bluesky is even more anger and politics, the last things I need or want. Threads hasn’t yet stuck for me, and it’s owned by an asshole. Twitter was an amazing product and community that was previously owned by assholes, and now it’s X, which has a strong but different tech community, and is owned by different assholes. So I’ve just been… nowhere, for the first time since the 1990s. It has been, once again, quite lonely. Sure, it’s been relaxing, too, but in the way a vacation is relaxing. It’s great to take breaks, but after a long vacation, I’m ready to turn my brain back on. Without participating in a large, broad tech community, I’m exposed to fewer new ideas, both of my main jobs (app development and podcasting) are much harder, the quality and reach of my work have suffered, my skills aren’t improving as much, my mind is less stimulated, and I feel disconnected from my professions and passions. My world has gotten smaller. I could keep going like this to prove a stubborn victory or purity test, but I’d mostly just be hurting myself. Communities change, conditions change, and we change. We’re all better off if we’re willing to change our minds, and if our communities let us, no matter how strongly our younger selves declared a previous position. So I’m going to wade back in and see how it goes, trying to find a better balance to hopefully have better results, including on Mastodon. My people are spread across multiple services, and I’m going to explore them all, despite most of them being owned by assholes. I’d love to participate in communities without any assholes benefiting from it, but in the modern world, that simply isn’t possible. The only alternative is to isolate myself and not have a community at all (or a job, or a smartphone, or a computer, or internet access, or a car, or most of my food, or my country). That’s neither a price I’m willing to pay nor a condition I’d try to impose on anyone else. I’d love to have your support, forgiveness, and grace as I try to find my people again, and I strive to offer the same to all of you. Everyone deserves to be part of a community. And as I grow, I’m learning to declare fewer absolutes in a complex, imperfect, and constantly changing world.
For the first half of my childhood in suburban Ohio, we lived next to Bob. Bob and his wife were a kind, reserved older couple who decorated their home mostly with crucifixes and talked about Jesus a lot. Bob helped my mom with a lot of common household tasks after my dad died. When she woke up to a small fire on our deck, she didn’t call the fire department — she called Bob.1 I never learned any of Bob’s political views, though I can make some guesses in retrospect. They never mattered. He was just our neighbor. We later moved, and hit the neighbor jackpot with Van. Van was deeply kind, generous, and friendly — the sort of neighbor everyone wishes for. In my lazy teenage years, he’d often mow our lawn or shovel our snow for us before I even woke up. As his mobility later declined, I was honored to return the favor. Van and his delightful wife passed away years ago, before I learned how either of them felt on any political issue. If they were my neighbors today, how likely would that be? * * * If I learn today that my neighbors have political views that my “side” considers unacceptable and unforgivable, should I be able to have dinner at their house, shovel their snow, or even greet them outside? Should I be condemned for supporting or associating with them in non-political contexts? Should I be condemned for not publicly condemning them myself? While privilege and power dynamics play a complex and unavoidable role in this, I don’t want to associate with any community that would insist on either of those.2 And I can’t help but feel that maybe we were better off before we knew everyone’s hot takes on everything. The town coffee shop was just a coffee shop. We weren’t publicly shamed for going there because one of the owners was an ass. It was just a place to hang out with our friends. Our casual work friends were just casual work friends. We didn’t expect them to take positions on the world’s most complicated conflicts. We could just talk about weather and TV shows, and maybe sometimes get a beer after work. And our neighbors were just our neighbors. That was the day I learned the word “SHIT!” ↩︎ Somewhere on Bob’s wall might’ve been a relevant statement from an influential political dissident about casting the first stone. ↩︎
As Apple celebrates its fiftieth birthday, we celebrate the spirit of its formation, when people who loved computers started making great computers to inspire more people to love computers. That spirit is difficult to find in the tech business today. Immense scale, soulless optimization, and an insatiable thirst for growth dominate its behavior and discourse, leaving little room for the spirit and principles embodied by Steve Jobs and Steve Wozniak. Apple still has this spirit, and I believe you do, too. But it’s not infinite or invincible. It’s under constant pressure, including from Apple itself. It seems likely that you’ll soon be leading Apple, which will place unfathomable responsibility on your shoulders. As you grow into the leader that we know you can be, I urge you, on behalf of everyone who loves computers as much as we do, to protect and cultivate this spirit of Apple’s founders as the company’s top priority: We love computers. We don’t hide that — we celebrate it! We use computers to enhance our minds, lives, and abilities — not to be controlled, restricted, tricked, placated, angered, or surveilled. Our computers work for us, with the utmost respect for our time, attention, money, data, and privacy. We are customers and owners — not resources to be harvested, annoyed, or badgered into ever more services and upsells. Apple leads the industry in these values, but leading doesn’t always mean excelling. Remaining true to these values requires constant diligence, honest evaluation, introspection, and the audacity and courage to effect change. Apple doesn’t settle for fine, functional, or good enough in its hardware (and thanks for your incredible work on that). We love making and using products that aren’t just great, but greater than they need to be, always raising the bar of greatness for its own sake. Software, services, revenue sources, and world impact need to be held to that same standard. Focus on making great computers with great user experiences above all else, and you can trust that every other major goal will follow: profit, market share, expansion, impact, and benefit to the world. Making great computers must remain Apple’s top responsibility, because if you don’t do it, nobody will.
Ten years ago, Apple’s Phil Schiller surprised Apple enthusiasts and developers by walking out on stage at John Gruber’s The Talk Show Live WWDC event and giving an open, human, honest interview to a somewhat jaded community. I wrote this in response: Both Apple and Phil Schiller himself took a huge risk in doing this. That they agreed at all is a noteworthy gift to this community of long-time enthusiasts, many of whom have felt under-appreciated as the company has grown. […] Phil’s appearance on the show was warm, genuine, informative, and entertaining. It was human. And humanizing the company and its decisions, especially to developers — remember, developer relations is all under Phil — might be worth the PR risk. This started a ten-year run of interviews by Apple executives on The Talk Show every year at WWDC that proved to be great, surprisingly safe PR for Apple. No executive ever said something they shouldn’t have (they’re pros), no sensational or negative news stories ever resulted from them, and Apple’s enthusiastic fans and developers felt seen, heard, and appreciated. * * * For unspecified reasons, Apple has declined to participate this year, ending what had become a beloved tradition in our community — and I can’t help but suspect that it won’t come back. (A lot has changed in the meantime.) Maybe Apple has good reasons. Maybe not. We’ll see what their WWDC PR strategy looks like in a couple of weeks. In the absence of any other information, it’s easy to assume that Apple no longer wants its executives to be interviewed in a human, unscripted, unedited context that may contain hard questions, and that Apple no longer feels it necessary to show their appreciation to our community and developers in this way. I hope that’s either not the case, or it doesn’t stay the case for long. This will be the first WWDC I’m not attending since 2009 (excluding the remote 2020 one, of course). Given my realizations about my relationship with Apple and how they view developers, I’ve decided that it’s best for me to take a break this year, gain some perspective, and decide what my future relationship should look like. Maybe Apple’s leaders are doing that, too.
More in programming
I’ve been taking a break from Mastodon as my social home on the web for about six weeks. It has been more difficult than I’d usually admit. I have an amazing local community in my town that I love and am deeply involved in. But nobody here is a programmer, or even a computer nerd, so it doesn’t serve my need for a tech community. I work alone, for myself, in my house. It’s a fortunate lifestyle with many upsides, but it means I don’t get that form of community at work, either. I’ve had this feeling before… as a teenager in the 1990s. It’s profoundly lonely. But when I got internet access, that started to change. I found my people. First in the Total Annihilation community (RIP, Annihilated), then the HardOCP Forum, then the Something Awful Forums, then Tumblr, then Twitter and our community of Apple bloggers, and then Mastodon. Communities form, change, and dissolve over time. None of those are, today, anything like what they were when I most enjoyed them. We change, too. I didn’t move on from most of those solely because they changed — it was usually because we both changed, and it was no longer a good fit. My break from Mastodon has been a relief in some ways, as I’ve felt increasing divergence between what I wanted and what I was getting from it, but I didn’t know where else to go. Instagram, TikTok, and YouTube are all more video-broadcast platforms than communities. All I’ve found on Bluesky is even more anger and politics, the last things I need or want. Threads hasn’t yet stuck for me, and it’s owned by an asshole. Twitter was an amazing product and community that was previously owned by assholes, and now it’s X, which has a strong but different tech community, and is owned by different assholes. So I’ve just been… nowhere, for the first time since the 1990s. It has been, once again, quite lonely. Sure, it’s been relaxing, too, but in the way a vacation is relaxing. It’s great to take breaks, but after a long vacation, I’m ready to turn my brain back on. Without participating in a large, broad tech community, I’m exposed to fewer new ideas, both of my main jobs (app development and podcasting) are much harder, the quality and reach of my work have suffered, my skills aren’t improving as much, my mind is less stimulated, and I feel disconnected from my professions and passions. My world has gotten smaller. I could keep going like this to prove a stubborn victory or purity test, but I’d mostly just be hurting myself. Communities change, conditions change, and we change. We’re all better off if we’re willing to change our minds, and if our communities let us, no matter how strongly our younger selves declared a previous position. So I’m going to wade back in and see how it goes, trying to find a better balance to hopefully have better results, including on Mastodon. My people are spread across multiple services, and I’m going to explore them all, despite most of them being owned by assholes. I’d love to participate in communities without any assholes benefiting from it, but in the modern world, that simply isn’t possible. The only alternative is to isolate myself and not have a community at all (or a job, or a smartphone, or a computer, or internet access, or a car, or most of my food, or my country). That’s neither a price I’m willing to pay nor a condition I’d try to impose on anyone else. I’d love to have your support, forgiveness, and grace as I try to find my people again, and I strive to offer the same to all of you. Everyone deserves to be part of a community. And as I grow, I’m learning to declare fewer absolutes in a complex, imperfect, and constantly changing world.
My now-traditional “summary” of the conference along with my own commentary and thoughts.
Every year, browser vendors collaborate via the Interop project to synchronize their efforts on new HTML/CSS/JS features. This year, there’s a ton of super exciting features being considered! In this blog post, we’ll explore the CSS of the future, to get a glimpse of what we’ll soon be able to do.
Imprint’s average engineer is merging 4x more PRs than 2 years ago. Yay. Aggregated behind the average is something even more interesting: we have two teams that are merging 10x their PRs over that period. One engineer is merging 25x the PRs per week of our average engineer two years ago, and a number are merging 10x. That’s pretty neat, and the slope of the line suggests our average in a year is going to look much closer to today’s p90 than today’s p50. Even more importantly, we aren’t just shipping more, our quality has improved: incidents are down 42% half-over-half, and critical incidents are down 78% half-over-half. I think we’ve always known the quality/velocity tradeoff wasn’t a particularly useful construct, but that’s undeniable today. As we’ve walked this path, we–like everyone else progressing down it–have run into the problem of pull request review. The p50 pull request review took about 33 minutes to complete 2 years ago, but now takes us about 77 minutes. The slope here correlates tightly with the number of pull requests getting merged. Tightly enough that addressing it became one of my priorities for the week. We solved it! Just kidding, we’ve absolutely not solved it, but I think our current approach is an interesting one, which we call the small-change. The work discussed here was shaped and done by many folks at Imprint including Dan, Joy, Daniel, Lenny, Taylor, Ted, and Annie, among many others. Why review pull requests? There are four reasons we review pull requests: Knowledge sharing between author and reviewer – software engineering is generally learned via apprenticeship, and reviews are a primary mechanism of experienced engineers teaching others what good software looks like. We believe the current volume means that this knowledge sharing will have to move towards operating on clusters of pull requests, rather than individual pull requests Preventing errors – review has always been an expensive time to catch errors, and we believe that effective CICD along with harness verification techniques like poteto’s pstack are more effective mechanisms for preventing errors than review Preventing malicious behavior – my experience is that even careful code review will generally miss competent malicious behavior, although fortunately most malicious behavior is incompetent. We believe both sorts of behavior are best identified using a combination of fixed rules (modifying these specific objects/classes/files requires review), anti-abuse mechanisms that we’ve refined in our LLM-based member support experiences, auditing, and audit log inspection Compliance obligations in our documented controls – these can be changed, but today they require that an independent human reviews changes in a subset of our repositories In the modern development paradigm, we think there is a better solution for each of these problems. But, we also want to make these changes very cautiously. Small steps with small-change We believe we move fast by finding narrow, contained ways to validate new practices, so to iterate our way into an effective solution, we’ve developed the concept of a small-change. Here’s how it works: When a pull request is marked ready, CICD determines whether it is eligible to be a small-change. This is represented as a passing Github check. There is also a label for visibility, but it is the check itself, managed by the build and not human modifiable, that determines eligibility. (The label is used for initial filtering, but the check is still independently verified.) Something is a small-change if it meets all of these criteria: It is less than 1,000 lines of code There are no p1 or p2 findings from our ai-code-review automated review bot The author has “experience” in the service or application, defined as 10+ prior merged pull requests into that service or application’s directories. For changes working across multiple services or applications, the author must have experience in all of them to qualify Impacts fewer than N distinct services or applications. N is 1 on our frontend monorepo, and N is 3 in our backend monorepo, where typical changes will modify both a gRPC service and a common HTTP gateway Shared libraries and utilities files are not eligible for small-change Each repository has a .small_change file which allows per-path customization while we continue to tune the behaviors. For example, a service dealing with money movement can be excluded Our Slack automation tags every reference to a pull request with reacji labeling the state of that pull request, for example if it has failing tests, if it has been approved but tests are still running, or if it is ready to merge. We added a small-change reacji that indicates which of those pull requests meet the small-change criteria In our internal skills repository, we created two skills that have been distributed across harnesses: review-small-change which takes a list of repositories and reviews all open, non-draft, small-changes, including providing approval if they meet the review’s criteria (all checks pass including small-change check, and not touching on sensitive topics such as payments and security) review-pr-cluster which reviews batches of pull requests to identify common themes that came up, and suggest opportunities for discussion and adding verifications We set the social expectation that any error caused by a small-change is a failure in our system, and has to be solved at the system level: improvements to ai-code-review, to our verification mechanisms, to our CICD, to our harness, etc. This took about two days of infrequent attention to implement and rollout, although I imagine we will be tuning the knobs here for a long time. Maybe forever. fleet: centrally orchestrated harness I’ll briefly mention our other approach to reducing code review volume, which anchors on our centrally-orchestrated harness, fleet. (Which is similar to Stripe’s Minions although not following the one-shot approach in particular). We rely on fleet to implement and verify software in a highly controlled harness running in the cloud, with independent evaluation mechanisms to identify the correct number of independent reviewers for any given change. That number can be as low as one (e.g. fleet writes, one human reviews) for changes evaluated as safe. Results / next steps / reception I’ll be honest, it’s too early to give a useful result here. From early results, about 10% of changes qualify as a small-change. Those are now eligible for very fast approval, and generally get it unless they trip a flag, generally due to touching a security or money adjacent topic. While 10% is a small percentage, the core concept here is working, and ultimately will succeed or fail based on our ability to expand that percentage over the next few months in tandem with expanding our verification mechanisms to be increasingly comprehensive. This is ultimately going to fail or succeed depending on our ability to improve our verifications, which does require more difficult sorts of techniques to accomplish in low scale-complexity, high domain-complexity software than in the high scale-complexity, low domain-complexity products where many techniques like incremental rollouts are exceptionally effective. (Don’t get me wrong, they still work, they just provide less coverage when most users have different configuration and behavior.) Folks have not been uniformly excited about this, it’s a substantial change from how we’ve learned to be good engineers, and it’s uncomfortable to reevaluate whether a long-held best practice has become merely dogma. The most energizing part of the current cycle is that we get to reevaluate these practices we grew up on every week; the most painful part of the current cycle is that we’re likely to fall behind if we ever stop reevaluating them.
