Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Don Melton
For reasons that will soon become obvious, I’m shutting the doors on this website. Everything will remain online for now, but I don’t plan on returning to write anything new here. Not that I’ve added any content in almost two years anyway. I still have a passion for making observations, telling stories and recording my thoughts as they happen. I’ll just be doing it elsewhere. Thank you for reading.
America is sick. And I don’t just mean with COVID-19. The bad news is that removing the ugly, orange tumor in the White House next week will not be enough to affect a cure. The malignancy has spread. It didn’t even start with the presidency. We’ve been brewing and self-dosing a toxic carcinogen for centuries. White supremacy is, of course, the not-so-secret ingredient. While responsibility for this has changed hands over the years, the modern Republican Party is obviously ensuring racism persists and even flourishes here in America. And for nothing more than electoral advantage. The GOP cannot maintain power if everyone is allowed to vote. We all know this. But even disenfranchising non-white, non-native and otherwise different people is not enough for them now. And their new strategy only starts by overturning election results they don’t like, achieved by lying to their lesser number of voters that they’ve been cheated. The violent insurrectionists at the Capitol last week weren’t really interested in elections, fair or otherwise. They wanted to install a dictator. And many in the GOP—federal, state and local officials as well as their voters—were encouraging that. It’s clear that Republicanism is no longer compatible with democracy. So we have a big problem. The reason so many of our fellow citizens are angry and demanding autocracy is more complicated than simple political disagreements. There’s something elemental going on here. An embrace of conspiracy theories, ignorance and grievance. And on a scale I’ve never seen. I’ll be honest that I have no idea how to cure this disease. I’m not even sure where to begin. But our nation needs some powerful medicine. Or the body will die.
After I posted that link to my latest podcast with Rene Ritchie, several folks alerted me via Twitter that all my colorful metaphors had been “bleeped” on the audio. I didn’t realize that because I hadn’t listened to the recording myself. And I don’t normally listen to my own podcasts because… that’s just sort of creepy, isn’t it? Obviously, that means I don’t mix the audio either. I don’t do that because 1) I don’t have relevant experience at it, 2) I’m really lazy and 3) fine folks elsewhere do all the hard work for me. My apologies if you didn’t get the whole “Melton” experience you were expecting. Rene tells me that episode was an accident and our next podcast won’t be censored. “Let Melton be Melton,” as he likes to say. Plus, we might just release an explicit version of the current show. Has everyone calmed the fuck down now?1 OK, here’s the thing—I was not upset at all about being censored. The show might be called “Melton” but that’s only because 1) Rene Ritchie is a generous man, 2) I’m vain and 3) we couldn’t think of a better name after we recorded the first episode. I consider the whole enterprise as something Rene and I do together. It’s our show. Not my show. If anything, I’m the co-host. This is exactly why I call Rene (and Kelly Guimont for our “Westworld” podcast) “boss.” I’m not trying to be funny, ironic or insult them. I’m reminding myself who really is in charge. And who does all the hard work. Seriously, I just talk into a microphone, folks. And it’s a microphone that Rene gave me! A really nice Røde Podcaster model, too. Talking is easy and I continue to be amazed that anyone out there cares about listening to what I have to say. I’m honored that all these nice people enable me to broadcast my various musings, opinions and rants. So if Rene and anyone else at iMore—or Jason Snell and anyone else at The Incomparable—decide to censor my many and frequent vulgarities, it’s their call. They’re the publishers. And being censored won’t damage my “brand”—whatever the hell that means. (Actually, it scares me thinking about what that means.) Yes, words matter. Exact words even. But the truth is that some people—whether they admit being offended or not—have difficulty listening to vulgarities. Especially at the pace I spew them. A friend of mine told me he’s sad that he can’t listen to my podcasts in his car anymore now that he has kids. I get it. There are valid reasons to hit the buzzer. As anyone who’s adventurous enough to follow me on Twitter knows, I’m saltier than most sailors. I don’t plan on changing that there or on this website. But if someone needs to filter me a bit elsewhere, I’m fine with that. A tired catchphrase which really needs retirement. I should know. ↩
More in programming
My now-traditional “summary” of the conference along with my own commentary and thoughts.
Every year, browser vendors collaborate via the Interop project to synchronize their efforts on new HTML/CSS/JS features. This year, there’s a ton of super exciting features being considered! In this blog post, we’ll explore the CSS of the future, to get a glimpse of what we’ll soon be able to do.
Imprint’s average engineer is merging 4x more PRs than 2 years ago. Yay. Aggregated behind the average is something even more interesting: we have two teams that are merging 10x their PRs over that period. One engineer is merging 25x the PRs per week of our average engineer two years ago, and a number are merging 10x. That’s pretty neat, and the slope of the line suggests our average in a year is going to look much closer to today’s p90 than today’s p50. Even more importantly, we aren’t just shipping more, our quality has improved: incidents are down 42% half-over-half, and critical incidents are down 78% half-over-half. I think we’ve always known the quality/velocity tradeoff wasn’t a particularly useful construct, but that’s undeniable today. As we’ve walked this path, we–like everyone else progressing down it–have run into the problem of pull request review. The p50 pull request review took about 33 minutes to complete 2 years ago, but now takes us about 77 minutes. The slope here correlates tightly with the number of pull requests getting merged. Tightly enough that addressing it became one of my priorities for the week. We solved it! Just kidding, we’ve absolutely not solved it, but I think our current approach is an interesting one, which we call the small-change. The work discussed here was shaped and done by many folks at Imprint including Dan, Joy, Daniel, Lenny, Taylor, Ted, and Annie, among many others. Why review pull requests? There are four reasons we review pull requests: Knowledge sharing between author and reviewer – software engineering is generally learned via apprenticeship, and reviews are a primary mechanism of experienced engineers teaching others what good software looks like. We believe the current volume means that this knowledge sharing will have to move towards operating on clusters of pull requests, rather than individual pull requests Preventing errors – review has always been an expensive time to catch errors, and we believe that effective CICD along with harness verification techniques like poteto’s pstack are more effective mechanisms for preventing errors than review Preventing malicious behavior – my experience is that even careful code review will generally miss competent malicious behavior, although fortunately most malicious behavior is incompetent. We believe both sorts of behavior are best identified using a combination of fixed rules (modifying these specific objects/classes/files requires review), anti-abuse mechanisms that we’ve refined in our LLM-based member support experiences, auditing, and audit log inspection Compliance obligations in our documented controls – these can be changed, but today they require that an independent human reviews changes in a subset of our repositories In the modern development paradigm, we think there is a better solution for each of these problems. But, we also want to make these changes very cautiously. Small steps with small-change We believe we move fast by finding narrow, contained ways to validate new practices, so to iterate our way into an effective solution, we’ve developed the concept of a small-change. Here’s how it works: When a pull request is marked ready, CICD determines whether it is eligible to be a small-change. This is represented as a passing Github check. There is also a label for visibility, but it is the check itself, managed by the build and not human modifiable, that determines eligibility. (The label is used for initial filtering, but the check is still independently verified.) Something is a small-change if it meets all of these criteria: It is less than 1,000 lines of code There are no p1 or p2 findings from our ai-code-review automated review bot The author has “experience” in the service or application, defined as 10+ prior merged pull requests into that service or application’s directories. For changes working across multiple services or applications, the author must have experience in all of them to qualify Impacts fewer than N distinct services or applications. N is 1 on our frontend monorepo, and N is 3 in our backend monorepo, where typical changes will modify both a gRPC service and a common HTTP gateway Shared libraries and utilities files are not eligible for small-change Each repository has a .small_change file which allows per-path customization while we continue to tune the behaviors. For example, a service dealing with money movement can be excluded Our Slack automation tags every reference to a pull request with reacji labeling the state of that pull request, for example if it has failing tests, if it has been approved but tests are still running, or if it is ready to merge. We added a small-change reacji that indicates which of those pull requests meet the small-change criteria In our internal skills repository, we created two skills that have been distributed across harnesses: review-small-change which takes a list of repositories and reviews all open, non-draft, small-changes, including providing approval if they meet the review’s criteria (all checks pass including small-change check, and not touching on sensitive topics such as payments and security) review-pr-cluster which reviews batches of pull requests to identify common themes that came up, and suggest opportunities for discussion and adding verifications We set the social expectation that any error caused by a small-change is a failure in our system, and has to be solved at the system level: improvements to ai-code-review, to our verification mechanisms, to our CICD, to our harness, etc. This took about two days of infrequent attention to implement and rollout, although I imagine we will be tuning the knobs here for a long time. Maybe forever. fleet: centrally orchestrated harness I’ll briefly mention our other approach to reducing code review volume, which anchors on our centrally-orchestrated harness, fleet. (Which is similar to Stripe’s Minions although not following the one-shot approach in particular). We rely on fleet to implement and verify software in a highly controlled harness running in the cloud, with independent evaluation mechanisms to identify the correct number of independent reviewers for any given change. That number can be as low as one (e.g. fleet writes, one human reviews) for changes evaluated as safe. Results / next steps / reception I’ll be honest, it’s too early to give a useful result here. From early results, about 10% of changes qualify as a small-change. Those are now eligible for very fast approval, and generally get it unless they trip a flag, generally due to touching a security or money adjacent topic. While 10% is a small percentage, the core concept here is working, and ultimately will succeed or fail based on our ability to expand that percentage over the next few months in tandem with expanding our verification mechanisms to be increasingly comprehensive. This is ultimately going to fail or succeed depending on our ability to improve our verifications, which does require more difficult sorts of techniques to accomplish in low scale-complexity, high domain-complexity software than in the high scale-complexity, low domain-complexity products where many techniques like incremental rollouts are exceptionally effective. (Don’t get me wrong, they still work, they just provide less coverage when most users have different configuration and behavior.) Folks have not been uniformly excited about this, it’s a substantial change from how we’ve learned to be good engineers, and it’s uncomfortable to reevaluate whether a long-held best practice has become merely dogma. The most energizing part of the current cycle is that we get to reevaluate these practices we grew up on every week; the most painful part of the current cycle is that we’re likely to fall behind if we ever stop reevaluating them.
The park near my house has lots of chestnuts and my kids have been collecting them like crazy (it’s actually their favorite post-school activity). We accumulated quite a hoard. I thought it would be fun to make a launcher so we had something to do with our new chestnut collection. I wanted to go with a very simple crossbow-style design (like the famous D2 item) using latex tubing and a leather pouch for holding the chestnuts. This is what I came up with. It launches chestnuts about 220 feet, and it’s simple enough to build in an afternoon. The frame is a 42 inch 2x2 and a spare IKEA bed slat for the crossbeam, glued and screwed together. A 3/8 inch eye bolt goes through each end of the crossbeam, and the latex tubing is tied to those and to the pouch from a leather shepherd’s sling. Tying the tubing was the fiddliest part. It has to be stretched while you wrap it with butcher’s twine, so I clamped a couple of scrap blocks to my workbench to hold it. I like this design because the tubing can be wrapped around the eye bolts to easily add more tension and power. The Trigger My favorite part of the design was that I decided to use an archery release like this one for the trigger. It provides a safe and reliable way to hold the tubing back, and it’s made to let go cleanly. I 3D printed a little housing that mounts the release securely to the back of the stock, and the release clips onto a loop of archery D-loop cord on the back of the pouch. Then I added a dowel for a rear grip and an old handle I had previously removed from my shop vac. How Well Does It Work? We tried it at the park and it works surprisingly well. Aimed at about 35 to 40 degrees, it sends our chestnuts (on the small side, about 10 to 15 grams) about 220 feet. Thanks to the archery release, we haven’t had a single misfire. I got curious and did some rough math. To go that far with air drag, the chestnuts are leaving at somewhere around 85 to 100 mph, which works out to about the same energy as a paintball. So safety glasses are important and you never point it at anyone. You can find the 3D print on Printables and a full build guide on Instructables. Make sure to always use it with adult supervision :)
