Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from heavymeta.org
Just after midnight Eastern Time on July 19, 2024, the enterprise cybersecurity company CrowdStrike YOLOed a software update to millions of Windows machines. Or as they put it: On July 19, 2024 at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems. That sensor configuration update caused the largest IT outage in history. Overnight, about 8.5 million computers blue screened, affecting hospitals, banks, 911 systems–as the New York Times put it, “It is more apt to ask what was not affected.” The answer is Linux, Macs, and phones. The outage highlighted a different kind of digital divide. On one side, gmail, Facebook, and Twitter kept running, letting us post photos of blue screens located on the other side: the Windows machines responsible for actually doing things in the world like making appointments, opening accounts, and dispatching police. They also run airlines. Here’s a visualization of the chaos that CrowdStrike caused for airlines from the New York Times: Airline cancellations is a good metric, but I want to look directly at air traffic: How many planes were in the air? How many planes should have been in the air? At about noon UTC, 8 hours after the CrowdStrike update hit, someone posted a video to Twitter that they made with FlightRadar24 showing air traffic over the United States. It was described as a 12-hour timelapse of American Airlines, Delta, and United plane traffic that showed the nationwide ground stop of the three airlines due to CrowdStrike. Here’s the video: It’s not a good visualization of the impact because there’s no basis for comparison. It clearly shows fewer planes flying at night, but that happens every day. Was that night different from any other night? There’s no way to tell. In Bellingcat’s “OSHIT: Seven Deadly Sins of Bad Open Source Research”, sin #4 is “Lacking Context for Occurrences, Common or Otherwise”. In this post I’ll show the effects CrowdStrike had on air traffic, with enough context to make the significance clear. Impact on U.S. Aviation CrowdStrike hit on July 19. This chart shows the number of aircraft that took off in the United States, hour by hour, on that day. It also shows the same numbers for July 12, the previous Friday. The same day one week previously seems to be a good basis for comparison–both days are Fridays, and there aren’t any major holidays on either day. I also plotted the stats for July 18, the day before CrowdStrike, but it was very similar so I’ll continue to compare to the previous week. Note that the chart is for all of aviation in the United States, including fire fighting aircraft, police, military, and general aviation as well as commercial aviation. From about 0600 to 1300 there seems to have been a small decrease in the number of flights, and then a small increase in the rest of the day. Looking at the cumulative statistics starting from 0400, when the CrowdStrike update was pushed, flights were up 2.6% compared to the same period on the previous Friday. This chart shows the percentage change in flights, comparing each hour on July 19 to the matching hour of the previous Friday as the baseline: This chart brings CrowdStrike’s effects into greater relief. The hour with the largest percent decrease was from 0800 to 0900, which had only 261 flights compared to the previous Friday’s 378 flights, a 31% reduction. Airline Statistics Now let’s look at the statistics for the top 4 U.S. airlines: Delta, United, American, and Southwest. Delta Air Lines Change during CrowdStrike: -1087 flights (-46%) United Airlines Change during CrowdStrike: -596 flights (-36%) American Airlines Change during CrowdStrike: -376 flights (-16%) Southwest Airlines Change during CrowdStrike: +101 flights (+3%) Airlines Summary Delta was hardest hit, then United, and to a significantly smaller degree American. Southwest didn’t seem to be affected at all. Apparently Southwest Airlines’ ingenious strategy of never upgrading from Windows 3.1 allowed it to remain unscathed. This seems to be false, BTW. This ABC News article says that Southwest wasn’t affected because they don’t use CrowdStrike.] Delta Air Lines took an extended time to recover, canceling thousands of flights in the days following the CrowdStrike update. Why were other airlines able to get back to normal so much faster than Delta? A terrible article from ABC News said this: The reason for the prolonged recovery from the outage was because the CrowdStrike update disruption required a manual fix at each individual computer system, experts told ABC News. While each fix can be completed in no more than 10 minutes, the vast number of Delta’s digital terminals required significant manpower to address, expert said. I’m reminded of sin #4 again–How is this different from any other airline? ABC News has no idea. A random redditor gave an unsourced explanation that might be wrong but at least attempts to answer the question “Why Delta so bad?” (DR = disaster recovery): These “experts” are completely wrong. The core issue was Delta did NOT have a proper DR plan ready and did NOT have a proper IT business continuity plan ready. UA, AA, and F9 recovered so fast because they had plans on stand-by and engaged them immediately. After the SWA IT problem, UA and AA put in robust DR plans staged everywhere from the server farms, to cloud solutions, to end-user stations at airports. They had plans on how to recover systems. DL outsources a lot of their IT. UA and AA engaged those plans quickly. They did not hold back paying OT for staff. UA and AA have just as much reliance on Windows as Delta. AA was recovered by end of data Friday and resumed normal operations Saturday. UA was about 12 hours behind them having it resolved by Saturday morning resuming normal schedules Saturday afternoon. The ONUS is 100% on DL C+ level in their IT decisions. Data and Analysis I took raw ADS-B data from ADS-B Exchange and processed it through my custom code to detect aircraft takeoffs. I’m assuming that a takeoff is roughly equivalent to a flight, which isn’t actually true but is close enough for these purposes. It tends to undercount the number of aircraft flying, e.g. in the case where an aircraft took off from a field outside of ADS-B Exchange’s coverage, but it does so in a systematic way that still allows for valid comparisons between time periods. That is, the absolute numbers of flights may be too low, but the percent changes in numbers are accurate. I counted takeoffs instead of counting flying aircraft because I already had code to detect takeoffs and didn’t want to write new code–this was just a quick weekend project.
More in technology
Here are the Vim forks I know about: Neovim: “a Vim-based text editor engineered for extensibility and usability, to encourage new applications and contributions.” Vim Classic: “a stable, long-term support fork of the ubiquitous text editor, maintained without the assistance of generative AI tools.” EVi: “a hard-fork of Vim v9.1.0 (Jan 2024) before AI was used in the project.” MacVim: “a macOS version of the Vim text editor, providing a graphical user interface for Vim, while tightly integrating with macOS and providing features specific to the platform.” This list doesn’t include Vim-inspired editors like Kakoune, Vim-style plugins for other editors like IdeaVim, or Vim “distributions” like LazyVim. Contact me if you know of other forks!
Thought I’d make a quick blog post about some weird behavior I’m seeing on iOS 27 and Dynamic Island live activities, that I initially thought was restricted to the Duo, but turns out affects everything. Yay. Maybe this post can save some others some time, and hopefully Apple can fix it. Long story short, ever since iOS 16.1 introduced “Live Activities”, where we can display handy little widgets in the Dynamic Island, there’s been a bunch of different configurations for showing things up there: something on the left, the right, when expanded, when two apps are showing things and yours gets a bit smaller, and on the lock screen. One interesting one is if you only need to show a small indicator in the Dynamic Island (screen recording indicator, electric vehicle charging status, etc.) you can do so, which just shows on the left on older iPhones, and on the Duo a single one below the camera cutout. Apple describes how to do this in their developer documentation (emphasis added): If you can’t find a solution to display views for leading and trailing compact presentations with limited width, provide a compact leading presentation that best represents your Live Activity, such as the view you use for the minimal presentation, and an empty view for the trailing compact presentation. The system hides the empty trailing view automatically. Unfortunately, in iOS 27 (and 27.1, and 27.2), this only half works, due to a new addition in iOS 27: isDynamicIslandLimitedInWidth. It’s a variable that basically tells you if the Dynamic Island is even narrower than normal, presumably because they knew the Duo was coming with its vertical Dynamic Island, but also because as of iOS 27, Live Activities now show in the Dynamic Island when the device is in landscape too. So it’s handy to know if you have a bit less space than usual. In this new narrow view however, it seems like they didn’t test all the configurations thoroughly, as if you provide EmptyView for that trailing compact presentation, it’s not empty at all, and in fact enforces a quite large minimum size so the Dynamic Island looks hilariously tall. In other words: The system hides the empty trailing view automatically Left: what it should look like. Right: reality You can imagine this is especially rough on the Duo, where navigation buttons, toolbar buttons, tab bar buttons, etc. are all competing for precious space on the right. But that’s only half true. It displays perfectly fine (hiding it) when isDynamicIslandLimitedInWidth is false (AKA how it always used to be), but in the new cases where it can be true, like the Duo and iPhones in landscape, that’s when it breaks. So tl;dr: In iOS 27, Dynamic Island Live Activities are improperly sized if you use the listed EmptyView configuration and isDynamicIslandLimitedInWidth is true. This makes it especially bad on the Duo, womp womp. Complications Now, you might be saying “That’s weird, I haven’t seen that behavior! This is nothing but hogwash!”. But it is not hogwash, just a little complicated. There’s basically three categories of apps. Apple apps Maps, for instance, displays this single, small Dynamic Island live activity perfectly well and sized, and has been beautifully compiled with the iOS 27.2 SDK. By all accounts it should show this bug, too! Apple apps don’t have to use the same APIs as us normies, so funnily enough through really cool tools like REA we can disassemble the iOS Simulator and see that it’s in fact true, the Apple Maps live activity just completely bypasses WidgetKit and instead just renders plain UIKit through MapsBannerViewController which shows the cute little cyan widget through a ManeuverAccessoryView and then for the hidden trailing view simply returns a DistanceAccessoryView with CGSizeZero, which iOS actually respects, unlike our paltry SwiftUI EmptyView. Third-party apps NOT updated to iOS 27 If your app hasn’t been compiled with any of the iOS 27 SDKs, it only knows about iOS 26, and thus doesn’t know about (or have to worry about) the new, troublesome isDynamicIslandLimitedInWidth property, so it just… displays perfectly well. Here’s the Rivian app, which hasn’t been updated for iOS 27 yet, which shows the Live Activity displaying perfectly sized on the Duo (indicating vehicle charging): (Technical disassembly detail for the nerds: iOS shows the app’s minimal config, a smaller config normally used when multiple apps are in the Dynamic Island together, rather than the normal compact config, so it bypasses this issue entirely.) Developers who updated their apps for iOS 27 Surprise! That picture from before? On the left is just me compiling with the old 26 SDK that works! Unfortunately if you are a good developer who updated their app for iOS 27 and all the new features, you kind of get punished through this bug, because your live activity displays the improperly sized way described above, and you can’t just use raw UIKit like Apple. So you’re kinda stuck where I am: do I ship a weirdly sized Dynamic Island experience for everyone? Do I shove something extraneous in the compactTrailing spot just so it doesn’t look as empty? Do I not update the app at all until Apple fixes this bug? Widgets are techincally a separate extension from your main app, so you could compile them with a different SDK (26) than your main app (27) which fixes this bug, but I’m not sure Apple approves of that. Apple On the bright side hopefully it’s an easy enough fix, treating EmptyView like you used to on iOS 26 and per the documentation (the part that is weirdly right beside the mention of the new API that introduces this craziness in the first place). Radar/feedback assistant ID: FB25301955 Easy reproduction code if you’re curious: import ActivityKit import SwiftUI import WidgetKit struct CircleAttributes: ActivityAttributes { struct ContentState: Codable, Hashable {} } struct BlueCircle: View { var body: some View { Circle() .fill(.blue) .frame(width: 20, height: 20) } } @main struct CircleLiveActivity: Widget { var body: some WidgetConfiguration { ActivityConfiguration(for: CircleAttributes.self) { _ in BlueCircle() } dynamicIsland: { _ in DynamicIsland { DynamicIslandExpandedRegion(.center) { BlueCircle() } } compactLeading: { BlueCircle() } compactTrailing: { EmptyView() // pls respect me } minimal: { BlueCircle() } } } }
Vicky, driving me home from O'Hare, got sixteen dollars for the best conversation I had in months. The last line is hers.
I’ve had “AI usage statements” in the back of my mind ever since I read Ethan Marcott’s blog post sharing a round-up of his favorite ones. I didn’t consider writing one for this site, since it had the smell of status-signaling. Like a quasi-political declaration of allegiance to some sort of in-group (be it pro-AI, anti-AI, etc). I couldn’t be less interested in that.1 But then something happened that caught me completely off-guard. I was reading a post on my friend’s personal blog. This was a guy that I knew personally and looked up to. I’ve been reading everything he had been writing for years. About two-thirds of the way through a long (and fairly insightful) blog post, I had a sudden realization that the post had been written by AI. I’m not the best at detecting this kind of thing, but once I saw it, I couldn’t unsee it. It was unmistakable. And the thing is, I didn’t feel indifferent or even disappointed, as I might have predicted. I felt betrayed. For somebody who considers themselves open-minded—even optimistic—towards the development of AI, I didn’t know where this feeling was coming from. I have theories… maybe it has something to do with the social contract between the writer and the reader. Whatever it is, I didn’t want anyone who was reading my blog (hi mom 👋) to feel the same thing. So for the sake of transparency, here is my AI usage statement: The content of this website is written by me. I do not use AI to generate the text you are reading because writing from scratch helps me process my thoughts (which is one of the main reasons I blog in the first place). I don’t currently use AI-powered writing tools (grammar-check, spell-check, dictation tools, etc), but I may experiment with them as long as they don’t undermine the thinking process. The code for this website was originally hand-written, but I have since used AI to make several useful updates. I expect to continue using AI to write code for this website and other personal coding projects. When I create a public project exclusively with AI, I mention that fact in the project README.2 I am experimenting and my AI usage may change. When it does, I’ll update the policy here on my /about page, which is where it will live long-term. To explain why, see Paul Graham's Keep your Identity Small. ↩ For an example, see s-tier ↩ Thanks for reading this post via RSS! Let me know your thoughts by leaving a comment on the original post or sending me an email.
