Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from heavymeta.org
Just after midnight Eastern Time on July 19, 2024, the enterprise cybersecurity company CrowdStrike YOLOed a software update to millions of Windows machines. Or as they put it: On July 19, 2024 at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems. That sensor configuration update caused the largest IT outage in history. Overnight, about 8.5 million computers blue screened, affecting hospitals, banks, 911 systems–as the New York Times put it, “It is more apt to ask what was not affected.” The answer is Linux, Macs, and phones. The outage highlighted a different kind of digital divide. On one side, gmail, Facebook, and Twitter kept running, letting us post photos of blue screens located on the other side: the Windows machines responsible for actually doing things in the world like making appointments, opening accounts, and dispatching police. They also run airlines. Here’s a visualization of the chaos that CrowdStrike caused for airlines from the New York Times: Airline cancellations is a good metric, but I want to look directly at air traffic: How many planes were in the air? How many planes should have been in the air? At about noon UTC, 8 hours after the CrowdStrike update hit, someone posted a video to Twitter that they made with FlightRadar24 showing air traffic over the United States. It was described as a 12-hour timelapse of American Airlines, Delta, and United plane traffic that showed the nationwide ground stop of the three airlines due to CrowdStrike. Here’s the video: It’s not a good visualization of the impact because there’s no basis for comparison. It clearly shows fewer planes flying at night, but that happens every day. Was that night different from any other night? There’s no way to tell. In Bellingcat’s “OSHIT: Seven Deadly Sins of Bad Open Source Research”, sin #4 is “Lacking Context for Occurrences, Common or Otherwise”. In this post I’ll show the effects CrowdStrike had on air traffic, with enough context to make the significance clear. Impact on U.S. Aviation CrowdStrike hit on July 19. This chart shows the number of aircraft that took off in the United States, hour by hour, on that day. It also shows the same numbers for July 12, the previous Friday. The same day one week previously seems to be a good basis for comparison–both days are Fridays, and there aren’t any major holidays on either day. I also plotted the stats for July 18, the day before CrowdStrike, but it was very similar so I’ll continue to compare to the previous week. Note that the chart is for all of aviation in the United States, including fire fighting aircraft, police, military, and general aviation as well as commercial aviation. From about 0600 to 1300 there seems to have been a small decrease in the number of flights, and then a small increase in the rest of the day. Looking at the cumulative statistics starting from 0400, when the CrowdStrike update was pushed, flights were up 2.6% compared to the same period on the previous Friday. This chart shows the percentage change in flights, comparing each hour on July 19 to the matching hour of the previous Friday as the baseline: This chart brings CrowdStrike’s effects into greater relief. The hour with the largest percent decrease was from 0800 to 0900, which had only 261 flights compared to the previous Friday’s 378 flights, a 31% reduction. Airline Statistics Now let’s look at the statistics for the top 4 U.S. airlines: Delta, United, American, and Southwest. Delta Air Lines Change during CrowdStrike: -1087 flights (-46%) United Airlines Change during CrowdStrike: -596 flights (-36%) American Airlines Change during CrowdStrike: -376 flights (-16%) Southwest Airlines Change during CrowdStrike: +101 flights (+3%) Airlines Summary Delta was hardest hit, then United, and to a significantly smaller degree American. Southwest didn’t seem to be affected at all. Apparently Southwest Airlines’ ingenious strategy of never upgrading from Windows 3.1 allowed it to remain unscathed. This seems to be false, BTW. This ABC News article says that Southwest wasn’t affected because they don’t use CrowdStrike.] Delta Air Lines took an extended time to recover, canceling thousands of flights in the days following the CrowdStrike update. Why were other airlines able to get back to normal so much faster than Delta? A terrible article from ABC News said this: The reason for the prolonged recovery from the outage was because the CrowdStrike update disruption required a manual fix at each individual computer system, experts told ABC News. While each fix can be completed in no more than 10 minutes, the vast number of Delta’s digital terminals required significant manpower to address, expert said. I’m reminded of sin #4 again–How is this different from any other airline? ABC News has no idea. A random redditor gave an unsourced explanation that might be wrong but at least attempts to answer the question “Why Delta so bad?” (DR = disaster recovery): These “experts” are completely wrong. The core issue was Delta did NOT have a proper DR plan ready and did NOT have a proper IT business continuity plan ready. UA, AA, and F9 recovered so fast because they had plans on stand-by and engaged them immediately. After the SWA IT problem, UA and AA put in robust DR plans staged everywhere from the server farms, to cloud solutions, to end-user stations at airports. They had plans on how to recover systems. DL outsources a lot of their IT. UA and AA engaged those plans quickly. They did not hold back paying OT for staff. UA and AA have just as much reliance on Windows as Delta. AA was recovered by end of data Friday and resumed normal operations Saturday. UA was about 12 hours behind them having it resolved by Saturday morning resuming normal schedules Saturday afternoon. The ONUS is 100% on DL C+ level in their IT decisions. Data and Analysis I took raw ADS-B data from ADS-B Exchange and processed it through my custom code to detect aircraft takeoffs. I’m assuming that a takeoff is roughly equivalent to a flight, which isn’t actually true but is close enough for these purposes. It tends to undercount the number of aircraft flying, e.g. in the case where an aircraft took off from a field outside of ADS-B Exchange’s coverage, but it does so in a systematic way that still allows for valid comparisons between time periods. That is, the absolute numbers of flights may be too low, but the percent changes in numbers are accurate. I counted takeoffs instead of counting flying aircraft because I already had code to detect takeoffs and didn’t want to write new code–this was just a quick weekend project.
More in technology
My favorite memory of my M1 Pro MacBook Pro was the whole sensation of “holy crap, you never hear the fans in this thing”, which was very novel in 2021. Four years later, this MacBook Pro is still a delight. It’s the longest I’ve ever owned a laptop, and while I’d love to pick up the new M4 goodness, this dang thing still seems to just shrug at basically anything I throw at it. Video editing, code compiling, CAD models, the works. (My desire to update is helped though by the fact I got the 2TB SSD, 32GB RAM option, and upgrading to those on new MacBooks is still eye wateringly expensive.) But my MacBook is starting to show its age in one area: it’s not quiet anymore. If you’re doing anything too intensive like compiling code for awhile, or converting something in Handbrake, the age of the fans being quiet is long past. The fans are properly loud. (And despite having two cats, it’s not them! I clean out the fans pretty regularly.) Enter the thermal paste Everyone online seems to point toward one thing: the thermal paste on computers tends to dry up over the years. What the heck is thermal paste? Well, components on your computer that generate a lot of heat are normally made to touch something like a copper heatsink that is really good at pulling that heat away from it. The issue is, when you press these two metal surfaces against each other, even the best machining isn’t perfect and you there’s microscopic gaps between them meaning there’s just air at those parts, and air is a terrible conductor of heat. The solution is to put a little bit of thermal paste (basically a special grey toothpaste gunk that is really good at transferring heat) between them, and it fills in any of those microscopic gaps. The problem with this solution is after hundreds and hundreds of days of intense heat, the paste can dry up into something closer to almost a powder, and it’s not nearly as good at filling in those gaps. Replacement time The logic board! MacBook thermal paste isn’t anything crazy (for the most part, see below), custom PC builders use thermal paste all the time so incredibly performant options are available online. I grabbed a tube of Noctua NT-H2 for about $10 and set to taking apart my MacBook to swap out the aging thermal paste. And thankfully, iFixit has a tremendous, in depth guide on the disassembly required, so I got to it. Indeed, that grey thermal paste looked quite old, but also above and below it (on the RAM chips) I noticed something that didn’t quite seem like thermal paste, it was far more… grainy almost? Spottiness is due to half of it being on the heatsink It turns out, ending with my generation of MacBooks (lucky me!) Apple used a very special kind of thermal compound often called “Carbon Black”, which is basically designed to be able to bridge an even thicker gap than traditional thermal paste. I thought about replacing it, but it seems really hard to come across that special thermal compound (and do not do it with normal thermal paste) and my RAM temperatures always seemed fine (65°C is fine… right?) so I just made sure to not touch that. For the regular grey thermal paste, I used some cotton swabs and isopropyl alcohol to remove the dried up existing thermal paste, then painted on a bit of the new stuff. Disaster To get to the underside of the CPU, you basically need to disassemble the entire MacBook. It’s honestly not that hard, but iFixit warned that the fan cables (which also need to be unclipped) are incredibly delicate. And they’re not wrong, seriously they have the structural integrity of the half-ply toilet paper available at gas stations. So, wouldn’t you know it, I moved the left fan’s cable a bit too hard and it completely tore in half. Gah. I found a replacement fan online (yeah you can’t just buy the cable, need a whole new fan) and in the meantime I just kept an eye on my CPU thermals. As long as I wasn’t doing anything too intensive it honestly always stayed around 65° which was warm, but not terrifying (MacBook Airs completely lack a fan, after all). Take two A few days later, the fans arrived, and I basically had to redo the entire disassembly process to get to the fans. At least I was a lot faster this time. The fan was incredibly easy to swap out (hats off there, Apple!) and I screwed everything back together and began reconnecting all the little connectors. Until I saw it: the tiny (made of the same half ply material as the fan cable) Touch ID sensor cable was inexpicably torn in half, the top half just hanging out. I didn’t even half to touch this thing really, and I hadn’t even got to the stage of reconnecting it (I was about to!), it comes from underneath the logic board and I guess just the movement of sliding the logic board back in sheared it in half. me Bah. I looked up if I could just grab another replacement cable here, and sure enough you can… but the Touch ID chip is cryptographically paired to your MacBook so you’d have to take it into an Apple Store. Estimates seemed to be in the hundreds of dollars, so if anyone has any experience there let me know, but for now I’m just going to live happily without a Touch ID sensor… or the button because the button also does not work. RIP little buddy (And yeah I’m 99.9% sure I can’t solder this back together, there’s a bunch of tiny lanes that make up the cable that you would need experience with proper micro-soldering to do.) Honestly, the disassembly process for my MacBook was surprisingly friendly and not very difficult, I just really wish they beefed up some of the cables even slightly so they weren’t so delicate. The results I was going to cackle if I went through all that just to have identical temperatures as before, but I’m very happy to say they actually improved a fair bit. I ran a Cinebench test before disassembling the MacBook the very first time to establish a baseline: Max CPU temperature: 102°C Max fan speed: 6,300 RPM Cinbench score: 12,252 After the new thermal paste (and the left fan being new): Max CPU temperature: 96°C Max fan speed: 4,700 RPM Cinbench score: 12,316 Now just looking at those scores you might be like… so? But let me tell you, dropping 1,600 RPM on the fan is a noticeable change, it goes from “Oh my god this is annoyingly loud” to “Oh look the fans kicked in”, and despite slower fan speeds there was still a decent drop in CPU temperature! And a 0.5% higher Cinebench score! But where I also really notice it is in idling: just writing this blog post my CPU was right at 46°C the whole time, where previously my computer idled right aroud 60°C. The whole computer just feels a bit healthier. So… should you do it? Honestly, unless you’re very used to working on small, delicate electronics, probably not. But if you do have that experience and are very careful, or have a local repair shop that can do it for a reasonable fee (and your MacBook is a few years old so as to warrant it) it’s honestly a really nice tweak that I feel will hopefully at least get me to the M5 generation. I do miss Touch ID, though.
Meet the Creators of Choplifter, Wizardry, Castle Wolfenstein, Zaxxon, Canyon Climber, and the Arcade Machine
We’re excited to invite you to a brand-new workshop created in collaboration with Amazon Web Services (AWS). Whether you’re modernizing factory operations or tinkering with your first industrial project, this hands-on workshop is your gateway to building cloud-connected PLCs that ship data – fast. At Arduino, we believe in making advanced technology more accessible. That’s […] The post New AWS x Arduino Opta Workshop: Connect your PLC to the Cloud in just a few steps appeared first on Arduino Blog.
The term “mmWave” refers to radio waves with wavelengths on the millimeter scale. When it comes to wireless communications technology, like 5G, mmWave allows for very fast data transfer — though that comes at the expense of range. But mmWave technology also has some very useful sensing and scanning applications, which you may have experienced […] The post Concept Bytes’ coffee table tracks people and walks itself across a room when called appeared first on Arduino Blog.
