More from axio.ms
A microcontroller Macintosh This all started from a conversation about the RP2040 MCU, and building a simple desktop/GUI for it. I’d made a comment along the lines of “or, just run some old OS”, and it got me thinking about the original Macintosh. The original Macintosh was released 40.5 years before this post, and is a pretty cool machine especially considering that the hardware is very simple. Insanely Great and folklore.org are fun reads, and give a glimpse into the Macintosh’s development. Memory was a squeeze; the original 128KB version was underpowered and only sold for a few months before being replaced by the Macintosh 512K, arguably a more appropriate amount of memory. But, the 128 still runs some real applications and, though it pre-dates MultiFinder/actual multitasking, I found it pretty charming. As a tourist. In 1984 the Mac cost roughly 1/3 as much as a VW Golf and, as someone who’s into old computers and old cars, it’s hard to decide which is more frustrating to use. So back to this £3.80 RPi Pico microcontroller board: The RP2040’s 264KB of RAM gives a lot to play with after carving out the Mac’s 128KB – how cool would it be to do a quick hack, and play with a Mac on it? Time passes. A lot of time. But I totally delivered on the janky hack front: You won’t believe that this quality item didn’t take that long to build. So the software was obviously the involved part, and turned into work on 3 distinct projects. This post is going to be a “development journey” story, as a kind of code/design/venting narrative. If you’re just here for the pictures, scroll along! What is pico-mac? A Raspberry Pi RP2040 microcontroller (on a Pico board), driving monochrome VGA video and taking USB keyboard/mouse input, emulating a Macintosh 128K computer and disc storage. The RP2040 has easily enough RAM to house the Mac’s memory, plus that of the emulator; it’s fast enough (with some tricks) to meet the performance of the real machine, has USB host capability, and the PIO department makes driving VGA video fairly uneventful (with some tricks). The basic Pico board’s 2MB of flash is plenty for a disc image with OS and software. Here’s the Pico MicroMac in action, ready for the paperless office of the future: The Pico MicroMac RISC CISC workstation of the future I hadn’t really used a Mac 128K much before; a few clicks on a museum machine once. But I knew they ran MacDraw, and MacWrite, and MacPaint. All three of these applications are pretty cool for a 128K machine; a largely WYSIWYG word processor with multiple fonts, and a vector drawing package. A great way of playing with early Macintosh system software, and applications of these wonderful machines is via https://infinitemac.org, which has shrinkwrapped running the Mini vMac emulator by emscriptening it to run in the browser. Highly recommended, lots to play with. As a spoiler, MicroMac does run MacDraw, and it was great to play with it on “real fake hardware”: (Do you find “Pico Micro Mac” doesn’t really scan? I didn’t think this taxonomy through, did I?) GitHub links are at the bottom of this page: the pico-mac repo has construction directions if you want to build your own! The journey Back up a bit. I wasn’t committed to building a Pico thing, but was vaguely interested in whether it was feasible, so started tinkering with building a Mac 128K emulator on my normal computer first. The three rules I had a few simple rules for this project: It had to be fun. It’s OK to hack stuff to get it working, it’s not as though I’m being paid for this. I like writing emulation stuff, but I really don’t want to learn 68K assembler, or much about the 68K. There’s a lot of love for 68K out there and that’s cool, but meh I don’t adore it as a CPU. So, right from the outset I wanted to use someone else’s 68K interpreter – I knew there were loads around. Similarly, there are a load of OSes whose innards I’d like to learn more about, but the shittiest early Mac System software isn’t high on the list. Get in there, emulate the hardware, boot the OS as a black box, done. I ended up breaking 2 of and sometimes all 3 of these rules during this project. The Mac 128K The machines are generally pretty simple, and of their time. I started with schematics and Inside Macintosh, PDFs of which covered various details of the original Mac hardware, memory map, mouse/keyboard, etc. https://tinkerdifferent.com/resources/macintosh-128k-512k-schematics.79/ https://vintageapple.org/inside_o/ Inside Macintosh Volumes I-III are particularly useful for hardware information; also Guide to Macintosh Family Hardware 2nd Edition. The Macintosh has: A Motorola 68000 CPU running at 7.whatever MHz roughly 8MHz Flat memory, decoded into regions for memory-mapped IO going to the 6522 VIA, the 8530 SCC, and the IWM floppy controller. (Some of the address decoding is a little funky, though.) Keyboard and mouse hang off the VIA/SCC chips. No external interrupt controller: the 68K has 3 IRQ lines, and there are 3 IRQ sources (VIA, SCC, programmer switch/NMI). “No slots” or expansion cards. No DMA controller: a simple autonomous PAL state machine scans video (and audio samples) out of DRAM. Video is fixed at 512x342 1BPP. The only storage is an internal FDD (plus an external drive), driven by the IWM chip. The first three Mac models are extremely similar: The Mac 128K and Mac 512K are the same machine, except for RAM. The Mac Plus added SCSI to a convenient space in the memory map and an 800K floppy drive, which is double-sided whereas the original was a single 400K side. The Mac Plus ROM also supports the 128K/512K, and was an upgrade to create the Macintosh 512Ke. ‘e’ for Extra ROM Goodness. The Mac Plus ROM supports the HD20 external hard disc, and HFS, and Steve Chamberlin has annotated a disassembly of it. This was the ROM to use: I was making a Macintosh 128Ke. Mac emulator: umac After about 8 minutes of research, I chose the Musashi 68K interpreter. It’s C, simple to interface to, and had a simple out-of-box example of a 68K system with RAM, ROM, and some IO. Musashi is structured to be embedded in bigger projects: wire in memory read/write callbacks, a function to raise an IRQ, call execute in a loop, done. I started building an emulator around it, which ultimately became the umac project. The first half (of, say, five halves) went pretty well: A simple commandline app loading the ROM image, allocating RAM, providing debug messages/assertions/logging, and configuring Musashi. Add address decoding: CPU reads/writes are steered to RAM, or ROM. The “overlay” register lets the ROM boot at 0x00000000 and then trampoline up to a high ROM mirror after setting up CPU exception vectors – this affects the address decoding. This is done by poking a VIA register, so decoded just that bit of that register for now. At this point, the ROM starts running and accessing more non-existent VIA and SCC registers. Added more decoding and a skeleton for emulating these devices elsewhere – the MMIO read/writes are just stubbed out. There are some magic addresses that the ROM accesses that “miss” documented devices: there’s a manufacturing test option that probes for a plugin (just thunk it), and then we witness the RAM size probing. The Mac Plus ROM is looking for up to 4MB of RAM. In the large region devoted to RAM, the smaller amount of actual RAM is mirrored over and over, so the probe writes a magic value at high addresses and spots where it starts to wrap around. RAM is then initialised and filled with a known pattern. This was an exciting point to get to because I could dump the RAM, convert the region used for the video framebuffer into an image, and see the “diagonal stripe” pattern used for RAM testing! “She’s alive!” Not all of the device code enjoyed reading all zeroes, so there was a certain amount of referring to the disassembly and returning, uh, 0xffffffff sometimes to push it further. The goal was to get it as far as accessing the IWM chip, i.e. trying to load the OS. After seeing some IWM accesses there and returning random rubbish values, the first wonderful moment was getting the “Unknown Disc” icon with the question mark – real graphics! The ROM was REALLY DOING SOMETHING! I think I hadn’t implemented any IRQs at this point, and found the ROM in an infinite loop: it was counting a few Vsyncs to delay the flashing question mark. Diversion into a better VIA, with callbacks for GPIO register read/write, and IRQ handling. This also needed to wire into Musashi’s IRQ functions. This was motivating to get to – remembering rule #1 – and “graphics”, even though via a manual memory dump/ImageMagick conversion, was great. I knew the IWM was an “interesting” chip, but didn’t know details. I planned to figure it out when I got there (rule #1). IWM, 68K, and disc drivers My god, I’m glad I put IWM off until this point. If I’d read the “datasheet” (vague register documentation) first, I’d’ve just gone to the pub instead of writing this shitty emulator. IWM is very clever, but very very low-level. The disc controllers in other contemporary machines, e.g. WD1770, abstract the disc physics. At one level, you can poke regs to step to track 17 and then ask the controller to grab sector 3. Not so with IWM: first, the discs are Constant Linear Velocity, meaning the angular rotation needs to change appropriate to whichever track you’re on, and second the IWM just gives the CPU a firehose of crap from the disc head (with minimal decoding). I spent a while reading through the disassembly of the ROM’s IWM driver (breaking rule #2 and rule #1): there’s some kind of servo control loop where the driver twiddles PWM values sent to a DAC to control the disc motor, measured against a VIA timer reference to do some sort of dynamic rate-matching to get the correct bitrate from the disc sectors. I think once it finds the track start it then streams the track into memory, and the driver decodes the symbols (more clever encoding) and selects the sector of interest. I was sad. Surely Basilisk II and Mini vMac etc. had solved this in some clever way – they emulated floppy discs. I learned they do not, and do the smart engineering thing instead: avoid the problem. The other emulators do quite a lot of ROM patching: the ROM isn’t run unmodified. You can argue that this then isn’t a perfect hardware emulation if you’re patching out inconvenient parts of the ROM, but so what. I suspect they were also abiding by a rule #1 too. I was going to do the same: I figured out a bit of how the Mac driver interface works (gah, rule #3!) and understood how the other emulators patched this. They use a custom paravirtualised 68K driver which is copied over the ROM’s IWM driver, servicing .Sony requests from the block layer and routing them to more convenient host-side code to manage the requests. Basilisk II uses some custom 68K opcodes and a simple driver, and Mini vMac a complex driver with trappy accesses to a custom region of memory. I reused the Basilisk II driver but converted to access a trappy region (easier to route: just emulate another device). The driver callbacks land in the host/C side and some cut-down Basilisk II code interprets the requests and copies data to/from the OS-provided buffers. Right now, all I needed was to read blocks from one disc: I didn’t need different formats (or even write support), or multiple drives, or ejecting/changing images. Getting the first block loaded from disc took waaaayyy longer than the first part. And, I’d had to learn a bit of 68K (gah), but just in the nick of time I got a Happy Mac icon as the System software started to load. This was still a simple Linux commandline application, with zero UI. No keyboard or mouse, no video. Time to wrap it in an SDL2 frontend (the unix_main test build in the umac project), and I could watch the screen redraw live. I hadn’t coded the 1Hz timer interrupt into the VIA, and after adding that it booted to a desktop! The first boot As an aside, I try to create a dual-target build for all my embedded projects, with a native host build for rapid prototyping/debugging; libSDL instead of an LCD. It means I don’t need to code at the MCU, so I can code in the garden. :) Next was mouse support. Inside Macintosh and the schematics show how it’s wired, to the VIA (good) and the SCC (a beast). The SCC is my second least-favourite chip in this machine; it’s complex and the datasheet/manual seems to be intentionally written to hide information, piss off readers, get one back at the world. (I didn’t go near the serial side, its main purpose, just external IRQ management. But, it’ll do all kinds of exciting 1980s line coding schemes, offloading bitty work from the CPU. It was key for supporting things like AppleTalk.) Life was almost complete at this point; with a working mouse I could build a new disc image (using Mini vMac, an exercise in itself) with Missile Command. This game is pretty fun for under 10KB on disc. So: Video works Boots from disc Mouse works, Missile Command I had no keyboard, but it’s largely working now. Time to start on sub-project numero due: Hardware and RP2040 Completely unrelated to umac, I built up a circuit and firmare with two goals: Display 512x342x1 video to VGA with minimal components, Get the TinyUSB HID example working and integrated. This would just display a test image copied to a framebuffer, and printf() keyboard/mouse events, as a PoC. The video portion was fun: I’d done some I2S audio PIO work before, but here I wanted to scan out video and arbitrarily control Vsync/Hsync. Well, to test I needed a circuit. VGA wants 0.7V max on the video R,G,B signals and (mumble, some volts) on the syncs. The R,G,B signals are 75Ω to ground: with some maths, a 3.3V GPIO driving all three through a 100Ω resistor is roughly right. The day I started soldering it together I needed a VGA connector. I had a DB15 but wanted it for another project, and felt bad about cutting up a VGA cable. But when I took a walk at lunchtime, no shitting you, I passed some street cables. I had a VGA cable – the rust helps with the janky aesthetic. Free VGA cable The VGA PIO side was pretty fun. It ended up as PIO reading config info dynamically to control Hsync width, display position, and so on, and then some tricks with DMA to scan out the config info interleaved with framebuffer data. By shifting the bits in the right direction and by using the byteswap option on the RP2040 DMA, the big-endian Mac framebuffer can be output directly without CPU-side copies or format conversion. Cool. This can be fairly easily re-used in other projects: see video.c. But. I ended up (re)writing the video side three times in total: First version had two DMA channels writing to the PIO TX FIFO. The first would transfer the config info, then trigger the second to transfer video data, then raise an IRQ. The IRQ handler would then have a short time (the FIFO depth!) to choose a new framebuffer address to read from, and reprogram DMA. It worked OK, but was highly sensitive to other activity in the system. First and most obvious fix is that any latency-sensitive IRQ handler must have the __not_in_flash_func() attribute so as to run out of RAM. But even with that, the design didn’t give much time to reconfigure the DMA: random glitches and blanks occurred when moving the mouse rapidly. Second version did double-buffering with the goal of making the IRQ handler’s job trivial: poke in a pre-prepared DMA config quickly, then after the critical rush calculate the buffer to use for next time. Lots better, but still some glitches under some high load. Even weirder, it’d sometimes just blank out completely, requiring a reset. This was puzzling for a while; I ended up printing out the PIO FIFO’s FDEBUG register to try to catch the bug in the act. I saw that the TXOVER overflow flag was set, and this should be impossible: the FIFOs pull data from DMA on demand with DMA requests and a credited flow-contr…OH WAIT. If credits get messed up or duplicated, too many transfers can happen, leading to an overflow at the receiver side. Well, I’d missed a subtle rule in the RP2040 DMA docs: Another caveat is that multiple channels should not be connected to the same DREQ. So the third version…… doesn’t break this rule, and is more complicated as a result: One DMA channel transfers to the PIO TX FIFO Another channel programs the first channel to send from the config data buffer A third channel programs the first to send the video data The programming of the first triggers the corresponding “next reprogram me” channel The nice thing – aside from no lock-ups or video corruption – is that this now triggers a Hsync IRQ during the video line scan-out, greatly relaxing the deadline of reconfiguring the DMA. I’d like to further improve this (with yet another DMA channel) to transfer without an IRQ per line, as the current IRQ overhead of about 1% of CPU time can be avoided. (It would’ve been simpler to just hardwire the VGA display timing in the PIO code, but I like (for future projects) being able to dynamically-reconfigure the video mode.) So now we have a platform and firmware framework to embed umac into, HID in and video out. The hardware’s done, fuggitthat’lldo, let’s throw it over to the software team: How it all works Back to emulating things A glance at the native umac binary showed a few things to fix before it could run on the Pico: Musashi constructed a huge opcode decode jumptable at runtime, in RAM. It’s never built differently, and never changes at runtime. I added a Musashi build-time generator so that this table could be const (and therefore live in flash). The disassembler was large, and not going to be used on the Pico, so another option to build without. Musashi tries to accurately count execution cycles for each instruction, with more large lookup tables. Maybe useful for console games, but the Mac doesn’t have the same degree of timing sensitivity. REMOVED. (This work is in my small-build branch.) pico-mac takes shape, with the ROM and disc image in flash, and enjoyably it now builds and runs on the Pico! With some careful attention to not shoving stuff in RAM, the RAM use is looking pretty good. The emulator plus HID code is using about 35-40KB on top of the Mac’s 128KB RAM area – there’s 95+KB of RAM still free. This was a good time to finish off adding the keyboard support to umac. The Mac keyboard is interfaced serially through the VIA ‘shift register’, a basic synchronous serial interface. This was logically simple, but frustrating because early attempts at replying to the ROM’s “init” command just were persistently ignored. The ROM disassembly was super-useful again: reading the keyboard init code, it looked like a race condition in interrupt acknowledgement if the response byte appears too soon after the request is sent. Shoved in a delay to hold off a reply until a later poll, and then it was just a matter of mapping keycodes (boooooorrrriiiiing). With a keyboard, the end-of-level MacWrite boss is reached: One problem though: it totally sucked. It was suuuuper slow. I added a 1Hz dump of instruction count, and it was doing about 300 KIPS. The 68000 isn’t an amazing CPU in terms of IPC. Okay, there are some instructions that execute in 4 cycles. But you want to use those extravagant addressing modes don’t you, and touching memory is spending those cycles all over the place. Not an expert, but targeting about 1 MIPS for an about 8MHz 68000 seems right. Only 3x improvement needed. Performance I didn’t say I wasn’t gonna cheat: let’s run that Pico at 250MHz instead of 125MHz. Okay better, but not 2x better. From memory, only about 30% better. Damn, no free lunch today. Musashi has a lot of configurable options. My first goal was to get its main loop (as seen from disassembly/post-compile end!) small: the Mac doesn’t report Bus Errors, so the registers don’t need copies for unwinding. The opcodes are always fetched from a 16b boundary, so don’t need alignment checking, and can use halfword loads (instead of two byte loads munged into a halfword!). For the Cortex-M0+/armv6m ISA, reordering some of the CPU context structure fields enabled immediate-offset access and better code. The CPU type, mysteriously, was dynamically-changeable and led to a bunch of runtime indirection. Looking better, maybe 2x improvement, but not enough. Missile Command was still janky and the mouse wasn’t smooth! Next, some naughty/dangerous optimisations: remove address alignment checking, because unaligned accesses don’t happen in this constrained environment. (Then, this work is in my umac-hacks branch.) But the real perf came from a different trick. First, a diversion! RP2040 memory access The RP2040 has fast RAM, which is multi-banked so as to allow generally single-cycle access to multiple users (2 CPUs, DMA, etc.). Out of the box, most code runs via XIP from external QSPI flash. The QSPI usually runs at the core clock (125MHz default), but has a latency of ~20 cycles for a random word read. The RP2040 uses a relatively simple 16KB cache in front of the flash to protect you from horrible access latency, but the more code you have the more likely you are to call a function and have to crank up QSPI. When overclocking to 250MHz, the QSPI can’t go that fast so stays at 125MHz (I think). Bear in mind, then, that your 20ish QSPI cycles on a miss become 40ish CPU cycles. The particular rock-and-a-hard-place here is that Musashi build-time generates a ton of code, a function for each of its 1968 opcodes, plus that 256KB opcode jumptable. Even if we make the inner execution loop completely free, the opcode dispatch might miss in the flash cache, and the opcode function itself too. (If we want to get 1 MIPS out of about 200 MIPS, a few of these delays are going to really add up.) The __not_in_flash_func() attribute can be used to copy a given function into RAM, guaranteeing fast execution. At the very minimum, the main loop and memory accessors are decorated: every instruction is going to access an opcode and most likely read or write RAM. This improves performance a few percent. Then, I tried decorating whole classes of opcodes: move is frequent, as are branches, so put ‘em in RAM. This helped a lot, but the remaining free RAM was used up very quickly, and I wasn’t at my goal of much above 1 MIPS. Remember that RISC architecture is gonna change everything? We want to put some of those 1968 68K opcodes into RAM to make them fast. What are the top 10 most often-used instructions? Top 100? By adding a 64K table of counters to umac, booting the Mac and running key applications (okay, playing Missile Command for a bit), we get a profile of dynamic instruction counts. It turns out that the 100 hottest opcodes (5% of the total) account for 89% of the execution. And the top 200 account for a whopping 98% of execution. Armed with this profile, the umac build post-processes the Musashi auto-generated code and decorates the top 200 functions with __not_in_flash_func(). This adds only 17KB of extra RAM usage (leaving 95KB spare), and hits about 1.4 MIPS! Party on! At last, the world can enjoy Missile Command’s dark subject matter in performant comfort: Missile Command on pico-mac What about MacPaint? Everyone loves MacPaint. Maybe you love MacPaint, and have noticed I’ve deftly avoided mentioning it. Okay, FINE: It doesn’t run on a Mac 128Ke, because the Mac Plus ROM uses more RAM than the original. :sad-face: I’d seen this thread on 68kMLA about a “Mac 256K”: https://68kmla.org/bb/index.php?threads/the-mythical-mac-256k.46149/ Chances are that the Mac 128K was really a Mac 256K in the lab (or maybe even intended to have 256K and cost-cut before release), as the OS functions fine with 256KB. I wondered, does the Mac ROM/OS need a power-of-two amount of RAM? If not, I have that 95K going spare. Could I make a “Mac 200K”, and then run precious MacPaint? Well, I tried a local hack that patches the ROM to update its global memTop variable based on a given memory size, and yes, System 3.2 is happy with non-power-of-2 sizes. I booted with 256K, 208K, and 192K. However, there were some additional problems to solve: the ROM memtest craps itself without a power-of-2 size (totally fair), and NOPping that out leads to other issues. These can be fixed, though also some parts of boot access off the end of RAM. A power-of-2 size means a cheap address mask wraps RAM accesses to the valid buffer, and that can’t be done with 192K. Unfortunately, when I then tested MacPaint it still wouldn’t run because it wanted to write a scratch file to the read-only boot volume. This is totally breaking rule #1 by this point, so we are staying with 128KB for now. However, a 256K MicroMac is extremely possible. We just need an MCU with, say, 300KB of RAM… Then we’d be cooking on gas. Goodbye, friend Well, dear reader, this has been a blast. I hope there’s been something fun here for ya. Ring off now, caller! The MicroMac! HDMI monitor, using a VGA-to-HDMI box umac screenshot System 3.2, Finder 5.3 Performance tuning Random disc image working OK Resources https://github.com/evansm7/umac https://github.com/evansm7/pico-mac https://www.macintoshrepository.org/7038-all-macintosh-roms-68k-ppc- https://winworldpc.com/product/mac-os-0-6/system-3x https://68kmla.org/bb/index.php?threads/macintosh-128k-mac-plus-roms.4006/ https://docs.google.com/spreadsheets/d/1wB2HnysPp63fezUzfgpk0JX_b7bXvmAg6-Dk7QDyKPY/edit#gid=840977089
In 1974, Gerald Popek and Robert Goldberg published a paper, “Formal Requirements for Virtualizable Third Generation Architectures”, giving a set of characteristics for correct full-machine virtualisation. Today, these characteristics remain very useful. Computer architects will informally cite this paper when debating Instruction Set Architecture (ISA) developments, with arguments like “but that’s not Popek & Goldberg-compliant!” In this post I’m looking at one aspect of computer architecture evolution since 1974, and observing how RISC-style atomic operations provide some potential virtualisation gotchas for both programmers and architects. Principles of virtualisation First, some virtualisation context, because it’s fun! A key P&G requirement is that of equivalence: it’s reasonable to expect software running under virtualisation to have the same behaviour as running it bare-metal! This property is otherwise known as correctness. :-) P&G classify instructions as being sensitive if they behave differently when running at a lower privilege level (i.e. the program can detect that it is being run in a different manner). An ISA is said to be classically virtualisable if: Sensitive instructions are privileged, and Privileged instructions executed at a lower privilege level can be trapped to a higher level of privilege. For a classically-virtualisable system, perfect equivalence can then be achieved by running software at a lower than usual level of privilege, trapping all privileged/sensitive instructions, and emulating their behaviour in a VMM. That is, if the design of the ISA ensures that all “sensitive” instructions can be trapped, it’s possible to ensure the logical execution of the software cannot be different to running bare-metal. This virtualisation technique is called “privilege compression”. Note: This applies recursively, running OS-level software with user privilege, or hypervisor-level software at OS/user privilege. Popek & Goldberg formalise this too, giving properties required for correct nested virtualisation. System/360 and PowerPC are both classically virtualisable, almost as though IBM thought about this. ;-) Equivalent virtualisation can be achieved by: Running an OS in user mode (privilege compression, for CPU virtualisation), Catching traps (to supervisor mode/HV) when the guest OS performs a privileged operation, In the hypervisor, operating on a software-maintained “shadow” of what would have been the guest OS’s privileged CPU state were it running bare-metal. Constructing shadow address translations (for memory virtualisation). Linux’s KVM support on PowerPC includes a “PR” feature, which does just this: for CPUs without hardware virtualisation, guests are run in user mode (or “PRoblem state” in IBM lingo). Note: It is key that the hypervisor can observe and control all of the guest’s state. Today, most systems address the performance impact of all of this trap-and-emulate by providing hardware CPU and memory virtualisation (e.g. user, OS and hypervisor execution privilege levels, with nested page tables). But, classically virtualisable ISA design remains important for clear reasoning about isolation between privilege levels and composability of behaviours. Computers in 1974 were ~all CISC All computers in 1974 were available in corduroy with a selection of Liberty-print input devices. All consoles had ashtrays (not even joking tbh). Architecture-wise, IBM was working on early RISC concepts leading to the 701, but most of the industry was on a full-steam trajectory to peak CISC (VAX) in the late 1970s. It’s fair to say that “CISC” wasn’t even a thing yet; instruction sets were just complex. P&G’s paper considered three contemporary computers: IBM System/360 Honeywell 6000 DEC PDP-10 CISC atomic operations and synchronisation primitives These machines had composite/”read-modify-write” atomic operations, similar to those in today’s x86 architectures. System/360 had compare-and-swap, locked operations (read-operate-write), test-and-set, and PDP-10 had EXCHange/swap. These kinds of instructions are not sensitive so, unless the addressed memory is privileged, atomic operations can be performed inside virtual machines without the hypervisor needing to know. Atomic operations in RISC machines Many RISC machines support multi-instruction synchronisation sequences built up around two instruction primitives: Load-and-set-reservation Store-conditional MIPS called these load-linked (LL) and store-conditional (SC), and I’ll use these terms. ARMv8 has LDXR/STXR. PowerPC has LWARX/STWCX. RISC-V has LR/SC. Many machines (such as ARMv8-LSE) also add composite operations such as CAS or atomic addition but still retain the base LL/SC mechanism, and sizes/acquire/release variants are often provided. The concept is that the LL simultaneously loads a value and sets a “reservation” covering the address in question, and a subsequent SC succeeds only if the reservation is still present. A conflicting write to the location (e.g. a store on another CPU) clears the reservation and the SC returns a failure value without modifying memory; LL/SC are performed in a loop to retry until the update succeeds. An LL/SC sequence can typically be arbitrarily complex – a lock routine might test a location is cleared and store a non-zero value if so, whereas an update might increment a counter or calculate a “next” value, and so on. Typically an ISA does not restrict what lies between LL and SC. Coming back to virtualisation requirements, the definition of a reservation is interesting because it’s effectively “hidden state” that the hypervisor cannot manage. Typically, a hypervisor cannot easily read whether a reservation exists, and it can’t be saved/restored1. CISC-like RmW atomic operations do not exhibit this property. Problem seen, problem felt Shall I get to the point? I saw an odd but legal guest code sequence that can be difficult to virtualise. I’ve been trying to run MacOS 9.2 in KVM-PR on a PowerPC G4, and observed the NanoKernel acquire-lock routine happens to use a sensitive instruction (mfsprg) between a lwarx and stwcx. This is strange, and guarantees a trap to the host between the LL and SC operations. Though the guest should not be doing weird stuff when acquiring a lock, it’s still an architecturally-correct program. This means that if the reservation isn’t preserved across the trap, the lock is never taken. Forward progress is never achieved and virtualisation equivalence is not maintained (because the guest livelocks). Specifically, if the reservation is always cleared on the trap, we have a problem. If it is sometimes kept, the guest program can progress. Since the state is hidden (the hypervisor can’t save/restore/re-create), correctness depends on two things: The hypervisor’s exception-emulation-return path not itself clearing the reservation every time for any possible trap The ISA and hardware implementation guaranteeing the reservation is not always cleared by hardware This potential issue isn’t limited to PPC or the MacOS guest. Software guarantees The hypervisor must guarantee two things: It must not intentionally clear reservations on all traps. It must not accidentally do so as a side-effect of a chosen activity: For example, using its own synchronisation primitives elsewhere, or by writing memory that would conflict with the guest’s reservation. This can be challenging: context switching must be avoided in the T&E handler (no sleep or pre-emption), and it can’t take locks. In my MacOS guest experiment, KVM-PR does not happen to currently use any synchronisation primitives on its emulation path, ew delicate – but I had tracing on, which does. The guest locked up. Hardware guarantees But does your CPU guarantee that reservations aren’t always cleared?2 That seems to depend. This morning’s light reading gives: PowerPC architecture PowerISA is comparatively clear on the behaviour (which isn’t surprising, as PowerISA is generally very clearly-specified). PowerISA v3.1 section 1.7.2.1 describes reservations, listing specific reasons for reservation loss. Some are the expected “lose the reservation if someone else hits the memory” reasons, but previous PowerISAs (e.g. 2.06) permitted embedded implementations to clear the reservation on all exceptions. This permission was removed by 3.1; in my opinion a good move. (I did just this, for reasons, in my homebrew PowerPC CPU, oops!) PowerISA does permit spontaneous reservation loss due to speculative behaviour, but is careful to require that forward progress is guaranteed (i.e. that an implementation doesn’t happen to clear the reservation every time for a given piece of code). Finally, it includes a virtualisation-related programming note stating a reservation may be lost if software executes a privileged instruction or utilizes a privileged facility (i.e. sensitive instructions). This expresses intent, but isn’t specification: it doesn’t criminalise a guest doing wrong things unless it’s a rule that was there from the dawn of time. At any rate, this post is going to be old news to the PowerISA authors. Nice doc, 8/10, good jokes, would read again. RISC-V architecture The lack of any guest legacy permits the problem to be solved from the other direction. Interestingly, the RISC-V ISA explicitly constrains the instruction sequences between LR/SC: "The dynamic code executed between the LR and SC instructions can only contain instructions from the base “I” instruction set, excluding loads, stores, backward jumps, taken backward branches, JALR, FENCE, FENCE.I, and SYSTEM instructions.“ This is a good move. Tacitly, this bans sensitive instructions in the critical region, and permits an absence of progress if the guest breaks the rules. Ruling out memory accesses is interesting too, because it can be useful for a hypervisor to be able to T&E any given page in the guest address space without repercussions. Reservation granule size An LL operation is usually architecturally permitted to set an address-based reservation with a size larger than the original access, called the “reservation granule”. A larger granule reduces tracking requirements but increases the risk of a kind of false sharing between locks where an unrelated CPU taking an unrelated lock could clear your CPU’s reservation. This is important to our hypervisor, because of guarantee #2 above: when emulating a sensitive instruction it must not access anything that always causes the reservation to clear. You would hope the guest doesn’t soil itself by executing an instruction against its interests, so we can assume the guest won’t intentionally direct the hypervisor to hit on shared addresses, but if hypervisor and guest memory could ever coexist within a reservation granule there is scope for conflict. PowerPC defines the largest granule as, effectively, the (small) page size. ARM defines it as 4KB (effectively, the same). It’s a reasonable architectural assumption that guest and host memory is disjoint at page size granularity. RISC-V permits the reservation granule to be unlimited, which isn’t great3 – but later notes that “a platform specification may constrain the size and shape of the reservation set. For example, the Unix platform is expected to require of main memory that the reservation set be of fixed size, contiguous, naturally aligned, and no greater than the virtual memory page size.” Conclusion An ISA cannot be classically virtualised if it permits some aspect of trapping or emulation (such as the exception itself) to always cause a reservation to be cleared, unless sensitive instructions are prohibited from any region dependent on a reservation. In terms of computer science, it’s quite unsatisfying if it were possible to have a sequence of RISC instructions that cannot be classically virtualised due to hidden state. In practical terms, trap-and-emulate is alive and well in systems supporting nested virtualisation. Although some ISAs provide a level of hardware support for NV, it tends to be assists to speed up use of privilege compression rather than more exception levels and more translation stages (which, to be fair, would be awful). Consequently there is always something hypervisor-privileged being trapped to the real hypervisor, i.e. T&E is used in anger. So, there are some hardware behaviours which must (continue to be) guaranteed and, unfortunately, some constraints on already-complex software which must be observed. I thought this small computer architecture safari might be interesting to others, and hope you enjoyed the read! Footnotes In theory an ISA could provide the hypervisor with a previous reservation’s address, but re-creating it with a later LL raises ordering model questions! ↩ Sorry for the double-negative, but this alludes to the possibility of architecture permissions (for example, statements like “X is permitted to spontaneously happen at any time”) leading to implementations taking convenient liberties such as “always do X when any cache line is fetched”. If these decisions were to exist, they would be impossible to avoid stepping on, even with a carefully-written hypervisor. ↩ It would be terrible to permit an implementation to allow all hypervisor memory accesses to clear the reservation! ↩
Built October 2018 I used to hate Cathode Ray Tubes. As a kid in Europe, everything flickered at 50Hz, or made a loud whistle at 15.625KHz (back when I could still hear it). CRTs just seemed crude, “electro-brutalist” contraptions from the valve era. They were heavy, and delicate, and distorted, and blurry, and whistled, and gave people electric shocks when they weren’t busy imploding and spreading glass shards around the place. When I saw the film Brazil, I remember getting anxious about exposed CRTs all over the place — seems I was the kind of kid who was more worried about someone touching the anode or electron gun than the totalitarian bureaucratic world they lived in. 🤷🏻♂️ As ever, I digress. Now in the 2020s, the CRT is pretty much gone. We have astonishing flat-panel LCD and OLED screens. Nothing flickers, everything’s pin-sharp, multi-megapixel resolutions, nothing whines (except me), and display life is pretty incredible for those of us old enough to remember green-screen computing (but young enough to still see the details). But, the march to betterness marches away from accessible: if you take apart a phone, the LCD is a magic glass rectangle, and that’s it. Maybe you can see some LEDs if you tear it apart, but it’s really not obvious how it works. CRTs are also magic, but in a pleasing 19th century top-hat-and-cane science kind of way. Invisible beams trace out images through a foot of empty space. They respond colourfully to magnets (also magic) held to their screens by curious children whose glee rapidly decays into panic, and trying to undo the effect using the other pole before their mother looks around and discovers what they’ve done (allegedly). The magnet-game is a clue: (most) CRTs use electromagnets that scans the invisible electron beam to light an image at the front. There’s something enjoyable about moving the beam yourself, with a magnet in hand, and you can kind of intuitively figure out how it works from doing this. (Remember the Left-hand Rule?) I started to warm to CRTs, maybe a fondness when I realised I hadn’t had to seriously use one for over a decade. I wanted to build something. I also like smol displays, and found an excellent source for a small CRT — a video camera viewfinder. Home cameras had tiny CRTs, roughly 1cm picture size, but I looked to find a higher-end professional viewfinder because they tended to have larger tubes for a higher-quality image. Eventually I found a Sony HVF-2000 viewfinder, from ca. 1980. This viewfinder contained a monochrome 1.5” CRT, and its drive circuitry on stinky 1970s phenolic resin PCBs. All it needs are two turntables and an 8V DC power supply and composite video input. It displays nice, sharp images on a cool white phosphor. I built this from it: Small CRT floating in a box I wanted to show the CRT from all angles, without hiding any of it, in the trusty “desktop curiosity” style. The idea was to show off this beautiful little obsolete glass thingy, in a way that you could sorta guess how it worked. Switching it on with a pleasing clack, it starts silently playing a selection of 1980s TV shows, over and over and over: I had this on my desk at work, and a Young PersonTM came into my office one day to ask about it. He hadn’t really seen a CRT close-up before, and we had a fun chat about how it worked (including waving a magnet at it – everyone has a spare magnet on their desk for these moments, don’t they? Hello…?). Yay! If you’re unfamiliar with CRTs, they work roughly like this: The glass envelope contains a vacuum. The neck contains a heating filament (like a lightbulb) which gives off electrons into the void. This “electron gun” is near some metal plates (with variously high positive and negative voltages), which act to focus the fizz of electrons into a narrow beam, directing it forward. The inside of the front face of the tube is covered by a phosphorescent material which lights up when hit with electrons. The front face is connected to the anode terminal, a high positive voltage. This attracts the beam of electrons, which accelerate to the front. The beam hits the front and creates light in a small spot. To create the picture, the beam is steered in rasters/lines using horizontal and vertical electromagnets wrapped around the neck of the tube. (The magnets are called the “yoke”.) For PAL at 50Hz, lines are drawn 15625 times a second. Relying on the principle of persistence of vision, this creates the illusion of a steady image. The tube is sealed and electron gun inside is largely invisible, but here you can see the malicious-looking thick anode wire, and how dainty the tube really is with the yoke removed: Note: the anode voltage for this tube is, from memory, about 2.5 kilovolts, so not particularly spicy. A large computer monitor will give you 25KV! Did I mention the X-rays? Circuit The original viewfinder was a two-board affair, fitting in a strange transverse shape for the viewfinder case. I removed a couple of controls and indicators unrelated to the CRT operation, and extended the wires slightly so they could be stacked. The viewfinder’s eyepiece looks onto a mirror, turning 90º to the CRT face — so the image is horizontally flipped. This was undone by swapping the horizontal deflection coil wires, reversing the field direction. The circuit’s pretty trivial. It just takes a DC input (9-12V) and uses two DC-DC converter modules to create an 8V supply for the CRT board and a 5V supply for a Raspberry Pi Zero layered at the bottom. The whole thing uses under 2W. The Pi’s composite output drops straight into the CRT board. The Pi starts up a simple shell script that picks a file to play. There’s a rotary encoder on the back, to change channel, but I haven’t wired it up yet. Case For me, the case was the best bit. I had just got (and since lost :((( ) access to a decent laser cutter, and wanted to make a dovetailed transparent case for the parts. It’s made from 3mm colourless and sky-blue acrylic. Rubber bands make the world go round The CRT is supported from two “hangers”, and two trays below hold the circuitry. These are fixed to the sides using a slot/tab approach, with captive nuts. In the close-up pictures you can see there are some hairline stress fractures around the corners of some of the tab cut-outs: they could evidently do with being a few hundred µm wider! The front/top/back/bottom faces are glued together, then the left/right sides are screwed into the shelves/hangers with captive M3 nuts. This sandwiches it all together. The back holds a barrel-style DC jack, power switch, and (as-yet unused) rotary encoder. The encoder was intended to eventually be a kind of “channel select”: The acrylic is a total magnet for fingerprints and dust, which is excellent if you’re into that kind of thing. There seems to also be little flecks filling the case, probably some aquadag flaking off the CRT. This technology just keeps on giving. OpenSCAD The case is designed in OpenSCAD, and is somewhat parameterised: the XYZ dimensions, dovetailing, spacing of shelves and so forth can be tweaked till it looks good. One nice OpenSCAD laser-cutting trick I saw is that 2D parts can be rendered into a “preview” 3D view, tweaked and fettled, and then re-rendered flat on a 2D plane to create a template for cutting. So, make a 3D prototype, change the parameters until it looks good (maybe printing stuff out to see whether the physical items actually fit!)… …then change the mode variable, and the same parts are laid out in 2D for cutting: Feel free to hack on and re-use this template. Resources OpenSCAD box sources Pics Tiny dmesg! Edmund Esq
Built in November 2015 (now-traditional multi-year writeup delay applied) A hat, bejewelled with 38 RGB LEDs Is this thing on..? It’s been a while since I’ve written one of these. So, the hat. It’s been on the writeup pile for almost 6 years, nagging away. Finally it’s its time to shine! NO PUN ESCAPES Anyway, the hat. It seemed like a good idea, and I even wore it out dancing. I know, so cool. This hat had been through at least two fancy-dress events, and had a natty band aftermarket mod even before the LEDs. Long story short, got a hat, put a battery, ARM Cortex-M0 microcontroller, accelerometer in it and a strip of full-colour RGB LEDs around it. The LEDs then react to movement, with an effect similar to a spirit level: as it tilts, a spark travels to the highest point. The spark rolls around, fading out nicely. Hardware Pretty much full bodge-city, and made in a real rush before a party. Parts: Charity shop Trilby (someone’s going to correct me that this is not an ISO standard Trilby and is in fact a Westcountry Colonel Chap Trilby, or something). Bugger it – a hat. A WS2812B strip of 38 LEDs. 38 is what would fit around the hat. Cheapo ADXL345 board. Cheapo STM32F030 board (I <3 these boards! So power, such price wow). Cheapo Li-Ion charging board and 5V step-up module all-in-one (AKA “powerbank board”). Li-Ion flat/pouch-style battery. Obviously some hot glue in there somewhere too. No schematic, sorry, it was quite freeform. The battery is attached to charging board. That connects to the rest of the system via a 0.1” header/disconnectable “power switch” cable. The 5V power then directly feeds the LED strip, from Cortex-M0 board (which then generates 3.3V itself). The ADXL345 accelerometer is joined directly to the the STM32 board at what was the UART header, which is configured for I2C: The STM32 board is also stripped of any unnecessary or especially pointy parts, such as jumpers/pin headers, to make it as flat and pain-free as possible. The LED strip is bent into a ring and soldered back onto itself. 5V and ground are linked at the join, whereas DI enters at the join and DO is left hanging. This is done for mechanical stability, and can’t hurt for power distribution too. Here’s the ring in testing: The electronics are mounted in an antistatic bag (with a hole for the power “switch” header pins, wires, etc.), and the bag sewn into the top of the hat: The LED ring is attached via a small hole, and sewn on with periodic thread loops: Software The firmware goes through an initial “which way is up?” calibration phase for the first few seconds, where it: Lights a simple red dotted pattern to warn the user it’s about to sample which way is up, so put it on quick and stand as naturally as you can with such exciting technology on your head, Lights a simple white dotted pattern, as it measures the “resting vector”, i.e. which way is up. This “resting vector” is thereafter used as the reference for determining whether the hat is tilted, and in which direction. Tilt direction vectors The main loop’s job is to regulate the rate of LED updates, read the accelerometer, calculate a position to draw a bright spark “blob”, and update the LEDs. The accelerometer returns a 3D vector of a force; when not being externally accelerated, the vector represents the direction of Earth’s gravity, i.e. ‘down’. Trigonometry is both fun and useful Roughly, the calculations that are performed are: Relative to “vertical” (approximated by the resting vector), calculate the hat’s tilt in terms of angle of the measured vector to vertical, and its bearing to “12 o’clock” in the horizontal (XY) plane. Convert the bearing of the vector into a position in the LED hoop. Use the radius of the vector in the XY plane as a crude magnitude, scaling up the spark intensity for a larger tilt. All this talk of tilt and gravity vectors assumes the hat isn’t being moved (i.e. worn by a human). It doesn’t correct for the fact that the hat is likely actually accelerating, rather than sitting static at a tilt but, hey, this is a hat with LEDs and not a rocket. It is incorrect and looks good. Floating-point I never use floating point in any of my embedded projects. I’m a die-hard fixed-point kind of guy. You know where you are with fixed point. Sooo anyway, the firmware uses the excellent Qfplib, from https://www.quinapalus.com/qfplib-m0-tiny.html. This provides tiny single-precision floating point routines, including the trigonometric routines I needed for the angle calculations. Bizarrely, with an embedded hat on, it was way easier using gosh-darnit real FP than it was to do the trigonometry in fixed point. Framebuffer The framebuffer is only one dimensional :) It’s a line of pixels representing the LEDs. Blobs are drawn into the framebuffer at given position, and start off “bright”. Every frame, the brightness of all pixels is decremented, giving a fade-out effect. The code drawing blobs uses a pre-calculated colour look-up table, to give a cool white-blue-purple transition to the spark. Driving the WS2812B RGB LEDs The WS2812B LEDs take a 1-bit stream of data encoding 24b of RGB data, in a fixed-time frame using relative timing of rising/falling edges to give a 0 or 1 bit. The code uses a timer in PWM mode to output a 1/0 data bit, refilled from a neat little DMA routine. Once a framebuffer has been drawn, the LEDs are refreshed. For each pixel in the line, the brightness bits are converted into an array of timer values each representing a PWM period (therefore a 0-time or a 1-time). A double-buffered DMA scheme is used to stream these values into the timer PWM register. This costs a few bytes of memory for the intermediate buffers, and is complicated, but has several advantages: It’s completely flicker-free and largely immune to any other interrupt/DMA activity compared to bitbanging approaches. It goes on in the background, freeing up CPU time to calculate the next frame. Though the CPU is pretty fast, this allows LEDHat to update at over 100Hz, giving incredibly fluid motion. Resources Firmware sourcecode: https://github.com/evansm7/LEDHat
More in technology
There is a long history to chemical and biological warfare—one that, we at least hope, ended in 1993. The paucity of chemical and biological warfare today is a result of long-standing objections that culminated in the 1993 Chemical Weapons Convention. It is striking that this formal end to most chemical weapons programs came within a year of the last US test of a nuclear weapon: similar sentiments, and a similar 1990s "end of history" context, motivated both. The similarity makes the differences more striking. Nuclear weapons are still part of the pop culture today, the quintessential "weapon of mass destruction" and a symbol of the destructive potential of modern warfare. But what of that term? Why "weapons of mass destruction?" The history of the phrase WMD is complex, and while it does predate nuclear weapons our modern use emerged in response to them. Still, it's a broader category. In today's language, WMD refers to nuclear, biological, or chemical weapons. Some of its most prominent invocations, namely in the Iraq War, referred to chemical rather than nuclear warfare. Still, to most Americans, chemical weapons are out of sight and out of mind. There are a few reasons: chemical weapons have been more or less prohibited in warfare for over one hundred years, so we have little experience with or memory of them. In fact, today, in the United States, there are none. That was the modern chemical war: the safe management, recovery, and destruction of an enormous stockpile of chemical weapons. This effort spanned military reservations across the United States, dismantling and burning chemical and biological munitions as life went on as usual outside of the fence. Well, at least, "as usual" for the Cold War: one of the effects of the stockpile was to maintain a sort of localized Cold War climate in those towns that fell downwind of it. Today, with the discovery of a dangerously poorly managed stockpile of Lead Azide that will require an exceptionally risky emergency measure to destroy, we are all acutely aware of the direction that the wind blows. The upcoming plume of lead over Iowa is far from the first time that dangerous compounds have blown away from military custody. There have been dozens of such incidents, with varying degrees of intentionality and risk to the public. Many were, at the time, classified. Others were simply covered up. While not exactly the future we wanted, it is perhaps a sign of progress that the upcoming detonation has been announced at all. The public notifications seem, to me, to be the result of an attitude shift that happened at the tail end of the era of US chemical warfare. Windblown plume forecasting models, originally developed for the fallout of nuclear war, were reapplied to risk analysis of chemical weapon stockpiles. The news was bad, so bad that civilian towns were hardened against attack: not enemy attack, but an attack of bad luck, an accident upwind. But first, we should consider how we got there. There are accounts of chemical and biological warfare dating to antiquity, although as with most early history it can be hard to tell what was speculative and what was actually done. By the 18th century, there had certainly been multiple uses of poisons and noxious chemicals on the battlefield. Chemical weapons in particular took a revolutionary step forward with the development of a more formal science of chemistry, during the 19th century. Several important advancements in chemistry occurred in the pursuit of poisonous gases, and just about every other advance in chemistry was applied to them. Still, even in this early era, there was more of an appetite for the development of chemical weapons than for their actual use. A particularly memorable incident in this history comes from Lyon Playfair, the very same who popularized the "Playfair Cipher" that makes a standard appearance in cryptography classes. In his role as a key scientific official during the Crimean War of the mid-1850s, he proposed the use of shells that would disperse a very poisonous form of Cyanide. Objections to the plan came not from pacifist do-gooders but from the weapons-builders themselves, who considered the whole thing morally questionable. Playfair's words in response are almost haunting in consideration of what would follow, outcomes of which he was simultaneously prescient and shockingly short-sighted. War is destruction, and the more destructive it can be made with the least suffering the sooner will be ended that barbarous method of protecting national rights. No doubt in time chemistry will be used to lessen the suffering of combatants, and even of criminals condemned to death. One of the reasons for the lackluster popularity of chemical weapons is that this notion of "easing the suffering" of the enemy, even if accepted as something other than madness, never really materialized. Particularly in the 1990s, the ethics of capital punishment in the United States came to be seen as sort of a technical problem. Maybe homicide by the state wasn't so bad if it was "painless," or "quick," or at least clinical in its appearance. The same idea pervaded chemical warfare in the 19th century, and in the 20th century it fell short in the same way that lethal injection has now. Objection to chemical weapons generally preceded their use. The Hague Convention of 1899 prohibited the form of chemical weapon that was then most obvious, the gas shell. That this prohibition applied specifically to artillery shells is significant: one of the complexities of chemical warfare is its interaction with domestic use. Like hollow-point bullets, tear gas, for example, is widely used in many nations' interiors—but it has been prohibited in warfare for many decades. This situation was already apparent with the first large-scale use of chemical weapons, at the onset of the First World War. Tear gas, a broad category of irritants that are not generally fatal but induce painful and disabling effects, was first developed in France. Parisian police struggled with organized criminal groups who would barricade themselves into buildings and successfully repel attempts at arrest. A commission was formed to investigate the matter in 1912, and it recommended the adoption of ethyl bromoacetate. Delivered to a barricaded position, it caused uncontrollable tearing, difficulty breathing, and considerable pain—enough to drive gangsters out of cover. That development in policing might have been controversial in its own right, but it was quickly overshadowed by the outbreak of the Great War. Tear gas was adapted to hand grenades, directed against the Germans at Alsace in 1914. In practice, this early chemical attack proved ineffective, but it opened Pandora's box. The Germans developed their own irritant shells, both sides justifying their use by leaning on the term "asphyxiating gas" in the Hague convention. Tear gases are unpleasant, but they are not asphyxiating as the term is defined. It was a slippery slope, though, and once the gears of war were in motion they would not be stopped. German chemists improved both the chemical agents and the means of delivery, such that during 1915 a drifting cloud of Chlorine gas was used to scatter French soldiers holding a defensive line. As it happened, the reticence of the Germans to march into their own poisonous cloud meant that they gained little advantage, but it proved the concept. Once again, the legality of the measure was questioned, but this time, the Germans emphasized the Hague convention's specific use of the word "shells"—no artillery shells having been involved in this particular weapon. Faced with this new threat, both sides developed countermeasures, and faced with those countermeasures, both developed new weapons. By the end of 1915 Phosgene was in wide use, and in 1917, the Germans introduced Mustard Gas. Mustard is a terrifying weapon, causing severe blistering of any exposed skin and extensive bleeding. Mustard Gas was extensively used by both sides, becoming part of routine infantry strategy, and along the way dooming tens of thousands of soldiers to a torturous ordeal in the overwhelmed military hospitals. Ironically, despite the extreme response it induced, Mustard Has was not particularly fatal. Many of its victims survived, disfigured after a long period of extreme suffering. These were the stories that came to be associated with chemical warfare. Far from Playfair's vision of "the least suffering," Mustard Gas became one of the war's main sources of it. Mustard-exposed soldiers convalescing in rows, their skin covered in blisters, their eyes sealed closed... these were the images that defined chemical warfare for decades to follow. While some use of chemical weapons continued after the First World War, Mustard Gas inspired a level of outrage that hung over the 1925 Geneva Conference. The Hague convention's prohibition on shells with asphyxiating gas was narrow enough that Mustard Gas arguably did not apply, so negotiators at Geneva agreed to stronger language. The Geneva Protocol, entering into force in 1928, broadly prohibited the use of chemical or biological agents in warfare between states. That could have been the end of the story. The Geneva Protocol was not perfectly observed, but it did prevent almost all use of such weapons. Despite significant technological advances made in Germany, including the development of several nerve agents during the 1930s, World War II elapsed with only limited use of chemical or biological weapons in combat. Most chemical attacks were by the Japanese, perhaps less affected by the experience of the First World War, and even then were directed mostly against other combatants in Asia. On the European front, only a handful of chemical attacks took place. Here, we arrive at the strangest contradiction in the history of chemical weapons: the near end of the use of chemical weapons, the Geneva Protocol, came before the peak of their development and manufacturing. Despite the limited chemical attacks during WWII, chemical weapons programs became larger than ever before. One reason was the Geneva Protocol's differentiation between combat use by state actors and other uses. To say that the Nazis made little use of chemical weapons, while correct in a certain technical sense, becomes absurd in the face of the Holocaust. Domestic use of chemical weapons only escalated, a dichotomy that still resonates today. This potential for domestic use or use against non-state adversaries (such as putting down revolutionary movements) justified a decision by most WWI combatants to continue development and manufacturing of chemical weapons. That created an ongoing possibility of their use, despite prohibition, which itself justified even more development. Consider the case of Australia, a nation that did not have a domestic chemical weapons program but still built massive stockpiles of imported chemical munitions during WWII. Why? For defense, of course: Australia was relatively close to Japan, the main chemical combatant, and felt the need to have a credible retaliation capability if Japan did decide to initiate all-out chemical war. This kind of reasoning, foreshadowing the doctrine that would demand nuclear weapons programs to the modern day, echoed around the world. Australia was importing those weapons principally from the US and UK, two countries that committed not to use chemical weapons, indeed never used them, and nevertheless operated major research centers and manufacturing plants throughout the war. The Germans clearly had a chemical inventory, which they so saliently demonstrated against their own population, and what if they decided to cast the treaty aside? Ought we to remain defenseless? The fear was not unfounded; Germany was known to have violated the Geneva Protocol on several occasions and some historians take the view that their use of chemical weapons was limited by technical issues rather than a policy decision. Churchill favored a chemical "first strike," although the suggestion didn't get much traction. In the US, as well, a notional retaliation capability was grounds for extensive R&D on rocket-delivered chemical warheads. Ironically, most of the US casualties to chemical weapons were self-inflicted. During the war the military manufactured stockpiles of Mustard Gas, Hydrogen Cyanide, and Cyanogen Chloride. These compounds were packaged into weapons and, in at least some cases, staged at the front in preparation for use. This program was a closely held secret and little is known of it even today. One major detail could not escape discovery: a German air raid on US Navy ships in Italy, in 1943, struck a ship carrying Mustard Gas. The resulting leakage at least contributed to the deaths of 69 people, and sickened hundreds. Already, perhaps the most dangerous thing about chemical weapons was possessing them. Chemical warfare programs only accelerated after WWII. The newly developed German nerve agents, including Sarin1, made it to the US as spoils of war. The research programs that formed around these new agents are some of the most infamous of the period, including exposure of thousands of soldiers to small doses of poisons to observe the effects, a program that continued at Edgewood Arsenal in Maryland until the public became aware—and the program collapsed in scandal—in 1975. The involvement of LSD in these experiments often sees them lumped together with MKULTRA, but the vast majority of the research was on less exotic substances: VX, Sarin, Mustard Gas, Tear Gas. Edgewood was also the manufacturing center for Mustard and Phosgene. What Edgewood couldn't provide was storage; it didn't have room for the well-separated bunkers required for munitions. The Army built new installations for the stockpile: Redstone Arsenal in Alabama, Rocky Mountain Arsenal in Colorado, Pine Bluff Arsenal in Arkansas, and Tooele Ordnance Depot in Utah. At each of these sites, hundreds of acres of earth-covered concrete sheds stored weapons at a distance that wouldn't set off their neighbors. In the very early years of the Cold War, the US Army weaponized a new chemical agent called BZ. BZ gained notoriety because of its psychological effects, which included hallucinations, and because of various accusations that it was covertly used by the US during the 1960s. These allegations don't generally hold up, but they illustrate the atmosphere of paranoia that chemical weapons created, and in response to which more chemical weapons were developed. BZ was produced at Swannanoa, North Carolina, and assembled into weapons at Pine Bluff Arsenal, where binary (two parts that are mixed together) nerve agents were produced as well. Phosphate was manufactured at Muscle Shoals, Alabama, and VX at Newport, Indiana. The Aberdeen Proving Ground handled Mustard gas manufacturing and filling. The Rocky Mountain Arsenal, though, just outside of Denver, was the center of the post-war program: Sarin, Mustard, and Sulfur Mustard were all produced and stored there. The Vietnam War proved a pivotal conflict for chemical warfare, as with so many other aspects of military history. Defoliants, such as Agent Orange, were not considered to be chemical weapons and saw heavy use in Vietnam. While targeted at plants, they were not safe for humans either, and played a major role in creating a generation of disabled veterans. In the meantime, the accidental dangers became more apparent: in 1968, an accident at Dugway Proving Ground in Utah released the nerve gas VX and killed thousands of sheep. The Army denied involvement, and then later admitted involvement but denied fault, a position it maintained into the '90s. Despite attempts to maintain secrecy, the Dugway sheep incident brought public attention to the chemical weapons program—which had maintained such a low profile that many didn't realize it existed. Press investigations after Dugway revealed further problems: disposal practices that had released large quantities of chemical agents into the ocean, handling accidents in overseas military installations, and of course numerous human experiments that fell far short of the ethical standards of the time. In short, the Dugway incident reminded the public, and Congress, that the US Army Chemical Corps existed. It had thousands of soldiers, dozens of facilities, and maintained an inventory of chemical weapons that spanned a national network of ammunition igloos. Ongoing manufacturing operations in the 1960s included BZ cluster bombs, VX shells, Hydrogen Cyanide, and others. Once the press latched on to the program, though, it was the beginning of the end. In 1969, President Nixon announced a "no first use" policy for chemical weapons and the complete prohibition of biological weapons—a stronger policy than has been adopted for nuclear weapons even to this day. As part of the 1969 declaration, Nixon also ended the biological weapons program entirely. I have not spoken as much of biological weapons here, as the scale of the US effort was always smaller than that for chemical agents, but I shouldn't understate it. A half dozen viral and bacterial infections, including Anthrax and Botulism, were weaponized in research laboratories. In part because of moral and practical concerns in the military, most of these were never manufactured at scale and delivery systems (such as artillery shells or warheads) were rarely developed to maturity. When Nixon ordered the program ended, its small scale made it relatively straightforward to destroy stockpiled material. Much of the laboratory-scale work was retained, anyway, for its value in researching defensive measures. Chemical weapons were a different matter entirely, because of the scale of the stockpiles. Continuing expansion of the chemical weapons program saw new stockpiles at Umatilla, Oregon; Pueblo, Colorado; Hawthorne, Nevada; Anniston, Alabama; and Blue Grass, Kentucky. At the peak, the United States disclosed an inventory of 31,100 metric tons of nerve agents and Mustard, stored across a dozen sites. Nearly half was concentrated in Tooele, Utah. Due to the old age of many chemical munitions, there had already been efforts to dispose of aged-out stockpiles in the 1960s. This program took the crude form of missions like "Operation CHASE," or "Cut Holes and Sink 'Em," in which Liberty ships full of munitions were towed off of the coast of Florida and scuttled. Only some carried chemical weapons, but well-founded concerns over the environmental impacts of such a chemical assault on the ocean led to legislation prohibiting this means of disposal in 1972. Other smaller-scale disposal efforts had taken forms like loading Phosphorus into barrels, rolling them into a lake, and then shooting at them until they sank, but besides the generally crude nature of this approach there were only so many lakes that could be so used. Congress had intervened in other ways, as well. Starting in 1970, the Office of the Surgeon General was tasked with reviewing the way that chemical weapons were handled and destroyed. This responsibility shifted around several agencies, ultimately involving the Centers for Disease Control, and brought a level of visibility into chemical weapons management that forced the Department of Defense to take a far more systematic and cautious approach to the remaining stockpile. During the 1970s, the Army's Chemical Corps took on a new focus: safe handling and disposal. The most concerning—and contentious—part of the stockpile was weapons that had been "forward staged" in Europe and Asia. These were intended to allow a fast retaliation to any chemical attack on Europe, but by this point they had become unpopular with host nations like Germany and Japan. This set up the first wave of chemical demilitarization: a months-long Army/Navy effort moved the chemical inventory from Okinawa, thousands of tons of munitions with nerve agents and Mustard Gas, to Johnston Atoll. While better known for its role in the nuclear weapons program, Johnston also served as a base for satellite programs (both intelligence collection and anti-satellite weapons) and a test site for biological and chemical weapons. Starting in 1971, storage and destruction of chemical weapons was made the installation's main purpose. In reaction to the safety incidents of the late '60s, legislation passed in the early 1970s that prohibited transfer of most chemical weapons from overseas to the United States. Instead, over the following decades, nearly all US chemical weapons that had been stored outside of North America were consolidated to Johnston Atoll. The inventory there in the 1990s included even German-made chemical shells that had been captured by the Allies and stored in Germany for study. In 1972, an Army command was formally established to manage the destruction of chemical weapons. The first priority was made the M34 cluster bomb. Designed in the 1950s, the M34 was the first Cold War-era chemical weapon and already considered obsolete. M34s were among the weapons destroyed by ocean disposal during Operation CHASE, but that method had since been prohibited. The Army had to find another way to handle more than 20,000 M34 bombs still stored by Denver, and because of the location of the weapons that method would have to actually neutralize the agents. For the Sarin-laden M34s, a hole was carefully drilled to drain the Sarin from each of the "bomblets" that made up the cluster munition. That Sarin liquid was then mixed with a neutralizing caustic chemical in a sealed reaction chamber. The resulting salt brine was dried, and the precipitated solids (no longer a nerve agent but still an environmental toxin) were buried in nearly 20,000 steel drums. In the meantime, Rocky Mountain Arsenal burned tens of thousands of tons of Mustard gas projectiles in an on-site incinerator, and began chemical neutralization of outdated VX munitions that had been relocated to the site for storage. These early disposal operations at Rocky Mountain Arsenal took place in part because of its proximity to a major city, and a political factor that had made it the top priority for site closure. That was also a liability: disposal operations added to existing environmental contamination, exacerbating a groundwater plume that killed nearby crops and creating a new set of toxic landfills. The weapons that were destroyed there in the '60s and '70s were the easy cases, intact weapons with chemical agents that had known neutralization techniques or were easily incinerated. The Army realized that many other chemical weapons, including older weapons that were already in poor condition and beginning to leak, would require a more complex approach with greater risk of exposure. To develop these techniques, the Chemical Corps established the Chemical Agent Munitions Disposal System (CAMDS) at the Deseret Chemical Depot, a new purpose-built installation adjacent to the Tooele Army Depot on the south side of the Great Salt Lake. CAMDS became the primary R&D site for disposal, and pioneered techniques that included robotic remote handling, semi-automated processing lines, and high-temperature incineration with extensive filtration and chemical treatment of the exhaust. Over the lifespan of CAMDS, more than 91 tons of chemical agents were treated, but most operations at CAMDS served as pilots for larger-scale facilities at other installations. In 1981, an ambitious construction program started at Johnston Atoll: the Johnston Atoll Chemical Agent Disposal System (JACADS). Based on techniques in use at CAMDS and built over five years, JACADS then went through four years of testing and validation before the first full-scale chemical weapons disposal line started operation in 1990. Nerve gas shells were carefully unloaded from storage pallets and set up on a belt. That carried them through a sort of de-production line, where the shells were opened, drained, and washed. The explosives and chemicals they contained were burned in a high-temperature incinerator, while the metal casings of the shells and other components were treated in a high-temperature kiln to ensure they were safe to handle. Simultaneously, a set of "auxiliary" incinerators and kilns processed packaging materials and handling waste. Monitoring systems continuously sampled emissions for any escaping chemical weapons, and periodically such leaks were discovered. This period of chemical weapons history is an odd one, because the political tide had turned against chemical weapons but the Army's inertia continued in the direction of chemical warfare. As JACADS was under construction, the Army built a set of new manufacturing plants at Pine Bluff Arsenal. This would specifically produce binary chemical weapons, which use two separate agents that are relatively safe until they are mixed to produce a nerve agent. Binary weapons can be much safer to handle, since the mixing doesn't occur until an in-flight "arming" stage. They were also seen as an escalation in chemical warfare, especially in the context of the Cold War. As often happens with controversial weapons systems, the Army's work to productionize binary weapons was also their undoing. Controversy around the new weapon designs, and the facilities for manufacturing them, led to a renewal in negotiations with the Soviet Union for some type of chemical weapons treaty. Domestically, the issue made its way to Washington, where in 1986 legislation passed requiring that all non-binary (and thus less safe to handle) weapons be safely destroyed. Lack of congressional enthusiasm (resulting in limited funding), the intrinsic complexity of safely handling dangerous chemicals and explosives, and standard military acquisition problems meant that the new binary weapons programs were slow in coming. Despite the start of construction on several binary weapons plants in 1982, only one was operational by the 1990s, producing the M20 and M21 cartridges which produced Sarin upon mixing in an artillery shell. Barely after M20/M21 production started, ongoing talks with the Soviet Union led to an early agreement to end chemical weapons production—a somewhat informal memorandum, but one that kickstarted negotiations for a formal treaty, complete with timelines and verification requirements. In 1991, Congress ordered the destruction of all material related to the chemical weapons program. This included binary materials, as well as waste, experimental samples, and chemical weapons waste buried in landfills. The United States had, at that point, officially ended its chemical weapons program—a position that would be reinforced by the 1997 ratification of the Chemical Weapons Convention, a complete demilitarization treaty with nearly every nation of the world, and most critically key opponent Russia, as signatories. In a certain strategic sense, the US was no longer a "chemical power," the entire stockpile considered retired. In a practical sense, there were still thousands of tons of chemical agents, packaged into hundreds of thousands of tons of munitions, stored in concrete bunkers from Alabama to Oregon. Throughout the last decade of that century, every chemical weapon storage site became a disposal operation. Congressional restrictions and political realities meant that chemical weapons could not generally be transported between sites. During the 1980s, Sarin-carrying Mk 116 "Weteye" bombs stored at Rocky Mountain Arsenal were dismantled there until safety concerns forced a decision to discontinue demilitarization operations at the edge of a major city. The remaining stockpile at Denver was small, but still included almost a thousand Weteye bombs, particularly the "difficult cases" that had begun to leak. The decision was made to ship these bombs to Tooele for safer storage and disposal at the planned CAMDS. The move did happen, but not without inspiring one of the largest political battles of the chemical weapons era, a media circus that considerably frayed relations between the Army and the state of Utah. Congress followed up by restricting domestic transportation, and by the time disarmament was fully underway it was clear that it would all need to be done "as is, where is." I put a lot of time into writing this, and I hope that you enjoy reading it. If you can spare a few dollars, consider supporting me on ko-fi. You'll receive an occasional extra, subscribers-only post, and defray the costs of providing artisanal, hand-built world wide web directly from Albuquerque, New Mexico. The US Army Chemical Demilitarization and Remediation Activity built the Tooele Chemical Agent Disposal Facility (scaled up from CAMDS), Anniston CADF, Umatilla CADF, and Newport CADF. Failures of several pilot programs to develop a practical alternative to high-temperature incineration led to the construction of the Aberdeen and Pine Bluff CADFs shortly after. Manufacturing facilities were converted, sometimes hastily, to dismantling facilities. The Chemical Weapons Convention's verification scheme started a comprehensive audit of military stockpiles, with storage sites physically searched for chemical inventory. A dream that seems so remote for nuclear weapons had been achieved: it was now military policy, and a major military objective, to find and destroy every last chemical weapon. At the turn of the millennium, JACADS completed its mission, having incinerated nearly half a million weapons—6% of the total US stockpile—stored at Johnston Atoll. Disposal facilities were operating at nearly every stockpile site, almost exclusively by incineration, although some neutralization operations continued at smaller scale. While most of the chemical inventory took the form of large quantities of finished weapons at depots, there was also a sort of "miscellaneous" category. Smaller supplies of weapons that had been moved for testing or R&D, stored agents that were never assembled into weapons, laboratory samples and waste, and other "etcetera" cases made up the category of "non-stockpile" chemical materials. For example, the military had manufactured over a hundred thousand "chemical agent identification sets," sort of portable laboratories and field guides for soldiers that might encounter chemical weapons in the field. These contained small samples of several chemical weapons, and were stored at several sites. They met their end via the Rapid Response System, a three-semi-trailer portable agent disposal facility that could perform a range of neutralization processes at small scale. After qualification testing at Tooele, the Rapid Response System made a sort of nationwide tour of each facility. A particularly hard case were those weapons that had begun to leak, or where the explosives had become unstable. These were dangerous to handle or transport, even over short distances. In response, Sandia National Laboratories developed the Explosive Destruction System. The EDS was a single semi trailer, but one that traveled with about a dozen supporting trucks. It was typically set up under a tent with a negative pressure air filtration system and an airlock for entry. Chemical munitions were set up in a tray, and shaped explosive charges placed across the top. This nightmare casserole is then loaded into a huge steel chamber, where the explosives are detonated to both "open" the weapons and thermally neutralize some of the contents. The rest of the chemical agent is destroyed by neutralizing chemicals injected into the chamber post-detonation, and the steel vessel is heated and rotated to mix everything up and complete the operation. During the '90s and '00s, multiple EDS units were used to handle small inventories of "discovered" chemical weapons, leaking and damaged weapons, and non-stockpile materials. After the Chemical Weapons Convention, the EDS convoy included a "treaty trailer" where staff from the international treaty enforcement body Organization for the Prohibition of Chemical Weapons (OPCW) independently observed the work. Major storage sites required industrial-scale facilities. At Umatilla, 1,000 concrete bunkers held artillery shells, rockets, bombs, landmines, sprayers, and storage containers. These weapons contained nerve agents Sarin and VX, as well as Mustard Gas and blister agents. 1980s upgrades to the facility included better sealing the igloos, fitting them with drains that would contain water from the bunkers for sampling, and adding air filtration on the bunker vents. These reduced the risk from leaking weapons, but as the stockpile aged, leakage was a more and more serious problem. Starting in 1998 the Army carefully recorded each leak; prior to that year most went unnoticed. Environmental personnel interviewed former employees and found multiple stories of leaks, cracks, and other incidents. In at least one case, a leaking munition described by a former worker was found, still stored, right in the bunker where he said it had sprayed its contents across the wall. Contracted to Raytheon, construction of the Umatilla Chemical Agent Disposal Facility (UMCDF) began in 1997. The plant went in to operation in 2004. With 819 miles of wire, 25,000 cubic yards of concrete, and 2,655 tons of steel, the Army boasted that it was "among the most advanced industrial facilities in the world." Destroying the full inventory at Umatilla would take years. A risk-prioritized approach was used in which large-quantity nerve agent weapons were destroyed first, and the relatively less dangerous munitions (such as those with only Mustard Gas or those with very small quantities of nerve agent) were left for later. Operated by contractor URS, the UMCDF consisted of a Container Handling Building where airtight, double-walled steel containers that had been packed and sealed in the bunkers were inspected and transferred to conveyors. From that point, they moved into the Munitions Demilitarization Building where the containers were opened, munitions were disassembled and drained, and the chemical agents and components were delivered to various incinerators. While a few workers with extensive protective equipment performed the most complex disassembly tasks, most operations within the Demilitarization Building were performed by a system of conveyors, hoists, and robots under remote control. Different components went into different incinerators. Liquid from washing weapons and containers went to the Liquid Incinerators, where it was mixed with air and natural gas and fed into a two-stage vortex burner system operating between 2,500 and 2,700 degrees Fahrenheit (1,370-1,480 C). Explosives and propellants, usually contaminated by residual chemical agents, went to the Deactivation Furnace System. A 32 foot long rotating drum kiln treated the waste at a minimum of 1,100 degrees Fahrenheit (590 C) for 7-10 minutes, followed by a discharge conveyor that kept them nearly as hot for another 15 minutes. Most of the material burned during this process, leaving ash (packaged for disposal in a chemical landfill) and flue gas that passed through a final stage of incineration in an afterburner chamber. Just about everything else, mostly the metal casings and components of weapons, went to the Metal Parts Furnace (MPF). The MPF was a three-stage incinerator, with a system of belts and airlocks to move material through it in an almost continuous stream. The three stages held materials for various time periods (depending on the type of waste) at temperatures from 1,450 F to 1,800 F (790 C to 980 C), until the materials arrive in the out-feed airlock where they are cooled. The air blown in for cooling was sampled on the way out and tested to ensure that it was free of chemical agents. All gases from the MPF were processed by a multi-stage afterburner incinerator. The UMCDF required an array of support facilities, like the Pollution Abatement System that received the exhaust gases from all of the incinerators. This exhaust, clean of chemical agents but still bearing a variety of interesting and toxic chemicals, was cooled in a quench tower and then put through a series of chemical scrubbers and filters before a final incinerator stage for good measure. Liquid brines, collected from the scrubber towers and parts of the Demilitarization Building process, went to a treatment plant where the water was evaporated off (and the vapors scrubbed) to leave dry crystallized salts. These were packaged for disposal at a hazardous waste landfill. Laboratory, utility, and office buildings made up the rest of the complex. Congressional scrutiny of chemical weapons also led to the Chemical Stockpile Emergency Preparedness Program. This legislation responded to accidents that had occurred at storage and demilitarization sites by requiring that the Army and FEMA take measures to protect nearby communities. The highest population concentration around a major depot, after the relatively early end of operations at Rocky Mountain Arsenal, was found in Oregon and Washington towns around Umatilla. In that region, FEMA funded the construction of a new emergency operations center, installation of sirens, and positive-pressure chemical filtration systems on schools. It was projected that residents of Umatilla would have to move into sealed shelter in less than an hour, were there to be an accident at the depot. A FEMA-funded computer modeling system, at the Emergency Operations Center, would project plume travel to inform evacuation and shelter decisions. The counties stockpiled protective suits and other equipment, and arranged military training on surviving chemical attack for firefighters. By 2000, residents of the area were provided with "tone alert" radios to sound an alarm in their home. Counties distributed "shelter in place kits," with tape and plastic sheeting, to offer residents something to do if the radios ever did sound. The East Oregonian described a county official saying that the kits wouldn't really seal a room, but "may buy more time for a family inside." Pressurization systems were made standard in new schools in the area, and added to the hospital and a nursing home in Umatilla. Working with the CDC, the State of Oregon trained medical personnel on chemical weapons response and distributed the nerve agent antidote Atropine to hospitals. FEMA funds were directed to school districts to buy more buses—a local transportation capability that could be conscripted for an evacuation. In 2008, the last nerve gas weapon, a VX land mine, passed through the plant. The last Mustard Gas weapon was processed in 2011, and evacuation sirens sounded their last Tuesday test. In 2013, the Army published a plan to remediate contamination at Umatilla, demolish the plant, and return the land to other use. Even this process was complex: many of the bunkers were contaminated and had to be carefully cleaned. Much of the demolition debris from the incinerator plant had to be handled as hazardous waste. Each and every bunker was swabbed for testing, soil samples were taken, buildings were knocked over and covered. At the building where workers changed into their protective equipment, septic leachate and dryer lint were collected as chemical waste. In 2023, the Army transferred the 9,511 acres that made up the installation to a coalition of local governments. Much of it will be used by the Oregon National Guard as a training site, other areas are slated to become industrial parks, and the tribal government of the Cayuse, Umatilla, and Walla Walla peoples will operate a large area as an environmental preserve—some of which will require an ongoing program of monitoring and institutional controls. The last chemical weapons in the US stockpile were located at Blue Grass Army Depot, where the Blue Grass Chemical Agent Disposal Facility processed the final Sarin-filled M55 rocket on July 7th, 2023—just ahead of the September 30th deadline set by the Chemical Weapons Convention. Measured from Congress's 1986 legislative mandate, the complete destruction of the US inventory of chemical and biological weapons took 37 years. Closure of the later disposal facilities is still ongoing, and the small-scale portable disposal systems will be retained for use on any chemical or biological weapons that might turn up in the future. An Assistant Secretary of Defense, Deborah Rosenblum, said that "this achievement demonstrates our credibility to the international community and has helped move the US government toward closing this particular chapter of US military history." There are few chapters of military history that ever close. The arc of the chemical weapons program, developed under wartime pressures but accelerated during the curious pressure of the Cold War, largely rejected by both the military and the public, and then wrapped up under international agreement at great expense... it is impossible not to compare it to that of the nuclear weapons program. Negotiations with the Soviet Union leading to a bilateral agreement, followed by an international convention, to completely disarm: the dream of generations of nuclear weapons protestors. It seems so far away, so impossible, for nuclear weapons that it's hard to remember that we've done it once before. So why can't we do it again? Well, the comparison isn't entirely fair. For one, the strategic significance of chemical weapons has never been on par with that of their nuclear peers. The story of World War I shows that chemical weapons are hard to use effectively outside of a few very specific situations, and that never really changed. They have a small area of effect, making them useful mostly to clear the way to advance, but they poison the land such that they can prevent that advance actually happening. Chemical disarmament was achieved, in good part, because the chemical weapons just weren't that important to begin with. One also wonders if the history of use is a factor: chemical weapons were widely used in the First World War, which both spread awareness of their horrific effects and made them more approachable, more understandable, as a part of combat. Nuclear weapons have seen such limited use that they remain more of a symbol than a real weapon, surrounded by such a mystique that their power seems both unlimited and unlikely to actually matter. During the 1899 Hague Convention, which banned the use of "shells containing asphyxiating gas," there was a sole vote against. That vote came from the United States, through its representative, Captain Alfred Mahan of the Navy. He explained his vote thus: The inventiveness of Americans should not be restricted in the development of new weapons. We do consider ourselves to be a powerful culture. The development and manufacturing of chemical weapons was a massive program, involving tens of thousands of workers across the military and private industry. The dismantling of those same weapons happened on the same scale, and often faced even more formidable technical challenges. Perhaps our inventiveness could be less restricted in the destruction of weapons, as well. German researchers gave nerve agents names such as Sarin and Tabun, while the US and NATO used two-letter codes like GB and GA. In fact, these are the same: Sarin is GB and Tabun is GA. I find the acronyms difficult to remember, and the German names are generally the ones used in popular media, so I use the German names when applicable. The problem is that nerve agents developed within NATO countries, such as VX, have no such names. The result will unfortunately be a mix of both conventions.↩
Between Microsoft's two-day outage and the FAA ground stop yesterday, there's plenty of ink spilled about who's responsible, why these things happened, etc. But it made me think back to some of the infrastructure and ops teams I've worked on, and similar (though much smaller scale) outages we encountered. Having an ops mindset is fundamentally different from a development or project management mindset. And it took me over a decade of software development to understand why.
I’ve been getting back into the politics fandom this past month.
A few weeks ago I was up late in my kitchen. I've always been a bad sleeper, starting from when I was 10 and would spend whole nights reading books and watching the clock slowly wind it way to when I had to get up and go to
The resiliency startup has slashed the amount it spends on AI; down from $4,500 per dev, per month, in August, to $1,600 now.