More from joshua stein
In 2019, I started watching Andreas Kling's programming videos, many of which were live-coding sessions writing his new operating system Serenity OS. He was always pretty fast at writing code, especially since his IDE, Qt Creator, had knowledge of all his C++ code and could auto-complete classes, variable names, and function signatures. Back then it seemed like he had much of the code written in his head and was limited just by his typing speed. Four years later, he switched to CLion which had early GitHub Copilot support for AI-assisted code completion. While initially skeptical of Copilot, he experimented with it and within months, Copilot was writing code blocks, "reading [his] mind", and writing whole functions. His IDE's auto-completion through AI seemed like it had finally caught up with his brain and he could get ideas out into code faster. While I found it fascinating watching Andreas speed up his development, none of it appealed to me for my work for years after. I didn't even use an IDE or an LSP in my editor. I was skeptical of AI and probably wrote it off too soon due to its early failures. I eventually tinkered with Claude in a web browser, asking it questions that I would have otherwise asked a search engine and been directed to a Stack Overflow answer. Once Claude Code came out, I was amazed at the power of giving it access to a directory of code on my laptop and letting it directly read and modify files. Its ability to completely (and quickly) reverse engineer a binary blob with zero context is still pretty awesome. Once local models got good and small enough, I started using Ollama on my M4 Mac Mini to do local offline work. Like many things, it's possible to like and use a technology, language, or tool while rejecting all of the hype, fraud, and negative communities that build up around them (like Rails, or Rust). Of course, you can also choose not to like or use any of those things. You can prefer to use a hand saw instead of a circular saw, or to drive a manual transmission car instead of a self-driving EV. But I think at this point it's naive to think of all AI technology as useless or bad. The AI industry on the other hand… In 2020, taking inspiration from Andreas, I started recording my own videos programming on an 8 Mhz Macintosh 512Ke (later a Macintosh Plus) in the THINK C 5 IDE. I had never actually used a classic Macintosh before and I wanted to learn what it was like to program for them as one would have 40 years ago. I made a rule for myself that I would only write code on the Macintosh itself, typing everything on its Apple M0110 keyboard. No modern computers would be involved in writing, editing, or compiling any of the code for my projects. Since starting that series, I've written (most of) an IMAP e-mail client, a revision control system, a Wikipedia reader, an IRC client, then a multi-user, multi-threaded BBS server. It all taught me a lot about the Macintosh and about C, and ignited my vintage computing hobby which has introduced me to many new friends. One thing about writing my e-mail client that bugged me was that it required a proxy server on a modern server to strip TLS and present plaintext to the poky Macintosh. So after adding Wi-Fi functionality to my Mac Plus, I added a TLS offloading device so my Mac could do TLS decryption independently. That provided the final giant piece of the puzzle needed to write a "modern" web browser (at least something newer than MacWeb that could parse HTML 5) for classic Mac OS, since everything is encrypted these days. I started working on that browser in 2024, adding other protocols such as Gopher (and begrudgingly, Gemini). A couple months ago, I was feeling particularly salty about computers and posted these notes: I have lost all enthusiasm for reading about new software projects, especially dealing with vintage computing. What would have sounded amazing a year or two ago and made me want to engage with the author, I now just assume was something AI spit out and feels hollow and boring so I skim past it. And writing software feels like being Marge in that go-kart race. Work slow and steady for a year writing something neat that had never been brought to a platform before, but someone can hear about it and use Claude to whiz by you in a week and make something with 10 times the functionality. Case in point, I saw these today: A modern web browser for Mac OS 9 with CSS, ES5 JavaScript, and TLS A 3D printer monitoring app for System 7 with video A year ago either would have sounded awesome but now I scroll past because I assume AI wrote it all Today someone posted this, a completely vibe-coded Gopher client for classic Mac OS. Claude even made a fancy website for it. And wouldn't you know it, but some of the code for it even comes from my own projects. The other week, Andreas tweeted this: Feels like “I actually enjoy writing code by hand” is about to become the big 2026 virtue signal for programmers. While I don't totally agree (and I think he's got a bit of audience capture staying on that site), it did cause me to question why I bother writing software for 40-year-old computers that only gets used by a handful of people and why I'm bothered by vibecoded software in this context. I write a lot of my software because I want it, but I'm also happy to make it for others in the community to use. Maybe I'm just being vain though, because I want my labor-of-love to be appreciated by people in the community I'm a part of and if a vibecoded version enters the chat with a ton of features and a slick interface, I'm afraid my work will get drowned out and it'll feel like a lot of wasted effort. My Comet will get Yahoo'd. So if a vibecoded tool is genuinely useful and available now while mine's been in development for years, maybe I should just be happy that other people in the community have something new to use. I can continue working on my version for myself and release it when it's ready. After all, Claude has to take that code from somewhere.
These are some notes for getting OpenBSD-current running on the Japanese-model Pomera DM250, DM250X, and DM250XY. Table of Contents Top Throat-clearing Making a backup Some DM250 hardware notes Making an OpenBSD installation SD card Installing OpenBSD Post-Installation Building a custom installation ramdisk Throat-clearing Much of my work has not yet been committed upstream so installation currently requires a custom kernel and U-Boot image which are provided here. OpenBSD support is still improving and may not be stable at any given time. Install at your own risk. These risks include: If the battery completely drains due to a software bug, the device may not power on correctly and will not be able to recharge its battery (see below). If the device cannot boot properly into U-Boot, you may need to recover it through a USB cable which may also require having to open the device (see recovery). Note: Do not use these instructions for the US-model DM250US yet. That device has a different power charging chip and keyboard layout. Making a backup This is optional but recommended. The tools and instructions at EKESETE.net should be used to make a full eMMC backup before doing anything. Some DM250 hardware notes The DM250's factory U-Boot detects Right Shift + Left Alt being pressed immediately after power-on and boots the recovery Linux kernel from its recovery_kernel partition instead of its normal kernel partition. The recovery kernel boots the initramfs image at the recovery_boot partition which mounts the SD card and executes any _sdboot.sh script found at its root directory. If you hold Shift + Alt too long at power-on, it will instead boot its graphical factory hardware test program. The recovery kernel and script do not show any output on the console while they boot and run. Hold down Right Shift + Left Alt + Power and count 3 seconds, then let go and it should boot the recovery kernel and _sdboot.sh from the SD card. Once the new U-Boot image is written to the eMMC (even before going through the OpenBSD installer), the DM250 will no longer be able to boot the original Linux kernel or recovery. You'll need to boot something via EFI to write the backup U-Boot image to eMMC to get back to the factory software. This can be done with the OpenBSD installation SD card created here by dropping to a shell. The DM250 will not fully power off with a USB-C cable connected to power (even with the factory Linux software). This means even if you halt -p and it powers itself off, it will immediately power back on. My U-Boot detects if the lid is closed when it first boots and if it's closed, it will halt the boot process so it doesn't boot into OpenBSD while it's closed on your desk trying to charge (though it will still be powered up to U-Boot). A working battery is required to be able to provide enough power even with a USB-C cable connected. If the battery is completely drained, it may not be able to boot up far enough to get to U-Boot for it to be able to tell the charging chip to go into a higher charge mode. By default it does a trickle charge but because it will always try to power on as soon as it has power connected, it may quickly drain the power it just added to the battery trying to boot. In short, don't let the battery drain completely. Making an OpenBSD installation SD card It is possible to install OpenBSD to the installation SD card and boot from it so the factory Linux software remains on the eMMC. It will run a bit slower (it runs slow even on the eMMC) and you will still need to write the new U-Boot image to eMMC, but you can easily revert back to the original U-Boot to resume booting Linux at a later time. These instructions assume you're running OpenBSD (any architecture). You'll need an SD card of a few GB, which is assumed to be sd1 here. Format an SD card with a GPT partition layout, making an EFI partition of at least 100MB and giving the rest to an OpenBSD partition. # fdisk -ygb 204800 sd1 # echo -e "a\n\n\n\n\nw\nx" | disklabel -E sd1 # newfs /dev/rsd1a # newfs_msdos /dev/rsd1i # mount /dev/sd1i /mnt U-Boot and OpenBSD will use that EFI partition as its firmware partition, but the original Pomera recovery software will see it as a normal MSDOS partition and run the /_sdboot.sh script contained on it. Fetch the standard OpenBSD armv7 EFI bootloader to the SD card's EFI partition as /efi/boot/BOOTARM.EFI. # mkdir -p /mnt/efi/boot # cd /mnt/efi/boot # ftp https://cdn.openbsd.org/pub/OpenBSD/snapshots/armv7/{BOOTARM.EFI,SHA256.sig} # signify -C -x SHA256.sig BOOTARM.EFI Fetch my uboot.img to the EFI partition as /uboot.img. This U-Boot image has an embedded copy of the binary device-tree built from my Linux tree for the DM250 (JP). # cd /mnt # ftp https://jcs.org/dm250/uboot.img Fetch this installation script to the EFI partition as /_sdboot.sh. This will get executed by the factory recovery partition's boot system once the DM250 is booted into recovery mode. The script mounts the EFI partition of the SD card, backs up all of the firmware files from the eMMC to the SD card, makes a backup of the current eMMC U-Boot partition to the SD card, and then writes the new U-Boot image from the SD card to the eMMC. # ftp https://jcs.org/dm250/_sdboot.sh Switch to the OpenBSD partition of the SD card and fetch the latest OpenBSD armv7 snapshot disk images to it. # cd / # umount /mnt # mount /dev/sd1a /mnt # cd /mnt # ftp https://cdn.openbsd.org/pub/OpenBSD/snapshots/armv7/{SHA256.sig,base79.tgz,comp79.tgz,game79.tgz,man79.tgz,xbase79.tgz,xfont79.tgz,xserv79.tgz,xshare79.tgz} # signify -C -x SHA256.sig *.tgz Fetch the current bwfm firmware so you can have working Wi-Fi after installation. # ftp http://firmware.openbsd.org/firmware/snapshots/{SHA256.sig,bwfm-firmware-20200316.1.3p5.tgz} # signify -C -x SHA256.sig bwfm-firmware-* Fetch my bsd.rd ramdisk image and bsd kernel. # ftp https://jcs.org/dm250/bsd{,.rd} Your SD card is now ready to go. Unmount it. # cd / # umount /mnt Installing OpenBSD Insert the prepped SD card into the DM250. If it's plugged into power, unplug it. If it's powered up, power it down. Hold down the Right Shift + Left Alt + Power buttons. As soon as the Pomera logo displays, wait about two seconds and let go. The Pomera logo will clear and the recovery kernel should now boot. It will execute the _sdboot.sh script on the SD card which will make a backup of the existing U-Boot partition to the SD card and write the new one. Nothing will be shown on the screen while it's working but it should only take about 30 seconds. Once it's done, it will reboot the device. Note that in OpenBSD, the default keyboard layout matches the Japanese keyboard on the DM250 so things like @ and * are not where they are on a US keyboard. At this point it will boot the new U-Boot with EFI support and video and keyboard drivers for the DM250. If it detected the eMMC and SD card properly, it will boot the OpenBSD EFI bootloader. No EFI variables loaded Loading Boot0000 'mmc 0' failed Booting: Label: mmc 1 Device path: /VenHw(...)/SD(1)/SD(0) disks: sd0* sd1 >> OpenBSD/armv7 BOOTARM 1.23 boot> Enter b bsd.rd to boot the installer. boot> b bsd.rd cannot open sd0a:/etc/random.seed: No such file or directory booting sd0a:bsd.rd: ... When prompted for the root disk, enter ? to see which is which. The eMMC should be sd1 showing 7.3G. Note: When installing to the eMMC, you'll need to manually setup the GPT partition table putting the EFI partition after ~16MB or so to leave room for U-Boot. This will not be needed in the future once the installer is able to recognize the DM250. Press Control + Z to get to a shell so you can manually fdisk. Use (W)hole disk or (E)dit the MBR? [whole] ^Z [1] + Suspended /install # cd /dev # sh MAKEDEV sd1 # fdisk -e -g sd1 Do you wish to write the new GPT? [n] y Writing GPT. Enter 'help' for information sd1: 1> Edit partition 0 which was auto-created, changing it to an ef partition at offset 32768 with size 8192. sd1: 1> e 0 #: type [ start: size ] ------------------------------------------------------------------------ 0: OpenBSD [ 64: 15269791 ] Partition id ('0' to disable) [01 - FF, <uuid>]: [A6] ef Partition offset [34 - 15269854]: [64] 32768 Partition size [1 - 15237087]: [15237087] 8192 Partition name: [OpenBSD Area] EFI sd1*: 1> Now create partition 1 as an OpenBSD (a6) type with the rest of the space. sd1*: 1> e 1 #: type [ start: size ] ------------------------------------------------------------------------ 1: Unused [ 0: 1 ] Partition id ('0' to disable) [01 - FF, <uuid>]: [00] (? for help) a6 Partition offset [34 - 15269854]: [40960] Partition size [1 - 15228895]: [15228895] Partition name: [] OpenBSD sd1*: 1> Print the table to make sure it matches, then w and q. sd1*: 1> p Disk: sd1 Usable LBA: 34 to 15269854 [15269888 Sectors] #: type [ start: size ] ------------------------------------------------------------------------ : Free [ 34: 32734 ] 0: EFI Sys [ 32768: 8192 ] 1: OpenBSD [ 40960: 15228895 ] sd1*: 1> w Writing GPT. sd1: 1> q # Now you can return to the installer, which is asking for disk info. Enter ? to get it to refresh disk information and it should now print out sd1. Enter e to edit the GPT (again). # fg install ? Disk: sd1 Usable LBA: 34 to 15269854 [15269888 Sectors] #: type [ start: size ] ------------------------------------------------------------------------ : Free [ 34: 32734 ] 0: EFI Sys [ 32768: 8192 ] 1: OpenBSD [ 40960: 15228895 ] Use (W)hole disk or (E)dit the GPT? [whole] e You'll now be back at fdisk, just q to quit. You can accept the auto-allocated partition layout or just make one big root and a swap partition. When prompted for the location of the sets, specify disk and then respond no that it's not mounted, and enter sd0 (the SD card), then partition a. The pathname to the sets will be just /. Continue installing, ignoring any errors from signify. The installer should automatically pick up the bwfm firmware and install it. At the end it will try to re-link the kernel which is using the upstream kernel object files, so it's important you copy the custom kernel back over /bsd. Exit to (S)hell, (H)alt or (R)eboot? [reboot] s To boot the new system, enter 'reboot' at the command prompt. # mount /dev/sd0a /mnt2 # cp /mnt2/bsd /mnt/bsd You'll also need to disable reorder_kernel which will run at each boot so it doesn't revert the kernel back to the upstream version. # mv /mnt/usr/libexec/reorder_kernel{,.disabled} # echo "#!/bin/sh" > /mnt/usr/libexec/reorder_kernel # chmod +x /mnt/usr/libexec/reorder_kernel (Note: the _ key is to the left of the up arrow, + is to the right of L.) You can now reboot. Post-Installation The first boot will report an error about bwfm and loadfirmware. You'll need to copy the firmware/nvram_ap6212a.txt file that was backed up to the SD card's EFI partition when U-Boot was flashed. # mount /dev/sd0i /mnt # cp /mnt/firmware/nvram_ap6212a.txt /etc/firmware/brcmfmac43430-sdio.rockchip,pomera-dm250.txt # umount /mnt U-Boot is compiled with boot logo support, so you can put a bitmap in the EFI partition's root directory as logo.bmp to get it to show at boot. To make it show OpenBSD puffy, you can use this image. # mount /dev/sd1i /mnt # cd /mnt # ftp https://jcs.org/dm250/logo.bmp # cd / # umount /mnt The two LEDs near the USB-C port can be addressed once booted. Create an /etc/rc.securelevel that sets them up so they are accessible from a normal kern.securelevel: # cat > /etc/rc.securelevel #!/bin/sh gpioctl -q gpio1 8 set out red_led gpioctl -q gpio1 12 set out green_led ^D # After rebooting, they can be addressed with the gpioctl utility: # gpioctl gpio1 8 1 pin 8: state 0 -> 1 Building a custom installation ramdisk If you want to tinker beyond installing, you'll need a Git clone of my OpenBSD rk3128 tree, my U-Boot pomera-dm250 tree with video and keyboard drivers, and my Linux tree which contains the DTB source files. This is optional and requires being able to compile on an armv7 machine (or cross-compile if you know how), or you can just download the latest image I've already built. Fetch an existing bsd.rd, extract its installation filesystem, build the RAMDISK kernel from my rk3128 tree, and then insert the filesystem into the new kernel. $ cd /tmp $ ftp https://cdn.openbsd.org/pub/OpenBSD/snapshots/armv7/bsd.rd $ rdsetroot -x bsd.rd root.fs $ cd /usr/src/sys/arch/armv7/compile/RAMDISK $ make config $ make $ cp -f obj/bsd /tmp/bsd.rd $ rdsetroot /tmp/bsd.rd /tmp/root.fs The /tmp/bsd.rd file is now a bootable installation image with a custom kernel.
A new release of my Wallops IRC client is available: wallops-2.2.sit (StuffIt 3 archive) adc2b473664ec0960b5b028335566831dd69f4a1de40f79e22bf1337292cca88 SHA1: 603c5080c41a0d5752cae49a08376db4e18628b1 This update includes a number of new features and bugfixes: Auto-connect with the last saved settings on startup, if they are available, unless the Command key is held down Add support for sending Pushover notifications when the user's nick is mentioned or directly messaged, when a screen saver is detected to be running; sends to Pushover's API over plain-text HTTP Improve nick highlighting when the user's nick is not at the beginning of the line Speed up parsing of nicknames when joining large channels Add support for the /monitor command which is sent directly to the server Fix bug that may have caused a crash when initially connecting, and when reconnecting
Last year I upgraded my Kindle Paperwhite to a Kindle Scribe to be able to write notes and draw diagrams while programming to help visualize things. One thing that bothered me about the Scribe was that its sleep screen images were pretty boring and because I'm now often reading PDFs or writing in a notebook, I couldn't benefit from the Kindle OS's new functionality that uses the cover of the book being read as the sleep screen image (which previously required a jailbreak and custom software). Since the process for installing custom sleep screen images is rather cumbersome and the information is scattered across old forum posts, I thought I'd document how I did it to possibly make it easier for someone else. Table of Contents Top Jailbreaking Disabling OTA Updates Installing KOReader Installing Custom Images Jailbreaking On my Kindle Paperwhite, I long-ago used a jailbreak to install a bunch of software packages that enabled custom sleep screen images. For the newer Scribe, this wasn't available until recently with the "WinterBreak" jailbreak, and the process to use custom images is now easier. To install WinterBreak on the Scribe, one needs a computer with an MTP application since the Scribe now uses MTP like an Android phone rather than attaching as a regular USB disk drive. On my Mac, I used OpenMTP. Follow the instructions at the WinterBreak page, which at the time of writing (and worked on my 1st generation Scribe running firmware 5.17.3) were: Download the current WinterBreak release Enable Airplane mode on the Kindle Connect the Scribe to the computer with a USB-C cable Extract the WinterBreak archive and copy all of the files (except .git*, README.md, and instructions.txt) to the Kindle's home directory, including the .active_content_sandbox directory which may not initially be visible in the MTP program Tap the "Disconnect" button on the Kindle and unplug it Swipe down from the top of the Kindle screen to open the settings drawer and tap "All Settings", then "Device options", then "Restart" Once restarted, tap the shopping cart icon on the Kindle to open the Kindle Store; when prompted to turn Airplane mode off, tap "Yes" Instead of the usual Kindle Store page, there will be a small icon labeled "Mesquito"; tap on it A bunch of debugging messages will print on the screen and it will say "Please install hotfix now." Plug the Kindle into the computer again with the USB-C cable Download the Hotfix .bin file and copy it to the Kindle's home directory with the MTP program Tap the "Disconnect" button on the Kindle and unplug it Swipe down from the top of the Kindle screen again and tap "All Settings", then tap the 3-dot menu button at the top and tap "Update your Kindle"; tap "Update" when prompted Once the Kindle restarts, there will be a new item in the "Your recent items" list named "KUAL" Disabling OTA Updates To prevent a future automatic update from disabling your jailbreak and possibly removing your custom software, you can install the "renameotabin" extension using the instructions shown there. Installing KOReader Once jailbroken, the KOReader application can be installed, which offers a different interface for reading books. I don't particularly like it (especially since it's a bit cumbersome to launch each time) but it's needed for the utilities included in it. Find the latest KOReader nightly release directory and download the kindlehf zip file in that directory Connect the Scribe to the computer with a USB-C cable and open the MTP program Extract the KOReader zip file and copy its koreader and extensions directories to the home directory on the Scribe (there will probably already be an extensions directory, so the extensions/koreader directory from the zip file needs to go in that extensions directory) Tap the "Disconnect" button on the Kindle and unplug it From the Kindle's home screen, tap "KUAL"; there should now be a "KOReader" entry in the list Installing Custom Images Sleep screen images for the Kindle Scribe (at least my 1st generation device) must be 8-bit grayscale PNG images, with a resolution of 1860x2480. I used GIMP to prepare some photos by scaling/cropping them to 1860x2480, converting to grayscale, and then exporting as a PNG. In GIMP's "Export Image as PNG" dialog, the "automatic.pixelformat" option should be changed to "8bpc GRAY". From what I've read, having images in a different format can cause crashes, so it's best to just force this format. When exporting your images, name them bg_ss00.png, bg_ss01.png, bg_ss02.png, etc. To install them: Connect the Scribe to the computer with a USB-C cable and open the MTP program Copy your bg_ss*.png images to the Kindle's home directory Tap the "Disconnect" button on the Kindle and unplug it From the Kindle's home screen, tap "KUAL" In KUAL, tap "KOReader", then the first "Start KOReader" option; the screen will go back to the Library and after a few seconds, KOReader will start Tap the "KOReader" header at the top to open its menu Tap the tools icon, then tap the ">" button to go to Page 2 Tap "More tools" Tap "Terminal emulator" and then "Open terminal session" You are now at a root shell, standing in an open field west of a white house, with a boarded front door; there is a small mailbox here You will need to delete the factory sleep screen images and then copy the bg_ss*.png images you transferred over to that directory. If you want to keep the factory images, don't delete anything and start your bg_ss*.png numbering at 07 since there are 7 factory sleep screen images (which start at 00). To modify the system sleep images, you'll need to remount / as read-write since it's read-only by default. In the Terminal emulator screen at the root # prompt, enter: mount -o remount,rw / If you want to delete the factory images, enter: rm /usr/share/blanket/screensaver/*.png To copy your custom images to the system directory, enter: cp /mnt/us/bg_ss* /usr/share/blanket/screensaver/ Once finished, tap the "X" button above the keyboard to close the terminal emulator. Once back at KOReader, tap the 3-line button at the top right and tap "Exit", then "Exit" again. You'll be returned to the Kindle home screen. Swipe down from the top of the Kindle screen to open the settings drawer and tap "All Settings", then "Screen and brightness". Make sure the "Show covers on lock screen" option is disabled. Tap "Device options", then "Restart" Once restarted, press the Kindle's sleep button and, if all went well, you should see a randomly-selected image from the custom ones you uploaded. A phoropter
More in technology
Well, well, well, well, well, well, well, well, well, well, well, well, well, well, well. We're back. Sorry. We've been watching the onslaught of vulnerabilities flood the internet. Every man, dog, and their grandmas (apparently?) are now using LLMs to find and reproduce vulnerabilities - it’
You want less of them. That’s the reason. You may find that it’s too hard to stop people from doing the thing, literally blood, sweat, and tears trying to prosecute people, but that’s a different thing.
Solitaire Alone Together I made a new game. It's called Solitaire Alone Together. It's Windows 98 solitaire, but you can play with everyone else on the internet. Read the full post on my blog! Here's a raw link, if you need it: https://eieio.games/blog/solitaire-alone-together
This post is a living diary of all the times I messed up something with my website in a funny way. I value those who have the confidence to own their mistakes and share the learning with others, and so this is me doing just that! That Time I Accidentally Made a Tarpit That Time I Accidentally Made Really Large Headers That Time I Accidentally Made a Tarpit Back to Top A "tarpit" is an unofficial term used in computing to describe an intentionally slow response to a request. In these modern times many people are using tarpits as a way to combat the relentless theft of data by AI companies, although there's little to no evidence of that actually being in any way effective. I don't use tarpits, at least not intentionally, but there was that one time when I accidentally created a tarpit and trapped all visitors in it. As I've shared previously, I refuse connections from IP addresses that are blocked or belong to a blocked subnet, and I enforce this firewall during the TCP handshake. The logic here is straightforward: there's no reason to waste resources doing a TLS handshake, accepting an HTTP request, and then rejecting the connection if I already know I'm going to reject it at the earliest step. At the time, the code worked like this: the HTTP server would repeatedly call the Accept() function below expecting a new connection. I've added some comments to help explain the logic. func (l *firewallListener) Accept() (net.Conn, error) { // Accept the connection from the TCP listener. This blocks until there is a connection to accept or the listner was closed. conn, err := l.l.AcceptTCP() if err != nil { return conn, err } // Separate the IP address out from the remote address (which includes the port) ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } // Check if it's blocked, if so close the connection and return a refuseError if IsBlocked(ip, true) { conn.Close() return nil, &refuseError{} } // Otherwise return the connection on to the HTTP server return conn, nil } If the incoming connection was from a blocked IP then I'd return a refuseError. I need to use a specific error interface because the HTTP server will halt if it encounters a non-temporary error from the call to Accept(), so I need to return an error that satisfies the definition of a temporary error. I defined refuseError like this: type refuseError struct{} func (e *refuseError) Error() string { return "." } func (e *refuseError) Timeout() bool { return true } func (e *refuseError) Temporary() bool { return true } func (e *refuseError) Is(err error) bool { return err == context.DeadlineExceeded } This did accomplish the goal of rejecting connections before the TLS handshake for blocked addresses, but it had one really unintended and difficult to track down side-effect. Accepting connections is done serially, after which servers typically then process that request on a dedicated thread (or in Go's case a goroutine). This means that any delays during the accept loop will block all incoming connection. What I had missed while reviewing the code for Go's HTTP server is that when it receives a temporary error from Accept() is that while it doesn't abort, it does sleep for up to a maximum of 1 second. This sleep blocks the entire server for all incoming connections. You can see a trimmed copy of the code that does this below, with some marks I've added which I will explain. // src/net/http/server.go // Copyright 2009 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. for { // (1) rw, err := l.Accept() if err != nil { if s.shuttingDown() { return ErrServerClosed } // (2) if ne, ok := err.(net.Error); ok && ne.Temporary() { if tempDelay == 0 { tempDelay = 5 * time.Millisecond } else { tempDelay *= 2 } if max := 1 * time.Second; tempDelay > max { tempDelay = max } s.logf("http: Accept error: %v; retrying in %v", err, tempDelay) // (3) time.Sleep(tempDelay) continue } return err } connCtx := ctx if cc := s.ConnContext; cc != nil { connCtx = cc(connCtx, rw) if connCtx == nil { panic("ConnContext returned nil") } } tempDelay = 0 c := s.newConn(rw) c.setState(c.rwc, StateNew, runHooks) // before Serve can return // (4) go c.serve(connCtx) } At mark 1 the server calls the Accept() function, this is the exact function that I defined above where I might return a temporary error. At mark 2 it checks if an error was returned, and if so if that error is temporary. If there was a temporary error, at mark 3 it sleeps for an increasing amount of time up-to 1 second, otherwise, at mark 4 it processes the connection on a dedicated goroutine, which allows the server to accept the next connection. I'm not entirely sure why the Go developers added this sleep delay and the change when it was introduced doesn't provide any meaningful insight. Regardless, it caused significant latency connecting to my website when a flood of rejected requests was coming in. It just goes to show how important it is to write meaningful commit messages, because you never know when somebody might come back years later wondering "why was this done?". I sure home I don't come to eat those words later. Coincidentally, you can actually see this happening if you look carefully at one of the metric graphs I shared in my first post about my server's security model: Securing My Web Infrastructure. This is the graph I shared in that blog post and while I didn't know it at the time, the fact that these request spikes all cap-out at around 60 requests per minute was not a coincidence. These requests were not being made with a limit in mind, attackers rarely ever care about things like that, instead it the accidental tarpit I had created. The downside to this was that while the malicious requests were being rate-limited, all requests were being rate-limited, up to a point of taking so long they timed out. The Fix Fixing the issue was relatively straightforward enough. Instead of returning a temporary error to the HTTP server during the accept loop, just don't return anything at all and wait for the next valid connection. func (l *firewallListener) Accept() (net.Conn, error) { for { conn, err := l.l.AcceptTCP() if err != nil { return conn, err } ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } if IsBlocked(ip, true) { conn.SetLinger(0) conn.Close() continue } return conn, nil } } Now, when the HTTP server calls Accept(), the only time it returns is with a connection from an IP that isn't blocked, or if there genuinely is an error. No more sleep delays, no more excessive timeouts. That Time I Accidentally Made Really Large Headers Back to Top For about 10 years now all major browsers have support for a security feature known as a Content Security Policy or CSP. A CSP is an HTTP header provided by the server that instructs the browser on where it can load assets from, this could be scripts, images, stylesheets, fonts, etc. The objective of using a CSP is to prevent against injected HTML that tries to load assets, such as a malicious Javascript file, from a remote source. With so much user-provided content being available online, it's very possible for this to happen without an attacker compromising the entire web server. CSP protects against that by saying "scripts can only be loaded from these domains". That's a really simplified way of looking at it, anyways. My web server supports injecting the CSP header automatically, but before I go on I need to explain a little bit about the structure of my web server. When an incoming HTTP request is accepted (having passed all firewall checks and assertions), we look at the destination host for the request. This can either be the value of the Host header or as specified during the TLS handshake. We then look at a map of hosts to apps. Apps are just an interface that accept a few methods: type App interface { Cleanup() ReloadConfig() ServeHTTP(rw http.ResponseWriter, r *http.Request) Setup(dataDir string) error Shutdown() } One of the apps is the Proxy app, which is a reverse proxy - it accepts the incoming HTTP request and then proxies it on to another host. This is a very common design, especially with increasingly complex TLS setups. Because each app is unique to a host, and different hosts have different requirements for CSP rules, the proxy app includes a CSP preset that we use to build the header value, or skip it entirely. When the proxy app was going to copy an HTTP request to the downstream host, it would build the CSP header, however there was a slight bug... func (a *App) ServeHTTP(rw http.ResponseWriter, inRequest *ht2.Request) { // --snip -- if a.CSP != nil { a.CSP.ConnectSrc += " " + inRequest.Origin } CopyHttpRequest(inRequest, outRequest, rw, CopyHttpRequestOptions{ Origin: inRequest.Origin, Csp: a.CSP, Cors: a.CORS, AddHeaders: !a.SkipHeaders, UseHTTP3: a.UseHTTP3, InsecureTLS: a.InsecureTLS, }) } I'm really unsure as to what I was doing with the line to append to the ConnectSrc, but the impact is that I'm appending to a variable that lives on the App, rather than a variable that is per-request. This meant that every time there was a request to the app, any request at all, the origin would be appended to the header value. This went on for quite a long time unnoticed and unresolved, largely because I am constantly tweaking and tinkering with my web server, after all, it's how I made having a website fun again. Each time I restarted the server process, the header value would be reset, but only for it to continue to grow and grow. Eventually, after a period of being busy with other matters, the server process stayed running for long enough that the header value grew too large and HTTP clients began to reject it. There is no defined maximum for an HTTP header value, however most HTTP clients use 100KiB, which is perfectly reasonable, and this header value would continue to grow well beyond that. Diagnosing this issue turned out to be difficult as tools like Curl would fail with errors relating to entities being too large, but stopped short of saying what specifically. I eventually used openssl s_client to send an HTTP request by hand and observed my terminal window being filled with a domain name repeated thousands of times. Looking at the commit history, it was really unclear why I added the culprit lines of code. The commit message just says "Improved CSP support". It just goes to show how important it is to write - hey look it's those words I'm now having to eat! The Fix The fix was to just delete those three lines of code. Yup, it really was that simple, and fixing this bug actually made a larger positive impact than I had expected, as it was immediately clear when I fixed the bug by looking at outbound network bytes: So much traffic was being wasted on excessive header sizes. You might look at these mistakes I've made and think "wow, Ian, these are some obvious mistakes, I never would have made them!" to which I say "good for you!" with the utmost sarcasm and disdain. I enjoy making and refining software, and making anything means making mistakes along the way. Each time I make mistakes such as the ones above, I improve my skills of investigation, diagnosing, and repair. Skills that, judging by my peers in the industry, seemingly everyone is quickly willing to throw away because a robot does it "better" than you. Header Image: "Car accident on the Ffestiniog to Bala road. Nobody was hurt" by Geoff Charles, CC BY-SA 4.0, via Wikimedia Commons.
Back in 2021, I wired up data from Buienalarm and Buienradar through Node-RED and a big pile of Jinja2 templates to get a rain forecast graph on my Apple Watch: eight Unicode block characters showing the next two hours in 15-minute chunks, glanceable without unlocking the phone. It worked, and I used it every day […] The post Buienwatch, a custom Home Assistant integration for Buienradar and Buienalarm graphs on your Apple Watch appeared first on Style over Substance.