Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
1

Omarchy on iMac

from Willem L. Middelkoop [alt+shift+b] in technology

After helping a family member migrate to a new computer, I found myself with an old iMac. The machine from 2017 was very slow running modern macOS, yet I have a soft spot for these all-in-one computers designed by Jony Ive and his team. Could I install Linux on it to give it a second life? iMac Probably one of the most important computers that Apple ever built is iMac, the original was critical in Steve Jobs' effort to save Apple from bankruptcy. I like its looks and how its design makes a separate box/tower unnecessary. This saves you a lot of cables that original desktop PC's need (like wires connecting the monitor and speakers). Migrating to a new iMac Using macOS's Migration Assistant I helped my family member move his files and apps to a new computer. His old iMac had become very slow running macOS. Newer macOS versions no longer run on this 2017 machine as Apple has dropped support for it. I hate planned obsolescence and offered to find out if I could give this machine a new purpose. Lessons from a takeaway plastic bag: My ever increasing antipathy to planned obsolescence (Jan. 8, 2018) Look, I ain't no saint when it comes to the environment. I have driven race cars and flown planes for fun and lunch, but I do care about throwing good things away for the wrong reasons. In the past I have used my Linux expertise to revitalise old hardware, like in these two earlier posts: Working Offline First: Learning from a 15-year old ThinkPad X200 (May 1, 2023) Helping people with free software: Installing Debian GNU/Linux on an old laptop (July 20, 2018) The old iMac is a 21-inch model from 2017 with an Intel i5 processor, 8GB RAM and a 1TB hard disk. Admittedly this machine won't break any speed records, but it looks nice and it runs almost completely silent. Its display has great colours and the glass and aluminium housing offers plenty of space for ports. It has USB-C/Thunderbolt, USB-A, gigabit ethernet, a headphone jack and a SD card reader....
21st Sep 2025

Stay updated

Get a weekly newsletter with the top 5 articles worth reading every week.

More from Willem L. Middelkoop

Twenty Times Faster

On 11 August I wrote in my notebook: the speed at which we move is about twenty times the old top speed. In one day I do what I used to do in twenty. This post is about that speed, what it does to the human in the loop, and the counterweight I found. It is the first post of September, written on a day of deliberately doing less. Twenty times The number is not a boast, it is a measurement. In August two problems on the cloud were analysed and fixed on the day they appeared, work that used to be a week of digging. It is mentally draining, this speed. Still I find it beautiful, because you shoot with precision and power. Last week was a normal week. A complete restaurant site. A proof of concept for an event venue, promised online within the week. The final touches on a bar site. New sections on two local news sites and a full site for a city square. A site and a members area for a medical publication. Printer support in the shop app, through Apple's review. Analyses of four competitors. A video call setup on the new laptop. And on Sunday evening a prototype for writing by hand, which is how these notes were written. In the same week: three customer visits, two video calls, school runs, football training and a match, a birthday present for my wife built as a little game, a sleepover guest. It is not surprising that we sometimes feel a bit full. Saturday night, 5 September: the terminal on the laptop, one of the sites of the week in the making The human in the loop Mid August, after a week of full throttle, I wrote: I am at my max. Head and body at the limit. The gear worked fantastically, and I was the limiting factor. That is the honest state of things in 2026: the agents do not tire, the logs do not sleep, the builds run at night. The bottleneck has moved, and it moved to me. On a campsite in France I had already written the paradox down: post AI, the challenge is simple to use, less is more. Ship less. A contradiction with agents that can make everything. And the people around you do not always keep up either. The pipelines give you speed, enough to ship at eighty percent and learn. But a customer will look at the breakthrough tomorrow, and a partner needs a week. I have to learn to accept that not everyone switches at light speed. Sometimes that delay is a gift: a quieter period, a little more time for a delivery. The partial eclipse over the trees on 12 August, an evening walk in the busiest week of the month The counterweight The answer I found is not slower agents. It is a routine for the body. We must not see sport as not working: the ideas, the space and the energy are crucial. So there is a shake before the coffee, a run on the odd days, the gym three times a week, and no more finishing the leftovers. A 10K alone on a Friday morning is a relief, clarity of mind through real silence. The best runs of the summer, along the Seine at dawn and on the coast of Brittany, were not fast or strong. They were where the thinking happened. And at the end of August the sharpest note of all: the overload creates the wish for rest and space. That space has to be protected. Sleep matters, and working late at night maybe not. With school, sport and football there is not enough left over to sustain that. I have known this since my first marathon: you can push the limit, but you cannot move it by wanting. The Seine at sunrise on 6 August, the run where the thinking happened The Eiffel tower from below at the turnaround, not fast or strong A marathon, not a sprint Something else happened this week. Sites that generate themselves are becoming common, I see comparable techniques appear around me. The distinguishing factor is no longer the public endpoint, it is the stack behind it: a customer who is happy with the app in her pocket, another who bounces because of his charts. That is much harder to replicate. And there is a harder question still, also for the big AI companies: how do you carry the responsibility of keeping an engine running well, for years? Generating something neat once is easy. Keeping it running, maintaining it, over the long term is another matter. This is a marathon, not a sprint. One of the week's deliveries on the phone: a site for a city square, fresh every day This week So this week we do it differently. Mandatory time for sport and away from the keyboard. A week of rolling out cannot hurt: ninety-nine percent reactive, an empty inbox, and some polish on our own tools. No new websites without a customer's mandate. No customer contact on my initiative. This morning I slept an hour and a half longer and ran five kilometers. The plan had been a half marathon. The body really needed the extra sleep, and I feel a lot better now. The things that are in motion can roll out on their own for a while. Sunday 6 September: sleep 86 percent, recovery 94 percent, strain 0.4, a day off by design Conclusion Twenty times faster is real, and so is the bill. The agents do not tire, I do, and that makes the pace mine to set. A run, a night of sleep and a week of doing less are not a break from the work. They are how the work stays good. Sunday evening: the prototype for writing by hand on the new laptop, these notes were written with it

2 weeks ago 2 votes
Innovation Is Software

In August my computer setup changed more than it had in years. And yet the lesson of the month was the opposite of new hardware: the existing gear has stretch, innovation is software. This post is about one brain, the machine in the homelab where my files and my agents live, the machines around it, and why infrastructure has become subordinate to concept. One brain On 2 August, from a campsite in France, I got remote control over the router at home. The plan was to set up a direct connection to the machine in the homelab in Limburg: one brain, one login, one place where files and agents work. Two days later the workshop server that hosted my projects was phased out. Every project moved home, checksum verified, and the homelab took over the full role through a direct mosh link. From the campsite I steered it from my phone, with speech to go faster. That machine in the homelab is the anchor. It holds the platform, the customer projects, the photo library, the backups of the cloud and the agents that work on all of it. The cloud servers run the sites, the agents work from home. Everything else is glass: a screen and a keyboard to look at the brain with. 2 August, from the campsite: a mosh session on the phone into the machine in the homelab, the one brain Logs are readable On 11 August two problems on the cloud, one in mail and one in the web server, were analysed and fixed on the same day. That is only possible through the integration of the anchor in the agentic flow, plus an understanding of the relevant logs and signals. The speed at which we move is about twenty times the old top speed. In one day I do what I used to do in twenty. It is mentally draining, but you shoot with precision and power. The agents do not all live in the cloud. A local model runs on the homelab with no data leaving the house. On 24 August it watched the footage of the doorbell and described, frame by frame, a man walking barefoot to the front door. It was me. Local agents, with access to the network, the files and the protocols, nothing in the way, and a local shell: that is what makes the anchor more than a computer. 24 August: the local model on the homelab describes the doorbell footage frame by frame, no data leaves the house Glass Mid August I watched the introduction of the new Omarchy release. Beautiful work, and the most important features I already had for years: a window manager and integrated tooling. The agent integration is nice, but not fundamentally different from what I do myself. Eye candy is nice, but not a money mover. The existing gear has stretch, innovation is software. Sixteen months ago I explained why I chose a Framework laptop for that gear. So why is there a second Framework on my desk, a twelve inch one? Because of a strict job description. It is not a second brain, it is the agentic workbench: persistent split screens for several agents at once, closed and reopened with one keystroke, and no local computing on it. Glass never becomes anchor. It runs Omarchy, it booted with a green W, and it cost a fraction of the machine it looks at. The Framework 12 open on the table: modular, repairable, and deliberately just glass First boot after setup on 25 August: a green W on the workbench The twin On 22 August I ordered a backup computer and an internet line for Amsterdam. Yes, a fixed line: nine years after the day I killed my LAN and went mobile only, the cable is back, because a machine that receives backups every night needs a connection that is always there. Although I am no fan of clutter, this was a smart step, crucial even, for the data. The anchor has become so important that we must do this. It became a Framework mainboard in a small case with an eight terabyte disk, built on the kitchen floor on 26 August and ready to travel to Amsterdam. From its first night there, the anchor pushes its backups to it. Two houses, two copies, one brain. 26 August, the kitchen floor: the Amsterdam twin being built, a mainboard in a small case with the console on a monitor Infrastructure becomes concept The future? Infrastructure becomes subordinate to concept. Control over routers, devices and connections is scriptable. Logs are readable and concepts are translatable. The artefacts were there before: my own tablet OS on a Surface, the Gran Fondo app. They were not good enough, and I abandoned them. Now it is different, because I can realise things much faster. A custom launcher for my phone, with all its compliance, I would never have built for fun. This month I did. That is the next post. Conclusion One brain, a few pieces of glass and a twin in Amsterdam. The hardware decisions of this month were the small part. The big part is that a machine with agents, logs and a local shell does in a day what used to take weeks, and that the same setup now stands in two houses. Innovation is software. So before you order a faster machine: give the one you have some agents and a look at its logs. It has more stretch than you think, ha! Back at work: the workbench with the mechanical keyboard, the brain out of sight

4 weeks ago 1 votes
Counting Agentic Traffic

This month I brought Lemmid Count back: the website statistics of my platform, rebuilt from scratch without AWStats and without a database. Its party piece is the dimension every other analytics tool throws away: the bots. In the agentic era the question "what did AI agents read on my site" matters as much as "how many humans visited". This post is about counting what pretends not to be there, and about putting the answer in your customer's pocket. Google Analytics is blind here Google Analytics works by placing a piece of JavaScript on your pages. A browser runs it, the script reports home, and that report is your statistics. I explained the difference with log-based tools back in 2018. In 2026 that difference has become the whole story. GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and their friends fetch your HTML and read it. They do not run your JavaScript. They never appear in Analytics. If the agentic web is where your next customers come from, your dashboard is showing you a shrinking world. The web server sees everything. Every request, every user agent, every status code, JavaScript or not. In July my site hartvaat.nl served 114,883 pageviews. 2,187 of them were humans. Analytics would have shown you the 2,187 and called it a quiet month. hartvaat.nl in July 2026: 114,883 pageviews, 2,187 by humans, 112,696 by bots; Analytics would have shown the small number Logs, not scripts Lemmid Count v3 reads the per-site logs of the web server directly, every hour, and folds them into small JSON files per site per day. No database anywhere, no tracking script on the pages, no third party at runtime: the visitor is never touched. That also fixes what killed the previous version: AWStats pumping a near-raw copy of the logs into MariaDB forever. Now the rotated logs are the raw store and the summaries are the product. Two principles. Pages, never people: no visitor counts, no uniqueness tricks, no IP addresses on disk, the browser language instead of geo. And auto configuration: a site with a log file has statistics, nobody has to register anything. Every site on the platform runs along automatically. Classify, do not filter Every request becomes either human or bot, and a bot gets a family and a group: search, ai, social, seo, monitor, feed, lib, scraper. Declared bots are named by their user agent. Instead of filtering them out, Count shows them as first class citizens: a "read by AI agents" card with the agents by name, and a toggle between all, humans and bots. That is new, and it makes people look up. On a restaurant site, in the first days of July, ChatGPT had read 75 pages and Claude 51, next to 185 human pageviews. All, humans or bots: one restaurant site in the first days of July, 185 human pageviews next to 1,171 by bots The humans view: top pages of a cafe site, filtered in one tap The pretenders Labeling is difficult, because some bots pretend to be human. A current Chrome user agent, HTTP/2, a plausible language header, one page per IP address, thousands of addresses from cloud providers. The user agent is worthless there. Behaviour is not. Real browsers load images. Real visitors arrive from search and links, and go to the same popular pages. A crowd that visits thousands of distinct pages, each exactly once, is not a crowd, it is a harvest. Hosting more than a hundred websites is the structural advantage: a signature that shows up on many sites within one hour is not a person. Add honey pots, pages no human would ask for, and address space from datacenters as a hint, never as a verdict. The classifier is versioned, and after every upgrade the last two weeks of history are recomputed. That matters, because the other side moves. One fleet answered a new rule within a day by fetching a single CSS file as an alibi. The next version asked for images. I will not list every signal here, that is the point of a honey pot. Read by AI agents, by name: chatgpt-user, claudebot and the stealth group of pretenders on the same site Full circle: in your pocket Here is the thing: detailed statistics are not unique. Anyone with logs and patience can count. The other half of the job is translating it back into something friendly, pocketable and actionable. Count is an app my customers carry in their pocket, next to the other Lemmid apps: one screen per site, the month in one chart, the top pages, the sources, and the agents by name. No login rituals, no setup, it simply appears for every site you own. The feedback is the best part. A customer showing the stats during the coffee round, exactly the use I had in mind years ago. A cafe owner seeing that ChatGPT read the menu seventy times this month, and drawing the obvious conclusion: put the menu where the agents read it. That is a statistic turned into an action, on a phone, in ten seconds. The pocket app: Manager, Count, Handler, User and Bill, one home screen for every site you own Conclusion The share of humans on the web will keep falling, and a tool that only counts humans will keep telling you the wrong story. Count the agents, name them, and hand the answer to the person who can act on it. High tech in the log parser, low tech on the screen. The bots read. You had better make sure they read the right thing, ha! Midnight on 6 July, the night Count came back, terminal on the iPad

6th Jul 2026 1 votes
Six Websites in Ten Days

Halfway June I came back from Amsterdam buzzing. A week and a half later six websites were live on a publisher that did not exist before: MP1, a static site generator that lives as a ZIP file inside my own CMS. This post is about what made that possible: automating the dull work with AI agents, so my time goes to the things that matter. It was exhausting, exhilarating and highly rewarding. Amsterdam energy Those days, and nights, were completely nuts, as I wrote in my notebook on 13 June. I had been in Amsterdam and was caught by the energy of real entrepreneurs, and by my own way of doing things from before the move to Limburg. It led to a list: a friend on board with five websites, another back on board with one, new connections with creative people in the Amsterdam nightlife, and Lemmid Handler, my order handling app, in production. Building sites for hospitality in Amsterdam is the Champions League of webdesign, I knew what I was getting into. The dull work Every website needs the same boring things: a web server configuration, a certificate, a place to upload, a way to publish. I have done this by hand a hundred times. In one weekend that became managed-web-configs: an nginx configuration with certificate, created in production fully automatically. Add a queue for downloads and the distribution of the engine itself, and the boring part is gone. Not because it is unimportant, but because it is the same every time. That is exactly what computers are for. MP1 The Lemmid Manager is the CMS I built over the past decade: content, images and settings for every site live there. The proof of concept for a new generation publisher, MP1, turns the generator into a ZIP file that is stored in the site's own configuration. Check it out, change it, check it in. On save the Manager unpacks the ZIP, runs the build and syncs the result to the server. No generator lives on a laptop anymore, the Manager holds the only copy. An AI agent works inside that same loop, on its own working copy, and reports back. This is the report I found on my terminal at 07:41 one morning: checked out, changed, verified, checked in, dress rehearsal passed. The video reel on the homepage now comes straight from the Manager at build time. The agent's report on my terminal at 07:41 on 13 June: checked out, changed, verified, checked in, dress rehearsal passed Rapid iteration The effect of embracing AI is visible: complex challenges can be settled step by step. It is the agentic loop at full speed: try, look, improve, again, while the agent does the checking. willem.com, sjocombinatie.nl, k2amsterdam.nl, fishfries.nl, centrumbar.nl and lammetje.com, in ten days. Each one a real site for real people, with its own design, photos and story. The best part is where the freed up time went: straight into the things that matter. The design, the words, the photos, the conversation with the owner about what the place is really about. That is the work only a human can do, and now there is more room for it. k2amsterdam.nl on MP1: two seasons, one party lammetje.com: come by 't Lammetje, photo wall included fishfries.nl: what the guests say, in English and Dutch centrumbar.nl: brown cafe by day, party by night (in development) The reaction of one of the owners, on the evening the site went live, says it all. And my own site got the same treatment: willem.com is on MP1 too, including the Dutch edition you are maybe reading right now. The reaction of one of the owners on the evening the site went live willem.com itself on the new publisher: Hallo, ik ben Willem Conclusion On 23 June, late at night, I wrote in my notebook: "What a week and a half! We thought up MP1 and built it. It is insane! Now sleep." That is the honest summary. It is exhausting, exhilarating and highly rewarding at the same time: delivering cool and amazing things to more people. The ambition has not changed since I wrote about staying a company of one: build systems, not projects. What changed is the speed. Automate the dull work and the rest of the day is yours, for the work that only you can do. Ha! The load monitor on the evening of 23 June, during the batch publish of six sites

23rd Jun 2026 1 votes
Elfstedentocht on Feel

Last Monday I rode the Fietselfstedentocht, the hottest edition ever, on my fixed gear bike. No bike computer, no GPS, no heart rate on my wrist. Just a mechanical watch, a stamp card and my WHOOP quietly recording in the background. Eighteen months ago I wondered whether I could ever reach peak performance without data. This ride was the answer, let me tell you about it. Warming up in a storm Two weeks before the tour I cycled from Amsterdam back home to Hegelsom, the reverse of the ride I described in 2023. Before I started I doubted for a moment whether I wanted a GPS track after all, maybe some power data. I decided not to and that was the right decision. It became my fastest crossing ever: 6 hours and 19 minutes, without a screen telling me so. On the ferries I talked to people: a guy in the same Rapha kit returning from a crash, two men who had cycled to Rome. The story, the photos and the WHOOP heart rate afterwards are more than a dataset. The feeling of my body on the bike became even more high definition than before. I knew I could accelerate up a bridge. Or into the wind. Or overtake someone. And that in intense rain. Numbers are followers: they describe reality after it happened. The feeling tells me more about what is possible next. Storm front over the dike road on the way from Amsterdam to Hegelsom, 13 May 2026 The Schindelhauer Siegfried Road after arrival: fastest crossing ever, 6h19m, no screen The tour I have ridden the Elfstedentocht many times, but this was the hottest one ever. A very intense ride, ridden entirely on feel. Many stretches fast, firm and long. Everything, the bike, the watch, the WHOOP and the drinking system worked within spec. On a fixed gear you have exactly one gear, I wrote about that before. Several times the cadence was the limit: the trains of riders went faster than I could spin, over a hundred revolutions per minute, without hammering through my stamina like a madman. Into the headwind speeds dropped and I could stay within my cadence. Funny enough, but not unexplainable, my folding bike with its wide gear range is in a way more powerful than my race bike. I think what I experienced is a limitation by design: not having certain things means, literally, that you can not do certain things. Within that frame I operated at the maximum. Faster than 80% of the people, slower than the carbon trains. Saddle, temperature and posture were the biggest limiting factors, and those are not improved by data, only by training. My conclusion: if I had been twice as fit, I would not have gone twice as fast. The stamp card of De Friese Fietselfstedentocht 2026, Frisian flag on the gloves Somewhere between the eleven cities, ridden on feel More data would not have helped More, or other, data would not have made a difference for the performance, at most for bragging rights. I know I cycled at over 30 km/h, but with the "kluun and chat time" (Dutch for walking your bike, and talking) the average even on the fast segments is 25 km/h-ish. For post-analysis a GPS track would have been helpful, a power curve maybe. But the stamp card holds the time of every checkpoint, so the segment speeds came out anyway, on paper. Last year I had all the data and I did not care about it. Now, without data, I was much more aware of my performance. I saw it during the ride too: trains of people riding fixated on, presumably, an arbitrary number on their screen. I passed them at 4 to 5 km/h faster. And not having to manage the data collection was wonderful! No charging, no pairing, no syncing. The WHOOP was not even empty after the trip, it recorded the whole day, the recovery and the journey home. Breathing turned out to be the most important indicator of what is sustainable, I can cruise on an "in-in-out-out" rhythm. Heat is the other one, as I learned running the Leiden Marathon. Temperature is something you train for too, and I have come a long way there. Segment speeds reconstructed from the stamp card times, no GPS needed My only instrument on the wrist: a mechanical watch (and the WHOOP on other arm) Conclusion At the finish tent, big, seasoned men gave me a fist bump and a compliment on the bike and the pace. They could not keep up and the performance was "with style", they said. That is worth more than any Strava segment. In Data versus Feeling I wrote that I hoped to know myself and my body well enough to perform without data. This tour was that moment. It is not that data is bad, it is that the feeling turned out to be the richer signal. Next up: a marathon in Cologne, on feel. Ha!

26th May 2026 1 votes

More in technology

Is This A Joke? In The Auth Header? (F5 BIG-IP UnAuth Heap-Overflow to RCE CVE-2026-94127)

Well, well, well, well, well, well, well, well, well, well, well, well, well, well, well. We're back. Sorry. We've been watching the onslaught of vulnerabilities flood the internet. Every man, dog, and their grandmas (apparently?) are now using LLMs to find and reproduce vulnerabilities - it’

15 hours ago 1 votes
The Reason You Prohibit Things

You want less of them. That’s the reason. You may find that it’s too hard to stop people from doing the thing, literally blood, sweat, and tears trying to prosecute people, but that’s a different thing.

17 hours ago
Solitaire Alone Together

Solitaire Alone Together I made a new game. It's called Solitaire Alone Together. It's Windows 98 solitaire, but you can play with everyone else on the internet. Read the full post on my blog! Here's a raw link, if you need it: https://eieio.games/blog/solitaire-alone-together

3 days ago 1 votes
All The Ways I Broke My Website

This post is a living diary of all the times I messed up something with my website in a funny way. I value those who have the confidence to own their mistakes and share the learning with others, and so this is me doing just that! That Time I Accidentally Made a Tarpit That Time I Accidentally Made Really Large Headers That Time I Accidentally Made a Tarpit Back to Top A "tarpit" is an unofficial term used in computing to describe an intentionally slow response to a request. In these modern times many people are using tarpits as a way to combat the relentless theft of data by AI companies, although there's little to no evidence of that actually being in any way effective. I don't use tarpits, at least not intentionally, but there was that one time when I accidentally created a tarpit and trapped all visitors in it. As I've shared previously, I refuse connections from IP addresses that are blocked or belong to a blocked subnet, and I enforce this firewall during the TCP handshake. The logic here is straightforward: there's no reason to waste resources doing a TLS handshake, accepting an HTTP request, and then rejecting the connection if I already know I'm going to reject it at the earliest step. At the time, the code worked like this: the HTTP server would repeatedly call the Accept() function below expecting a new connection. I've added some comments to help explain the logic. func (l *firewallListener) Accept() (net.Conn, error) { // Accept the connection from the TCP listener. This blocks until there is a connection to accept or the listner was closed. conn, err := l.l.AcceptTCP() if err != nil { return conn, err } // Separate the IP address out from the remote address (which includes the port) ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } // Check if it's blocked, if so close the connection and return a refuseError if IsBlocked(ip, true) { conn.Close() return nil, &refuseError{} } // Otherwise return the connection on to the HTTP server return conn, nil } If the incoming connection was from a blocked IP then I'd return a refuseError. I need to use a specific error interface because the HTTP server will halt if it encounters a non-temporary error from the call to Accept(), so I need to return an error that satisfies the definition of a temporary error. I defined refuseError like this: type refuseError struct{} func (e *refuseError) Error() string { return "." } func (e *refuseError) Timeout() bool { return true } func (e *refuseError) Temporary() bool { return true } func (e *refuseError) Is(err error) bool { return err == context.DeadlineExceeded } This did accomplish the goal of rejecting connections before the TLS handshake for blocked addresses, but it had one really unintended and difficult to track down side-effect. Accepting connections is done serially, after which servers typically then process that request on a dedicated thread (or in Go's case a goroutine). This means that any delays during the accept loop will block all incoming connection. What I had missed while reviewing the code for Go's HTTP server is that when it receives a temporary error from Accept() is that while it doesn't abort, it does sleep for up to a maximum of 1 second. This sleep blocks the entire server for all incoming connections. You can see a trimmed copy of the code that does this below, with some marks I've added which I will explain. // src/net/http/server.go // Copyright 2009 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. for { // (1) rw, err := l.Accept() if err != nil { if s.shuttingDown() { return ErrServerClosed } // (2) if ne, ok := err.(net.Error); ok && ne.Temporary() { if tempDelay == 0 { tempDelay = 5 * time.Millisecond } else { tempDelay *= 2 } if max := 1 * time.Second; tempDelay > max { tempDelay = max } s.logf("http: Accept error: %v; retrying in %v", err, tempDelay) // (3) time.Sleep(tempDelay) continue } return err } connCtx := ctx if cc := s.ConnContext; cc != nil { connCtx = cc(connCtx, rw) if connCtx == nil { panic("ConnContext returned nil") } } tempDelay = 0 c := s.newConn(rw) c.setState(c.rwc, StateNew, runHooks) // before Serve can return // (4) go c.serve(connCtx) } At mark 1 the server calls the Accept() function, this is the exact function that I defined above where I might return a temporary error. At mark 2 it checks if an error was returned, and if so if that error is temporary. If there was a temporary error, at mark 3 it sleeps for an increasing amount of time up-to 1 second, otherwise, at mark 4 it processes the connection on a dedicated goroutine, which allows the server to accept the next connection. I'm not entirely sure why the Go developers added this sleep delay and the change when it was introduced doesn't provide any meaningful insight. Regardless, it caused significant latency connecting to my website when a flood of rejected requests was coming in. It just goes to show how important it is to write meaningful commit messages, because you never know when somebody might come back years later wondering "why was this done?". I sure home I don't come to eat those words later. Coincidentally, you can actually see this happening if you look carefully at one of the metric graphs I shared in my first post about my server's security model: Securing My Web Infrastructure. This is the graph I shared in that blog post and while I didn't know it at the time, the fact that these request spikes all cap-out at around 60 requests per minute was not a coincidence. These requests were not being made with a limit in mind, attackers rarely ever care about things like that, instead it the accidental tarpit I had created. The downside to this was that while the malicious requests were being rate-limited, all requests were being rate-limited, up to a point of taking so long they timed out. The Fix Fixing the issue was relatively straightforward enough. Instead of returning a temporary error to the HTTP server during the accept loop, just don't return anything at all and wait for the next valid connection. func (l *firewallListener) Accept() (net.Conn, error) { for { conn, err := l.l.AcceptTCP() if err != nil { return conn, err } ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } if IsBlocked(ip, true) { conn.SetLinger(0) conn.Close() continue } return conn, nil } } Now, when the HTTP server calls Accept(), the only time it returns is with a connection from an IP that isn't blocked, or if there genuinely is an error. No more sleep delays, no more excessive timeouts. That Time I Accidentally Made Really Large Headers Back to Top For about 10 years now all major browsers have support for a security feature known as a Content Security Policy or CSP. A CSP is an HTTP header provided by the server that instructs the browser on where it can load assets from, this could be scripts, images, stylesheets, fonts, etc. The objective of using a CSP is to prevent against injected HTML that tries to load assets, such as a malicious Javascript file, from a remote source. With so much user-provided content being available online, it's very possible for this to happen without an attacker compromising the entire web server. CSP protects against that by saying "scripts can only be loaded from these domains". That's a really simplified way of looking at it, anyways. My web server supports injecting the CSP header automatically, but before I go on I need to explain a little bit about the structure of my web server. When an incoming HTTP request is accepted (having passed all firewall checks and assertions), we look at the destination host for the request. This can either be the value of the Host header or as specified during the TLS handshake. We then look at a map of hosts to apps. Apps are just an interface that accept a few methods: type App interface { Cleanup() ReloadConfig() ServeHTTP(rw http.ResponseWriter, r *http.Request) Setup(dataDir string) error Shutdown() } One of the apps is the Proxy app, which is a reverse proxy - it accepts the incoming HTTP request and then proxies it on to another host. This is a very common design, especially with increasingly complex TLS setups. Because each app is unique to a host, and different hosts have different requirements for CSP rules, the proxy app includes a CSP preset that we use to build the header value, or skip it entirely. When the proxy app was going to copy an HTTP request to the downstream host, it would build the CSP header, however there was a slight bug... func (a *App) ServeHTTP(rw http.ResponseWriter, inRequest *ht2.Request) { // --snip -- if a.CSP != nil { a.CSP.ConnectSrc += " " + inRequest.Origin } CopyHttpRequest(inRequest, outRequest, rw, CopyHttpRequestOptions{ Origin: inRequest.Origin, Csp: a.CSP, Cors: a.CORS, AddHeaders: !a.SkipHeaders, UseHTTP3: a.UseHTTP3, InsecureTLS: a.InsecureTLS, }) } I'm really unsure as to what I was doing with the line to append to the ConnectSrc, but the impact is that I'm appending to a variable that lives on the App, rather than a variable that is per-request. This meant that every time there was a request to the app, any request at all, the origin would be appended to the header value. This went on for quite a long time unnoticed and unresolved, largely because I am constantly tweaking and tinkering with my web server, after all, it's how I made having a website fun again. Each time I restarted the server process, the header value would be reset, but only for it to continue to grow and grow. Eventually, after a period of being busy with other matters, the server process stayed running for long enough that the header value grew too large and HTTP clients began to reject it. There is no defined maximum for an HTTP header value, however most HTTP clients use 100KiB, which is perfectly reasonable, and this header value would continue to grow well beyond that. Diagnosing this issue turned out to be difficult as tools like Curl would fail with errors relating to entities being too large, but stopped short of saying what specifically. I eventually used openssl s_client to send an HTTP request by hand and observed my terminal window being filled with a domain name repeated thousands of times. Looking at the commit history, it was really unclear why I added the culprit lines of code. The commit message just says "Improved CSP support". It just goes to show how important it is to write - hey look it's those words I'm now having to eat! The Fix The fix was to just delete those three lines of code. Yup, it really was that simple, and fixing this bug actually made a larger positive impact than I had expected, as it was immediately clear when I fixed the bug by looking at outbound network bytes: So much traffic was being wasted on excessive header sizes. You might look at these mistakes I've made and think "wow, Ian, these are some obvious mistakes, I never would have made them!" to which I say "good for you!" with the utmost sarcasm and disdain. I enjoy making and refining software, and making anything means making mistakes along the way. Each time I make mistakes such as the ones above, I improve my skills of investigation, diagnosing, and repair. Skills that, judging by my peers in the industry, seemingly everyone is quickly willing to throw away because a robot does it "better" than you. Header Image: "Car accident on the Ffestiniog to Bala road. Nobody was hurt" by Geoff Charles, CC BY-SA 4.0, via Wikimedia Commons.

5 days ago
Buienwatch, a custom Home Assistant integration for Buienradar and Buienalarm graphs on your Apple Watch

Back in 2021, I wired up data from Buienalarm and Buienradar through Node-RED and a big pile of Jinja2 templates to get a rain forecast graph on my Apple Watch: eight Unicode block characters showing the next two hours in 15-minute chunks, glanceable without unlocking the phone. It worked, and I used it every day […] The post Buienwatch, a custom Home Assistant integration for Buienradar and Buienalarm graphs on your Apple Watch appeared first on Style over Substance.

a week ago 0 votes
📚 BoredReading

You seem to be enjoying this.

Join free to unlock everything.

Create free account

Already have an account? Sign in