Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Tony Finch's blog
How can something that “just works” be so annoying? situation We live in Cambridge off a little road down a drive in shared ownership between us and our neighbouring houses. All the utilities are buried under this drive, including the phone line. anticipation Over the last few years we have been canvassed repeatedly by CityFibre saying that they can deliver fibre all way to our house. I saw them digging trenches and leaving tails of purple fibre cladding along nearby roads, ready to hook up all the houses. I thought they would need to do something similar to deliver fibre to us. So when they turned up and knocked on our door, I talked to their salesbods and walked them up and down the drive and pointed out where the existing BT line goes. Then they gave up trying to sell to us. This happened about three times. disaffection We were not eager enough for an upgrade to deal with these impediments. notification A few months ago we were told that CityFibre would soon come and do the upgrade, since there’s a nationwide deadline for turning off the copper phone network at the end of the year. We expected that this would force them to actually plan some digging works, so we talked to our neighbours about it. We were all ready for some huge faff to follow the next visit by the CityFibre bods. installation CityFibre turned up on the promised morning bright and early. To our enormous surprise, a brown fibre housing was already poking out of the ground next to our copper phone line. It had been fed through 50 metres of 5cm duct without us being aware they were even working on the street. Within a couple of hours, the technicians had drilled through our wall, installed the ONT, blown fibre through the unexpected pipe, plugged in the CPE (superficially identical to the old one), and left telling us to anticipate that it might not work properly until tomorrow. activation Around lunch time, the copper phone line stopped working completely. Some faff ensued, switching all our devices over to the new WiFi network. For a while we thought this was the death of our land line, but in the course of debugging other issues, I realised that the router has a built-in VoIP adapter (I don’t think we were told it has a built-in VoIP adapter) so I plugged the phone in and it Just Worked: they had ported our phone number across and everything. Flawless. I was seriously impressed. rumination It has been a few weeks since the switchover, and apart from a couple of horrible Clown-afflicted IoT devices, it has been fairly smooth. What prompted me to write this up was realising that we delayed this upgrade for years because the sales people were not given enough technical information about how the installation process works: the fact that houses typically have a 5cm duct containing the copper lines (probably standard for the last 40 years) and the fact that fibre can be shoved through a few tens of metres without difficulty. And worse, the sales people didn’t have an esclation path for difficult cases: they just gave up instead. From a technical point of view, the installation was impeccable. (I guess the loose 24 hour window for the cutover time was because OpenReach and CityFibre don’t have tight requirements on ISP reconfiguration schedules.) From the sales point of view, it was crap. Maybe it would have gone faster if we offered to switch early without asking if the drive would be a problem? But I guess the difference between “yes!” and “yes, but will this be a problem?” is too much to expect from a minimum-wage door-to-door salesbod whose employer didn’t give them enough information or any escalation path.
Many years ago I wrote about how to convert Gregorian dates to Julian Day numbers or similar counts such as rata die as used in Calendrical Calculations. This algorithm is the core of C’s mktime() function that converts a broken-down date-time into linear time_t. I recently learned from Ben Joffe that I was missing a few tricks, and my old code wasn’t as good as it could have been. Here’s a better version (using conventional not C numbering): if m > 2 { m -= 2; } else { m += 10; y -= 1; } y*365 + y/4 - y/100 + y/400 + m*979/32 + d - 336 the main idea Julian years Gregorian correction the month pattern the epoch domains and ranges leap year test length of month the main idea There’s a helpful coincidence in the Gregorian calendar. Although the month lengths aren’t obviously regular, there’s a repeating 5 month pattern that becomes easier to see when you start from March, as illustrated by the table below. This pattern resets at the end of February, midway through its third repeat, coincidentally at the same point that leap days occur. Thus the first line of the code above adjusts the month and year numbers so that January and February are counted at the end of the previous year, and the coincidental alignment occurs at the boundary between the adjusted year numbers. I’ll explain the details of the adjustment as I discuss the relevant parts of the second line 31 days April 30 days May 31 days June 30 days July 31 days 31 days September 30 days October 31 days November 30 days December 31 days 31 days February 28 or 29 Julian years The first part of the main formula counts the number of days before the start of year y, in terms of normal years and leap days. y * 365 + y / 4 The adjustment subtracts one from the year in January and February. The effect is that the leap day in year 4 is counted as a day before the start of the adjusted beginning of year 4, i.e. before March, i.e. exactly the right place. I previously combined this part of the expression into a single term, y * 1461 / 4 Ben Joffe pointed out that when it is written this way the function is only able to make use of 25% of the range of its output data type, because the multiplication overflows for very large year numbers. And on modern CPUs it isn’t actually faster to eliminate the addition. Gregorian correction The next part corrects the number of leap years before the current year. - y/100 + y/400 It works in basically the same way as the Julian leap year calculation, but whereas y/4 trivially compiles to a simple shift operation, this needs a bit more cleverness. As Hacker’s Delight explains, a modern compiler will turn y/100 into a multiply-and-shift: y * (1<<N) * (1/25) >> (N+2) That is, the compiler uses a fixed-point representation of the reciprocal of the divisor. Then it uses common subexpression elimination to suppress the second multiplication by 1/25. So these two divisions are turned into a wide multiply and two shifts. Neat. the month pattern The next part counts the number of days in this (adjusted) year before the start of month m. m * 979 / 32 I previously wrote it using the number of days in the repeating pattern of 5 months, m * 153 / 5 But this is relatively difficult for compilers to optimize well (clang uses two multiplications instead of one), and they don’t know the range of m is limited, so we can do better by turning it into a multiply-and-shift by hand. 979/32 == 30.59375 which is close enough to the exact value 153/5 == 30.6 Either of these expressions produce the right 5 month long/short pattern, but the pattern doesn’t necessarily line up with the normal month numbering. (The two expressions above need different adjustments.) We move March to number 1, just before the start of the pattern. When counting the days before April, we get 31 more than the count for March; when counting the days before May, we get 30 more than the count for April, etc. January is adjusted to follow December to match the adjusted year numbering. m *979/32 diff -------------------- 1 30 March 2 61 31 April 3 91 30 May 4 122 31 5 152 30 6 183 31 7 214 31 8 244 30 9 275 31 10 305 30 December 11 336 31 January 12 367 31 13 397 30 the epoch Because calendars count from 1, the Gregorian date 0001-01-01 gets numbered rata die 1. The adjustments turn January into month 11, and so (as in the second column in the table above) we count 336 days in the adjusted year 0 before January. We need to subtract those extra days to compensate for the adjustment. We can change the offset to choose a different epoch, e.g. the MJD epoch 1858-11-17 is r.d. 678576, and the Unix epoch 1970-01-01 is r.d. 719163. domains and ranges In my old C code I casually used int, which misleadingly implied that it worked with proleptic Gregorian calendar dates before year 1. However signed division and modulus on common CPUs and low-level programming languages truncates towards zero, but this algorithm needs Euclidean or flooring division (which are equivalent for positive divisors). So it’s better to use u32 for these calculations. (Compilers also do a better job when this code uses unsigned integers.) To support negative years, a multiple of 400 years can be added to move year 0 to the middle of the u32 range, and subtracted from the return value to produce a signed count of days. leap year test Ben Joffe also examined fast leap year tests. My new favourite one is I think the neatest if not the fastest: if y % 25 == 0 { y % 16 == 0 } else { y % 4 == 0 } Note that 25 * 16 == 400, so it’s a leap year if it’s divisible by both 25 and by 16, else if it’s divisible by 4 but not by 25. The classic version of this check first tests divisibility by 100. CPUs that rely on branch prediction will correctly predict it 99% of the time: much better than the 75% you get from testing divisibilty by 4 first! But nowadays this if is compiled into a CMOV or CSEL (so branch prediction doesn’t matter), and divisibility by 25 is easier to compile than divisibility by 100. length of month What prompted me to revisit this code was the idea that it’s possible to work out a simpler multiply-and-shift optimization when the values have limited ranges (as in m*979/32 above) and/or when we don’t depend on the exact result of the multiplication. I previously wrote this code for calculating the length of a month: if m == 2 { 28 + is_leap_year(y) as u32 } else { 30 + (m * 275 % 9 > 3) as u32 } The multiply-and-shift idea led me to this replacement for months other than February: 30 + (m * 7 % 16 < 9) as u32 I found it by writing a brute force program that tries successive bitmask widths and multipliers, until it finds a case where all the long months produce results greater than all the short months, or vice versa (as in the winner). But there’s a neater expression, apparently due to Dr Matthias Kretz: 30 | (m ^ (m >> 3)) This uses two tricks: The 1 bit of the month number matches the odd/even long/short pattern in the months before August (month 8), when the phase flips. The flip is done by using the 8 bit to toggle the 1 bit. Bitwise or with 30 sets bits 2, 4, 8, 16 so the higher bits of the month number don’t matter. It compiles to just two ARM instructions: eor w0, w0, w0, lsr #3 orr w0, w0, #0x1e It’s so sweet I actually love how much better it is than my attempt!
A few weeks ago I was enjoying a couple of boiled eggs (in the shell, with plenty of salt and pepper, and buttery fingers of toast to dunk into the runny yolk) and pondering how fiddly it is to cut off one end of the shell after boiling compared to eating a poached egg. And I was annoyed because (I thought) I didn’t know how to poach eggs. misconceptions For decades I have been under the impression that poached eggs are difficult, because cheffy bods on the telly make such a fuss over cooking them. They led me to believe two falsehoods, both of which have a grain of truth, but it turns out they are not the overwhelming obstacles I thought. I decided to see what happens if I just don’t do any of the chef tricks. How bad could it be? If the poached eggs turned out to be a disaster, I would at least have confirmed what I believed. If not, I have added some delicious food to my repertoire. What did I believe? And what did I learn… If you just break an egg into boiling water, it’ll dissolve into a soupy mess? Yes the egg will spread out and the water will get messy, but almost all of the egg will hold together neatly by itself. If you don’t do the cheffy faff, your poached eggs will be inedible? In fact the fuss is mostly about levelling up from basic to restaurant-standard presentation. A no-fuss but frilly egg is still nice to eat. basic tricks The key trick for boiling an egg is to have plenty of boiling water in the pan before adding the eggs. That gives you a stable temperature and therefore predictable cooking times. To boil large hen eggs from room temperature, I aim for about 4 minutes for a runny yolk, or 8 minutes for a slightly fudgy hard-boiled yolk. For poached eggs, the water should be at a gentle simmer to avoid agitating the wispy white more than necessary. I add plenty of salt for seasoning. A cooking time of 3 minutes is about right. The key trick for poaching eggs is not to worry about the wispy whites in the water. It might be messy but it’ll be fine. Break the eggs near the surface of the water so they aren’t agitated too much from plunging in. When the surface of the white has started coagulating, give them a nudge to make sure they are moving enough to cook evenly and aren’t stuck to the bottom of the pan. Unlike a boiled egg, I can lift the poached egg out of the water with a slotted spoon and jiggle it to judge when it is ready, which is better than relying on my oven timer that can only be set in increments of a whole minute, and handy when I forget to set it… cheffy faff There are about half a dozen ways to improve the presentation of poached eggs. Use very fresh eggs, because they hold their shape better with less wispy white. I only have supermarket eggs, so egg age is not something I can control. Use a fine mesh strainer to separate the loose wispy white from the firm inner white. I think watching this trick taught me that raw eggs are a lot more cohesive and robust than I thought, and they don’t just dissolve into water. Get the water spinning like a vortex before adding the egg. Helen Rennie has a video on poaching eggs in which she discusses why this method is better in a restaurant (after about 6m10s). It requires a very large pot so it isn’t ideal for cooking a few portions at home. Break the egg into a small dish or ladle, so it can be introduced to the water more gently. This is worth doing but nevertheless I don’t bother :-) Add a little vinegar to the water. I don’t believe this has any effect on how the whites spread. It’s possible to use vinegar to coagulate the whites before poaching, but this requires a lot of vinegar and takes a long time, and harms the flavour of the eggs. Most of the spreading of a poached egg is due to turbulence when it plunges into the water, and it happens far too fast to be affected by a little vinegar. Wrap the egg in cling film. I don’t care enough about how neat my eggs are to fiddle with throwaway plastic and risk spilling egg in a clumsy mess. We have some reusable silicone things that cook eggs in a style somewhere between poached and coddled. We almost never use them because they are fiddly and tend to undercook the tops of the eggs even with a lid to keep the steam in. Trim the egg with a knife after cooking. Almost as wasteful as the strainer method! Or… don’t! thoughts In retrospect it’s curious that I was discouraged from even trying to poach eggs for such a long time, and that it took so little to discourage me. I suppose it illustrates how offputting extra steps can be to a beginner. It wasn’t clear to me which steps were optional and what were the consequences of omitting them. It’s something to keep in mind when writing documentation, I guess :-)
A while back I wrote about the linear rate limit algorithms leaky bucket and GCRA. Since then I have been vexed by how common it is to implement rate limiting using complicated and wasteful algorithms (for example). But linear (and exponential) rate limiters have a disadvantage: they can be slow to throttle clients whose request rate is above the limit but not super fast. And I just realised that this disadvantage can be unacceptable in some situations, when it’s imperative that no more than some quota of requests is accepted within a window of time. In this article I’ll explore a way to enforce rate limit quotas more precisely, without undue storage costs, and without encouraging clients to oscillate between bursts and pauses. However I’m not sure it’s a good idea. linear reaction time fixed window quota resets hybrid quota-linear algorithm discussion opinion linear reaction time How many requests does a linear rate limiter allow before throttling? The parameters for a rate limiter are: q, the permitted quota of requests w, the accounting time window So the maximum permitted rate is q/w. Let’s consider a client whose rate is some multiple a > 1 of the permitted rate (a for abuse factor) c = a * q/w I’ll model the rate limiter as a token bucket which starts off with q tokens at time 0. The bucket accumulates tokens at the permitted rate and the client consumes them at its request rate. (It is capped at q tokens but we can ignore that detail when a > 1.) b(t) = q + t*q/w - t*a*q/w The time taken for n requests is t(n) = n/c = (n*w) / (a*q) After n requests the bucket contains b(n) = q + n/a - n The rate limter throttles the client when the bucket is empty. b(t) = 0 = q + t * (1 - a) * q/w 0 = 1 - t * (a - 1) / w t = w / (a - 1) b(n) = 0 = q + n * (1/a - 1) 0 = q - n * (a - 1) / a n = q * a / (a - 1) For example, if the client is running at twice the permitted rate, a=2, they will be allowed q*2 requests within w seconds before they are throttled. That’s a bit slow. The problem is that the bucket starts with a full quota of tokens, and during the first window of time another quota-full is added. So a linear rate limiter seems to be more generous in its startup phase than its parameters suggest. fixed window quota resets This is a fairly common rate limit algorithm that enforces quotas more precisely than linear rate limiters. It is similar to a token bucket, but it resets the bucket to the full quota q after each window w. The disadvantage of periodically resetting the bucket is that careless clients are likely to send a fast burst of requests every window. (A linear rate limiter will tend to smooth out requests from careless clients, though it can’t prevent deliberately abusive burstiness.) And a quota-reset rate limiter uses more space than a linear rate limiter: it needs a separate bucket counter as well as a timestamp. each client has the time when its window started and a bucket of tokens if c == NULL c = new Client c.bucket = quota c.time = now reset the bucket when the window has expired if c.time + window <= now c.bucket = quota c.time = now the request is allowed if the client has a token to spend if c.bucket >= 1 c.bucket -= 1 return ALLOW else return DENY hybrid quota-linear algorithm The idea is to have two operating modes: Low-traffic clients are allowed to make bursts of requests, because that’s good for interactive latency. High-traffic clients are required to smooth out their requests to an even rate. The algorithm switches to smooth mode when a client consumes its entire quota and the bucket reaches zero, and switches back to bursty mode when the bucket recovers to a full quota. Bursty mode avoids being too generous by adding at most one quota of tokens to the bucket per window. Smooth mode also avoids being too generous by starting with an empty bucket. I’ve written this pseudocode in a repetitive style because that makes each paragraph more independent of its context, though it is still somewhat stateful and the order of the clauses matters. the rate limit is derived from the quota and window rate = quota / window a client that returns after a long absence is reinitialized like a new client; in bursty mode the time is the start of a fixed window and the bucket contains a whole number of tokens; we subtract one from the quota to account for the current request fn reset() c.bucket = quota - 1 c.time = now c.mode = BURSTY return ALLOW initialize the state of a new client if c == NULL c = new Client return reset() reset the state when a bursty client’s window has expired if c.mode == BURSTY and c.time + window <= now return reset() when a client consumes its last token, switch modes; in smooth mode the time is updated for every request and the bucket can hold fractional tokens; apply a negative penalty so we don’t allow any over-quota requests before the end of the fixed window; add one to allow the next request at the start of the next window if c.mode == BURSTY and c.bucket == 1 remaining = c.time + window - now c.bucket = -remaining * rate + 1 c.time = now c.mode = SMOOTH return ALLOW smooth mode accumulates tokens proportional to the time since the client’s previous request; update the request time so that tokens accumulate at the same rate whether the request is allowed or denied if c.mode == SMOOTH c.bucket += (now - c.time) * rate c.time = now when the bucket has refilled, switch back to bursty mode if c.mode == SMOOTH and c.bucket >= quota return reset() we have dealt with all the special cases; in either mode the request is allowed if the client has a token to spend if c.bucket >= 1 c.bucket -= 1 return ALLOW else return DENY discussion This hybrid algorithm has a similar effect to running both a quota-reset and a linear rate limiter in parallel, but it uses less space. The precision of quota enforcement in smooth mode is maybe arguable: It guarantees that the client remains below the limit on average over the whole time it is in smooth mode. But if it slows down for a while (but not long enough to return to bursty mode) it can speed up again and make more requests than its quota within a window. opinion I think it’s a mistake to try to treat each time window in strict isolation when assessing a client’s request quota. A linear rate limiter only seems to be unduly generous on startup if you ignore the fact that the client was quiet in the previous window. As well as being more expensive than necessary (in some cases disgracefully wasteful), algorithms like sliding-window and quota-reset encourage clients into cyclic burst-pause behaviour which is unhealthy for servers. And I’ve seen developers complaining about how annoying it is to use glut/famine rate limiters. By contrast, rate limiters that measure longer-term average behaviour by keeping state across multiple windows can naturally encourage clients to smooth out their requests. So on balance I think that instead of using this hybrid quota-linear rate limiter, you should reframe your problem so that you can use a simple linear rate limiter like GCRA.
More in programming
This post is a living diary of all the times I messed up something with my website in a funny way. I value those who have the confidence to own their mistakes and share the learning with others, and so this is me doing just that! That Time I Accidentally Made a Tarpit That Time I Accidentally Made Really Large Headers That Time I Accidentally Made a Tarpit Back to Top A "tarpit" is an unofficial term used in computing to describe an intentionally slow response to a request. In these modern times many people are using tarpits as a way to combat the relentless theft of data by AI companies, although there's little to no evidence of that actually being in any way effective. I don't use tarpits, at least not intentionally, but there was that one time when I accidentally created a tarpit and trapped all visitors in it. As I've shared previously, I refuse connections from IP addresses that are blocked or belong to a blocked subnet, and I enforce this firewall during the TCP handshake. The logic here is straightforward: there's no reason to waste resources doing a TLS handshake, accepting an HTTP request, and then rejecting the connection if I already know I'm going to reject it at the earliest step. At the time, the code worked like this: the HTTP server would repeatedly call the Accept() function below expecting a new connection. I've added some comments to help explain the logic. func (l *firewallListener) Accept() (net.Conn, error) { // Accept the connection from the TCP listener. This blocks until there is a connection to accept or the listner was closed. conn, err := l.l.AcceptTCP() if err != nil { return conn, err } // Separate the IP address out from the remote address (which includes the port) ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } // Check if it's blocked, if so close the connection and return a refuseError if IsBlocked(ip, true) { conn.Close() return nil, &refuseError{} } // Otherwise return the connection on to the HTTP server return conn, nil } If the incoming connection was from a blocked IP then I'd return a refuseError. I need to use a specific error interface because the HTTP server will halt if it encounters a non-temporary error from the call to Accept(), so I need to return an error that satisfies the definition of a temporary error. I defined refuseError like this: type refuseError struct{} func (e *refuseError) Error() string { return "." } func (e *refuseError) Timeout() bool { return true } func (e *refuseError) Temporary() bool { return true } func (e *refuseError) Is(err error) bool { return err == context.DeadlineExceeded } This did accomplish the goal of rejecting connections before the TLS handshake for blocked addresses, but it had one really unintended and difficult to track down side-effect. Accepting connections is done serially, after which servers typically then process that request on a dedicated thread (or in Go's case a goroutine). This means that any delays during the accept loop will block all incoming connection. What I had missed while reviewing the code for Go's HTTP server is that when it receives a temporary error from Accept() is that while it doesn't abort, it does sleep for up to a maximum of 1 second. This sleep blocks the entire server for all incoming connections. You can see a trimmed copy of the code that does this below, with some marks I've added which I will explain. // src/net/http/server.go // Copyright 2009 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. for { // (1) rw, err := l.Accept() if err != nil { if s.shuttingDown() { return ErrServerClosed } // (2) if ne, ok := err.(net.Error); ok && ne.Temporary() { if tempDelay == 0 { tempDelay = 5 * time.Millisecond } else { tempDelay *= 2 } if max := 1 * time.Second; tempDelay > max { tempDelay = max } s.logf("http: Accept error: %v; retrying in %v", err, tempDelay) // (3) time.Sleep(tempDelay) continue } return err } connCtx := ctx if cc := s.ConnContext; cc != nil { connCtx = cc(connCtx, rw) if connCtx == nil { panic("ConnContext returned nil") } } tempDelay = 0 c := s.newConn(rw) c.setState(c.rwc, StateNew, runHooks) // before Serve can return // (4) go c.serve(connCtx) } At mark 1 the server calls the Accept() function, this is the exact function that I defined above where I might return a temporary error. At mark 2 it checks if an error was returned, and if so if that error is temporary. If there was a temporary error, at mark 3 it sleeps for an increasing amount of time up-to 1 second, otherwise, at mark 4 it processes the connection on a dedicated goroutine, which allows the server to accept the next connection. I'm not entirely sure why the Go developers added this sleep delay and the change when it was introduced doesn't provide any meaningful insight. Regardless, it caused significant latency connecting to my website when a flood of rejected requests was coming in. It just goes to show how important it is to write meaningful commit messages, because you never know when somebody might come back years later wondering "why was this done?". I sure home I don't come to eat those words later. Coincidentally, you can actually see this happening if you look carefully at one of the metric graphs I shared in my first post about my server's security model: Securing My Web Infrastructure. This is the graph I shared in that blog post and while I didn't know it at the time, the fact that these request spikes all cap-out at around 60 requests per minute was not a coincidence. These requests were not being made with a limit in mind, attackers rarely ever care about things like that, instead it the accidental tarpit I had created. The downside to this was that while the malicious requests were being rate-limited, all requests were being rate-limited, up to a point of taking so long they timed out. The Fix Fixing the issue was relatively straightforward enough. Instead of returning a temporary error to the HTTP server during the accept loop, just don't return anything at all and wait for the next valid connection. func (l *firewallListener) Accept() (net.Conn, error) { for { conn, err := l.l.AcceptTCP() if err != nil { return conn, err } ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } if IsBlocked(ip, true) { conn.SetLinger(0) conn.Close() continue } return conn, nil } } Now, when the HTTP server calls Accept(), the only time it returns is with a connection from an IP that isn't blocked, or if there genuinely is an error. No more sleep delays, no more excessive timeouts. That Time I Accidentally Made Really Large Headers Back to Top For about 10 years now all major browsers have support for a security feature known as a Content Security Policy or CSP. A CSP is an HTTP header provided by the server that instructs the browser on where it can load assets from, this could be scripts, images, stylesheets, fonts, etc. The objective of using a CSP is to prevent against injected HTML that tries to load assets, such as a malicious Javascript file, from a remote source. With so much user-provided content being available online, it's very possible for this to happen without an attacker compromising the entire web server. CSP protects against that by saying "scripts can only be loaded from these domains". That's a really simplified way of looking at it, anyways. My web server supports injecting the CSP header automatically, but before I go on I need to explain a little bit about the structure of my web server. When an incoming HTTP request is accepted (having passed all firewall checks and assertions), we look at the destination host for the request. This can either be the value of the Host header or as specified during the TLS handshake. We then look at a map of hosts to apps. Apps are just an interface that accept a few methods: type App interface { Cleanup() ReloadConfig() ServeHTTP(rw http.ResponseWriter, r *http.Request) Setup(dataDir string) error Shutdown() } One of the apps is the Proxy app, which is a reverse proxy - it accepts the incoming HTTP request and then proxies it on to another host. This is a very common design, especially with increasingly complex TLS setups. Because each app is unique to a host, and different hosts have different requirements for CSP rules, the proxy app includes a CSP preset that we use to build the header value, or skip it entirely. When the proxy app was going to copy an HTTP request to the downstream host, it would build the CSP header, however there was a slight bug... func (a *App) ServeHTTP(rw http.ResponseWriter, inRequest *ht2.Request) { // --snip -- if a.CSP != nil { a.CSP.ConnectSrc += " " + inRequest.Origin } CopyHttpRequest(inRequest, outRequest, rw, CopyHttpRequestOptions{ Origin: inRequest.Origin, Csp: a.CSP, Cors: a.CORS, AddHeaders: !a.SkipHeaders, UseHTTP3: a.UseHTTP3, InsecureTLS: a.InsecureTLS, }) } I'm really unsure as to what I was doing with the line to append to the ConnectSrc, but the impact is that I'm appending to a variable that lives on the App, rather than a variable that is per-request. This meant that every time there was a request to the app, any request at all, the origin would be appended to the header value. This went on for quite a long time unnoticed and unresolved, largely because I am constantly tweaking and tinkering with my web server, after all, it's how I made having a website fun again. Each time I restarted the server process, the header value would be reset, but only for it to continue to grow and grow. Eventually, after a period of being busy with other matters, the server process stayed running for long enough that the header value grew too large and HTTP clients began to reject it. There is no defined maximum for an HTTP header value, however most HTTP clients use 100KiB, which is perfectly reasonable, and this header value would continue to grow well beyond that. Diagnosing this issue turned out to be difficult as tools like Curl would fail with errors relating to entities being too large, but stopped short of saying what specifically. I eventually used openssl s_client to send an HTTP request by hand and observed my terminal window being filled with a domain name repeated thousands of times. Looking at the commit history, it was really unclear why I added the culprit lines of code. The commit message just says "Improved CSP support". It just goes to show how important it is to write - hey look it's those words I'm now having to eat! The Fix The fix was to just delete those three lines of code. Yup, it really was that simple, and fixing this bug actually made a larger positive impact than I had expected, as it was immediately clear when I fixed the bug by looking at outbound network bytes: So much traffic was being wasted on excessive header sizes. You might look at these mistakes I've made and think "wow, Ian, these are some obvious mistakes, I never would have made them!" to which I say "good for you!" with the utmost sarcasm and disdain. I enjoy making and refining software, and making anything means making mistakes along the way. Each time I make mistakes such as the ones above, I improve my skills of investigation, diagnosing, and repair. Skills that, judging by my peers in the industry, seemingly everyone is quickly willing to throw away because a robot does it "better" than you. Header Image: "Car accident on the Ffestiniog to Bala road. Nobody was hurt" by Geoff Charles, CC BY-SA 4.0, via Wikimedia Commons.
I owe a lot of my professional identity and success to CSS-Tricks. CSS-Tricks repeatedly gave me the opportunity to write for them. In doing so, they helped to both socialize and normalize accessibility as a mainstream frontend concern. I’m deeply thankful to them for this. The team was also a joy to work with, notably Geoff Graham. He’s a mensch, and one of the nicest people you can interact with in the frontend web space. If you have not been following the news about the site, Kevin Powell has a good video about the whole situation: Content skipped. I’m not speaking on behalf of Geoff, Chris, or others involved with running the current version of CSS-Tricks. I’ve got skin in the game as an author. This is my personal opinion, born of my feelings and beliefs. I think a lot of the web’s infrastructure should be co-ops, and CSS-Tricks is knowledge infrastructure. To that point, I should also point out that the website covers far more than just CSS. The corporate model of ownership can be a risk. If infrastructure is not part of a corporation’s core strategy, it is not a priority. As Kevin’s video touched on, it seems like promotion via owning the frontend content space isn’t part of Digital Ocean’s strategy anymore. It is not that CSS-Tricks does not have value. It is that Digital Ocean cannot see it. It is deeply, tragically ironic to me that Digital Ocean allowed this to transpire. This is because I know for a fact that the techniques and philosophies shared by CSS-Trick authors helped to shape iterations of their product’s UI. Some may be quick to point out that this knowledge now—illegally—exists inside of LLM training data, so the risk of the website going away is mitigated. To this, know that we should be striving to keep resources like CSS-Tricks going. Human creativity is the force that creates new techniques, strategies, and technologies. The web will calcify without voices sharing what they know, forever locking us into endless permutations of a fixed point in time. Unlike corporations, co-ops don’t have to be motivated by profit. By not needing to prioritize growth at all costs it means co-ops can instead prioritize and incentivise things like preservation and cultivation. It is also a successful model of operation, one that even already exists, and flourishes in the tech space. Collective ownership can also serve as checks and balances for, and protection against hierarchical decision-making. I only need to point to the chaotic and aberrant decisions many CEOs in the technology space have been making as of late to demonstrate the value of this approach. Paddy Srinivasan, if you somehow wind up reading this: Save some face and take a big swing. Give CSS-Tricks back to the people who love it.
How can something that “just works” be so annoying? situation We live in Cambridge off a little road down a drive in shared ownership between us and our neighbouring houses. All the utilities are buried under this drive, including the phone line. anticipation Over the last few years we have been canvassed repeatedly by CityFibre saying that they can deliver fibre all way to our house. I saw them digging trenches and leaving tails of purple fibre cladding along nearby roads, ready to hook up all the houses. I thought they would need to do something similar to deliver fibre to us. So when they turned up and knocked on our door, I talked to their salesbods and walked them up and down the drive and pointed out where the existing BT line goes. Then they gave up trying to sell to us. This happened about three times. disaffection We were not eager enough for an upgrade to deal with these impediments. notification A few months ago we were told that CityFibre would soon come and do the upgrade, since there’s a nationwide deadline for turning off the copper phone network at the end of the year. We expected that this would force them to actually plan some digging works, so we talked to our neighbours about it. We were all ready for some huge faff to follow the next visit by the CityFibre bods. installation CityFibre turned up on the promised morning bright and early. To our enormous surprise, a brown fibre housing was already poking out of the ground next to our copper phone line. It had been fed through 50 metres of 5cm duct without us being aware they were even working on the street. Within a couple of hours, the technicians had drilled through our wall, installed the ONT, blown fibre through the unexpected pipe, plugged in the CPE (superficially identical to the old one), and left telling us to anticipate that it might not work properly until tomorrow. activation Around lunch time, the copper phone line stopped working completely. Some faff ensued, switching all our devices over to the new WiFi network. For a while we thought this was the death of our land line, but in the course of debugging other issues, I realised that the router has a built-in VoIP adapter (I don’t think we were told it has a built-in VoIP adapter) so I plugged the phone in and it Just Worked: they had ported our phone number across and everything. Flawless. I was seriously impressed. rumination It has been a few weeks since the switchover, and apart from a couple of horrible Clown-afflicted IoT devices, it has been fairly smooth. What prompted me to write this up was realising that we delayed this upgrade for years because the sales people were not given enough technical information about how the installation process works: the fact that houses typically have a 5cm duct containing the copper lines (probably standard for the last 40 years) and the fact that fibre can be shoved through a few tens of metres without difficulty. And worse, the sales people didn’t have an esclation path for difficult cases: they just gave up instead. From a technical point of view, the installation was impeccable. (I guess the loose 24 hour window for the cutover time was because OpenReach and CityFibre don’t have tight requirements on ISP reconfiguration schedules.) From the sales point of view, it was crap. Maybe it would have gone faster if we offered to switch early without asking if the drive would be a problem? But I guess the difference between “yes!” and “yes, but will this be a problem?” is too much to expect from a minimum-wage door-to-door salesbod whose employer didn’t give them enough information or any escalation path.
I listen to a lot of podcasts, and I like how they fit around other tasks. I press play, lock my phone, and put it down. I’m free to wash the dishes, fold the laundry, or shop for groceries. Unfortunately, more and more information is only published as a video. Technical talks, conference sessions, video essays – they don’t work in an audio-only podcast app. I could convert these videos to MP3 files, but that breaks down the moment a video isn’t pure spoken word. If a speaker says, “Look at this slide” or holds up a diagram, an audio-only file leaves me stranded. I don’t want to give up the podcast player I like, nor stare at a screen for an hour – but I do want the information in these videos. To solve this, I’m abusing my podcast player’s chapter support. This gives me the best of both worlds: I can listen to a video as audio-first, and glance at my lock screen if I need a moment of visual context. The idea: Chapters every few seconds MP3 files can have ID3 metadata, and ID3 metadata can include chapters. A chapter covers a particular time range, and it can have an associated title, description, and cover art. My podcast app of choice is Overcast, which can’t play videos, but it does have robust chapter support. I can jump between chapters, navigate a table of contents, and see per-chapter cover art. To get videos into Overcast, I’m creating MP3 files with a new chapter every few seconds, and the per-chapter cover art is a corresponding frame from the video. As I play the file, I get a slow, stop-motion-like rendition of the original video. If my phone is locked, I can glance at my lock screen and see the current frame in the Now Playing screen. Overcast is developed by Marco Arment, and I got this idea from Forecast, his app for adding chapters to podcasts. In particular, I was struck by its ability to create chapters that don’t display in the chapter list – ideal if I don’t want a table of contents with hundreds of entries. As I was developing my script, I compared my output to the output from Forecast to ensure I was creating the chapters correctly. The code: FFmpeg and Mutagen There are three steps in this process: Convert a video file to an MP3 Extract images from the video at a fixed interval Insert the images as hidden chapters in the MP3 file Let’s go through each in turn. 1. Convert a video file to an MP3 Converting a video file to an MP3 is a single FFmpeg command: ffmpeg -i video.mp4 audio.mp3 This is consistently the slowest step of the process, and I do wonder if I could use different settings or an alternative encoder to make it go faster – but it’s not slow enough to be worth further investigation. 2. Extract images from the video at a fixed interval Extracting images from a video needs a more complicated FFmpeg command: ffmpeg -i video.mp4 \ -vf 'fps=1/5,scale=iw*sar:ih,scale=min(iw\,945):min(ih\,945):force_original_aspect_ratio=decrease' \ thumbnail_%04d.jpg This extracts an image every 5 seconds, downscales any image larger than 945 pixels square (while preserving the original aspect ratio), and saves the results as sequentially numbered JPEG images (thumbnail_0001.png, thumbnail_0002.png, and so on). The key is the -vf flag, which defines two FFmpeg filters: The fps filter selects one frame every 5 seconds (fps=1/5). The first scale filter scales the width based on the sample aspect ratio (scale=iw*sar:ih). Without this filter, frames can be stretched and distorted. The second scale filter scales the input video, preserving the original aspect ratio (force_original_aspect_ratio=decrease), and ensuring the output images fit within 945×945px or the size of the input video, whichever is smaller. My limit is 945 pixels because that’s the largest size that cover art is shown on my iPhone. This filter still isn’t completely correct – it sometimes creates images from portrait videos that are smaller than I’m expecting – but it’s good enough. These are only thumbnails for glancing at, and if I want to change it later, I can always do the image resizing outside FFmpeg. 3. Insert the images as hidden chapters in the MP3 file Inserting the chapters into the MP3 file is more complicated. Although FFmpeg has basic support for ID3 metadata, as far as I know, it can’t insert chapters with per-chapter artwork. Instead, I’m going to reach for Python and the Mutagen library. Here’s the code to add a chapter to an MP3 file: from mutagen.id3 import APIC, CHAP, ID3, PictureType audio = ID3("audio.mp3") with open("thumbnail_0001.jpg", "rb") as f: img_data = f.read() image_frame = APIC(mime="image/jpeg", type=PictureType.OTHER, data=img_data) chapter_frame = CHAP( element_id="chp1", start_time=0, end_time=5 * 1000, sub_frames=[image_frame] ) audio.add(chapter_frame) audio.save() This creates a single chapter that lasts the first 5 seconds (0 to 5000 milliseconds), and the per-chapter cover art is thumbnail_0001.jpg. If we ran this in a loop, we could add images for every 5 second slice of the original video. This code is inserting two frames into the ID3 metadata: The CHAP (chapter) frame contains the timing information, and it can have subframes for metadata like title, chapter art, or associated URL. The APIC (attached picture) subframe contains information about a picture, which can either be a blob of image data or a URL to an image on the web. Normally, you’d also insert a CTOC frame which defines a table of contents, but I don’t want a TOC with hundreds of 5-second chapters, so I’m deliberately not doing this here. This is allowed by the ID3 spec – you’re not required to insert a CTOC frame if you’re using chapters, and you can have chapters that aren’t listed in your table of contents. To work out which frames I needed, I used Forecast to create some chapters by hand, and I inspected their frames. In particular, loading an MP3 and calling Mutagen’s pprint() method shows a human-readable list of frames, and then I could drill into the individual fields: from mutagen.id3 import ID3 audio = ID3("audio.mp3") print(audio.pprint()) I wrapped all this code in a project called glancecast, which allows you to convert a video file with a single command, with optional flags to set the frame length and chapter art size: $ python3 glancecast.py interesting_talk.mp4 interesting_talk.mp3 The process takes a minute or so to complete, most of which is spent transcoding the video file to MP3. The resulting MP3s are usually 40 to 50 MB in size, which is very reasonable. The outcome: How it looks in practice Here’s what one of these “glanceable” podcasts looks like in Overcast and on my lock screen: Maggie Appleton presented this talk over two years ago and it’s been on my “talks to watch” list ever since. Once I put it in Overcast? I listened to it in less than a day. It’s not a lot of extra information, but enough that I can quickly glance down and get the gist of what a speaker is saying. Both views update with a new frame every few seconds, or I can put my phone in my pocket and ignore the screen. I’ve used this approach for half a dozen videos so far, and I’m happy with the results. I expect to keep using it, because I have a long queue of videos I’ve been meaning to watch. If you’d like to try this, check out glancecast for the full code and instructions. [If the formatting of this post looks odd in your feed reader, visit the original article]
