Full Width 
Shortcuts 
Sign Up 
Log In 
Articles 
Reading List More from Freewheeling Apps
What is text? This blog post reminds me of the time 10 years ago when I tried to do away with Linux's desktop environment because it seemed overly complex and hard to wrap my head around. I had a lot of trouble running wifi without the desktop environment, and it left me quite bitter. Linux had seemed like a hacker's OS when I first started out. But it was trying so hard to compete with Windows that it was starting to seem like Windows in all the wrong ways, it seemed. That experience contributed to me walking off into the wilderness to build Mu. I've gone back and forth between two tendencies in my life. On the one hand, I used to brag about a minimalist setup living in text mode, only running startx as needed, long after the world had moved to fully graphical desktop environments. On the other hand, I was frustrated early on with the limitation of pure text, creating a little bit of Vimscript to view and edit drawings inside a code fold inside code comments. (I don't remember what format I used for the drawings. Maybe Inkscape? I always liked its format and found it quite easy to hack manually.) And these poles have continued to attract me in turn. I wanted to build Mu with just text mode for a long time, and now in the past couple of years I seem to have tried to minimize my use of terminals, with all their ugly historical and path-dependent hacks, and to do more on a graphical canvas (of a game engine). Trying to look back dispassionately, I seem to operate by: Structured data. Not too much. Mostly text. But then, last year, I had a fascinating and also slightly frustrating conversation with a friend where we realized we had very different associations for the term "text". When I said I wanted to keep something as "just text" I meant "viewable with cat on a vt100." (But with Unicode.) When they said they wanted text they meant, "everything you could do in the tradition of book publishing before the advent of computers." (But maybe not images? Equations and line drawings, certainly. And video is right out.) Another dimension of our disagreement: I was thinking of text in terms of compatibility. What can you do regardless of what device you have on you right now. They were thinking in terms of "what's possible with an extremely low-power reader without regard to compatibility." Also, I imagined authoring text and reading it as happening on the same device. Perhaps they imagined authoring on high-powered devices and reading on low-powered ones. So, 30 years after I started programming, I still don't really understand what I want. I try to clear my mind of the cobwebs created by the hacky, historically path-dependent past. But I'm not there yet.
As I start teaching the kids programming, I’ve been thinking about how to motivate it to them. They do a lot of drawing, so that seemed like a potential hook to show how drawing on the computer may make some things easier (erasing, manipulating, repeating) and other things harder (e.g. drawing well). Demo of a little bare-bones fork of Lua Carousel for a computer with a large screen and a mouse. The large screen is divided up between a code editor and a drawing area. There are a few drawing tools. When you make a stroke the code for it is inserted above the cursor in the code editor. You can also edit the code directly and run it to make changes to the drawing area. There is no support for grabbing points or shapes in the drawing area. All you can do there is emit shapes with coordinates. Everything else you gotta program the old-fashioned way. Download (120KB). Inspired by Ronin and Sketch-n-Sketch.
As an experiment, I've machine-translated the help docs in Lua Carousel into 7 languages: Spanish (Peninsular and Latin American) German French Japanese Chinese Portuguese Hopefully it is helpful for someone to no longer have to put up with English. I put each language in a separate download to reduce the amount of data you need to download. Other languages are machine-translated from the English version. I've had friends spot-check it for obvious issues, but there are likely to be problems I missed. Please report any issues you spot. If you want to manually translate pages to improve wording and tone for your culture, contributions would be most welcome. No support yet for right-to-left languages, or languages with combining characters. Lua Carousel still assumes every character on screen is atomic. For example, I can support Chinese and Japanese but not Korean, Arabic or Indian languages. I don't know what I'm doing when it comes to multilingual keyboard support. My rough sense is that that's something to configure for your device. But if there's something I can change in Lua Carousel to make your life easier, please let me know. I've included different fonts for Chinese and Japanese, which blow up the download size for these languages. The source code inside each zip file download is exactly the same, only the font files are different. The larger font files also seem to take up to 10 seconds to load.
I've been reading The Nature of Order by Christopher Alexander, and it's inspired me to experiment with this representation to communicate the global structure of the source code of the Game of Snake: The whole program is arranged on a 2D surface within a series of nested boxes. You can hit ‘run’, and it extracts the code from the surface and runs it and you get to play a game of Snake. (Only on a computer; even though the app runs on a phone, you can't play this particular game of Snake without arrow keys.) So it can be viewed as a sort of Literate Program, even though the representation I'm tangling from has a richer UI than is typical, a surface you can pan and zoom around with a mouse wheel or touchscreen. (It doesn't support editing yet. This is a purely reading experience for now.) I wasn't trying to build yet another Literate Programming experience, though. What was on my mind was Christopher Alexander. Christopher Alexander (CA) was an architect, and he indirectly inspired Design Patterns in the software world. His magnum opus was The Nature of Order, (NoO) a book I'm not sure I'll ever finish. Not because it's long (it's true that it's 4 volumes, but it has a lot of pictures) but because I often need to put it down to think about the chapter I just read for a day. And then the day turns into a week or a month or a year. The first thing to know about The Nature of Order is it defeats attempts at summarization. With that out of the way, here's a summary of the fragments I've read: People build tools, and then the tools change people. In unanticipated ways, for better and for worse. Tools and buildings (remember, CA is an architect) that contribute to more functional societies have certain objective properties. The properties are hard to state or quantify, but have an objective reality analogous to the fundamental forces in Physics. In his earlier books, CA called this property "the quality without a name." In NoO, he calls this property "wholeness" or "life". The degree of life of an arrangement of matter is visible in its geometric form. Function matters, but CA rejects the cartesian separation between form and function. If something has good consequences, it's going to also look good. This sense of "looking good" is not subjective. People often have greater or lesser sensitivity to it, but we all inherently have the capacity to see that one arrangement has more life than another. The degree of life in an arrangement is a holistic property. You don't compute the life of all the parts and add them up. One poor property can destroy the life of the whole. As a best guess, life stems from approximately 15 properties. These properties establish a field of centers that integrate object with environment, foreground with background, inside with outside, form with function, matter with emptiness. Centers are made of centers, recursively, ad infinitum. Any attempt to create life by assembling homogeneous centers is doomed to fail. Instead, nature and the universe creates wholeness and life by starting with a seed and successively unfolding it, differentiating parts of it into centers, each step of the process intensifying existing centers, bringing new centers into being, and increasing the life of the whole. I'm not sure I buy it to the hilt, but it feels actionable even so. I don't need to believe in the objective reality of the vector field of centers to consider the possibility that it might point us in the direction of better quality as perceived by a lot of people. The image above is a very preliminary step at creating a more wholesome arrangement of the source code for the game of Snake, though CA would likely find it sophomoric. I'm trying to use some well-known ideas in software design. There's a top-down decomposition here. The game of Snake consists of two screens, one where you play the game and one when the player dies. The screen for the game itself (where the player spends the bulk of time) decomposes into 3 concepts: space, snake and food. At this point, however, things get muddy from a classic CS perspective. These concepts are not completely separate. There are places in the program where they coil into each other. For example, I can't entirely explain how a snake grows without reference to food, and I can't explain food without reference to its purpose: growing the snake. In response, I'm duplicating this code in multiple places on the surface, with slightly different emphasis: To me this evokes CA's property of interlock/ambiguity. The centers of space, snake and food don't have clear boundaries here, the boundary is ambiguous and the centers kinda intertwine. Another case of interlock/ambiguity: the screens for game and game over that I mentioned earlier are not completely separate. When the game ends, there's a great human need for some sort of conclusion to be derived from it, often in the form of a score. Did I do well or poorly? Set a personal best? Defeat my friend's score? Here the score takes into account the number of times the snake turned. You can imagine alternatives, but no matter how you calculate the score, it must make reference to the game that was just played. The two screens must share data. Here I am experimenting with indicating this permeable boundary by creating a gap in multiple levels of centers, close to uses of Nsteps, to show that the centers aren't crisply isolated from their surroundings. I spent some time on other touches. Rather than arrange code in a vertical rectangle as we usually do, I chose to break it up horizontally to make the boundaries between definitions more easily apparent. Viewed charitably, this might be seen as an example of what CA calls "alternating repetition", an attempt at a rhythm that resonates with the reader. Interspersing prose with code is another repetition, as are the columns. However, I tried to not make the columns or interspersed prose too monotonously regular. I tried to keep it "rough", and to hand-craft the spacing at each point rather than try to enforce some global policies for spacing and positioning. If you'd like to read a little more about Christopher Alexander and The Nature of Order, here are a couple more summaries: Beautiful Software is perhaps the most comprehensive archive of CA lore. In particular, see this early article about the fifteen properties. Dorian Taylor's obituary. A worked out example of the fifteen properties. Beware, you may get sucked into the full 4-volume opus.
More in technology
Well, well, well, well, well, well, well, well, well, well, well, well, well, well, well. We're back. Sorry. We've been watching the onslaught of vulnerabilities flood the internet. Every man, dog, and their grandmas (apparently?) are now using LLMs to find and reproduce vulnerabilities - it’
You want less of them. That’s the reason. You may find that it’s too hard to stop people from doing the thing, literally blood, sweat, and tears trying to prosecute people, but that’s a different thing.
Solitaire Alone Together I made a new game. It's called Solitaire Alone Together. It's Windows 98 solitaire, but you can play with everyone else on the internet. Read the full post on my blog! Here's a raw link, if you need it: https://eieio.games/blog/solitaire-alone-together
This post is a living diary of all the times I messed up something with my website in a funny way. I value those who have the confidence to own their mistakes and share the learning with others, and so this is me doing just that! That Time I Accidentally Made a Tarpit That Time I Accidentally Made Really Large Headers That Time I Accidentally Made a Tarpit Back to Top A "tarpit" is an unofficial term used in computing to describe an intentionally slow response to a request. In these modern times many people are using tarpits as a way to combat the relentless theft of data by AI companies, although there's little to no evidence of that actually being in any way effective. I don't use tarpits, at least not intentionally, but there was that one time when I accidentally created a tarpit and trapped all visitors in it. As I've shared previously, I refuse connections from IP addresses that are blocked or belong to a blocked subnet, and I enforce this firewall during the TCP handshake. The logic here is straightforward: there's no reason to waste resources doing a TLS handshake, accepting an HTTP request, and then rejecting the connection if I already know I'm going to reject it at the earliest step. At the time, the code worked like this: the HTTP server would repeatedly call the Accept() function below expecting a new connection. I've added some comments to help explain the logic. func (l *firewallListener) Accept() (net.Conn, error) { // Accept the connection from the TCP listener. This blocks until there is a connection to accept or the listner was closed. conn, err := l.l.AcceptTCP() if err != nil { return conn, err } // Separate the IP address out from the remote address (which includes the port) ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } // Check if it's blocked, if so close the connection and return a refuseError if IsBlocked(ip, true) { conn.Close() return nil, &refuseError{} } // Otherwise return the connection on to the HTTP server return conn, nil } If the incoming connection was from a blocked IP then I'd return a refuseError. I need to use a specific error interface because the HTTP server will halt if it encounters a non-temporary error from the call to Accept(), so I need to return an error that satisfies the definition of a temporary error. I defined refuseError like this: type refuseError struct{} func (e *refuseError) Error() string { return "." } func (e *refuseError) Timeout() bool { return true } func (e *refuseError) Temporary() bool { return true } func (e *refuseError) Is(err error) bool { return err == context.DeadlineExceeded } This did accomplish the goal of rejecting connections before the TLS handshake for blocked addresses, but it had one really unintended and difficult to track down side-effect. Accepting connections is done serially, after which servers typically then process that request on a dedicated thread (or in Go's case a goroutine). This means that any delays during the accept loop will block all incoming connection. What I had missed while reviewing the code for Go's HTTP server is that when it receives a temporary error from Accept() is that while it doesn't abort, it does sleep for up to a maximum of 1 second. This sleep blocks the entire server for all incoming connections. You can see a trimmed copy of the code that does this below, with some marks I've added which I will explain. // src/net/http/server.go // Copyright 2009 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. for { // (1) rw, err := l.Accept() if err != nil { if s.shuttingDown() { return ErrServerClosed } // (2) if ne, ok := err.(net.Error); ok && ne.Temporary() { if tempDelay == 0 { tempDelay = 5 * time.Millisecond } else { tempDelay *= 2 } if max := 1 * time.Second; tempDelay > max { tempDelay = max } s.logf("http: Accept error: %v; retrying in %v", err, tempDelay) // (3) time.Sleep(tempDelay) continue } return err } connCtx := ctx if cc := s.ConnContext; cc != nil { connCtx = cc(connCtx, rw) if connCtx == nil { panic("ConnContext returned nil") } } tempDelay = 0 c := s.newConn(rw) c.setState(c.rwc, StateNew, runHooks) // before Serve can return // (4) go c.serve(connCtx) } At mark 1 the server calls the Accept() function, this is the exact function that I defined above where I might return a temporary error. At mark 2 it checks if an error was returned, and if so if that error is temporary. If there was a temporary error, at mark 3 it sleeps for an increasing amount of time up-to 1 second, otherwise, at mark 4 it processes the connection on a dedicated goroutine, which allows the server to accept the next connection. I'm not entirely sure why the Go developers added this sleep delay and the change when it was introduced doesn't provide any meaningful insight. Regardless, it caused significant latency connecting to my website when a flood of rejected requests was coming in. It just goes to show how important it is to write meaningful commit messages, because you never know when somebody might come back years later wondering "why was this done?". I sure home I don't come to eat those words later. Coincidentally, you can actually see this happening if you look carefully at one of the metric graphs I shared in my first post about my server's security model: Securing My Web Infrastructure. This is the graph I shared in that blog post and while I didn't know it at the time, the fact that these request spikes all cap-out at around 60 requests per minute was not a coincidence. These requests were not being made with a limit in mind, attackers rarely ever care about things like that, instead it the accidental tarpit I had created. The downside to this was that while the malicious requests were being rate-limited, all requests were being rate-limited, up to a point of taking so long they timed out. The Fix Fixing the issue was relatively straightforward enough. Instead of returning a temporary error to the HTTP server during the accept loop, just don't return anything at all and wait for the next valid connection. func (l *firewallListener) Accept() (net.Conn, error) { for { conn, err := l.l.AcceptTCP() if err != nil { return conn, err } ip := utils.SocketStringToIPAddress(conn.RemoteAddr().String()) if ip == nil { return nil, nil } if IsBlocked(ip, true) { conn.SetLinger(0) conn.Close() continue } return conn, nil } } Now, when the HTTP server calls Accept(), the only time it returns is with a connection from an IP that isn't blocked, or if there genuinely is an error. No more sleep delays, no more excessive timeouts. That Time I Accidentally Made Really Large Headers Back to Top For about 10 years now all major browsers have support for a security feature known as a Content Security Policy or CSP. A CSP is an HTTP header provided by the server that instructs the browser on where it can load assets from, this could be scripts, images, stylesheets, fonts, etc. The objective of using a CSP is to prevent against injected HTML that tries to load assets, such as a malicious Javascript file, from a remote source. With so much user-provided content being available online, it's very possible for this to happen without an attacker compromising the entire web server. CSP protects against that by saying "scripts can only be loaded from these domains". That's a really simplified way of looking at it, anyways. My web server supports injecting the CSP header automatically, but before I go on I need to explain a little bit about the structure of my web server. When an incoming HTTP request is accepted (having passed all firewall checks and assertions), we look at the destination host for the request. This can either be the value of the Host header or as specified during the TLS handshake. We then look at a map of hosts to apps. Apps are just an interface that accept a few methods: type App interface { Cleanup() ReloadConfig() ServeHTTP(rw http.ResponseWriter, r *http.Request) Setup(dataDir string) error Shutdown() } One of the apps is the Proxy app, which is a reverse proxy - it accepts the incoming HTTP request and then proxies it on to another host. This is a very common design, especially with increasingly complex TLS setups. Because each app is unique to a host, and different hosts have different requirements for CSP rules, the proxy app includes a CSP preset that we use to build the header value, or skip it entirely. When the proxy app was going to copy an HTTP request to the downstream host, it would build the CSP header, however there was a slight bug... func (a *App) ServeHTTP(rw http.ResponseWriter, inRequest *ht2.Request) { // --snip -- if a.CSP != nil { a.CSP.ConnectSrc += " " + inRequest.Origin } CopyHttpRequest(inRequest, outRequest, rw, CopyHttpRequestOptions{ Origin: inRequest.Origin, Csp: a.CSP, Cors: a.CORS, AddHeaders: !a.SkipHeaders, UseHTTP3: a.UseHTTP3, InsecureTLS: a.InsecureTLS, }) } I'm really unsure as to what I was doing with the line to append to the ConnectSrc, but the impact is that I'm appending to a variable that lives on the App, rather than a variable that is per-request. This meant that every time there was a request to the app, any request at all, the origin would be appended to the header value. This went on for quite a long time unnoticed and unresolved, largely because I am constantly tweaking and tinkering with my web server, after all, it's how I made having a website fun again. Each time I restarted the server process, the header value would be reset, but only for it to continue to grow and grow. Eventually, after a period of being busy with other matters, the server process stayed running for long enough that the header value grew too large and HTTP clients began to reject it. There is no defined maximum for an HTTP header value, however most HTTP clients use 100KiB, which is perfectly reasonable, and this header value would continue to grow well beyond that. Diagnosing this issue turned out to be difficult as tools like Curl would fail with errors relating to entities being too large, but stopped short of saying what specifically. I eventually used openssl s_client to send an HTTP request by hand and observed my terminal window being filled with a domain name repeated thousands of times. Looking at the commit history, it was really unclear why I added the culprit lines of code. The commit message just says "Improved CSP support". It just goes to show how important it is to write - hey look it's those words I'm now having to eat! The Fix The fix was to just delete those three lines of code. Yup, it really was that simple, and fixing this bug actually made a larger positive impact than I had expected, as it was immediately clear when I fixed the bug by looking at outbound network bytes: So much traffic was being wasted on excessive header sizes. You might look at these mistakes I've made and think "wow, Ian, these are some obvious mistakes, I never would have made them!" to which I say "good for you!" with the utmost sarcasm and disdain. I enjoy making and refining software, and making anything means making mistakes along the way. Each time I make mistakes such as the ones above, I improve my skills of investigation, diagnosing, and repair. Skills that, judging by my peers in the industry, seemingly everyone is quickly willing to throw away because a robot does it "better" than you. Header Image: "Car accident on the Ffestiniog to Bala road. Nobody was hurt" by Geoff Charles, CC BY-SA 4.0, via Wikimedia Commons.
