More from dthompson
I'm happy to announce that guile-websocket 0.2.0 has been released! Guile-websocket is an implementation of the WebSocket protocol, both the client and server sides, for Guile Scheme. This release introduces breaking changes that overhaul the client and server implementations in order to support non-blocking I/O and TLS encrypted connections. source tarball: https://files.dthompson.us/guile-websocket/guile-websocket-0.2.0.tar.gz signature: https://files.dthompson.us/guile-websocket/guile-websocket-0.2.0.tar.gz.asc See the guile-websocket project page for more information. Bug reports, bug fixes, feature requests, and patches are welcomed.
Wasm GC is a wonderful thing that is now available in all major web browsers since slowpoke Safari/WebKit finally shipped it in December. It provides a hierarchy of heap allocated reference types and a set of instructions to operate on them. Wasm GC enables managed memory languages to take advantage of the advanced garbage collectors inside web browser engines. It’s now possible to implement a managed memory language without having to ship a GC inside the binary. The result is smaller binaries, better performance, and better integration with the host runtime. However, Wasm GC has some serious drawbacks when compared to linear memory. I enjoy playing around with realtime graphics programming in my free time, but I was disappointed to discover that Wasm GC just isn’t a good fit for that right now. I decided to write this post because I’d like to see Wasm GC on more or less equal footing with linear memory when it comes to binary data manipulation. Hello triangle For starters, let's take a look at what a “hello triangle” WebGL demo looks like with Wasm GC. I’ll use Hoot, the Scheme to Wasm compiler that I work on, to build it. Below is a Scheme program that declares imports for the subset of the WebGL, HTML5 Canvas, etc. APIs that are necessary and then renders a single triangle: (use-modules (hoot ffi)) ;; Document (define-foreign get-element-by-id "document" "getElementById" (ref string) -> (ref null extern)) ;; Element (define-foreign element-width "element" "width" (ref extern) -> i32) (define-foreign element-height "element" "height" (ref extern) -> i32) ;; Canvas (define-foreign get-canvas-context "canvas" "getContext" (ref extern) (ref string) -> (ref null extern)) ;; WebGL (define GL_VERTEX_SHADER 35633) (define GL_FRAGMENT_SHADER 35632) (define GL_COMPILE_STATUS 35713) (define GL_LINK_STATUS 35714) (define GL_ARRAY_BUFFER 34962) (define GL_STATIC_DRAW 35044) (define GL_COLOR_BUFFER_BIT 16384) (define GL_TRIANGLES 4) (define GL_FLOAT 5126) (define-foreign gl-create-shader "gl" "createShader" (ref extern) i32 -> (ref extern)) (define-foreign gl-delete-shader "gl" "deleteShader" (ref extern) (ref extern) -> none) (define-foreign gl-shader-source "gl" "shaderSource" (ref extern) (ref extern) (ref string) -> none) (define-foreign gl-compile-shader "gl" "compileShader" (ref extern) (ref extern) -> none) (define-foreign gl-get-shader-parameter "gl" "getShaderParameter" (ref extern) (ref extern) i32 -> i32) (define-foreign gl-get-shader-info-log "gl" "getShaderInfoLog" (ref extern) (ref extern) -> (ref string)) (define-foreign gl-create-program "gl" "createProgram" (ref extern) -> (ref extern)) (define-foreign gl-delete-program "gl" "deleteProgram" (ref extern) (ref extern) -> none) (define-foreign gl-attach-shader "gl" "attachShader" (ref extern) (ref extern) (ref extern) -> none) (define-foreign gl-link-program "gl" "linkProgram" (ref extern) (ref extern) -> none) (define-foreign gl-use-program "gl" "useProgram" (ref extern) (ref extern) -> none) (define-foreign gl-get-program-parameter "gl" "getProgramParameter" (ref extern) (ref extern) i32 -> i32) (define-foreign gl-get-program-info-log "gl" "getProgramInfoLog" (ref extern) (ref extern) -> (ref string)) (define-foreign gl-create-buffer "gl" "createBuffer" (ref extern) -> (ref extern)) (define-foreign gl-delete-buffer "gl" "deleteBuffer" (ref extern) (ref extern) -> (ref extern)) (define-foreign gl-bind-buffer "gl" "bindBuffer" (ref extern) i32 (ref extern) -> none) (define-foreign gl-buffer-data "gl" "bufferData" (ref extern) i32 (ref eq) i32 -> none) (define-foreign gl-enable-vertex-attrib-array "gl" "enableVertexAttribArray" (ref extern) i32 -> none) (define-foreign gl-vertex-attrib-pointer "gl" "vertexAttribPointer" (ref extern) i32 i32 i32 i32 i32 i32 -> none) (define-foreign gl-draw-arrays "gl" "drawArrays" (ref extern) i32 i32 i32 -> none) (define-foreign gl-viewport "gl" "viewport" (ref extern) i32 i32 i32 i32 -> none) (define-foreign gl-clear-color "gl" "clearColor" (ref extern) f64 f64 f64 f64 -> none) (define-foreign gl-clear "gl" "clear" (ref extern) i32 -> none) (define (compile-shader gl type source) (let ((shader (gl-create-shader gl type))) (gl-shader-source gl shader source) (gl-compile-shader gl shader) (unless (= (gl-get-shader-parameter gl shader GL_COMPILE_STATUS) 1) (let ((info (gl-get-shader-info-log gl shader))) (gl-delete-shader gl shader) (error "shader compilation failed" info))) shader)) (define (link-shader gl vertex-shader fragment-shader) (let ((program (gl-create-program gl))) (gl-attach-shader gl program vertex-shader) (gl-attach-shader gl program fragment-shader) (gl-link-program gl program) (unless (= (gl-get-program-parameter gl program GL_LINK_STATUS) 1) (let ((info (gl-get-program-info-log gl program))) (gl-delete-program gl program) (error "program linking failed" info))) program)) ;; Setup GL context (define canvas (get-element-by-id "canvas")) (define gl (get-canvas-context canvas "webgl")) (when (external-null? gl) (error "unable to create WebGL context")) ;; Compile shader (define vertex-shader-source "attribute vec2 position; attribute vec3 color; varying vec3 fragColor; void main() { gl_Position = vec4(position, 0.0, 1.0); fragColor = color; }") (define fragment-shader-source "precision mediump float; varying vec3 fragColor; void main() { gl_FragColor = vec4(fragColor, 1); }") (define vertex-shader (compile-shader gl GL_VERTEX_SHADER vertex-shader-source)) (define fragment-shader (compile-shader gl GL_FRAGMENT_SHADER fragment-shader-source)) (define shader (link-shader gl vertex-shader fragment-shader)) ;; Create vertex buffer (define stride (* 4 5)) (define buffer (gl-create-buffer gl)) (gl-bind-buffer gl GL_ARRAY_BUFFER buffer) (gl-buffer-data gl GL_ARRAY_BUFFER #f32(-1.0 -1.0 1.0 0.0 0.0 1.0 -1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0) GL_STATIC_DRAW) ;; Draw (gl-viewport gl 0 0 (element-width canvas) (element-height canvas)) (gl-clear gl GL_COLOR_BUFFER_BIT) (gl-use-program gl shader) (gl-enable-vertex-attrib-array gl 0) (gl-vertex-attrib-pointer gl 0 2 GL_FLOAT 0 stride 0) (gl-enable-vertex-attrib-array gl 1) (gl-vertex-attrib-pointer gl 1 3 GL_FLOAT 0 stride 8) (gl-draw-arrays gl GL_TRIANGLES 0 3) Note that in Scheme, the equivalent of a Uint8Array is a bytevector. Hoot uses a packed array, an (array i8) specifically, for the contents of a bytevector. And here is the JavaScript code necessary to boot the resulting Wasm binary: window.addEventListener("load", async () => { function bytevectorToUint8Array(bv) { let len = reflect.bytevector_length(bv); let array = new Uint8Array(len); for (let i = 0; i < len; i++) { array[i] = reflect.bytevector_ref(bv, i); } return array; } let mod = await SchemeModule.fetch_and_instantiate("triangle.wasm", { reflect_wasm_dir: 'reflect-wasm', user_imports: { document: { getElementById: (id) => document.getElementById(id) }, element: { width: (elem) => elem.width, height: (elem) => elem.height }, canvas: { getContext: (elem, type) => elem.getContext(type) }, gl: { createShader: (gl, type) => gl.createShader(type), deleteShader: (gl, shader) => gl.deleteShader(shader), shaderSource: (gl, shader, source) => gl.shaderSource(shader, source), compileShader: (gl, shader) => gl.compileShader(shader), getShaderParameter: (gl, shader, param) => gl.getShaderParameter(shader, param), getShaderInfoLog: (gl, shader) => gl.getShaderInfoLog(shader), createProgram: (gl, type) => gl.createProgram(type), deleteProgram: (gl, program) => gl.deleteProgram(program), attachShader: (gl, program, shader) => gl.attachShader(program, shader), linkProgram: (gl, program) => gl.linkProgram(program), useProgram: (gl, program) => gl.useProgram(program), getProgramParameter: (gl, program, param) => gl.getProgramParameter(program, param), getProgramInfoLog: (gl, program) => gl.getProgramInfoLog(program), createBuffer: (gl) => gl.createBuffer(), deleteBuffer: (gl, buffer) => gl.deleteBuffer(buffer), bindBuffer: (gl, target, buffer) => gl.bindBuffer(target, buffer), bufferData: (gl, buffer, data, usage) => { let bv = new Bytevector(reflect, data); gl.bufferData(buffer, bytevectorToUint8Array(bv), usage); }, enableVertexAttribArray: (gl, index) => gl.enableVertexAttribArray(index), vertexAttribPointer: (gl, index, size, type, normalized, stride, offset) => { gl.vertexAttribPointer(index, size, type, normalized, stride, offset); }, drawArrays: (gl, mode, first, count) => gl.drawArrays(mode, first, count), viewport: (gl, x, y, w, h) => gl.viewport(x, y, w, h), clearColor: (gl, r, g, b, a) => gl.clearColor(r, g, b, a), clear: (gl, mask) => gl.clear(mask) } } }); let reflect = await mod.reflect({ reflect_wasm_dir: 'reflect-wasm' }); let proc = new Procedure(reflect, mod.get_export("$load").value); proc.call(); }); Hello problems There are two major performance issues with this program. One is visible in the source above, the other is hidden in the language implementation. Heap objects are opaque on the other side Wasm GC heap objects are opaque to the host. Likewise, heap objects from the host are opaque to the Wasm guest. Thus the contents of an (array i8) object are not visible from JavaScript and the contents of a Uint8Array are not visible from Wasm. This is a good security property in the general case, but it’s a hinderance in this specific case. Let’s say we have an (array i8) full of vertex data we want to put into a WebGL buffer. To do this, we must make one JS->Wasm call for each byte in the array and store it into a Uint8Array. This is what the bytevectorToUint8Array function above is doing. Copying any significant amount of data per frame is going to tank performance. Hope you aren’t trying to stream vertex data! Contrast the previous paragraph with Wasm linear memory. A WebAssembly.Memory object can be easily accessed from JavaScript as an ArrayBuffer. To get a blob of vertex data out of a memory object, you just need to know the byte offset and length and you’re good to go. There are many Wasm linear memory applications using WebGL successfully. Manipulating multi-byte binary data is inefficient To read a multi-byte number such as an unsigned 32-bit integer from an (array i8), you have to fetch each individual byte and combine them together. Here’s a self-contained example that uses Guile-flavored WAT format: (module (type $bytevector (array i8)) (data $init #u32(123456789)) (func (export "main") (result i32) (local $a (ref $bytevector)) (local.set $a (array.new_data $bytevector $init (i32.const 0) (i32.const 4))) (array.get_u $bytevector (local.get $a) (i32.const 0)) (i32.shl (array.get_u $bytevector (local.get $a) (i32.const 1)) (i32.const 8)) (i32.or) (i32.shl (array.get_u $bytevector (local.get $a) (i32.const 2)) (i32.const 16)) (i32.or) (i32.shl (array.get_u $bytevector (local.get $a) (i32.const 3)) (i32.const 24)) (i32.or))) By contrast, Wasm linear memory needs but a single i32.load instruction: (module (memory 1) (func (export "main") (result i32) (i32.store (i32.const 0) (i32.const 123456789)) (i32.load (i32.const 0)))) Easy peasy. Not only is it less code, it's a lot more efficient. Unsatisfying workarounds There’s no way around the multi-byte problem at the moment, but for byte access from JavaScript there are some things we could try to work with what we have been given. Spoiler alert: None of them are pleasant. Use Uint8Array from the host This approach makes all binary operations from within the Wasm binary slow since we’d have to cross the Wasm->JS bridge for each read/write. Since most of the binary data manipulation is happening in the Wasm module, this approach will just make things slower overall. Use linear memory for bytevectors This would require a little malloc/free implementation and a way to reclaim memory for GC'd bytevectors. You could register every bytevector in a FinalizationRegistry in order to be notified upon GC and free the memory. Now you have to deal with memory fragmentation. This is Wasm GC, we shouldn’t have to do any of this! Use linear memory as a scratch space This avoids crossing the Wasm/JS boundary for each byte, but still involves a byte-by-byte copy from (array i8) to linear memory within the Wasm module. So far this feels like the least worst option, but the extra copy is still going to greatly reduce throughput. Wasm GC needs some fixin' I’ve used realtime graphics as an example because it’s a use case that is very sensitive to performance issues, but this unfortunate need to copy binary data byte-by-byte is also the reason why strings are trash on Wasm GC right now. Stringref is a good proposal and the Wasm community group made a mistake by rejecting it. Anyway, there has been some discussion about both multi-byte and ArrayBuffer access on GitHub, but as far as I can tell neither issue is anywhere close to a resolution. Can these things be implemented efficiently? How can the need for direct access to packed arrays from JS be reconciled with Wasm heap object opaqueness? I hope the Wasm community group can arrive at solutions sooner than later because it will take a long time to get the proposal(s) to phase 4 and shipped in all browsers, perhaps years. It would be a shame to be effectively shut out from using WebGPU when it finally reaches stable browser releases.
I'm pleased to announce that the very first release of guile-bstructs, version 0.1.0, has been released! This is a library I've been working on for quite some time and after more than one rewrite and many smaller refactors I think it's finally ready to release publicly. Let's hope I'm not wrong about that! About guile-bstructs Guile-bstructs is a library that provides structured read/write access to binary data for Guile. A bstruct (short for “binary structure”) is a data type that encapsulates a bytevector and a byte offset which interprets that bytevector based on a specified layout. Some use cases for bstructs are: manipulating C structs when using the foreign function interface packing GPU vertex buffers when using graphics APIs such as OpenGL implementing data types that benefit from Guile's unboxed math optimizations such as vectors and matrices This library was initially inspired by guile-opengl's define-packed-struct syntax but is heavily based on "Ftypes: Structured foreign types" by Andy Keep and R. Kent Dybvig. The resulting interface is quite similar but the implementation is completely original. This library provides a syntax-heavy interface; nearly all of the public API is syntax. This is done to ensure that bstruct types are static and well-known at compile time resulting in efficient bytecode and minimal runtime overhead. A subset of the interface deals in raw bytevector access for accessing structured data in bytevectors directly without going through an intermediary bstruct wrapper. This low-level interface is useful for certain batch processing situations where the overhead of creating wrapper bstructs would hinder throughput. Example Here are some example type definitions to give you an idea of what it’s like to use guile-bstructs: ;; Struct (define-bstruct <vec2> (padded (struct (x float) (y float)))) ;; Type group with a union (define-bstruct (<mouse-move-event> (struct (type uint8) (x int32) (y int32))) (<mouse-button-event> (struct (type uint8) (button uint8) (state uint8) (x int32) (y int32))) (<event> (union (type uint8) (mouse-move <mouse-move-event>) (mouse-button <mouse-button-event>)))) ;; Array (define-bstruct <matrix4> (array 16 float)) ;; Bit fields (define-bstruct <date> (bits (year 32 s) (month 4 u) (day 5 u))) ;; Pointer (define-bstruct (<item> (struct (type int))) (<chest> (struct (opened? uint8) (item (* <item>))))) ;; Packed struct modifier (define-bstruct <enemy> (packed (struct (type uint8) (health uint32)))) ;; Endianness modifier (define-bstruct <big-float> (endian big float)) ;; Recursive type (define-bstruct <node> (struct (item int) (next (* <node>)))) ;; Mutually recursive type group (define-bstruct (<forest> (struct (children (* <tree>)))) (<tree> (struct (value int) (forest (* <forest>)) (next (* <tree>))))) ;; Opaque type (define-bstruct SDL_GPUTexture) Download Source tarball: guile-bstructs-0.1.0.tar.gz GPG signature: guile-bstructs-0.1.0.tar.gz.asc This release was signed with this GPG key. See the guile-bstructs project page for more information.
The Spring Lisp Game Jam 2024 ended one week ago. 48 games were submitted, a new record for the jam! This past week has been a time for participants to play and rate each other’s games. As I explored the entries, I noticed two distinct meta-patterns in how people approached building games with Lisp. I think these patterns apply more broadly to all applications of Lisp. Let’s talk about these patterns in some detail, with examples. But first! Here’s the breakdown of the jam submissions by language: lang entries % (rounded) ---- ------- ----------- guile 15 31 fennel 10 21 clojure 5 10 cl 5 10 racket 4 8 elisp 4 8 s7 3 6 kawa 1 2 owl 1 2 I haven’t rolled up the various Schemes (Guile, Racket, S7, Kawa) into a general scheme category because Scheme is so minimally specified and they are all very distinct implementations for different purposes, not to mention that Racket has a lot more going on than just Scheme. For the first time ever, Guile came out on top with the most submissions! There’s a very specific reason for this outcome. 11 out of the 15 Guile games were built for the web with Hoot, a Scheme-to-WebAssembly compiler that I work on at the Spritely Institute. 2 of those 11 were official Spritely projects. We put out a call for people to try making games with Hoot before the jam started, and a lot of people took us up on it! Very cool! The next most popular language, which is typically the most popular language in these jams, is Fennel. Fennel is a Lisp that compiles to Lua. It’s very cool, too! Also of note, at least to me as a Schemer, is that three games used S7. Hmm, there might be something relevant to this post going on there. The patterns I’m about to talk about could sort of be framed as “The Guile Way vs. The Fennel Way”, but I don’t want to do that. It's not an “us vs. them” thing. It’s wonderful that there are so many flavors of Lisp these days that anyone can find a great implementation that suits their preferences. Not only that, but many of these implementations can be used to make games that anyone can easily play in their web browser! That was not the case several years ago. Incredible! I want to preface the rest of this post by saying that both patterns are valid, and while I prefer one over the other, that is not to say that the other is inferior. I'll also show how these patterns can be thought of as two ends of a spectrum and how, in the end, compromises must be made. Okay, let’s get into it! Lisp as icing The icing pattern is using Lisp as a “scripting” language on top of a cake that is made from C, Rust, and other static languages. The typical way to do this is by embedding a Lisp interpreter into the larger program. If you’re most interested in writing the high-level parts of an application in Lisp then this pattern is the fastest way to get there. All you need is a suitable interpreter/compiler and a way to add the necessary hooks into your application. Since the program is mainly C/Rust/whatever, you can then use emscripten to compile it to WebAssembly and deploy to the web. Instant gratification, but strongly tied to static languages and their toolchains. S7 is an example of an embeddable Scheme. Guile is also used for extending C programs, though typically that involves dynamically linking to libguile rather than embedding the interpreter into the program’s executable. Fennel takes a different approach, recognizing that there are many existing applications that are already extensible through Lua, and provides a lispy language that compiles to Lua. Lisp as cake The cake pattern is using Lisp to implement as much of the software stack as possible. It’s Lisp all the way down... sorta. Rather than embedding Lisp into a non-Lisp program, the cake pattern does the inverse: the majority of the program is written in Lisp. When necessary, shared libraries can be called via a foreign function interface, but this should be kept to a minimum. This approach takes longer to yield results. Time is spent implementing missing libraries for your Lisp of choice and writing wrappers around the C shared libraries you can’t avoid using. Web deployment gets trickier, too, since the project is not so easily emscriptenable. (You may recognize this as the classic embed vs. extend debate. You’re correct! I'm just adding my own thoughts and applying it specifically to some real-world Lisp projects.) I mentioned Guile as an option for icing, but Guile really shines best as cake. The initial vision for Guile was to Emacsify other programs by adding a Scheme interpreter to them. These days, the best practice is to write your program in Scheme to begin with. Common Lisp is probably the best example, though. Implementations like SBCL have good C FFIs and can compile efficient native executables, minimizing the desire to use some C for performance reasons. Case studies Let’s take a look at some of the languages and libraries used for the Lisp Game Jam and evaluate their icing/cake-ness. Fennel + love2d love2d has been a popular choice for solo or small team game development for many years. It is a C++ program that embeds a Lua interpreter, which means it’s a perfect target for Fennel. Most Linux distributions package love2d, so it’s easy to run .love files natively. Additionally, thanks to emscripten, love2d games can be deployed to the web. Thus most Fennel games use love2d. ./soko.bin and Gnomic Vengeance are two games that use this stack. Fennel + love2d is a perfect example of Lisp as icing. Fennel sits at the very top of the stack, but there’s not really a path to spread Lisp into the layers below. It is also the most successful Lisp game development stack to date. S7 + raylib This stack is new to me, but two games used it this time around: GhostHop and Life Predictor. (You really gotta play GhostHop, btw. It’s a great little puzzle game and it is playable on mobile devices.) Raylib is a C library with bindings for many higher-level languages that has become quite popular in recent years. S7 is also implemented in C and is easily embeddable. This makes the combination easy to deploy on the web with emscripten. S7 + raylib is another example of Lisp as icing. I’m curious to see if this stack becomes more popular in future jams. Guile + Chickadee This is the stack that I helped build. Chickadee is a game library for Guile that implements almost all of the interesting parts in Scheme, including rendering. Two games were built with Chickadee in the most recent jam: Turbo Racer 3000 and Bloatrunner. Guile + Chickadee is an example of Lisp as cake. Chickadee wraps some C libraries for low-level tasks such as loading images, audio, and fonts, but it is written in pure Scheme. All the matrix and vector math is in Scheme. Chickadee comes with a set of rendering primitives comparable to love2d and raylib but they’re all implemented in Scheme. I’ve even made progress on rendering vector graphics with Scheme, whereas most other Lisp game libraries use a C library such as nanosvg. Chickadee has pushed the limits of Guile’s compiler and virtual machine, and Guile has been improved as a result. But it’s the long road. Chickadee is mostly developed by me, alone, in my very limited spare time. It is taking a long time to reach feature parity with more popular game development libraries, but it works quite well for what it is. Hoot + HTML5 canvas I also helped build this one. Hoot is a Scheme-to-WebAssembly compiler. Rather than compile the Guile VM (written in C) to Wasm using emscripten, Hoot implements a complete Wasm toolchain and a new backend for Guile’s compiler that emits Wasm directly. Hoot is written entirely in Scheme. Unlike C programs compiled with emscripten that target Wasm 1.0 with linear memory, Hoot targets Wasm 2.0 with GC managed heap types. This gives Hoot a significant advantage: Hoot binaries do not ship a garbage collector and thus are much smaller than Lisp runtimes compiled via emscripten. The Wasm binary for my game weighs in at < 2MiB whereas the love2d game I checked had a nearly 6MiB love.wasm. Hoot programs can also easily interoperate with JavaScript. Scheme objects can easily be passed to JavaScript, and vice versa, as they are managed in the same heap. With all of the browser APIs just a Wasm import away, an obvious choice for games was the built-in HTML5 canvas API for easy 2D rendering. 11 games used Hoot in the jam, including (shameless plug) Cirkoban and Lambda Dungeon. Hoot + HTML5 canvas is mostly dense cake with a bit of icing. On one hand, it took a year and significant funding to boot Hoot. We said “no” to emscripten, built our own toolchain, and extended Guile’s compiler. It's Lisp all the way until you hit the browser runtime! We even have a Wasm interpreter that runs on the Guile VM! Hoot rules! It was a risk but it paid off. On the other hand, the canvas API is very high-level. The more cake thing to do would be to use Hoot’s JS FFI to call WebGL and/or WebGPU. Indeed, this is the plan for the future! Wasm GC needs some improvements to make this feasible, but my personal goal is to get Chickadee ported to Hoot. I want Chickadee games to be easy to play natively and in browsers, just like love2d games. The cake/icing spectrum I must acknowledge the limitations of the cake approach. We’re not living in a world of Lisp machines, but a world of glorified PDP-11s. Even the tallest of Lisp cakes sits atop an even larger cake made mostly of C. All modern Lisp systems bottom out at some point. Emacs rests on a C core. Guile’s VM is written in C. Hoot runs on mammoth JavaScript engines written in C++ like V8. Games on Hoot currently render with HTML5 canvas rather than WebGL/WebGPU. Good luck using OpenGL without libGL; Chickadee uses guile-opengl which uses the C FFI to call into libGL. Then there’s libpng, FreeType, and more. Who the heck wants to rewrite all this in Lisp? Who even has the resources? Does spending all this time taking the scenic route matter at all, or are we just deluding ourselves because we have fun writing Lisp code? I think it does matter. Every piece of the stack that can be reclaimed from the likes of C is a small victory. The parts written in Lisp are much easier to hack on, and some of those things become live hackable while our programs are running. They are also memory safe, typically, thanks to GC managed runtimes. Less FFI calls means less overhead from traversing the Lisp/C boundary and more safety. As more of the stack becomes Lisp, it starts looking less like icing and more like cake. Moving beyond games, we can look to the Guix project as a great example of just how tasty the cake can get. Guix took the functional packaging model from the Nix project and made a fresh implementation, replacing the Nix language with Guile. Why? For code staging, code sharing, and improved hackability. Guix also uses an init system written in Guile rather than systemd. Why? For code staging, code sharing, and improved hackability. These are real advantages that make the trade-off of not using the industry-standard thing worth it. I’ve been using Guix since the early days, and back then it was easy to make the argument that Guix was just reinventing wheels for no reason. But now, over 10 years later, the insistence on maximizing the usage of Lisp has been key to the success of the project. As a user, once you learn the Guix idioms and a bit of Guile, you unlock extraordinary power to craft your OS to your liking. It’s the closest thing you can get to a Lisp machine on modern hardware. The cake approach paid off for Guix, and it could pay off for other projects, too. If Common Lisp is more your thing, and even if it isn’t, you’ll be amazed by the Trial game engine and how much of it is implemented in Common Lisp rather than wrapping C libraries. There’s also projects like Pre-Scheme that give me hope that one day the layers below the managed GC runtime can be implemented in Lisp. Pre-Scheme was developed and successfully used for Scheme 48 and I am looking forward to a modern revival of it thanks to an NLnet grant. I'm a cake boy That’s right, I said it: I’m a cake boy. I want to see projects continue to push the boundaries of what Lisp can do. When it comes to the Lisp Game Jam, what excites me most are not the games themselves, but the small advances made to reclaim another little slice of the cake from stale, dry C. I intend to keep pushing the limits for Guile game development with my Chickadee project. It’s not a piece of cake to bake a lispy cake, and the way is often hazy, but I know we can’t be lazy and just do the cooking by the book. Rewrite it in Rust? No way! Rewrite it in Lisp!
More in programming
I have added syntax highlighting to my blog using tree-sitter. Here are some notes about what I learned, with some complaining. static site generator markdown ingestion highlighting incompatible?! highlight names class names styling code results future work frontmatter templates feed style highlight quality static site generator I moved my blog to my own web site a few years ago. It is produced using a scruffy Rust program that converts a bunch of Markdown files to HTML using pulldown-cmark, and produces complete pages from Handlebars templates. Why did I write another static site generator? Well, partly as an exercise when learning Rust. Partly, since I wrote my own page templates, I’m not going to benefit from a library of existing templates. On the contrary, it’s harder to create new templates that work with a general-purpose SSG than write my own simpler site-specific SSG. It’s miserable to write programs in template languages. My SSG can keep the logic in the templates to a minimum, and do all the fiddly stuff in Rust. (Which is not very fiddly, because my site doesn’t have complicated navigation – compared to the multilevel menus on www.dns.cam.ac.uk for instance.) markdown ingestion There are a few things to do to each Markdown file: split off and deserialize the YAML frontmatter find the <cut> or <toc> marker that indicates the end of the teaser / where the table of contents should be inserted augment headings with self-linking anchors (which are also used by the ToC) Before this work I was using regexes to do all these jobs, because that allowed me to treat pulldown-cmark as a black box: Markdown in, HTML out. But for syntax highlighting I had to be able to find fenced code blocks. It was time to put some code into the pipeline between pulldown-cmark’s parser and renderer. And if I’m using a proper parser I can get rid of a few regexes: after some hacking, now only the YAML frontmatter is handled with a regex. Sub-heading linkification and ToC construction are fiddly and more complicated than they were before. But they are also less buggy: markup in headings actually works now! Compared to the ToC, it’s fairly simple to detect code blocks and pass them through a highlighter. You can look at my Markdown munger here. (I am not very happy with the way it uses state, but it works.) highlighting As well as the tree-sitter-highlight documentation I used femark as an example implementation. I encountered a few problems. incompatible?! I could not get the latest tree-sitter-highlight to work as described in its documentation. I thought the current tree-sitter crates were incompatible with each other! For a while I downgraded to an earlier version, but eventually I solved the problem. Where the docs say, let javascript_language = tree_sitter_javascript::language(); They should say: let javascript_language = tree_sitter::Language::new( tree_sitter_javascript::LANGUAGE ); highlight names I was offended that tree-sitter-highlight seems to expect me to hardcode a list of highlight names, without explaining where they come from or what they mean. I was doubly offended that there’s an array of STANDARD_CAPTURE_NAMES but it isn’t exported, and doesn’t match the list in the docs. You mean I have to copy and paste it? Which one?! There’s some discussion of highlight names in the tree-sitter manual’s “syntax highlighting” chapter, but that is aimed at people who are writing a tree-sitter grammar, not people who are using one. Eventually I worked out that tree_sitter_javascript::HIGHLIGHT_QUERY in the tree-sitter-highlight example corresponds to the contents of a highlights.scm file. Each @name in highlights.scm is a highlight name that I might be interested in. In principle I guess different tree-sitter grammars should use similar highlight names in their highlights.scm files? (Only to a limited extent, it turns out.) I decided the obviously correct list of highlight names is the list of every name defined in the HIGHLIGHT_QUERY. The query is just a string so I can throw a regex at it and build an array of the matches. This should make the highlighter produce <span> wrappers for as many tokens as possible in my code, which might be more than necessary but I don’t have to style them all. class names The tree-sitter-highlight crate comes with a lightly-documented HtmlRenderer, which does much of the job fairly straightforwardly. The fun part is the attribute_callback. When the HtmlRenderer is wrapping a token, it emits the start of a <span then expects the callback to append whatever HTML attributes it thinks might be appropriate. Uh, I guess I want a class="..." here? Well, the highlight names work a little bit like class names: they have dot-separated parts which tree-sitter-highlight can match more or less specifically. (However I am telling it to match all of them.) So I decided to turn each dot-separated highlight name into a space-separated class attribute. The nice thing about this is that my Rust code doesn’t need to know anything about a language’s tree-sitter grammar or its highlight query. The grammar’s highlight names become CSS class names automatically. styling code Now I can write some simple CSS to add some colours to my code. I can make type names green, code span.hilite.type { color: #aca; } If I decide builtin types should be cyan like keywords I can write, code span.hilite.type.builtin, code span.hilite.keyword { color: #9cc; } results You can look at my tree-sitter-highlight wrapper here. Getting it to work required a bit more creativity than I would have preferred, but it turned out OK. I can add support for a new language by adding a crate to Cargo.toml and a couple of lines to hilite.rs – and maybe some CSS if I have not yet covered its highlight names. (Like I just did to highlight the CSS above!) future work While writing this blog post I found myself complaining about things that I really ought to fix instead. frontmatter I might simplify the per-page source format knob so that I can use pulldown-cmark’s support for YAML frontmatter instead of a separate regex pass. This change will be easier if I can treat the html pages as Markdown without mangling them too much (is Markdown even supposed to be idempotent?). More tricky are a couple of special case pages whose source is Handlebars instead of Markdown. templates I’m not entirely happy with Handlebars. It’s a more powerful language than I need – I chose Handlebars instead of Mustache because Handlebars works neatly with serde. But it has a dynamic type system that makes the templates more error-prone than I would like. Perhaps I can find a more static Rust template system that takes advantage of the close coupling between my templates and the data structure that describes the web site. However, I like my templates to be primarily HTML with a sprinkling of insertions, not something weird that’s neither HTML nor Rust. feed style There’s no CSS in my Atom feed, so code blocks there will remain unstyled. I don’t know if feed readers accept <style> tags or if it has to be inline styles. (That would make a mess of my neat setup!) highlight quality I’m not entirely satisfied with the level of detail and consistency provided by the tree-sitter language grammars and highlight queries. For instance, in the CSS above the class names and property names have the same colour because the CSS highlights.scm gives them the same highlight name. The C grammar is good at identifying variables, but the Rust grammar is not. Oh well, I guess it’s good enough for now. At least it doesn’t involve Javascript.
Simplify complex decisions by separating upsides from downsides, investing in upsides, vetoing with downsides, and using an appropriate decision framework.
I've been running Linux, Neovim, and Framework for a year now, but it easily feels like a decade or more. That's the funny thing about habits: They can be so hard to break, but once you do, they're also easily forgotten. That's how it feels having left the Apple realm after two decades inside the walled garden. It was hard for the first couple of weeks, but since then, it’s rarely crossed my mind. Humans are rigid in the short term, but flexible in the long term. Blessed are the few who can retain the grit to push through that early mental resistance and reach new maxima. That is something that gets harder with age. I can feel it. It takes more of me now to wipe a mental slate clean and start over. To go back to being a beginner. But the reward for learning something new is as satisfying as ever. But it's also why I've tried to be modest with the advocacy. I don't know if most developers are better off on Linux. I mean, I believe they are, at some utopian level, especially if they work for the web, using open source tooling. But I don't know if they are as humans with limited will or capacity for change. Of course, it's fair to say that one doesn't want to. Either because one remain a fan of Apple, in dire need of the remaining edge MacBooks retain on efficiency/battery, or simply content inside the ecosystem. There are plenty of reasons why someone might not want to change. It's not just about rigidity. Besides, it's a dead end trying to convince anyone of an alternative with the sharp end of a religious argument. That kind of crusading just seeds resentment and stubbornness. I know that all too well. What I've found to work much better is planting seeds and showing off your plowshare. Let whatever curiosity that blooms find its own way towards your blue sky. The mimetic engine of persuasion runs much cleaner anyway. And for me, it's primarily about my personal computing workbench regardless of what the world does or doesn't. It was the same with finding Ruby. It's great when others come along for the ride, but I'd also be happy taking the trip solo too. So consider this a postcard from a year into the Linux, Neovim, and Framework journey. The sun is still shining, the wind is in my hair, and the smile on my lips hasn't been this big since the earliest days of OS X.
Yesterday I gave a talk at Monki Gras 2025. This year, the theme is Sustaining Software Development Craft, and here’s the description from the conference website: The big question we want to explore is – how can we keep doing the work we do, when it sustains us, provides meaning and purpose, and sometimes pays the bills? We’re in a period of profound change, technically, politically, socially, economically, which has huge implications for us as practitioners, the makers and doers, but also for the culture at large. I did a talk about the first decade of my career, which I’ve spent working on projects that are designed to last. I’m pleased with my talk, and I got a lot of nice comments. Monki Gras is always a pleasure to attend and speak at – it’s such a lovely, friendly vibe, and the organisers James Governor and Jessica West do a great job of making it a nice day. When I left yesterday, I felt warm and fuzzy and appreciated. I also have a front-row photo of me speaking, courtesy of my dear friend Eriol Fox. Naturally, I chose my outfit to match my slides (and this blog post!). Key points How do you create something that lasts? You can’t predict the future, but there are patterns in what lasts People skills sustain a career more than technical skills Long-lasting systems cannot grow without bound; they need weeding Links/recommended reading Sibyl Schaefer presented a paper Energy, Digital Preservation, and the Climate at iPres 2024, which is about how digital preservation needs to change in anticipation of the climate crisis. This was a major inspiration for this talk. Simon Willison gave a talk Coping strategies for the serial project hoarder at DjangoCon US in 2022, which is another inspiration for me. I’m not as prolific as Simon, but I do see parallels between his approach and what I remember of Metaswitch. Most of the photos in the talk come from the Flickr Commons, a collection of historical photographs from over 100 international cultural heritage organisations. You can learn more about the Commons, browse the photos, and see who’s involved using the Commons Explorer https://commons.flickr.org/. (Which I helped to build!) Slides and notes Photo: dry stone wall building in South Wales. Taken by Wikimedia Commons user TR001, used under CC BY‑SA 3.0. [Make introductory remarks; name and pronouns; mention slides on my website] I’ve been a software developer for ten years, and I’ve spent my career working on projects that are designed to last – first telecoms and networking, now cultural heritage – so when I heard this year’s theme “sustaining craft”, I thought about creating things that last a long time. The key question I want to address in this talk is how do you create something that lasts? I want to share a few thoughts I’ve had from working on decade- and century-scale projects. Part of this is about how we sustain ourselves as software developers, as the individuals who create software, especially with the skill threat of AI and the shifting landscape of funding software. I also want to go broader, and talk about how we sustain the craft, the skill, the projects. Let’s go through my career, and see what we can learn. Photo: women working at a Bell System telephone switchboard. From the U.S. National Archives, no known copyright restrictions. My first software developer job was at a company called Metaswitch. Not a household name, they made telecoms equipment, and you’d probably have heard of their customers. They sold equipment to carriers like AT&T, Vodafone, and O2, who’d use that equipment to sell you telephone service. Telecoms infrastructure is designed to last a long time. I spent most of my time at Metaswitch working with BGP, a routing protocol designed on a pair of napkins in 1989. BGP is sometimes known as the "two-napkin protocol", because of the two napkins on which Kirk Lougheed and Yakov Rekhter wrote the original design. From the Computer History Museum. These are those napkins. This design is basically still the backbone of the Internet. A lot of the building blocks of the telephone network and the Internet are fundamentally the same today as when they were created. I was working in a codebase that had been actively developed for most of my life, and was expected to outlast me. This was my first job so I didn’t really appreciate it at the time, but Metaswitch did a lot of stuff designed to keep that codebase going, to sustain it into the future. Let’s talk about a few of them. Photo: a programmer testing electronic equipment. From the San Diego Air & Space Museum Archives, no known copyright restrictions. Metaswitch was very careful about adopting new technologies. Most of their code was written in C, a little C++, and Rust was being adopted very slowly. They didn’t add new technology quickly. Anything they add, they have to support for a long time – so they wanted to pick technologies that weren’t a flash in the pan. I learnt about something called “the Lindy effect” – this is the idea that any technology is about halfway through its expected life. An open-source library that’s been developed for decades? That’ll probably be around a while longer. A brand new JavaScript framework? That’s a riskier long-term bet. The Lindy effect is about how software that’s been around a long time has already proven its staying power. And talking of AI specifically – I’ve been waiting for things to settle. There’s so much churn and change in this space, if I’d learnt a tool six months ago, most of that would be obsolete today. I don’t hate AI, I love that people are trying all these new tools – but I’m tired and I learning new things is exhausting. I’m waiting for things to calm down before really diving deep on these tools. Metaswitch was very cautious about third-party code, and they didn’t have much of it. Again, anything they use will have to be supported for a long time – is that third-party code, that open-source project stick around? They preferred to take the short-term hit of writing their own code, but then having complete control over it. To give you some idea of how seriously they took this: every third-party dependency had to be reviewed and vetted by lawyers before it could be added to the codebase. Imagine doing that for a modern Node.js project! They had a lot of safety nets. Manual and automated testing, a dedicated QA team, lots of checks and reviews. These were large codebases which had to be reliable. Long-lived systems can’t afford to “move fast and break things”. This was a lot of extra work, but it meant more stability, less churn, and not much risk of outside influences breaking things. This isn’t the only way to build software – Metaswitch is at one extreme of a spectrum – but it did seem to work. I think this is a lesson for building software, but also in what we choose to learn as individuals. Focusing on software that’s likely to last means less churn in our careers. If you learn the fundamentals of the web today, that knowledge will still be useful in five years. If you learn the JavaScript framework du jour? Maybe less so. How do you know what’s going to last? That’s the key question! It’s difficult, but it’s not impossible. This is my first thought for you all: you can’t predict the future, but there are patterns in what lasts. I’ve given you some examples of coding practices that can help the longevity of a codebase, these are just a few. Maybe I have rose-tinted spectacles, but I’ve taken the lessons from Metaswitch and brought them into my current work, and I do like them. I’m careful about external dependencies, I write a lot of my own code, and I create lots of safety nets, and stuff doesn’t tend to churn so much. My code lasts because it isn’t constantly being broken by external forces. Photo: a child in nursery school cutting a plank of wood with a saw. From the Community Archives of Belleville and Hastings County, no known copyright restrictions. So that’s what the smart people were doing at Metaswitch. What was I doing? I joined Metaswitch when I was a young and twenty-something graduate, so I knew everything. I knew software development was easy, these old fuddy-duddies were making it all far too complicated, and I was gonna waltz in and show them how it was done. And obviously, that happened. (Please imagine me reading that paragraph in a very sarcastic voice.) I started doing the work, and it was a lot harder than I expected – who knew that software development was difficult? But I was coming from a background as a solo dev who’d only done hobby projects. I’d never worked in a team before. I didn’t know how to say that I was struggling, to ask for help. I kept making bold promises about what I could do, based on how quickly I thought I should be able to do the work – but I was making promises my skills couldn’t match. I kept missing self-imposed deadlines. You can do that once, but you can’t make it a habit. About six months before I left, my manager said to me “Alex, you have a reputation for being unreliable”. Photo: a boy with a pudding bowl haircut, photographed by Elinor Wiltshire, 1964. From the National Library of Ireland, no known copyright restrictions. He was right! I had such a history of making promises that I couldn’t keep, people stopped trusting me. I didn’t get to work on interesting features or the exciting projects, because nobody trusted me to deliver. That was part of why I left that job – I’d ploughed my reputation into the ground, and I needed to reset. Photo: the library stores at Wellcome Collection. Taken by Thomas SG Farnetti used under CC BY‑NC 4.0. I got that reset at Wellcome Collection, a London museum and library that some of you might know. I was working a lot with their collections, a lot of data and metadata. Wellcome Collection is building on long tradition of libraries and archives, which go back thousands of years. Long-term thinking is in their DNA. To give you one example: there’s stuff in the archive that won’t be made public until the turn of the century. Everybody who works there today will be long gone, but they assume that those records will exist in some shape or form form when that time comes, and they’re planning for those files to eventually be opened. This is century-scale thinking. Photo: Bob Hoover. From the San Diego Air & Space Museum Archives, no known copyright restrictions. When I started, I sat next to a guy called Chris. (I couldn’t find a good picture of him, but I feel like this photo captures his energy.) Chris was a senior archivist. He’d been at Wellcome Collection about twenty-five years, and there were very few people – if anyone – who knew more about the archive than he did. He absolutely knew his stuff, and he could have swaggered around like he owned the place. But he didn’t. Something I was struck by, from my very first day, was how curious and humble he was. A bit of a rarity, if you work in software. He was the experienced veteran of the organisation, but he cared about what other people had to say and wanted to learn from them. Twenty-five years in, and he still wanted to learn. He was a nice guy. He was a pleasure to work with, and I think that’s a big part of why he was able to stay in that job as long as he did. We were all quite disappointed when he left for another job! This is my second thought for you: people skills sustain a career more than technical ones. Being a pleasure to work with opens so many doors and opportunities than technical skill alone cannot. We could do another conference just on what those people skills are, but for now I just want to give you a few examples to think about. Photo: Lt.(jg.) Harriet Ida Pickens and Ens. Frances Wills, first Negro Waves to be commissioned in the US Navy. From the U.S. National Archives, no known copyright restrictions. Be a respectful and reliable teammate. You want to be seen as a safe pair of hands. Reliability isn’t about avoiding mistakes, it’s about managing expectations. If you’re consistently overpromising and underdelivering, people stop trusting you (which I learnt the hard way). If you want people to trust you, you have to keep your promises. Good teammates communicate early when things aren’t going to plan, they ask for help and offer it in return. Good teammates respect the work that went before. It’s tempting to dismiss it as “legacy”, but somebody worked hard on it, and it was the best they knew how to do – recognise that effort and skill, don’t dismiss it. Listen with curiosity and intent. My colleague Chris had decades of experience, but he never acted like he knew everything. He asked thoughtful questions and genuinely wanted to learn from everyone. So many of us aren’t really listening when we’re “listening” – we’re just waiting for the next silence, where we can interject with the next thing we’ve already thought of. We aren’t responding to what other people are saying. When we listen, we get to learn, and other people feel heard – and that makes collaboration much smoother and more enjoyable. Finally, and this is a big one: don’t give people unsolicited advice. We are very bad at this as an industry. We all have so many opinions and ideas, but sometimes, sharing isn’t caring. Feedback is only useful when somebody wants to hear it – otherwise, it feels like criticism, it feels like an attack. Saying “um, actually” when nobody asked for feedback isn’t helpful, it just puts people on the defensive. Asking whether somebody wants feedback, and what sort of feedback they want, will go a long way towards it being useful. So again: people skills sustain a career more than technical skills. There aren’t many truly solo careers in software development – we all have to work with other people – for many of us, that’s the joy of it! If you’re a nice person to work with, other people will want to work with you, to collaborate on projects, they’ll offer you opportunities, it opens doors. Your technical skills won’t sustain your career if you can’t work with other people. Photo: "The Keeper", an exhibition at the New Museum in New York. Taken by Daniel Doubrovkine, used under CC BY‑NC‑SA 4.0. When I went to Wellcome Collection, it was my first time getting up-close and personal with a library and archive, and I didn’t really know how they worked. If you’d asked me, I’d have guessed they just keep … everything? And it was gently explained to me that “No Alex, that’s hoarding.” “Your overflowing yarn stash does not count as an archive.” Big collecting institutions are actually super picky – they have guidelines about what sort of material they collect, what’s in scope, what isn’t, and they’ll aggressively reject anything that isn’t a good match. At Wellcome Collection, their remit was “the history of health and human experience”. You have medical papers? Definitely interesting! Your dad’s old pile of car magazines? Less so. Photo: a dumpster full of books that have been discarded. From brewbooks on Flickr, used under CC BY‑SA 2.0. Collecting institutions also engage in the practice of “weeding” or “deaccessioning”, which is removing material, pruning the collection. For example, in lending libraries, books will be removed from the shelves if they’ve become old, damaged, or unpopular. They may be donated, or sold, or just thrown away – but whatever happens, they’re gotten rid of. That space is reclaimed for other books. Getting rid of material is a fundamental part of professional collecting, because professionals know that storing something has an ongoing cost. They know they can’t keep everything. Photo: a box full of printed photos. From Miray Bostancı on Pexels, used under the Pexels license. This is something I think about in my current job as well. I currently work at the Flickr Foundation, where we’re thinking about how to keep Flickr’s pictures visible for 100 years. How do we preserve social media, how do we maintain our digital legacy? When we talk to people, one thing that comes up regularly is that almost everybody has too many photos. Modern smartphones have made it so easy to snap, snap, snap, and we end up with enormous libraries with thousands of images, but we can’t find the photos we care about. We can’t find the meaningful memories. We’re collecting too much stuff. Digital photos aren’t expensive to store, but we feel the cost in other ways – the cognitive load of having to deal with so many images, of having to sift through a disorganised collection. Photo: a wheelbarrow in a garden. From Hans Middendorp on Pexels, used under the Pexels license. I think there’s a lesson here for the software industry. What’s the cost of all the code that we’re keeping? We construct these enormous edifices of code, but when do we turn things off? When do we delete code? We’re more focused on new code, new ideas, new features. I’m personally quite concerned by how much generative AI has focused on writing more code, and not on dealing with the code we already have. Code is text, so it’s cheap to store, but it still has a cost – it’s more cognitive load, more maintenance, more room for bugs and vulnerabilities. We can keep all our software forever, but we shouldn’t. Photo: Open Garbage Dump on Highway 112, North of San Sebastian. Taken by John Vachon, 1973. From the U.S. National Archives no known copyright restrictions. I think this is going to become a bigger issue for us. We live in an era of abundance, where we can get more computing resources at the push of a button. But that can’t last forever. What happens when our current assumptions about endless compute no longer hold? The climate crisis – where’s all our electricity and hardware coming from? The economics of AI – who’s paying for all these GPU-intensive workloads? And politics – how many of us are dependent on cloud computing based in the US? How many of us feel as good about that as we did three months ago? Libraries are good at making a little go a long way, about eking out their resources, about deciding what’s a good use of resources and what’s waste. Often the people who are good with money are the people who don’t have much of it, and we have a lot of money. It’s easier to make decisions about what to prune and what to keep when things are going well – it’s harder to make decisions in an emergency. This is my third thought for you: long-lasting systems cannot grow without bound; they need weeding. It isn’t sustainable to grow forever, because eventually you get overwhelmed by the weight of everything that came before. We need to get better at writing software efficiently, at turning things off that we don’t need. It’s a skill we’ve neglected. We used to be really good at it – when computers were the size of the room, programmers could eke out every last bit of performance. We can’t do that any more, but it’s so important when building something to last, and I think it’s a skill we’ll have to re-learn soon. Photo: Val Weaver and Vera Askew running in a relay race, Brisbane, 1939. From the State Library of Queensland no known copyright restrictions. Weeding is a term that comes from the preservation world, so let’s stay there. When you talk to people who work in digital preservation, we often describe it as a relay race. There is no permanent digital media, there’s no digital parchment or stone tablets – everything we have today will be unreadable in a few decades. We’re constantly migrating from one format to another, trying to stay ahead of obsolete technology. Software is also a bit of a relay race – there is no “write it once and you’re done”. We’re constantly upgrading, editing, improving. And that can be frustrating, but it also means have regular opportunities to learn and improve. We have that chance to reflect, to do things better. Photo: Broken computer monitor found in the woods. By Jeff Myers on Flickr, used under CC BY‑NC 2.0. I think we do our best reflections when computers go bust. When something goes wrong, we spring into action – we do retrospectives, root cause analysis, we work out what went wrong and how to stop it happening again. This is a great way to build software that lasts, to make it more resilient. It’s a period of intense reflection – what went wrong, how do we stop it happening again? What I’ve noticed is that the best systems are doing this sort of reflection all the time – they aren’t waiting for something to go wrong. They know that prevention is better than cure, and they embody it. They give themselves regular time to reflect, to think about what’s working and what’s not – and when we do, great stuff can happen. Photo: Statue of Astrid Lindgren. By Tobias Barz on Flickr, used under CC BY‑ND 2.0. I want to give you one more example. As a sidebar to my day job, I’ve been writing a blog for thirteen years. It’s the longest job – asterisk – I’ve ever had. The indie web is still cool! A lot of what I write, especially when I was starting, was sharing bits of code. “Here’s something I wrote, here’s what it does, here’s how it works and why it’s cool.” Writing about my code has been an incredible learning experience. You might know have heard the saying “ask a developer to review 5 lines of code, she’ll find 5 issues, ask her to review 500 lines and she’ll say it looks good”. When I sit back and deeply read and explain short snippets of my code, I see how to do things better. I get better at programming. Writing this blog has single-handedly had the biggest impact on my skill as a programmer. Photo: Midnight sun in Advent Bay, Spitzbergen, Norway. From the Library of Congress, no known copyright restrictions. There are so many ways to reflect on our work, opportunities to look back and ask how we can do better – but we have to make the most of them. I think we are, in some ways, very lucky that our work isn’t set in stone, that we do keep doing the same thing, that we have the opportunity to do better. Writing this talk has been, in some sense, a reflection on the first decade of my career, and it’s made me think about what I want the next decade to look like. In this talk, I’ve tried to distill some of those things, tried to give you some of the ideas that I want to keep, that I think will help my career and my software to last. Be careful about what you create, what you keep, and how you interact with other people. That care, that process of reflection – that is what creates things that last. [If the formatting of this post looks odd in your feed reader, visit the original article]