Full Width [alt+shift+f] Shortcuts [alt+shift+k] TRY SIMPLE MODE
Sign Up [alt+shift+s] Log In [alt+shift+l]
168
I’ve been interested in functional reactive programming (FRP) for about a decade now. I even wrote a couple of blog posts back in 2014 describing my experiments. My initial source of inspiration was Elm, the Haskell-like language for the web that once had FRP as a core part of the language. Evan Czaplicki’s Strange Loop 2013 talk really impressed me, especially that Mario demo. From there, I explored the academic literature on the subject. Ultimately, I created and then abandoned a library that focused on FRP for games. It was a neat idea, but the performance was terrible. The overhead of my kinda-sorta FRP system was part of the problem, but mostly it was my own inexperience. I didn’t know how to optimize effectively and my implementation language, Guile, did not have as many optimization passes as it does now. Also, realtime simulations like games require much more careful use of heap allocation. I found that, overhead aside, FRP is a bad fit for things like scripting...
a year ago

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from dthompson

Guile-websocket 0.2.0 released

I'm happy to announce that guile-websocket 0.2.0 has been released! Guile-websocket is an implementation of the WebSocket protocol, both the client and server sides, for Guile Scheme. This release introduces breaking changes that overhaul the client and server implementations in order to support non-blocking I/O and TLS encrypted connections. source tarball: https://files.dthompson.us/guile-websocket/guile-websocket-0.2.0.tar.gz signature: https://files.dthompson.us/guile-websocket/guile-websocket-0.2.0.tar.gz.asc See the guile-websocket project page for more information. Bug reports, bug fixes, feature requests, and patches are welcomed.

6 months ago 86 votes
Wasm GC isn’t ready for realtime graphics

Wasm GC is a wonderful thing that is now available in all major web browsers since slowpoke Safari/WebKit finally shipped it in December. It provides a hierarchy of heap allocated reference types and a set of instructions to operate on them. Wasm GC enables managed memory languages to take advantage of the advanced garbage collectors inside web browser engines. It’s now possible to implement a managed memory language without having to ship a GC inside the binary. The result is smaller binaries, better performance, and better integration with the host runtime. However, Wasm GC has some serious drawbacks when compared to linear memory. I enjoy playing around with realtime graphics programming in my free time, but I was disappointed to discover that Wasm GC just isn’t a good fit for that right now. I decided to write this post because I’d like to see Wasm GC on more or less equal footing with linear memory when it comes to binary data manipulation. Hello triangle For starters, let's take a look at what a “hello triangle” WebGL demo looks like with Wasm GC. I’ll use Hoot, the Scheme to Wasm compiler that I work on, to build it. Below is a Scheme program that declares imports for the subset of the WebGL, HTML5 Canvas, etc. APIs that are necessary and then renders a single triangle: (use-modules (hoot ffi)) ;; Document (define-foreign get-element-by-id "document" "getElementById" (ref string) -> (ref null extern)) ;; Element (define-foreign element-width "element" "width" (ref extern) -> i32) (define-foreign element-height "element" "height" (ref extern) -> i32) ;; Canvas (define-foreign get-canvas-context "canvas" "getContext" (ref extern) (ref string) -> (ref null extern)) ;; WebGL (define GL_VERTEX_SHADER 35633) (define GL_FRAGMENT_SHADER 35632) (define GL_COMPILE_STATUS 35713) (define GL_LINK_STATUS 35714) (define GL_ARRAY_BUFFER 34962) (define GL_STATIC_DRAW 35044) (define GL_COLOR_BUFFER_BIT 16384) (define GL_TRIANGLES 4) (define GL_FLOAT 5126) (define-foreign gl-create-shader "gl" "createShader" (ref extern) i32 -> (ref extern)) (define-foreign gl-delete-shader "gl" "deleteShader" (ref extern) (ref extern) -> none) (define-foreign gl-shader-source "gl" "shaderSource" (ref extern) (ref extern) (ref string) -> none) (define-foreign gl-compile-shader "gl" "compileShader" (ref extern) (ref extern) -> none) (define-foreign gl-get-shader-parameter "gl" "getShaderParameter" (ref extern) (ref extern) i32 -> i32) (define-foreign gl-get-shader-info-log "gl" "getShaderInfoLog" (ref extern) (ref extern) -> (ref string)) (define-foreign gl-create-program "gl" "createProgram" (ref extern) -> (ref extern)) (define-foreign gl-delete-program "gl" "deleteProgram" (ref extern) (ref extern) -> none) (define-foreign gl-attach-shader "gl" "attachShader" (ref extern) (ref extern) (ref extern) -> none) (define-foreign gl-link-program "gl" "linkProgram" (ref extern) (ref extern) -> none) (define-foreign gl-use-program "gl" "useProgram" (ref extern) (ref extern) -> none) (define-foreign gl-get-program-parameter "gl" "getProgramParameter" (ref extern) (ref extern) i32 -> i32) (define-foreign gl-get-program-info-log "gl" "getProgramInfoLog" (ref extern) (ref extern) -> (ref string)) (define-foreign gl-create-buffer "gl" "createBuffer" (ref extern) -> (ref extern)) (define-foreign gl-delete-buffer "gl" "deleteBuffer" (ref extern) (ref extern) -> (ref extern)) (define-foreign gl-bind-buffer "gl" "bindBuffer" (ref extern) i32 (ref extern) -> none) (define-foreign gl-buffer-data "gl" "bufferData" (ref extern) i32 (ref eq) i32 -> none) (define-foreign gl-enable-vertex-attrib-array "gl" "enableVertexAttribArray" (ref extern) i32 -> none) (define-foreign gl-vertex-attrib-pointer "gl" "vertexAttribPointer" (ref extern) i32 i32 i32 i32 i32 i32 -> none) (define-foreign gl-draw-arrays "gl" "drawArrays" (ref extern) i32 i32 i32 -> none) (define-foreign gl-viewport "gl" "viewport" (ref extern) i32 i32 i32 i32 -> none) (define-foreign gl-clear-color "gl" "clearColor" (ref extern) f64 f64 f64 f64 -> none) (define-foreign gl-clear "gl" "clear" (ref extern) i32 -> none) (define (compile-shader gl type source) (let ((shader (gl-create-shader gl type))) (gl-shader-source gl shader source) (gl-compile-shader gl shader) (unless (= (gl-get-shader-parameter gl shader GL_COMPILE_STATUS) 1) (let ((info (gl-get-shader-info-log gl shader))) (gl-delete-shader gl shader) (error "shader compilation failed" info))) shader)) (define (link-shader gl vertex-shader fragment-shader) (let ((program (gl-create-program gl))) (gl-attach-shader gl program vertex-shader) (gl-attach-shader gl program fragment-shader) (gl-link-program gl program) (unless (= (gl-get-program-parameter gl program GL_LINK_STATUS) 1) (let ((info (gl-get-program-info-log gl program))) (gl-delete-program gl program) (error "program linking failed" info))) program)) ;; Setup GL context (define canvas (get-element-by-id "canvas")) (define gl (get-canvas-context canvas "webgl")) (when (external-null? gl) (error "unable to create WebGL context")) ;; Compile shader (define vertex-shader-source "attribute vec2 position; attribute vec3 color; varying vec3 fragColor; void main() { gl_Position = vec4(position, 0.0, 1.0); fragColor = color; }") (define fragment-shader-source "precision mediump float; varying vec3 fragColor; void main() { gl_FragColor = vec4(fragColor, 1); }") (define vertex-shader (compile-shader gl GL_VERTEX_SHADER vertex-shader-source)) (define fragment-shader (compile-shader gl GL_FRAGMENT_SHADER fragment-shader-source)) (define shader (link-shader gl vertex-shader fragment-shader)) ;; Create vertex buffer (define stride (* 4 5)) (define buffer (gl-create-buffer gl)) (gl-bind-buffer gl GL_ARRAY_BUFFER buffer) (gl-buffer-data gl GL_ARRAY_BUFFER #f32(-1.0 -1.0 1.0 0.0 0.0 1.0 -1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0) GL_STATIC_DRAW) ;; Draw (gl-viewport gl 0 0 (element-width canvas) (element-height canvas)) (gl-clear gl GL_COLOR_BUFFER_BIT) (gl-use-program gl shader) (gl-enable-vertex-attrib-array gl 0) (gl-vertex-attrib-pointer gl 0 2 GL_FLOAT 0 stride 0) (gl-enable-vertex-attrib-array gl 1) (gl-vertex-attrib-pointer gl 1 3 GL_FLOAT 0 stride 8) (gl-draw-arrays gl GL_TRIANGLES 0 3) Note that in Scheme, the equivalent of a Uint8Array is a bytevector. Hoot uses a packed array, an (array i8) specifically, for the contents of a bytevector. And here is the JavaScript code necessary to boot the resulting Wasm binary: window.addEventListener("load", async () => { function bytevectorToUint8Array(bv) { let len = reflect.bytevector_length(bv); let array = new Uint8Array(len); for (let i = 0; i < len; i++) { array[i] = reflect.bytevector_ref(bv, i); } return array; } let mod = await SchemeModule.fetch_and_instantiate("triangle.wasm", { reflect_wasm_dir: 'reflect-wasm', user_imports: { document: { getElementById: (id) => document.getElementById(id) }, element: { width: (elem) => elem.width, height: (elem) => elem.height }, canvas: { getContext: (elem, type) => elem.getContext(type) }, gl: { createShader: (gl, type) => gl.createShader(type), deleteShader: (gl, shader) => gl.deleteShader(shader), shaderSource: (gl, shader, source) => gl.shaderSource(shader, source), compileShader: (gl, shader) => gl.compileShader(shader), getShaderParameter: (gl, shader, param) => gl.getShaderParameter(shader, param), getShaderInfoLog: (gl, shader) => gl.getShaderInfoLog(shader), createProgram: (gl, type) => gl.createProgram(type), deleteProgram: (gl, program) => gl.deleteProgram(program), attachShader: (gl, program, shader) => gl.attachShader(program, shader), linkProgram: (gl, program) => gl.linkProgram(program), useProgram: (gl, program) => gl.useProgram(program), getProgramParameter: (gl, program, param) => gl.getProgramParameter(program, param), getProgramInfoLog: (gl, program) => gl.getProgramInfoLog(program), createBuffer: (gl) => gl.createBuffer(), deleteBuffer: (gl, buffer) => gl.deleteBuffer(buffer), bindBuffer: (gl, target, buffer) => gl.bindBuffer(target, buffer), bufferData: (gl, buffer, data, usage) => { let bv = new Bytevector(reflect, data); gl.bufferData(buffer, bytevectorToUint8Array(bv), usage); }, enableVertexAttribArray: (gl, index) => gl.enableVertexAttribArray(index), vertexAttribPointer: (gl, index, size, type, normalized, stride, offset) => { gl.vertexAttribPointer(index, size, type, normalized, stride, offset); }, drawArrays: (gl, mode, first, count) => gl.drawArrays(mode, first, count), viewport: (gl, x, y, w, h) => gl.viewport(x, y, w, h), clearColor: (gl, r, g, b, a) => gl.clearColor(r, g, b, a), clear: (gl, mask) => gl.clear(mask) } } }); let reflect = await mod.reflect({ reflect_wasm_dir: 'reflect-wasm' }); let proc = new Procedure(reflect, mod.get_export("$load").value); proc.call(); }); Hello problems There are two major performance issues with this program. One is visible in the source above, the other is hidden in the language implementation. Heap objects are opaque on the other side Wasm GC heap objects are opaque to the host. Likewise, heap objects from the host are opaque to the Wasm guest. Thus the contents of an (array i8) object are not visible from JavaScript and the contents of a Uint8Array are not visible from Wasm. This is a good security property in the general case, but it’s a hinderance in this specific case. Let’s say we have an (array i8) full of vertex data we want to put into a WebGL buffer. To do this, we must make one JS->Wasm call for each byte in the array and store it into a Uint8Array. This is what the bytevectorToUint8Array function above is doing. Copying any significant amount of data per frame is going to tank performance. Hope you aren’t trying to stream vertex data! Contrast the previous paragraph with Wasm linear memory. A WebAssembly.Memory object can be easily accessed from JavaScript as an ArrayBuffer. To get a blob of vertex data out of a memory object, you just need to know the byte offset and length and you’re good to go. There are many Wasm linear memory applications using WebGL successfully. Manipulating multi-byte binary data is inefficient To read a multi-byte number such as an unsigned 32-bit integer from an (array i8), you have to fetch each individual byte and combine them together. Here’s a self-contained example that uses Guile-flavored WAT format: (module (type $bytevector (array i8)) (data $init #u32(123456789)) (func (export "main") (result i32) (local $a (ref $bytevector)) (local.set $a (array.new_data $bytevector $init (i32.const 0) (i32.const 4))) (array.get_u $bytevector (local.get $a) (i32.const 0)) (i32.shl (array.get_u $bytevector (local.get $a) (i32.const 1)) (i32.const 8)) (i32.or) (i32.shl (array.get_u $bytevector (local.get $a) (i32.const 2)) (i32.const 16)) (i32.or) (i32.shl (array.get_u $bytevector (local.get $a) (i32.const 3)) (i32.const 24)) (i32.or))) By contrast, Wasm linear memory needs but a single i32.load instruction: (module (memory 1) (func (export "main") (result i32) (i32.store (i32.const 0) (i32.const 123456789)) (i32.load (i32.const 0)))) Easy peasy. Not only is it less code, it's a lot more efficient. Unsatisfying workarounds There’s no way around the multi-byte problem at the moment, but for byte access from JavaScript there are some things we could try to work with what we have been given. Spoiler alert: None of them are pleasant. Use Uint8Array from the host This approach makes all binary operations from within the Wasm binary slow since we’d have to cross the Wasm->JS bridge for each read/write. Since most of the binary data manipulation is happening in the Wasm module, this approach will just make things slower overall. Use linear memory for bytevectors This would require a little malloc/free implementation and a way to reclaim memory for GC'd bytevectors. You could register every bytevector in a FinalizationRegistry in order to be notified upon GC and free the memory. Now you have to deal with memory fragmentation. This is Wasm GC, we shouldn’t have to do any of this! Use linear memory as a scratch space This avoids crossing the Wasm/JS boundary for each byte, but still involves a byte-by-byte copy from (array i8) to linear memory within the Wasm module. So far this feels like the least worst option, but the extra copy is still going to greatly reduce throughput. Wasm GC needs some fixin' I’ve used realtime graphics as an example because it’s a use case that is very sensitive to performance issues, but this unfortunate need to copy binary data byte-by-byte is also the reason why strings are trash on Wasm GC right now. Stringref is a good proposal and the Wasm community group made a mistake by rejecting it. Anyway, there has been some discussion about both multi-byte and ArrayBuffer access on GitHub, but as far as I can tell neither issue is anywhere close to a resolution. Can these things be implemented efficiently? How can the need for direct access to packed arrays from JS be reconciled with Wasm heap object opaqueness? I hope the Wasm community group can arrive at solutions sooner than later because it will take a long time to get the proposal(s) to phase 4 and shipped in all browsers, perhaps years. It would be a shame to be effectively shut out from using WebGPU when it finally reaches stable browser releases.

7 months ago 84 votes
Guile-Bstructs 0.1.0 released

I'm pleased to announce that the very first release of guile-bstructs, version 0.1.0, has been released! This is a library I've been working on for quite some time and after more than one rewrite and many smaller refactors I think it's finally ready to release publicly. Let's hope I'm not wrong about that! About guile-bstructs Guile-bstructs is a library that provides structured read/write access to binary data for Guile. A bstruct (short for “binary structure”) is a data type that encapsulates a bytevector and a byte offset which interprets that bytevector based on a specified layout. Some use cases for bstructs are: manipulating C structs when using the foreign function interface packing GPU vertex buffers when using graphics APIs such as OpenGL implementing data types that benefit from Guile's unboxed math optimizations such as vectors and matrices This library was initially inspired by guile-opengl's define-packed-struct syntax but is heavily based on "Ftypes: Structured foreign types" by Andy Keep and R. Kent Dybvig. The resulting interface is quite similar but the implementation is completely original. This library provides a syntax-heavy interface; nearly all of the public API is syntax. This is done to ensure that bstruct types are static and well-known at compile time resulting in efficient bytecode and minimal runtime overhead. A subset of the interface deals in raw bytevector access for accessing structured data in bytevectors directly without going through an intermediary bstruct wrapper. This low-level interface is useful for certain batch processing situations where the overhead of creating wrapper bstructs would hinder throughput. Example Here are some example type definitions to give you an idea of what it’s like to use guile-bstructs: ;; Struct (define-bstruct <vec2> (padded (struct (x float) (y float)))) ;; Type group with a union (define-bstruct (<mouse-move-event> (struct (type uint8) (x int32) (y int32))) (<mouse-button-event> (struct (type uint8) (button uint8) (state uint8) (x int32) (y int32))) (<event> (union (type uint8) (mouse-move <mouse-move-event>) (mouse-button <mouse-button-event>)))) ;; Array (define-bstruct <matrix4> (array 16 float)) ;; Bit fields (define-bstruct <date> (bits (year 32 s) (month 4 u) (day 5 u))) ;; Pointer (define-bstruct (<item> (struct (type int))) (<chest> (struct (opened? uint8) (item (* <item>))))) ;; Packed struct modifier (define-bstruct <enemy> (packed (struct (type uint8) (health uint32)))) ;; Endianness modifier (define-bstruct <big-float> (endian big float)) ;; Recursive type (define-bstruct <node> (struct (item int) (next (* <node>)))) ;; Mutually recursive type group (define-bstruct (<forest> (struct (children (* <tree>)))) (<tree> (struct (value int) (forest (* <forest>)) (next (* <tree>))))) ;; Opaque type (define-bstruct SDL_GPUTexture) Download Source tarball: guile-bstructs-0.1.0.tar.gz GPG signature: guile-bstructs-0.1.0.tar.gz.asc This release was signed with this GPG key. See the guile-bstructs project page for more information.

7 months ago 86 votes
Lisp: Icing or Cake?

The Spring Lisp Game Jam 2024 ended one week ago. 48 games were submitted, a new record for the jam! This past week has been a time for participants to play and rate each other’s games. As I explored the entries, I noticed two distinct meta-patterns in how people approached building games with Lisp. I think these patterns apply more broadly to all applications of Lisp. Let’s talk about these patterns in some detail, with examples. But first! Here’s the breakdown of the jam submissions by language: lang entries % (rounded) ---- ------- ----------- guile 15 31 fennel 10 21 clojure 5 10 cl 5 10 racket 4 8 elisp 4 8 s7 3 6 kawa 1 2 owl 1 2 I haven’t rolled up the various Schemes (Guile, Racket, S7, Kawa) into a general scheme category because Scheme is so minimally specified and they are all very distinct implementations for different purposes, not to mention that Racket has a lot more going on than just Scheme. For the first time ever, Guile came out on top with the most submissions! There’s a very specific reason for this outcome. 11 out of the 15 Guile games were built for the web with Hoot, a Scheme-to-WebAssembly compiler that I work on at the Spritely Institute. 2 of those 11 were official Spritely projects. We put out a call for people to try making games with Hoot before the jam started, and a lot of people took us up on it! Very cool! The next most popular language, which is typically the most popular language in these jams, is Fennel. Fennel is a Lisp that compiles to Lua. It’s very cool, too! Also of note, at least to me as a Schemer, is that three games used S7. Hmm, there might be something relevant to this post going on there. The patterns I’m about to talk about could sort of be framed as “The Guile Way vs. The Fennel Way”, but I don’t want to do that. It's not an “us vs. them” thing. It’s wonderful that there are so many flavors of Lisp these days that anyone can find a great implementation that suits their preferences. Not only that, but many of these implementations can be used to make games that anyone can easily play in their web browser! That was not the case several years ago. Incredible! I want to preface the rest of this post by saying that both patterns are valid, and while I prefer one over the other, that is not to say that the other is inferior. I'll also show how these patterns can be thought of as two ends of a spectrum and how, in the end, compromises must be made. Okay, let’s get into it! Lisp as icing The icing pattern is using Lisp as a “scripting” language on top of a cake that is made from C, Rust, and other static languages. The typical way to do this is by embedding a Lisp interpreter into the larger program. If you’re most interested in writing the high-level parts of an application in Lisp then this pattern is the fastest way to get there. All you need is a suitable interpreter/compiler and a way to add the necessary hooks into your application. Since the program is mainly C/Rust/whatever, you can then use emscripten to compile it to WebAssembly and deploy to the web. Instant gratification, but strongly tied to static languages and their toolchains. S7 is an example of an embeddable Scheme. Guile is also used for extending C programs, though typically that involves dynamically linking to libguile rather than embedding the interpreter into the program’s executable. Fennel takes a different approach, recognizing that there are many existing applications that are already extensible through Lua, and provides a lispy language that compiles to Lua. Lisp as cake The cake pattern is using Lisp to implement as much of the software stack as possible. It’s Lisp all the way down... sorta. Rather than embedding Lisp into a non-Lisp program, the cake pattern does the inverse: the majority of the program is written in Lisp. When necessary, shared libraries can be called via a foreign function interface, but this should be kept to a minimum. This approach takes longer to yield results. Time is spent implementing missing libraries for your Lisp of choice and writing wrappers around the C shared libraries you can’t avoid using. Web deployment gets trickier, too, since the project is not so easily emscriptenable. (You may recognize this as the classic embed vs. extend debate. You’re correct! I'm just adding my own thoughts and applying it specifically to some real-world Lisp projects.) I mentioned Guile as an option for icing, but Guile really shines best as cake. The initial vision for Guile was to Emacsify other programs by adding a Scheme interpreter to them. These days, the best practice is to write your program in Scheme to begin with. Common Lisp is probably the best example, though. Implementations like SBCL have good C FFIs and can compile efficient native executables, minimizing the desire to use some C for performance reasons. Case studies Let’s take a look at some of the languages and libraries used for the Lisp Game Jam and evaluate their icing/cake-ness. Fennel + love2d love2d has been a popular choice for solo or small team game development for many years. It is a C++ program that embeds a Lua interpreter, which means it’s a perfect target for Fennel. Most Linux distributions package love2d, so it’s easy to run .love files natively. Additionally, thanks to emscripten, love2d games can be deployed to the web. Thus most Fennel games use love2d. ./soko.bin and Gnomic Vengeance are two games that use this stack. Fennel + love2d is a perfect example of Lisp as icing. Fennel sits at the very top of the stack, but there’s not really a path to spread Lisp into the layers below. It is also the most successful Lisp game development stack to date. S7 + raylib This stack is new to me, but two games used it this time around: GhostHop and Life Predictor. (You really gotta play GhostHop, btw. It’s a great little puzzle game and it is playable on mobile devices.) Raylib is a C library with bindings for many higher-level languages that has become quite popular in recent years. S7 is also implemented in C and is easily embeddable. This makes the combination easy to deploy on the web with emscripten. S7 + raylib is another example of Lisp as icing. I’m curious to see if this stack becomes more popular in future jams. Guile + Chickadee This is the stack that I helped build. Chickadee is a game library for Guile that implements almost all of the interesting parts in Scheme, including rendering. Two games were built with Chickadee in the most recent jam: Turbo Racer 3000 and Bloatrunner. Guile + Chickadee is an example of Lisp as cake. Chickadee wraps some C libraries for low-level tasks such as loading images, audio, and fonts, but it is written in pure Scheme. All the matrix and vector math is in Scheme. Chickadee comes with a set of rendering primitives comparable to love2d and raylib but they’re all implemented in Scheme. I’ve even made progress on rendering vector graphics with Scheme, whereas most other Lisp game libraries use a C library such as nanosvg. Chickadee has pushed the limits of Guile’s compiler and virtual machine, and Guile has been improved as a result. But it’s the long road. Chickadee is mostly developed by me, alone, in my very limited spare time. It is taking a long time to reach feature parity with more popular game development libraries, but it works quite well for what it is. Hoot + HTML5 canvas I also helped build this one. Hoot is a Scheme-to-WebAssembly compiler. Rather than compile the Guile VM (written in C) to Wasm using emscripten, Hoot implements a complete Wasm toolchain and a new backend for Guile’s compiler that emits Wasm directly. Hoot is written entirely in Scheme. Unlike C programs compiled with emscripten that target Wasm 1.0 with linear memory, Hoot targets Wasm 2.0 with GC managed heap types. This gives Hoot a significant advantage: Hoot binaries do not ship a garbage collector and thus are much smaller than Lisp runtimes compiled via emscripten. The Wasm binary for my game weighs in at < 2MiB whereas the love2d game I checked had a nearly 6MiB love.wasm. Hoot programs can also easily interoperate with JavaScript. Scheme objects can easily be passed to JavaScript, and vice versa, as they are managed in the same heap. With all of the browser APIs just a Wasm import away, an obvious choice for games was the built-in HTML5 canvas API for easy 2D rendering. 11 games used Hoot in the jam, including (shameless plug) Cirkoban and Lambda Dungeon. Hoot + HTML5 canvas is mostly dense cake with a bit of icing. On one hand, it took a year and significant funding to boot Hoot. We said “no” to emscripten, built our own toolchain, and extended Guile’s compiler. It's Lisp all the way until you hit the browser runtime! We even have a Wasm interpreter that runs on the Guile VM! Hoot rules! It was a risk but it paid off. On the other hand, the canvas API is very high-level. The more cake thing to do would be to use Hoot’s JS FFI to call WebGL and/or WebGPU. Indeed, this is the plan for the future! Wasm GC needs some improvements to make this feasible, but my personal goal is to get Chickadee ported to Hoot. I want Chickadee games to be easy to play natively and in browsers, just like love2d games. The cake/icing spectrum I must acknowledge the limitations of the cake approach. We’re not living in a world of Lisp machines, but a world of glorified PDP-11s. Even the tallest of Lisp cakes sits atop an even larger cake made mostly of C. All modern Lisp systems bottom out at some point. Emacs rests on a C core. Guile’s VM is written in C. Hoot runs on mammoth JavaScript engines written in C++ like V8. Games on Hoot currently render with HTML5 canvas rather than WebGL/WebGPU. Good luck using OpenGL without libGL; Chickadee uses guile-opengl which uses the C FFI to call into libGL. Then there’s libpng, FreeType, and more. Who the heck wants to rewrite all this in Lisp? Who even has the resources? Does spending all this time taking the scenic route matter at all, or are we just deluding ourselves because we have fun writing Lisp code? I think it does matter. Every piece of the stack that can be reclaimed from the likes of C is a small victory. The parts written in Lisp are much easier to hack on, and some of those things become live hackable while our programs are running. They are also memory safe, typically, thanks to GC managed runtimes. Less FFI calls means less overhead from traversing the Lisp/C boundary and more safety. As more of the stack becomes Lisp, it starts looking less like icing and more like cake. Moving beyond games, we can look to the Guix project as a great example of just how tasty the cake can get. Guix took the functional packaging model from the Nix project and made a fresh implementation, replacing the Nix language with Guile. Why? For code staging, code sharing, and improved hackability. Guix also uses an init system written in Guile rather than systemd. Why? For code staging, code sharing, and improved hackability. These are real advantages that make the trade-off of not using the industry-standard thing worth it. I’ve been using Guix since the early days, and back then it was easy to make the argument that Guix was just reinventing wheels for no reason. But now, over 10 years later, the insistence on maximizing the usage of Lisp has been key to the success of the project. As a user, once you learn the Guix idioms and a bit of Guile, you unlock extraordinary power to craft your OS to your liking. It’s the closest thing you can get to a Lisp machine on modern hardware. The cake approach paid off for Guix, and it could pay off for other projects, too. If Common Lisp is more your thing, and even if it isn’t, you’ll be amazed by the Trial game engine and how much of it is implemented in Common Lisp rather than wrapping C libraries. There’s also projects like Pre-Scheme that give me hope that one day the layers below the managed GC runtime can be implemented in Lisp. Pre-Scheme was developed and successfully used for Scheme 48 and I am looking forward to a modern revival of it thanks to an NLnet grant. I'm a cake boy That’s right, I said it: I’m a cake boy. I want to see projects continue to push the boundaries of what Lisp can do. When it comes to the Lisp Game Jam, what excites me most are not the games themselves, but the small advances made to reclaim another little slice of the cake from stale, dry C. I intend to keep pushing the limits for Guile game development with my Chickadee project. It’s not a piece of cake to bake a lispy cake, and the way is often hazy, but I know we can’t be lazy and just do the cooking by the book. Rewrite it in Rust? No way! Rewrite it in Lisp!

a year ago 163 votes

More in programming

The future of large files in Git is Git

.title {text-wrap:balance;} #content > p:first-child {text-wrap:balance;} If Git had a nemesis, it’d be large files. Large files bloat Git’s storage, slow down git clone, and wreak havoc on Git forges. In 2015, GitHub released Git LFS—a Git extension that hacked around problems with large files. But Git LFS added new complications and storage costs. Meanwhile, the Git project has been quietly working on large files. And while LFS ain’t dead yet, the latest Git release shows the path towards a future where LFS is, finally, obsolete. What you can do today: replace Git LFS with Git partial clone Git LFS works by storing large files outside your repo. When you clone a project via LFS, you get the repo’s history and small files, but skip large files. Instead, Git LFS downloads only the large files you need for your working copy. In 2017, the Git project introduced partial clones that provide the same benefits as Git LFS: Partial clone allows us to avoid downloading [large binary assets] in advance during clone and fetch operations and thereby reduce download times and disk usage. – Partial Clone Design Notes, git-scm.com Git’s partial clone and LFS both make for: Small checkouts – On clone, you get the latest copy of big files instead of every copy. Fast clones – Because you avoid downloading large files, each clone is fast. Quick setup – Unlike shallow clones, you get the entire history of the project—you can get to work right away. What is a partial clone? A Git partial clone is a clone with a --filter. For example, to avoid downloading files bigger than 100KB, you’d use: git clone --filter='blobs:size=100k' <repo> Later, Git will lazily download any files over 100KB you need for your checkout. By default, if I git clone a repo with many revisions of a noisome 25 MB PNG file, then cloning is slow and the checkout is obnoxiously large: $ time git clone https://github.com/thcipriani/noise-over-git Cloning into '/tmp/noise-over-git'... ... Receiving objects: 100% (153/153), 1.19 GiB real 3m49.052s Almost four minutes to check out a single 25MB file! $ du --max-depth=0 --human-readable noise-over-git/. 1.3G noise-over-git/. $ ^ 🤬 And 50 revisions of that single 25MB file eat 1.3GB of space. But a partial clone side-steps these problems: $ git config --global alias.pclone 'clone --filter=blob:limit=100k' $ time git pclone https://github.com/thcipriani/noise-over-git Cloning into '/tmp/noise-over-git'... ... Receiving objects: 100% (1/1), 24.03 MiB real 0m6.132s $ du --max-depth=0 --human-readable noise-over-git/. 49M noise-over-git/ $ ^ 😻 (the same size as a git lfs checkout) My filter made cloning 97% faster (3m 49s → 6s), and it reduced my checkout size by 96% (1.3GB → 49M)! But there are still some caveats here. If you run a command that needs data you filtered out, Git will need to make a trip to the server to get it. So, commands like git diff, git blame, and git checkout will require a trip to your Git host to run. But, for large files, this is the same behavior as Git LFS. Plus, I can’t remember the last time I ran git blame on a PNG 🙃. Why go to the trouble? What’s wrong with Git LFS? Git LFS foists Git’s problems with large files onto users. And the problems are significant: 🖕 High vendor lock-in – When GitHub wrote Git LFS, the other large file systems—Git Fat, Git Annex, and Git Media—were agnostic about the server-side. But GitHub locked users to their proprietary server implementation and charged folks to use it.1 💸 Costly – GitHub won because it let users host repositories for free. But Git LFS started as a paid product. Nowadays, there’s a free tier, but you’re dependent on the whims of GitHub to set pricing. Today, a 50GB repo on GitHub will cost $40/year for storage. In contrast, storing 50GB on Amazon’s S3 standard storage is $13/year. 😰 Hard to undo – Once you’ve moved to Git LFS, it’s impossible to undo the move without rewriting history. 🌀 Ongoing set-up costs – All your collaborators need to install Git LFS. Without Git LFS installed, your collaborators will get confusing, metadata-filled text files instead of the large files they expect. The future: Git large object promisors Large files create problems for Git forges, too. GitHub and GitLab put limits on file size2 because big files cost more money to host. Git LFS keeps server-side costs low by offloading large files to CDNs. But the Git project has a new solution. Earlier this year, Git merged a new feature: large object promisers. Large object promisors aim to provide the same server-side benefits as LFS, minus the hassle to users. This effort aims to especially improve things on the server side, and especially for large blobs that are already compressed in a binary format. This effort aims to provide an alternative to Git LFS – Large Object Promisors, git-scm.com What is a large object promisor? Large object promisors are special Git remotes that only house large files. In the bright, shiny future, large object promisors will work like this: You push a large file to your Git host. In the background, your Git host offloads that large file to a large object promisor. When you clone, the Git host tells your Git client about the promisor. Your client will clone from the Git host, and automagically nab large files from the promisor remote. But we’re still a ways off from that bright, shiny future. Git large object promisors are still a work in progress. Pieces of large object promisors merged to Git in March of 2025. But there’s more to do and open questions yet to answer. And so, for today, you’re stuck with Git LFS for giant files. But once large object promisors see broad adoption, maybe GitHub will let you push files bigger than 100MB. The future of large files in Git is Git. The Git project is thinking hard about large files, so you don’t have to. Today, we’re stuck with Git LFS. But soon, the only obstacle for large files in Git will be your half-remembered, ominous hunch that it’s a bad idea to stow your MP3 library in Git. Edited by Refactoring English Later, other Git forges made their own LFS servers. Today, you can push to multiple Git forges or use an LFS transfer agent, but all this makes set up harder for contributors. You’re pretty much locked-in unless you put in extra effort to get unlocked.↩︎ File size limits: 100MB for GitHub, 100MB for GitLab.com↩︎

yesterday 7 votes
Just a Little More Context Bro, I Promise, and It’ll Fix Everything

Conrad Irwin has an article on the Zed blog “Why LLMs Can't Really Build Software”. He says it boils down to: the distinguishing factor of effective engineers is their ability to build and maintain clear mental models We do this by: Building a mental model of what you want to do Building a mental model of what the code does Reducing the difference between the two It’s kind of an interesting observation about how we (as humans) problem solve vs. how we use LLMs to problem solve: With LLMs, you stuff more and more information into context until it (hopefully) has enough to generate a solution. With your brain, you tweak, revise, or simplify your mental model more and more until the solution presents itself. One adds information — complexity you might even say — to solve a problem. The other eliminates it. You know what that sort of makes me think of? NPM driven development. Solving problems with LLMs is like solving front-end problems with NPM: the “solution” comes through installing more and more things — adding more and more context, i.e. more and more packages. LLM: Problem? Add more context. NPM: Problem? There’s a package for that. Contrast that with a solution that comes through simplification. You don’t add more context. You simplify your mental model so you need less to solve a problem — if you solve it at all, perhaps you eliminate the problem entirely! Rather than install another package to fix what ails you, you simplify your mental model which often eliminates the problem you had in the first place; thus eliminating the need to solve any problem at all, or to add any additional context or complexity (or dependency). As I’m typing this, I’m thinking of that image of the evolution of the Raptor engine, where it evolved in simplicity: This stands in contrast to my working with LLMs, which often wants more and more context from me to get to a generative solution: I know, I know. There’s probably a false equivalence here. This entire post started as a note and I just kept going. This post itself needs further thought and simplification. But that’ll have to come in a subsequent post, otherwise this never gets published lol. Email · Mastodon · Bluesky

yesterday 4 votes
How to Leverage the CPU’s Micro-Op Cache for Faster Loops

Measuring, analyzing, and optimizing loops using Linux perf, Top-Down Microarchitectural Analysis, and the CPU’s micro-op cache

yesterday 6 votes
Omarchy micro-forks Chromium

You can just change things! That's the power of open source. But for a lot of people, it might seem like a theoretical power. Can you really change, say, Chrome? Well, yes! We've made a micro fork of Chromium for Omarchy (our new 37signals Linux distribution). Just to add one feature needed for live theming. And now it's released as a package anyone can install on any flavor of Arch using the AUR (Arch User Repository). We got it all done in just four days. From idea, to solicitation, to successful patch, to release, to incorporation. And now it'll be part of the next release of Omarchy. There are no speed limits in open source. Nobody to ask for permission. You have the code, so you can make the change. All you need is skill and will (and maybe, if you need someone else to do it for you, a $5,000 incentive 😄).

2 days ago 4 votes
Choosing Tools To Make Websites

Jan Miksovsky lays out his idea for website creation as content transformation. He starts by talking about tools that hide what’s happening “under the hood”: A framework’s marketing usually pretends it is unnecessary for you to understand how its core transformation works — but without that knowledge, you can’t achieve the beautiful range of results you see in the framework’s sample site gallery. This is a great callout. Tools will say, “You don’t have to worry about the details.” But the reality is, you end up worrying about the details — at least to some degree. Why? Because what you want to build is full of personalization. That’s how you differentiate yourself, which means you’re going to need a tool that’s expressive enough to help you. So the question becomes: how hard is it to understand the details that are being intentionally hidden away? A lot of the time those details are not exposed directly. Instead they’re exposed through configuration. But configuration doesn’t really help you learn how something works. I mean, how many of you have learned how typescript works under the hood by using tsconfig.json? As Jan says: Configuration can lead to as many problems as it solves Nailed it. He continues: Configuring software is itself a form of programming, in fact a rather difficult and often baroque form. It can take more data files or code to configure a framework’s transformation than to write a program that directly implements that transformation itself. I’m not a Devops person, but that sounds like Devops in a nutshell right there. (It also perfectly encapsulates my feelings on trying to setup configuration in GitHub Actions.) Jan moves beyond site creation to also discuss site hosting. He gives good reasons for keeping your website’s architecture simple and decoupled from your hosting provider (something I’ve been a long time proponent of): These site hosting platforms typically charge an ongoing subscription fee. (Some offer a free tier that may meet your needs.) The monthly fee may not be large, but it’s forever. Ten years from now you’ll probably still want your content to be publicly available, but will you still be happy paying that monthly fee? If you stop paying, your site disappears. In subscription pricing, any price (however small) is recurring. Stated differently: pricing is forever. Anyhow, it’s a good read from Jan and lays out his vision for why he’s building Web Origami: a tool for that encourages you to understand (and customize) how you transform content to a website. He just launched version 0.4.0 which has some exciting stuff I’m excited to try out further (I’ll have to write about all that later). Email · Mastodon · Bluesky

3 days ago 5 votes