Full Width [alt+shift+f] Shortcuts [alt+shift+k]
Sign Up [alt+shift+s] Log In [alt+shift+l]
157
The Spring Lisp Game Jam 2024 ended one week ago. 48 games were submitted, a new record for the jam! This past week has been a time for participants to play and rate each other’s games. As I explored the entries, I noticed two distinct meta-patterns in how people approached building games with Lisp. I think these patterns apply more broadly to all applications of Lisp. Let’s talk about these patterns in some detail, with examples. But first! Here’s the breakdown of the jam submissions by language: lang entries % (rounded) ---- ------- ----------- guile 15 31 fennel 10 21 clojure 5 10 cl 5 10 racket 4 8 elisp 4 8 s7 3 6 kawa 1 2 owl 1 2 I haven’t rolled up the various Schemes (Guile, Racket, S7, Kawa) into a general scheme category because Scheme is so minimally specified and they are all very distinct implementations for different...
a year ago

Improve your reading experience

Logged in users get linked directly to articles resulting in a better reading experience. Please login for free, it takes less than 1 minute.

More from dthompson

Guile-websocket 0.2.0 released

I'm happy to announce that guile-websocket 0.2.0 has been released! Guile-websocket is an implementation of the WebSocket protocol, both the client and server sides, for Guile Scheme. This release introduces breaking changes that overhaul the client and server implementations in order to support non-blocking I/O and TLS encrypted connections. source tarball: https://files.dthompson.us/guile-websocket/guile-websocket-0.2.0.tar.gz signature: https://files.dthompson.us/guile-websocket/guile-websocket-0.2.0.tar.gz.asc See the guile-websocket project page for more information. Bug reports, bug fixes, feature requests, and patches are welcomed.

5 months ago 77 votes
Wasm GC isn’t ready for realtime graphics

Wasm GC is a wonderful thing that is now available in all major web browsers since slowpoke Safari/WebKit finally shipped it in December. It provides a hierarchy of heap allocated reference types and a set of instructions to operate on them. Wasm GC enables managed memory languages to take advantage of the advanced garbage collectors inside web browser engines. It’s now possible to implement a managed memory language without having to ship a GC inside the binary. The result is smaller binaries, better performance, and better integration with the host runtime. However, Wasm GC has some serious drawbacks when compared to linear memory. I enjoy playing around with realtime graphics programming in my free time, but I was disappointed to discover that Wasm GC just isn’t a good fit for that right now. I decided to write this post because I’d like to see Wasm GC on more or less equal footing with linear memory when it comes to binary data manipulation. Hello triangle For starters, let's take a look at what a “hello triangle” WebGL demo looks like with Wasm GC. I’ll use Hoot, the Scheme to Wasm compiler that I work on, to build it. Below is a Scheme program that declares imports for the subset of the WebGL, HTML5 Canvas, etc. APIs that are necessary and then renders a single triangle: (use-modules (hoot ffi)) ;; Document (define-foreign get-element-by-id "document" "getElementById" (ref string) -> (ref null extern)) ;; Element (define-foreign element-width "element" "width" (ref extern) -> i32) (define-foreign element-height "element" "height" (ref extern) -> i32) ;; Canvas (define-foreign get-canvas-context "canvas" "getContext" (ref extern) (ref string) -> (ref null extern)) ;; WebGL (define GL_VERTEX_SHADER 35633) (define GL_FRAGMENT_SHADER 35632) (define GL_COMPILE_STATUS 35713) (define GL_LINK_STATUS 35714) (define GL_ARRAY_BUFFER 34962) (define GL_STATIC_DRAW 35044) (define GL_COLOR_BUFFER_BIT 16384) (define GL_TRIANGLES 4) (define GL_FLOAT 5126) (define-foreign gl-create-shader "gl" "createShader" (ref extern) i32 -> (ref extern)) (define-foreign gl-delete-shader "gl" "deleteShader" (ref extern) (ref extern) -> none) (define-foreign gl-shader-source "gl" "shaderSource" (ref extern) (ref extern) (ref string) -> none) (define-foreign gl-compile-shader "gl" "compileShader" (ref extern) (ref extern) -> none) (define-foreign gl-get-shader-parameter "gl" "getShaderParameter" (ref extern) (ref extern) i32 -> i32) (define-foreign gl-get-shader-info-log "gl" "getShaderInfoLog" (ref extern) (ref extern) -> (ref string)) (define-foreign gl-create-program "gl" "createProgram" (ref extern) -> (ref extern)) (define-foreign gl-delete-program "gl" "deleteProgram" (ref extern) (ref extern) -> none) (define-foreign gl-attach-shader "gl" "attachShader" (ref extern) (ref extern) (ref extern) -> none) (define-foreign gl-link-program "gl" "linkProgram" (ref extern) (ref extern) -> none) (define-foreign gl-use-program "gl" "useProgram" (ref extern) (ref extern) -> none) (define-foreign gl-get-program-parameter "gl" "getProgramParameter" (ref extern) (ref extern) i32 -> i32) (define-foreign gl-get-program-info-log "gl" "getProgramInfoLog" (ref extern) (ref extern) -> (ref string)) (define-foreign gl-create-buffer "gl" "createBuffer" (ref extern) -> (ref extern)) (define-foreign gl-delete-buffer "gl" "deleteBuffer" (ref extern) (ref extern) -> (ref extern)) (define-foreign gl-bind-buffer "gl" "bindBuffer" (ref extern) i32 (ref extern) -> none) (define-foreign gl-buffer-data "gl" "bufferData" (ref extern) i32 (ref eq) i32 -> none) (define-foreign gl-enable-vertex-attrib-array "gl" "enableVertexAttribArray" (ref extern) i32 -> none) (define-foreign gl-vertex-attrib-pointer "gl" "vertexAttribPointer" (ref extern) i32 i32 i32 i32 i32 i32 -> none) (define-foreign gl-draw-arrays "gl" "drawArrays" (ref extern) i32 i32 i32 -> none) (define-foreign gl-viewport "gl" "viewport" (ref extern) i32 i32 i32 i32 -> none) (define-foreign gl-clear-color "gl" "clearColor" (ref extern) f64 f64 f64 f64 -> none) (define-foreign gl-clear "gl" "clear" (ref extern) i32 -> none) (define (compile-shader gl type source) (let ((shader (gl-create-shader gl type))) (gl-shader-source gl shader source) (gl-compile-shader gl shader) (unless (= (gl-get-shader-parameter gl shader GL_COMPILE_STATUS) 1) (let ((info (gl-get-shader-info-log gl shader))) (gl-delete-shader gl shader) (error "shader compilation failed" info))) shader)) (define (link-shader gl vertex-shader fragment-shader) (let ((program (gl-create-program gl))) (gl-attach-shader gl program vertex-shader) (gl-attach-shader gl program fragment-shader) (gl-link-program gl program) (unless (= (gl-get-program-parameter gl program GL_LINK_STATUS) 1) (let ((info (gl-get-program-info-log gl program))) (gl-delete-program gl program) (error "program linking failed" info))) program)) ;; Setup GL context (define canvas (get-element-by-id "canvas")) (define gl (get-canvas-context canvas "webgl")) (when (external-null? gl) (error "unable to create WebGL context")) ;; Compile shader (define vertex-shader-source "attribute vec2 position; attribute vec3 color; varying vec3 fragColor; void main() { gl_Position = vec4(position, 0.0, 1.0); fragColor = color; }") (define fragment-shader-source "precision mediump float; varying vec3 fragColor; void main() { gl_FragColor = vec4(fragColor, 1); }") (define vertex-shader (compile-shader gl GL_VERTEX_SHADER vertex-shader-source)) (define fragment-shader (compile-shader gl GL_FRAGMENT_SHADER fragment-shader-source)) (define shader (link-shader gl vertex-shader fragment-shader)) ;; Create vertex buffer (define stride (* 4 5)) (define buffer (gl-create-buffer gl)) (gl-bind-buffer gl GL_ARRAY_BUFFER buffer) (gl-buffer-data gl GL_ARRAY_BUFFER #f32(-1.0 -1.0 1.0 0.0 0.0 1.0 -1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0) GL_STATIC_DRAW) ;; Draw (gl-viewport gl 0 0 (element-width canvas) (element-height canvas)) (gl-clear gl GL_COLOR_BUFFER_BIT) (gl-use-program gl shader) (gl-enable-vertex-attrib-array gl 0) (gl-vertex-attrib-pointer gl 0 2 GL_FLOAT 0 stride 0) (gl-enable-vertex-attrib-array gl 1) (gl-vertex-attrib-pointer gl 1 3 GL_FLOAT 0 stride 8) (gl-draw-arrays gl GL_TRIANGLES 0 3) Note that in Scheme, the equivalent of a Uint8Array is a bytevector. Hoot uses a packed array, an (array i8) specifically, for the contents of a bytevector. And here is the JavaScript code necessary to boot the resulting Wasm binary: window.addEventListener("load", async () => { function bytevectorToUint8Array(bv) { let len = reflect.bytevector_length(bv); let array = new Uint8Array(len); for (let i = 0; i < len; i++) { array[i] = reflect.bytevector_ref(bv, i); } return array; } let mod = await SchemeModule.fetch_and_instantiate("triangle.wasm", { reflect_wasm_dir: 'reflect-wasm', user_imports: { document: { getElementById: (id) => document.getElementById(id) }, element: { width: (elem) => elem.width, height: (elem) => elem.height }, canvas: { getContext: (elem, type) => elem.getContext(type) }, gl: { createShader: (gl, type) => gl.createShader(type), deleteShader: (gl, shader) => gl.deleteShader(shader), shaderSource: (gl, shader, source) => gl.shaderSource(shader, source), compileShader: (gl, shader) => gl.compileShader(shader), getShaderParameter: (gl, shader, param) => gl.getShaderParameter(shader, param), getShaderInfoLog: (gl, shader) => gl.getShaderInfoLog(shader), createProgram: (gl, type) => gl.createProgram(type), deleteProgram: (gl, program) => gl.deleteProgram(program), attachShader: (gl, program, shader) => gl.attachShader(program, shader), linkProgram: (gl, program) => gl.linkProgram(program), useProgram: (gl, program) => gl.useProgram(program), getProgramParameter: (gl, program, param) => gl.getProgramParameter(program, param), getProgramInfoLog: (gl, program) => gl.getProgramInfoLog(program), createBuffer: (gl) => gl.createBuffer(), deleteBuffer: (gl, buffer) => gl.deleteBuffer(buffer), bindBuffer: (gl, target, buffer) => gl.bindBuffer(target, buffer), bufferData: (gl, buffer, data, usage) => { let bv = new Bytevector(reflect, data); gl.bufferData(buffer, bytevectorToUint8Array(bv), usage); }, enableVertexAttribArray: (gl, index) => gl.enableVertexAttribArray(index), vertexAttribPointer: (gl, index, size, type, normalized, stride, offset) => { gl.vertexAttribPointer(index, size, type, normalized, stride, offset); }, drawArrays: (gl, mode, first, count) => gl.drawArrays(mode, first, count), viewport: (gl, x, y, w, h) => gl.viewport(x, y, w, h), clearColor: (gl, r, g, b, a) => gl.clearColor(r, g, b, a), clear: (gl, mask) => gl.clear(mask) } } }); let reflect = await mod.reflect({ reflect_wasm_dir: 'reflect-wasm' }); let proc = new Procedure(reflect, mod.get_export("$load").value); proc.call(); }); Hello problems There are two major performance issues with this program. One is visible in the source above, the other is hidden in the language implementation. Heap objects are opaque on the other side Wasm GC heap objects are opaque to the host. Likewise, heap objects from the host are opaque to the Wasm guest. Thus the contents of an (array i8) object are not visible from JavaScript and the contents of a Uint8Array are not visible from Wasm. This is a good security property in the general case, but it’s a hinderance in this specific case. Let’s say we have an (array i8) full of vertex data we want to put into a WebGL buffer. To do this, we must make one JS->Wasm call for each byte in the array and store it into a Uint8Array. This is what the bytevectorToUint8Array function above is doing. Copying any significant amount of data per frame is going to tank performance. Hope you aren’t trying to stream vertex data! Contrast the previous paragraph with Wasm linear memory. A WebAssembly.Memory object can be easily accessed from JavaScript as an ArrayBuffer. To get a blob of vertex data out of a memory object, you just need to know the byte offset and length and you’re good to go. There are many Wasm linear memory applications using WebGL successfully. Manipulating multi-byte binary data is inefficient To read a multi-byte number such as an unsigned 32-bit integer from an (array i8), you have to fetch each individual byte and combine them together. Here’s a self-contained example that uses Guile-flavored WAT format: (module (type $bytevector (array i8)) (data $init #u32(123456789)) (func (export "main") (result i32) (local $a (ref $bytevector)) (local.set $a (array.new_data $bytevector $init (i32.const 0) (i32.const 4))) (array.get_u $bytevector (local.get $a) (i32.const 0)) (i32.shl (array.get_u $bytevector (local.get $a) (i32.const 1)) (i32.const 8)) (i32.or) (i32.shl (array.get_u $bytevector (local.get $a) (i32.const 2)) (i32.const 16)) (i32.or) (i32.shl (array.get_u $bytevector (local.get $a) (i32.const 3)) (i32.const 24)) (i32.or))) By contrast, Wasm linear memory needs but a single i32.load instruction: (module (memory 1) (func (export "main") (result i32) (i32.store (i32.const 0) (i32.const 123456789)) (i32.load (i32.const 0)))) Easy peasy. Not only is it less code, it's a lot more efficient. Unsatisfying workarounds There’s no way around the multi-byte problem at the moment, but for byte access from JavaScript there are some things we could try to work with what we have been given. Spoiler alert: None of them are pleasant. Use Uint8Array from the host This approach makes all binary operations from within the Wasm binary slow since we’d have to cross the Wasm->JS bridge for each read/write. Since most of the binary data manipulation is happening in the Wasm module, this approach will just make things slower overall. Use linear memory for bytevectors This would require a little malloc/free implementation and a way to reclaim memory for GC'd bytevectors. You could register every bytevector in a FinalizationRegistry in order to be notified upon GC and free the memory. Now you have to deal with memory fragmentation. This is Wasm GC, we shouldn’t have to do any of this! Use linear memory as a scratch space This avoids crossing the Wasm/JS boundary for each byte, but still involves a byte-by-byte copy from (array i8) to linear memory within the Wasm module. So far this feels like the least worst option, but the extra copy is still going to greatly reduce throughput. Wasm GC needs some fixin' I’ve used realtime graphics as an example because it’s a use case that is very sensitive to performance issues, but this unfortunate need to copy binary data byte-by-byte is also the reason why strings are trash on Wasm GC right now. Stringref is a good proposal and the Wasm community group made a mistake by rejecting it. Anyway, there has been some discussion about both multi-byte and ArrayBuffer access on GitHub, but as far as I can tell neither issue is anywhere close to a resolution. Can these things be implemented efficiently? How can the need for direct access to packed arrays from JS be reconciled with Wasm heap object opaqueness? I hope the Wasm community group can arrive at solutions sooner than later because it will take a long time to get the proposal(s) to phase 4 and shipped in all browsers, perhaps years. It would be a shame to be effectively shut out from using WebGPU when it finally reaches stable browser releases.

6 months ago 73 votes
Guile-Bstructs 0.1.0 released

I'm pleased to announce that the very first release of guile-bstructs, version 0.1.0, has been released! This is a library I've been working on for quite some time and after more than one rewrite and many smaller refactors I think it's finally ready to release publicly. Let's hope I'm not wrong about that! About guile-bstructs Guile-bstructs is a library that provides structured read/write access to binary data for Guile. A bstruct (short for “binary structure”) is a data type that encapsulates a bytevector and a byte offset which interprets that bytevector based on a specified layout. Some use cases for bstructs are: manipulating C structs when using the foreign function interface packing GPU vertex buffers when using graphics APIs such as OpenGL implementing data types that benefit from Guile's unboxed math optimizations such as vectors and matrices This library was initially inspired by guile-opengl's define-packed-struct syntax but is heavily based on "Ftypes: Structured foreign types" by Andy Keep and R. Kent Dybvig. The resulting interface is quite similar but the implementation is completely original. This library provides a syntax-heavy interface; nearly all of the public API is syntax. This is done to ensure that bstruct types are static and well-known at compile time resulting in efficient bytecode and minimal runtime overhead. A subset of the interface deals in raw bytevector access for accessing structured data in bytevectors directly without going through an intermediary bstruct wrapper. This low-level interface is useful for certain batch processing situations where the overhead of creating wrapper bstructs would hinder throughput. Example Here are some example type definitions to give you an idea of what it’s like to use guile-bstructs: ;; Struct (define-bstruct <vec2> (padded (struct (x float) (y float)))) ;; Type group with a union (define-bstruct (<mouse-move-event> (struct (type uint8) (x int32) (y int32))) (<mouse-button-event> (struct (type uint8) (button uint8) (state uint8) (x int32) (y int32))) (<event> (union (type uint8) (mouse-move <mouse-move-event>) (mouse-button <mouse-button-event>)))) ;; Array (define-bstruct <matrix4> (array 16 float)) ;; Bit fields (define-bstruct <date> (bits (year 32 s) (month 4 u) (day 5 u))) ;; Pointer (define-bstruct (<item> (struct (type int))) (<chest> (struct (opened? uint8) (item (* <item>))))) ;; Packed struct modifier (define-bstruct <enemy> (packed (struct (type uint8) (health uint32)))) ;; Endianness modifier (define-bstruct <big-float> (endian big float)) ;; Recursive type (define-bstruct <node> (struct (item int) (next (* <node>)))) ;; Mutually recursive type group (define-bstruct (<forest> (struct (children (* <tree>)))) (<tree> (struct (value int) (forest (* <forest>)) (next (* <tree>))))) ;; Opaque type (define-bstruct SDL_GPUTexture) Download Source tarball: guile-bstructs-0.1.0.tar.gz GPG signature: guile-bstructs-0.1.0.tar.gz.asc This release was signed with this GPG key. See the guile-bstructs project page for more information.

6 months ago 81 votes
Functional reactive user interfaces with propagators

I’ve been interested in functional reactive programming (FRP) for about a decade now. I even wrote a couple of blog posts back in 2014 describing my experiments. My initial source of inspiration was Elm, the Haskell-like language for the web that once had FRP as a core part of the language. Evan Czaplicki’s Strange Loop 2013 talk really impressed me, especially that Mario demo. From there, I explored the academic literature on the subject. Ultimately, I created and then abandoned a library that focused on FRP for games. It was a neat idea, but the performance was terrible. The overhead of my kinda-sorta FRP system was part of the problem, but mostly it was my own inexperience. I didn’t know how to optimize effectively and my implementation language, Guile, did not have as many optimization passes as it does now. Also, realtime simulations like games require much more careful use of heap allocation. I found that, overhead aside, FRP is a bad fit for things like scripting sequences of actions in a game. I don’t want to give up things like coroutines that make it easy. I’ve learned how different layers of a program may call for different programming paradigms. Functional layers rest upon imperative foundations. Events are built on top of polling. Languages with expression trees run on machines that only understand linear sequences. You get the idea. A good general-purpose language will allow you to compose many paradigms in the same program. I’m still a big fan of functional programming, but single paradigm languages do not appeal to me. Fast forward 10 years, I find myself thinking about FRP again in a new context. I now work for the Spritely Institute where we’re researching and building the next generation of secure, distributed application infrastructure. We want to demo our tech through easy-to-use web applications, which means we need to do some UI programming. So, the back burner of my brain has been mulling over the possibilities. What’s the least painful way to build web UIs? Is this FRP thing worth revisiting? The reason why FRP is so appealing to me (on paper, at least) is that it allows for writing interactive programs declaratively. With FRP, I can simply describe the relationships between the various time-varying components, and the system wires it all together for me. I’m spared from callback hell, one of the more frightening layers of hell that forces programs to be written in a kind of continuation-passing style where timing and state bugs consume more development time as the project grows. What about React? In the time during and since I last experimented with FRP, a different approach to declarative UI programming has swept the web development world: React. From React, many other similar libraries emerged. On the minimalist side there are things like Mithril (a favorite of mine), and then there are bigger players like Vue. The term “reactive” has become overloaded, but in the mainstream software world it is associated with React and friends. FRP is quite different, despite sharing the declarative and reactive traits. Both help free programmers from callback hell, but they achieve their results differently. The React model describes an application as a tree of “components”. Each component represents a subset of the complete UI element tree. For each component, there is a template function that takes some inputs and returns the new desired state of the UI. This function is called whenever an event occurs that might change the state of the UI. The template produces a data structure known as a “virtual DOM”. To realize this new state in the actual DOM, React diffs the previous tree with the new one and updates, creates, and deletes elements as necessary. With FRP, you describe your program as an acyclic graph of nodes that contain time-varying values. The actual value of any given node is determined by a function that maps the current values of some input nodes into an output value. The system is bootstrapped by handling a UI event and updating the appropriate root node, which kicks off a cascade of updates throughout the graph. At the leaf nodes, side-effects occur that realize the desired state of the application. Racket’s FrTime is one example of such a system, which is based on Greg Cooper’s 2008 PhD dissertation “Integrating Dataflow Evaluation into a Practical Higher-Order Call-by-Value Language”. In FrTime, time-varying values are called “signals”. Elm borrowed this language, too, and there’s currently a proposal to add signals to JavaScript. Research into FRP goes back quite a bit further. Notably, Conal Elliot and Paul Hudak wrote “Functional Reactive Animation” in 1997. On jank The scope of potential change for any given event is larger in React than FRP. An FRP system flows data through an acyclic graph, updating only the nodes affected by the event. React requires re-evaluating the template for each component that needs refreshing and applying a diff algorithm to determine what needs changing in the currently rendered UI. The virtual DOM diffing process can be quite wasteful in terms of both memory usage and processing time, leading to jank on systems with limited resources like phones. Andy Wingo has done some interesting analyses of things like React Native and Flutter and covers the subject of jank well. So, while I appreciate the greatly improved developer experience of React-likes (I wrote my fair share of frontend code in the jQuery days), I’m less pleased by the overhead that it pushes onto each user’s computer. React feels like an important step forward on the declarative UI trail, but it’s not the destination. FRP has the potential for less jank because side-effects (the UI widget state updates) can be more precise. For example, if a web page has a text node that displays the number of times the user has clicked a mouse button, an FRP system could produce a program that would do the natural thing: Register a click event handler that replaces the text node with a new one containing the updated count. We don’t need to diff the whole page, nor do we need to create a wrapper component to update a subset of the page. The scope is narrow, so we can apply smaller updates. No virtual DOM necessary. There is, of course, overhead to maintaining the graph of time-varying values. The underlying runtime is free to use mutable state, but the user layer must take care to use pure functions and persistent, immutable data structures. This has a cost, but the per-event cost to refresh the UI feels much more reasonable when compared to React. From here on, I will stop talking about React and start exploring if FRP might really offer a more expressive way to do declarative UI without too much overhead. But first, we need to talk about a serious problem. FRP is acyclic FRP is no silver bullet. As mentioned earlier, FRP graphs are typically of the acyclic flavor. This limits the set of UIs that are possible to create with FRP. Is this the cost of declarativeness? To demonstrate the problem, consider a color picker tool that has sliders for both the red-green-blue and hue-saturation-value representations of color: In this program, updating the sliders on the RGB side should change the values of the sliders on the HSV side, and vice versa. This forms a cycle between the two sets of sliders. It’s possible to express cycles like this with event callbacks, though it’s messy and error-prone to do manually. We’d like a system built on top of event callbacks that can do the right thing without strange glitches or worse, unbounded loops. Propagators to the rescue Fortunately, I didn’t create that diagram above. It’s from Alexey Radul’s 2009 PhD dissertation: “Propagation Networks: A Flexible and Expressive Substrate for Computation”. This paper dedicates a section to explaining how FRP can be built on top a more general paradigm called propagation networks, or just propagators for short. The paper is lengthy, naturally, but it is written in an approachable style. There isn’t any terse math notation and there are plenty of code examples. As far as PhD dissertations go, this one is a real page turner! Here is a quote from section 5.5 about FrTime (with emphasis added by me): FrTime is built around a custom propagation infrastructure; it nicely achieves both non-recomputation and glitch avoidance, but unfortunately, the propagation system is nontrivially complicated, and specialized for the purpose of supporting functional reactivity. In particular, the FrTime system imposes the invariant that the propagation graph be acyclic, and guarantees that it will execute the propagators in topological-sort order. This simplifies the propagators themselves, but greatly complicates the runtime system, especially because it has to dynamically recompute the sort order when the structure of some portion of the graph changes (as when the predicate of a conditional changes from true to false, and the other branch must now be computed). That complexity, in turn, makes that runtime system unsuitable for other kinds of propagation, and even makes it difficult for other kinds of propagation to interoperate with it. So, the claim is that FRP-on-propagators can remove the acyclic restriction, reduce complexity, and improve interoperability. But what are propagators? I like how the book “Software Design for Flexibility” (2021) defines them (again, with emphasis added by me): “The propagator model is built on the idea that the basic computational elements are propagators, autonomous independent machines interconnected by shared cells through which they communicate. Each propagator machine continuously examines the cells it is connected to, and adds information to some cells based on computations it can make from information it can get from others. Cells accumulate information and propagators produce information.” Research on propagators goes back a long way (you’ll even find a form of propagators in the all-time classic “Structure and Interpretation of Computer Programs”), but it was Alexey Radul that discovered how to unify many different types of special-purpose propagation systems so that they could share a generic substrate and interoperate. Perhaps the most exciting application of the propagator model is AI, where it can be used to create “explainable AI” that keeps track of how a particular output was computed. This type of AI stands in stark contrast to the much hyped mainstream machine learning models that hoover up our planet’s precious natural resources to produce black boxes that generate impressive bullshit. But anyway! The diagram above can also be found in section 5.5 of the dissertation. Here’s the description: “A network for a widget for RGB and HSV color selection. Traditional functional reactive systems have qualms about the circularity, but general-purpose propagation handles it automatically.” This color picker felt like a good, achievable target for a prototype. The propagator network is small and there are only a handful of UI elements, yet it will test if the FRP system is working correctly. The prototype I read Alexey Radul’s dissertation, and then read chapter 7 of Software Design for Flexibility, which is all about propagators. Both use Scheme as the implementation language. The latter makes no mention of FRP, and while the former explains how FRP can be implemented in terms of propagators, there is (understandably) no code included. So, I had to implement it for myself to test my understanding. Unsurprisingly, I had misunderstood many things along the way and my iterations of broken code let me know that. Implementation is the best teacher. After much code fiddling, I was able to create a working prototype of the color picker. Here it is below: This prototype is written in Scheme and uses Hoot to compile it to WebAssembly so I can embed it right here in my blog. Sure beats a screenshot or video! This prototype contains a minimal propagator implementation that is sufficient to bootstrap a similarly minimal FRP implementation. Propagator implementation Let’s take a look at the code and see how it all works, starting with propagation. There are two essential data types: Cells and propagators. Cells accumulate information about a value, ranging from nothing, some form of partial information, or a complete value. The concept of partial information is Alexey Radul’s major contribution to the propagator model. It is through partial information structures that general-purpose propagators can be used to implement logic programming, probabilistic programming, type inference, and FRP, among others. I’m going to keep things as simple as possible in this post (it’s a big enough infodump already), but do read the propagator literature if phrases like “dependency directed backtracking” and “truth maintenance system” sound like a good time to you. Cells start out knowing nothing, so we need a special, unique value to represent nothing: (define-record-type <nothing> (make-nothing) %nothing?) (define nothing (make-nothing)) (define (nothing? x) (eq? x nothing)) Any unique (as in eq?) object would do, such as (list ’nothing), but I chose to use a record type because I like the way it prints. In addition to nothing, the propagator model also has a notion of contradictions. If one source of information says there are four lights, but another says there are five, then we have a contradiction. Propagation networks do not fall apart in the presence of contradictory information. There’s a data type that captures information about them and they can be resolved in a context-specific manner. I mention contradictions only for the sake of completeness, as a general-purpose propagator system needs to handle them. This prototype does not create any contradictions, so I won’t mention them again. Now we can define a cell type: (define-record-type <cell> (%make-cell relations neighbors content strongest equivalent? merge find-strongest handle-contradiction) cell? (relations cell-relations) (neighbors cell-neighbors set-cell-neighbors!) (content cell-content set-cell-content!) (strongest cell-strongest set-cell-strongest!) ;; Dispatch table: (equivalent? cell-equivalent?) (merge cell-merge) (find-strongest cell-find-strongest) (handle-contradiction cell-handle-contradiction)) The details of how a cell does things like merge old information with new information is left intentionally unanswered at this level of abstraction. Different systems built on propagators will want to handle things in different ways. In the propagator literature, you’ll see generic procedures used extensively for this purpose. For the sake of simplicity, I use a dispatch table instead. It would be easy to layer generic merge on top later, if we wanted. The constructor for cells sets the default contents to nothing: (define default-equivalent? equal?) ;; But what about partial information??? (define (default-merge old new) new) (define (default-find-strongest content) content) (define (default-handle-contradiction cell) (values)) (define* (make-cell name #:key (equivalent? default-equivalent?) (merge default-merge) (find-strongest default-find-strongest) (handle-contradiction default-handle-contradiction)) (let ((cell (%make-cell (make-relations name) '() nothing nothing equivalent? merge find-strongest handle-contradiction))) (add-child! (current-parent) cell) cell)) The default procedures used for the dispatch table are either no-ops or trivial. The default merge doesn’t merge at all, it just clobbers the old with the new. It’s up to the layers on top to provide more useful implementations. Cells can have neighbors (which will be propagators): (define (add-cell-neighbor! cell neighbor) (set-cell-neighbors! cell (lset-adjoin eq? (cell-neighbors cell) neighbor))) Since cells accumulate information, there are procedures for adding new content and finding the current strongest value contained within: (define (add-cell-content! cell new) (match cell (($ <cell> _ neighbors content strongest equivalent? merge find-strongest handle-contradiction) (let ((content* (merge content new))) (set-cell-content! cell content*) (let ((strongest* (find-strongest content*))) (cond ;; New strongest value is equivalent to the old one. No need ;; to alert propagators. ((equivalent? strongest strongest*) (set-cell-strongest! cell strongest*)) ;; Uh oh, a contradiction! Call handler. ((contradiction? strongest*) (set-cell-strongest! cell strongest*) (handle-contradiction cell)) ;; Strongest value has changed. Alert the propagators! (else (set-cell-strongest! cell strongest*) (for-each alert-propagator! neighbors)))))))) Next up is the propagator type. Propagators can be activated to create information using content stored in cells and store their results in some other cells, forming a graph. Data flow is not forced to be directional. Cycles are not only permitted, but very common in practice. So, propagators keep track of both their input and output cells: (define-record-type <propagator> (%make-propagator relations inputs outputs activate) propagator? (relations propagator-relations) (inputs propagator-inputs) (outputs propagator-outputs) (activate propagator-activate)) Propagators can be alerted to schedule themselves to be re-evaluted: (define (alert-propagator! propagator) (queue-task! (propagator-activate propagator))) The constructor for propagators adds the new propagator as a neighbor to all input cells and then calls alert-propagator! to bootstrap it: (define (make-propagator name inputs outputs activate) (let ((propagator (%make-propagator (make-relations name) inputs outputs activate))) (add-child! (current-parent) propagator) (for-each (lambda (cell) (add-cell-neighbor! cell propagator)) inputs) (alert-propagator! propagator) propagator)) There are two main classes of propagators that will be used: primitive propagators and constraint propagators. Primitive propagators are directional; they apply a function to the values of some input cells and write the result to an output cell: (define (unusable-value? x) (or (nothing? x) (contradiction? x))) (define (primitive-propagator name f) (match-lambda* ((inputs ... output) (define (activate) (let ((args (map cell-strongest inputs))) (unless (any unusable-value? args) (add-cell-content! output (apply f args))))) (make-propagator name inputs (list output) activate)))) We can use primitive-propagator to lift standard Scheme procedures into the realm of propagators. Here’s how we’d make and use an addition propagator: (define p:+ (primitive-propagator +)) (define a (make-cell)) (define b (make-cell)) (define c (make-cell)) (p:+ a b c) (add-cell-content! a 1) (add-cell-content! b 3) ;; After the scheduler runs… (cell-strongest c) ;; => 4 It is from these primitive propagators that we can build more complicated, compound propagators. Compound propagators compose primitive propagators (or other compound propagators) and lazily construct their section of the network upon first activation: (define (compound-propagator name inputs outputs build) (let ((built? #f)) (define (maybe-build) (unless (or built? (and (not (null? inputs)) (every unusable-value? (map cell-strongest inputs)))) (parameterize ((current-parent (propagator-relations propagator))) (build) (set! built? #t)))) (define propagator (make-propagator name inputs outputs maybe-build)) propagator)) By this point you may be wondering what all the references to current-parent are about. It is for tracking the parent/child relationships of the cells and propagators in the network. This could be helpful for things like visualizing the network, but we aren’t going to do anything with it today. I’ve omitted all of the other relation code for this reason. Constraint propagators are compound propagators whose inputs and outputs are the same, which results in bidirectional propagation: (define (constraint-propagator name cells build) (compound-propagator name cells cells build)) Using primitive propagators for addition and subtraction, we can build a constraint propagator for the equation a + b = c: (define p:+ (primitive-propagator +)) (define p:- (primitive-propagator -)) (define (c:sum a b c) (define (build) (p:+ a b c) (p:- c a b) (p:- c b a)) (constraint-propagator 'sum (list a b c) build)) (define a (make-cell)) (define b (make-cell)) (define c (make-cell)) (c:sum a b c) (add-cell-content! a 1) (add-cell-content! c 4) ;; After the scheduler runs… (cell-strongest b) ;; => 3 With a constraint, we can populate any two cells and the propagation system will figure out the value of the third. Pretty cool! This is a good enough propagation system for the FRP prototype. FRP implementation If you’re familiar with terminology from other FRP systems like “signals” and “behaviors” then set that knowledge aside for now. We need some new nouns. But first, a bit about the problems that need solving in order to implement FRP on top of general-purpose propagators: The propagator model does not enforce any ordering of when propagators will be re-activated in relation to each other. If we’re not careful, something in the network could compute a value using a mix of fresh and stale data, resulting in a momentary “glitch” that could be noticeable in the UI. The presence of cycles introduce a crisis of identity. It’s not sufficient for every time-varying value to be treated as its own self. In the color picker, the RGB values and the HSV values are two representations of the same thing. We need a new notion of identity to capture this and prevent unnecessary churn and glitches in the network. For starters, we will create a “reactive identifier” (needs a better name) data type that serves two purposes: To create shared identity between different information sources that are logically part of the same thing To create localized timestamps for values associated with this identity (define-record-type <reactive-id> (%make-reactive-id name clock) reactive-id? (name reactive-id-name) (clock reactive-id-clock set-reactive-id-clock!)) (define (make-reactive-id name) (%make-reactive-id name 0)) (define (reactive-id-tick! id) (let ((t (1+ (reactive-id-clock id)))) (set-reactive-id-clock! id t) `((,id . ,t)))) Giving each logical identity in the FRP system its own clock eliminates the need for a global clock, avoiding a potentially troublesome source of centralization. This is kind of like how Lamport timestamps are used in distributed systems. We also need a data type that captures the value of something at a particular point in time. Since the cruel march of time is unceasing, these are ephemeral values: (define-record-type <ephemeral> (make-ephemeral value timestamps) ephemeral? (value ephemeral-value) ;; Association list mapping identity -> time (timestamps ephemeral-timestamps)) Ephemerals are boxes that contain some arbitrary data with a bunch of shipping labels slapped onto the outside explaining from whence they came. This is the partial information structure that our propagators will manipulate and add to cells. Here’s how to say “the mouse position was (1, 2) at time 3” in code: (define mouse-position (make-reactive-id ’mouse-position)) (make-ephemeral #(1 2) `((,mouse-position . 3))) We need to perform a few operations with ephemerals: Test if one ephemeral is fresher (more recent) than another Merge two ephemerals when cell content is added Compose the timestamps from several inputs to form an aggregate timestamp for an output, but only if all timestamps for each distinct identifier match (no mixing of fresh and stale values) (define (ephemeral-fresher? a b) (let ((b-inputs (ephemeral-timestamps b))) (let lp ((a-inputs (ephemeral-timestamps a))) (match a-inputs (() #t) (((key . a-time) . rest) (match (assq-ref b-inputs key) (#f (lp rest)) (b-time (and (> a-time b-time) (lp rest))))))))) (define (merge-ephemerals old new) (cond ((nothing? old) new) ((nothing? new) old) ((ephemeral-fresher? new old) new) (else old))) (define (merge-ephemeral-timestamps ephemerals) (define (adjoin-keys alist keys) (fold (lambda (key+value keys) (match key+value ((key . _) (lset-adjoin eq? keys key)))) keys alist)) (define (check-timestamps id) (let lp ((ephemerals ephemerals) (t #f)) (match ephemerals (() t) ((($ <ephemeral> _ timestamps) . rest) (match (assq-ref timestamps id) ;; No timestamp for this id in this ephemeral. Continue. (#f (lp rest t)) (t* (if t ;; If timestamps don't match then we have a mix of ;; fresh and stale values, so return #f. Otherwise, ;; continue. (and (= t t*) (lp rest t)) ;; Initialize timestamp and continue. (lp rest t*)))))))) ;; Build a set of all reactive identifiers across all ephemerals. (let ((ids (fold (lambda (ephemeral ids) (adjoin-keys (ephemeral-timestamps ephemeral) ids)) '() ephemerals))) (let lp ((ids ids) (timestamps '())) (match ids (() timestamps) ((id . rest) ;; Check for consistent timestamps. If they are consistent ;; then add it to the alist and continue. Otherwise, return ;; #f. (let ((t (check-timestamps id))) (and t (lp rest (cons (cons id t) timestamps))))))))) Example usage: (define e1 (make-ephemeral #(3 4) `((,mouse-position . 4)))) (define e2 (make-ephemeral #(1 2) `((,mouse-position . 3)))) (ephemeral-fresher? e1 e2) ;; => #t (merge-ephemerals e1 e2) ;; => e1 (merge-ephemeral-timestamps (list e1 e2)) ;; => #f (define (mouse-max-coordinate e) (match e (($ <ephemeral> #(x y) timestamps) (make-ephemeral (max x y) timestamps)))) (define e3 (mouse-max-coordinate e1)) (merge-ephemeral-timestamps (list e1 e3)) ;; => ((mouse-position . 4)) Now we can build a primitive propagator constructor that lifts ordinary Scheme procedures so that they work with ephemerals: (define (ephemeral-wrap proc) (match-lambda* ((and ephemerals (($ <ephemeral> args) ...)) (match (merge-ephemeral-timestamps ephemerals) (#f nothing) (timestamps (make-ephemeral (apply proc args) timestamps)))))) (define* (primitive-reactive-propagator name proc) (primitive-propagator name (ephemeral-wrap proc))) Reactive UI implementation Now we need some propagators that live at the edges of our network that know how to interact with the DOM and can do the following: Sync a DOM element attribute with the value of a cell Create a two-way data binding between an element’s value attribute and a cell Render the markup in a cell and place it into the DOM tree in the right location Syncing an element attribute is a directional operation and the easiest to implement: (define (r:attribute input elem attr) (let ((attr (symbol->string attr))) (define (activate) (match (cell-strongest input) (($ <ephemeral> val) (attribute-set! elem attr (obj->string val))) ;; Ignore unusable values. (_ (values)))) (make-propagator 'r:attribute (list input) '() activate))) Two-way data binding is more involved. First, a new data type is used to capture the necessary information: (define-record-type <binding> (make-binding id cell default group) binding? (id binding-id) (cell binding-cell) (default binding-default) (group binding-group)) (define* (binding id cell #:key (default nothing) (group '())) (make-binding id cell default group)) And then a reactive propagator applies that binding to a specific DOM element: (define* (r:binding binding elem) (match binding (($ <binding> id cell default group) (define (update new) (unless (nothing? new) (let ((timestamp (reactive-id-tick! id))) (add-cell-content! cell (make-ephemeral new timestamp)) ;; Freshen timestamps for all cells in the same group. (for-each (lambda (other) (unless (eq? other cell) (match (cell-strongest other) (($ <ephemeral> val) (add-cell-content! other (make-ephemeral val timestamp))) (_ #f)))) group)))) ;; Sync the element's value with the cell's value. (define (activate) (match (cell-strongest cell) (($ <ephemeral> val) (set-value! elem (obj->string val))) (_ (values)))) ;; Initialize element value with the default value. (update default) ;; Sync the cell's value with the element's value. (add-event-listener! elem "input" (procedure->external (lambda (event) (update (string->obj (value elem)))))) (make-propagator 'r:binding (list cell) '() activate)))) A simple method for rendering to the DOM is to replace some element with a newly created element based on the current ephemeral value of a cell: (define (r:dom input elem) (define (activate) (match (cell-strongest input) (($ <ephemeral> exp) (let ((new (sxml->dom exp))) (replace-with! elem new) (set! elem new))) (_ (values)))) (make-propagator 'dom (list input) '() activate)) The sxml->dom procedure deserves some further explanation. To create a subtree of new elements, we have two options: Use something like the innerHTML element property to insert arbitrary HTML as a string and let the browser parse and build the elements. Use a Scheme data structure that we can iterate over and make the relevant document.createTextNode, document.createElement, etc. calls. Option 1 might be a shortcut and would be fine for a quick prototype, but it would mean that to generate the HTML we’d be stuck using raw strings. While string-based templating is commonplace, we can certainly do better in Scheme. Option 2 is actually not too much work and we get to use Lisp’s universal templating system, quasiquote, to write our markup. Thankfully, SXML already exists for this purpose. SXML is an alternative XML syntax that uses s-expressions. Since Scheme uses s-expression syntax, it’s a natural fit. Example: '(article (h1 "SXML is neat") (img (@ (src "cool.jpg") (alt "cool image"))) (p "Yeah, SXML is " (em "pretty neato!"))) Instead of using it to generate HTML text, we’ll instead generate a tree of DOM elements. Furthermore, because we’re now in full control of how the element tree is built, we can build in support for reactive propagators! Check it out: (define (sxml->dom exp) (match exp ;; The simple case: a string representing a text node. ((? string? str) (make-text-node str)) ((? number? num) (make-text-node (number->string num))) ;; A cell containing SXML (or nothing) ((? cell? cell) (let ((elem (cell->elem cell))) (r:dom cell elem) elem)) ;; An element tree. The first item is the HTML tag. (((? symbol? tag) . body) ;; Create a new element with the given tag. (let ((elem (make-element (symbol->string tag)))) (define (add-children children) ;; Recursively call sxml->dom for each child node and ;; append it to elem. (for-each (lambda (child) (append-child! elem (sxml->dom child))) children)) (match body ((('@ . attrs) . children) (for-each (lambda (attr) (match attr (((? symbol? name) (? string? val)) (attribute-set! elem (symbol->string name) val)) (((? symbol? name) (? number? val)) (attribute-set! elem (symbol->string name) (number->string val))) (((? symbol? name) (? cell? cell)) (r:attribute cell elem name)) ;; The value attribute is special and can be ;; used to setup a 2-way data binding. (('value (? binding? binding)) (r:binding binding elem)))) attrs) (add-children children)) (children (add-children children))) elem)))) Notice the calls to r:dom, r:attribute, and r:binding. A cell can be used in either the context of an element (r:dom) or an attribute (r:attribute). The value attribute gets the additional superpower of r:binding. We will make use of this when it is time to render the color picker UI! Color picker implementation Alright, I’ve spent a lot of time explaining how I built a minimal propagator and FRP system from first principles on top of Hoot-flavored Scheme. Let’s finally write the dang color picker! First we need some data types to represent RGB and HSV colors: (define-record-type <rgb-color> (rgb-color r g b) rgb-color? (r rgb-color-r) (g rgb-color-g) (b rgb-color-b)) (define-record-type <hsv-color> (hsv-color h s v) hsv-color? (h hsv-color-h) (s hsv-color-s) (v hsv-color-v)) And procedures to convert RGB to HSV and vice versa: (define (rgb->hsv rgb) (match rgb (($ <rgb-color> r g b) (let* ((cmax (max r g b)) (cmin (min r g b)) (delta (- cmax cmin))) (hsv-color (cond ((= delta 0.0) 0.0) ((= cmax r) (let ((h (* 60.0 (fmod (/ (- g b) delta) 6.0)))) (if (< h 0.0) (+ h 360.0) h))) ((= cmax g) (* 60.0 (+ (/ (- b r) delta) 2.0))) ((= cmax b) (* 60.0 (+ (/ (- r g) delta) 4.0)))) (if (= cmax 0.0) 0.0 (/ delta cmax)) cmax))))) (define (hsv->rgb hsv) (match hsv (($ <hsv-color> h s v) (let* ((h' (/ h 60.0)) (c (* v s)) (x (* c (- 1.0 (abs (- (fmod h' 2.0) 1.0))))) (m (- v c))) (define-values (r' g' b') (cond ((<= 0.0 h 60.0) (values c x 0.0)) ((<= h 120.0) (values x c 0.0)) ((<= h 180.0) (values 0.0 c x)) ((<= h 240.0) (values 0.0 x c)) ((<= h 300.0) (values x 0.0 c)) ((<= h 360.0) (values c 0.0 x)))) (rgb-color (+ r' m) (+ g' m) (+ b' m)))))) We also need some procedures to convert colors into the hexadecimal representations we’re used to seeing: (define (uniform->byte x) (inexact->exact (round (* x 255.0)))) (define (rgb->int rgb) (match rgb (($ <rgb-color> r g b) (+ (* (uniform->byte r) (ash 1 16)) (* (uniform->byte g) (ash 1 8)) (uniform->byte b))))) (define (rgb->hex-string rgb) (list->string (cons #\# (let lp ((i 0) (n (rgb->int rgb)) (out '())) (if (= i 6) out (lp (1+ i) (ash n -4) (cons (integer->char (let ((digit (logand n 15))) (+ (if (< digit 10) (char->integer #\0) (- (char->integer #\a) 10)) digit))) out))))))) (define (rgb-hex->style hex) (string-append "background-color: " hex ";")) Now we can lift the color API into primitive reactive propagator constructors: (define-syntax-rule (define-primitive-reactive-propagator name proc) (define name (primitive-reactive-propagator 'name proc))) (define-primitive-reactive-propagator r:rgb-color rgb-color) (define-primitive-reactive-propagator r:rgb-color-r rgb-color-r) (define-primitive-reactive-propagator r:rgb-color-g rgb-color-g) (define-primitive-reactive-propagator r:rgb-color-b rgb-color-b) (define-primitive-reactive-propagator r:hsv-color hsv-color) (define-primitive-reactive-propagator r:hsv-color-h hsv-color-h) (define-primitive-reactive-propagator r:hsv-color-s hsv-color-s) (define-primitive-reactive-propagator r:hsv-color-v hsv-color-v) (define-primitive-reactive-propagator r:rgb->hsv rgb->hsv) (define-primitive-reactive-propagator r:hsv->rgb hsv->rgb) (define-primitive-reactive-propagator r:rgb->hex-string rgb->hex-string) (define-primitive-reactive-propagator r:rgb-hex->style rgb-hex->style) From those primitive propagators, we can build the necessary constraint propagators: (define (r:components<->rgb r g b rgb) (define (build) (r:rgb-color r g b rgb) (r:rgb-color-r rgb r) (r:rgb-color-g rgb g) (r:rgb-color-b rgb b)) (constraint-propagator 'r:components<->rgb (list r g b rgb) build)) (define (r:components<->hsv h s v hsv) (define (build) (r:hsv-color h s v hsv) (r:hsv-color-h hsv h) (r:hsv-color-s hsv s) (r:hsv-color-v hsv v)) (constraint-propagator 'r:components<->hsv (list h s v hsv) build)) (define (r:rgb<->hsv rgb hsv) (define (build) (r:rgb->hsv rgb hsv) (r:hsv->rgb hsv rgb)) (constraint-propagator 'r:rgb<->hsv (list rgb hsv) build)) At long last, we are ready to define the UI! Here it is: (define (render exp) (append-child! (document-body) (sxml->dom exp))) (define* (slider id name min max default #:optional (step "any")) `(div (@ (class "slider")) (label (@ (for ,id)) ,name) (input (@ (id ,id) (type "range") (min ,min) (max ,max) (step ,step) (value ,default))))) (define (uslider id name default) ; [0,1] slider (slider id name 0 1 default)) (define-syntax-rule (with-cells (name ...) body . body*) (let ((name (make-cell 'name #:merge merge-ephemerals)) ...) body . body*)) (with-cells (r g b rgb h s v hsv hex style) (define color (make-reactive-id 'color)) (define rgb-group (list r g b)) (define hsv-group (list h s v)) (r:components<->rgb r g b rgb) (r:components<->hsv h s v hsv) (r:rgb<->hsv rgb hsv) (r:rgb->hex-string rgb hex) (r:rgb-hex->style hex style) (render `(div (h1 "Color Picker") (div (@ (class "preview")) (div (@ (class "color-block") (style ,style))) (div (@ (class "hex")) ,hex)) (fieldset (legend "RGB") ,(uslider "red" "Red" (binding color r #:default 1.0 #:group rgb-group)) ,(uslider "green" "Green" (binding color g #:default 0.0 #:group rgb-group)) ,(uslider "blue" "Blue" (binding color b #:default 1.0 #:group rgb-group))) (fieldset (legend "HSV") ,(slider "hue" "Hue" 0 360 (binding color h #:group hsv-group)) ,(uslider "saturation" "Saturation" (binding color s #:group hsv-group)) ,(uslider "value" "Value" (binding color v #:group hsv-group)))))) Each color channel (R, G, B, H, S, and V) has a cell which is bound to a slider (<input type="range">). All six sliders are identified as color, so adjusting any of them increments color’s timestamp. The R, G, and B sliders form one input group, and the H, S, and V sliders form another. By grouping the related sliders together, whenever one of the sliders is moved, all members of the group will have their ephemeral value refreshed with the latest timestamp. This behavior is crucial because otherwise the r:components<->rgb and r:components<->hsv propagators would see that one color channel has a fresher value than the other two and do nothing. Since the underlying propagator infrastructure does not enforce activation order, reactive propagators must wait until their inputs reach a consistent state where the timestamps for a given reactive identifier are all the same. With this setup, changing a slider on the RGB side will cause a new color value to propagate over to the HSV side. Because the relationship is cyclical, the HSV side will then attempt to propagate an equivalent color value back to the RGB side! This could be bad news, but since the current RGB value is equally fresh (same timestamp), the propagation stops right there. Redundant work is minimized and an unbounded loop is avoided. And that’s it! Phew! Complete source code can be found here. Reflections I think the results of this prototype are promising. I’d like to try building some larger demos to see what new problems arise. Since propagation networks include cycles, they cannot be garbage collected until there are no references to any part of the network from the outside. Is this acceptable? I didn’t optimize, either. A more serious implementation would want to do things like use case-lambda for all n-ary procedures to avoid consing an argument list in the common cases of 1, 2, 3, etc. arguments. There is also a need for a more pleasing domain-specific language, using Scheme’s macro system, for describing FRP graphs. Alexey Radul’s dissertation was published in 2009. Has anyone made a FRP system based on propagators since then that’s used in real software? I don’t know of anything but it’s a big information superhighway out there. Update: Since publishing, I have learned about the following: Holograph: A visual editor for propagator networks! Amazing! Scoped Propagators: A WIP propagator system with some notable differences from “traditional” propagators. I wish I had read Alexey Radul's disseration 10 years ago when I was first exploring FRP. It would have saved me a lot of time spent running into problems that have already been solved that I was not equipped to solve on my own. I have even talked to Gerald Sussman (a key figure in propagator research) in person about the propagator model. That conversation was focused on AI, though, and I didn’t realize that propagators could also be used for FRP. It wasn’t until more recently that friend and colleague Christine Lemmer-Webber, who was present for the aforementioned conversation with Sussman, told me about it. Christine has her own research project for propagators. There are so many interesting things to learn out there, but I am also so tired. Better late than never, I guess! Anyway, if you made it this far then I hope you have enjoyed reading about propagators and FRP. ’Til next time!

a year ago 160 votes

More in programming

Increase software sales by 50% or more

This is re-post of How to Permanently Increase Your Sales by 50% or More in Only One Day article by Steve Pavlina Of all the things you can do to increase your sales, one of the highest leverage activities is attempting to increase your products’ registration rate. Increasing your registration rate from 1.0% to 1.5% means that you simply convince one more downloader out of every 200 to make the decision to buy. Yet that same tiny increase will literally increase your sales by a full 50%. If you’re one of those developers who simply slapped the ubiquitous 30-day trial incentive on your shareware products without going any further than that, then I think a 50% increase in your registration rate is a very attainable goal you can achieve if you spend just one full day of concentrated effort on improving your product’s ability to sell. My hope is that this article will get you off to a good start and get you thinking more creatively. And even if you fail, your result might be that you achieve only a 25% or a 10% increase. How much additional money would that represent to you over the next five years of sales? What influence, if any, did the title of this article have on your decision to read it? If I had titled this article, “Registration Incentives,” would you have been more or less likely to read it now? Note that the title expresses a specific and clear benefit to you. It tells you exactly what you can expect to gain by reading it. Effective registration incentives work the same way. They offer clear, specific benefits to the user if a purchase is made. In order to improve your registration incentives, the first thing you need to do is to adopt some new beliefs that will change your perspective. I’m going to introduce you to what I call the “lies of success” in the shareware industry. These are statements that are not true at all, but if you accept them as true anyway, you’ll achieve far better results than if you don’t. Rule 1: What you are selling is merely the difference between the shareware and the registered versions, not the registered version itself. Note that this is not a true statement, but if you accept it as true, you’ll immediately begin to see the weaknesses in your registration incentives. If there are few additional benefits for buying the full version vs. using the shareware version, then you aren’t offering the user strong enough incentives to make the full purchase. Rule 2: The sole purpose of the shareware version is to close the sale. This is our second lie of success. Note the emphasis on the word “close.” Your shareware version needs to act as a direct sales vehicle. It must be able to take the user all the way to the point of purchase, i.e. your online order form, ideally with nothing more than a few mouse clicks. Anything that detracts from achieving a quick sale is likely to hurt sales. Rule 3: The customer’s perspective is the only one that matters. Defy this rule at your peril. Customers don’t care that you spent 2000 hours creating your product. Customers don’t care that you deserve the money for your hard work. Customers don’t care that you need to do certain things to prevent piracy. All that matters to them are their own personal wants and needs. Yes, these are lies of success. Some customers will care, but if you design your registration incentives assuming they only care about their own self-interests, your motivation to buy will be much stronger than if you merely appeal to their sense of honesty, loyalty, or honor. Assume your customers are all asking, “What’s in it for me if I choose to buy? What will I get? How will this help me?” I don’t care if you’re selling to Fortune 500 companies. At some point there will be an individual responsible for causing the purchase to happen, and that individual is going to consider how the purchase will affect him/her personally: “Will this purchase get me fired? Will it make me look good in front of my peers? Will this make my job easier or harder?” Many shareware developers get caught in the trap of discriminating between honest and dishonest users, believing that honest users will register and dishonest ones won’t. This line of thinking will ultimately get you nowhere, and it violates the third lie of success. When you make a purchase decision, how often do you use honesty as the deciding factor? Do you ever say, “I will buy this because I’m honest?” Or do you consider other more selfish factors first, such as how it will make you feel to purchase the software? The truth is that every user believes s/he is honest, so no user applies the honesty criterion when making a purchase decision. Thinking of your users in terms of honest ones vs. dishonest ones is a complete waste of time because that’s not how users primarily view themselves. Rule 4: Customers buy on emotion and justify with fact. If you’re honest with yourself, you’ll see that this is how you make most purchase decisions. Remember the last time you bought a computer. Is it fair to say that you first became emotionally attached to the idea of owning a new machine? For me, it’s the feeling of working faster, owning the latest technology, and being more productive that motivates me to go computer shopping. Once I’ve become emotionally committed, the justifications follow: “It’s been two years since I’ve upgraded, it will pay for itself with the productivity boost I gain, I can easily afford it, I’ve worked hard and I deserve a new machine, etc.” You use facts to justify the purchase. Once you understand how purchase decisions are made, you can see that your shareware products need to first get the user emotionally invested in the purchase, and then you give them all the facts they need to justify it. Now that we’ve gotten these four lies of success out of the way, let’s see how we might apply them to create some compelling registration incentives. Let’s start with Rule 1. What incentives can be spawned from this rule? The common 30-day trial is one obvious derivative. If you are only selling the difference between the shareware and registered versions, then a 30-day trial implies that you are selling unlimited future days of usage of the program after the trial period expires. This is a powerful incentive, and it’s been proven effective for products that users will continue to use month after month. 30-day trials are easy for users to understand, and they’re also easy to implement. You could also experiment with other time periods such as 10 days, 14 days, or 90 days. The only way of truly knowing which will work best for your products is to experiment. But let’s see if we can move a bit beyond the basic 30-day trial here by mixing in a little of Rule 3. How would the customer perceive a 30-day trial? In most cases 30 days is plenty of time to evaluate a product. But in what situations would a 30-day trial have a negative effect? A good example is when the user downloads, installs, and briefly checks out a product s/he may not have time to evaluate right away. By the time the user gets around to fully evaluating it, the shareware version has already expired, and a sale may be lost as a result. To get around this limitation, many shareware developers have started offering 30 days of actual program usage instead of 30 consecutive days. This allows the user plenty of time to try out the program at his/her convenience. Another possibility would be to limit the number of times the program can be run. The basic idea is that you are giving away limited usage and selling unlimited usage of the program. This incentive definitely works if your product is one that will be used frequently over a long period of time (much longer than the trial period). The flip side of usage limitation is to offer an additional bonus for buying within a certain period of time. For instance, in my game Dweep, I offer an extra 5 free bonus levels to everyone who buys within the first 10 days. In truth I give the bonus levels to everyone who buys, but the incentive is real from the customer’s point of view. Remember Rule 3 - it doesn’t matter what happens on my end; it only matters what the customer perceives. Any customer that buys after the first 10 days will be delighted anyway to receive a bonus they thought they missed. So if your product has no time-based incentives at all, this is the first place to start. When would you pay your bills if they were never due, and no interest was charged on late payments? Use time pressure to your advantage, either by disabling features in the shareware version after a certain time or by offering additional bonuses for buying sooner rather than later. If nothing else and if it’s legal in your area, offer a free entry in a random monthly drawing for a small prize, such as one of your other products, for anyone who buys within the first X days. Another logical derivative of Rule 1 is the concept of feature limitation. On the crippling side, you can start with the registered version and begin disabling functionality to create the shareware version. Disabling printing in a shareware text editor is a common strategy. So is corrupting your program’s output with a simple watermark. For instance, your shareware editor could print every page with your logo in the background. Years ago the Association of Shareware Professionals had a strict policy against crippling, but that policy was abandoned, and crippling has been recognized as an effective registration incentive. It is certainly possible to apply feature limitation without having it perceived as crippling. This is especially easy for games, which commonly offer a limited number of playable levels in the shareware version with many more levels available only in the registered version. In this situation you offer the user a seemingly complete experience of your product in the shareware version, and you provide additional features on top of that for the registered version. Time-based incentives and feature-based incentives are perhaps the two most common strategies used by shareware developers for enticing users to buy. Which will work best for you? You will probably see the best results if you use both at the same time. Imagine you’re the end user for a moment. Would you be more likely to buy if you were promised additional features and given a deadline to make the decision? I’ve seen several developers who were using only one of these two strategies increase their registration rates dramatically by applying the second strategy on top of the first. If you only use time-based limitations, how could you apply feature limitation as well? Giving the user more reasons to buy will translate to more sales per download. One you have both time-based and feature-based incentives to buy, the next step is to address the user’s perceived risk by applying a risk-reversal strategy. Fortunately, the shareware model already reduces the perceived risk of purchasing significantly, since the user is able to try before buying. But let’s go a little further, keeping Rule 3 in mind. What else might be a perceived risk to the user? What if the user reaches the end of the trial period and still isn’t certain the product will do what s/he needs? What if the additional features in the registered version don’t work as the user expects? What can we do to make the decision to purchase safer for the user? One approach is to offer a money-back guarantee. I’ve been offering a 60-day unconditional money-back guarantee on all my products since January 2000. If someone asks for their money back for any reason, I give them a full refund right away. So what is my return rate? Well, it’s about 8%. Just kidding! Would it surprise you to learn that my return rate at the time of this writing is less than 0.2%? Could you handle two returns out of every 1000 sales? My best estimate is that this one technique increased my sales by 5-10%, and it only took a few minutes to implement. When I suggest this strategy to other shareware developers, the usual reaction is fear. “But everyone would rip me off,” is a common response. I suggest trying it for yourself on an experimental basis; a few brave souls have already tried it and are now offering money-back guarantees prominently. Try putting it up on your web site for a while just to convince yourself it works. You can take it down at any time. After a few months, if you’re happy with the results, add the guarantee to your shareware products as well. I haven’t heard of one bad outcome yet from those who’ve tried it. If you use feature limitation in your shareware products, another important component of risk reversal is to show the user exactly what s/he will get in the full version. In Dweep I give away the first five levels in the demo version, and purchasing the full version gets you 147 more levels. When I thought about this from the customer’s perspective (Rule 3), I realized that a perceived risk is that s/he doesn’t know if the registered version levels will be as fun as the demo levels. So I released a new demo where you can see every level but only play the first five. This lets the customer see all the fun that awaits them. So if you have a feature-limited product, show the customer how the feature will work. For instance, if your shareware version has printing disabled, the customer could be worried that the full version’s print capability won’t work with his/her printer or that the output quality will be poor. A better strategy is to allow printing, but to watermark the output. This way the customer can still test and verify the feature, and it doesn’t take much imagination to realize what the output will look like without the watermark. Our next step is to consider Rule 2 and include the ability close the sale. It is imperative that you include an “instant gratification” button in your shareware products, so the customer can click to launch their default web browser and go directly to your online order form. If you already have a “buy now” button in your products, go a step further. A small group of us have been finding that the more liberally these buttons are used, the better. If you only have one or two of these buttons in your shareware program, you should increase the count by at least an order of magnitude. The current Dweep demo now has over 100 of these buttons scattered throughout the menus and dialogs. This makes it extremely easy for the customer to buy, since s/he never has to hunt around for the ordering link. What should you label these buttons? “Buy now” or “Register now” are popular, so feel free to use one of those. I took a slightly different approach by trying to think like a customer (Rule 3 again). As a customer the word “buy” has a slightly negative association for me. It makes me think of parting with my cash, and it brings up feelings of sacrifice and pressure. The words “buy now” imply that I have to give away something. So instead, I use the words, “Get now.” As a customer I feel much better about getting something than buying something, since “getting” brings up only positive associations. This is the psychology I use, but at present, I don’t know of any hard data showing which is better. Unless you have a strong preference, trust your intuition. Make it as easy as possible for the willing customer to buy. The more methods of payment you accept, the better your sales will be. Allow the customer to click a button to print an order form directly from your program and mail it with a check or money order. On your web order form, include a link to a printable text order form for those who are afraid to use their credit cards online. If you only accept two or three major credit cards, sign up with a registration service to handle orders for those you don’t accept. So far we’ve given the customer some good incentives to buy, minimized perceived risk, and made it easy to make the purchase. But we haven’t yet gotten the customer emotionally invested in making the purchase decision. That’s where Rule 4 comes in. First, we must recognize the difference between benefits and features. We need to sell the sizzle, not the steak. Features describe your product, while benefits describe what the user will get by using your product. For instance, a personal information manager (PIM) program may have features such as daily, weekly, and monthly views; task and event timers; and a contact database. However, the benefits of the program might be that it helps the user be more organized, earn more money, and enjoy more free time. For a game, the main benefit might be fun. For a nature screensaver, it could be relaxation, beauty appreciation, or peace. Features are logical; benefits are emotional. Logical features are an important part of the sale, but only after we’ve engaged the customer’s emotions. Many products do a fair job of getting the customer emotionally invested during the trial period. If you have an addictive program or one that’s fun to use, such as a game, you may have an easy time getting the customer emotionally attached to using it because the experience is already emotional in nature. But whatever your product is, you can increase your sales by clearly illustrating the benefits of making the purchase. A good place to do this is in your nag screens. I use nag screens both before and after the program runs to remind the user of the benefits of buying the full version. At the very least, include a nag screen when the customer exits the program, so the last thing s/he sees will be a reminder of the product’s benefits. Take this opportunity to sell the user on the product. Don’t expect features like “customizable colors” to motivate anyone to buy. Paint a picture of what benefits the user will obtain with the full version. Will I save time? Will I have more fun? Will I live longer, save money, or feel better? The simple change from feature-oriented selling to benefit-oriented selling can easily double or triple your sales. Be sure to use this approach on your web site as well if you don’t already. Developers who’ve recently made the switch have been reporting some amazing results. If you’re drawing a blank when trying to come up with benefits for your products, the best thing you can do is to email some of your old customers and ask them why they bought your program. What did it do for them? I’ve done this and was amazed at the answers I got back. People were buying my games for reasons I’d never anticipated, and that told me which benefits I needed to emphasize in my sales pitch. The next key is to make your offer irresistible to potential customers. Find ways to offer the customer so much value that it would be harder to say no than to say yes. Take a look at your shareware product as if you were a potential customer who’d never seen it before. Being totally honest with yourself, would you buy this program if someone else had written it? If not, don’t stop here. As a potential customer, what additional benefits or features would put you over the top and convince you to buy? More is always better than less. In the original version of Dweep, I offered ten levels in the demo and thirty in the registered version. Now I offer only five demo levels and 152 in the full version, plus a built-in level editor. Originally, I offered the player twice the value of the demo; now I’m offering over thirty times the value. I also offer free hints and solutions to every level; the benefit here is that it minimizes player frustration. As I keep adding bonuses for purchasing, the offer becomes harder and harder to resist. What clever bonuses can you throw in for registering? Take the time to watch an infomercial. Notice that there is always at least one “FREE” bonus thrown in. Consider offering a few extra filters for an image editor, ten extra images for a screensaver, or extra levels for a game. What else might appeal to your customers? Be creative. Your bonus doesn’t even have to be software-based. Offer a free report about building site traffic with your HTML editor, include an essay on effective time management with your scheduling program, or throw in a small business success guide with your billing program. If you make such programs, you shouldn’t have too much trouble coming up with a few pages of text that would benefit your customers. Keep working at it until your offer even looks irresistible to you. If all the bonuses you offer can be delivered electronically, how many can you afford to include? If each one only gains one more customer in a thousand (0.1%), would it be worth the effort over the lifetime of your sales? So how do you know if your registration incentives are strong enough? And how do you know if your product is over-crippled? Where do you draw the line? These are tough issues, but there is a good way to handle them if your product is likely to be used over a long period of time, particularly if it’s used on a daily basis. Simply make your program gradually increase its registration incentives over time. One easy way to do this is with a delay timer on your nag screens that increases each time the program is run. Another approach is to disable certain features at set intervals. You begin by disabling non-critical features and gradually move up to disabling key functionality. The program becomes harder and harder to continue using for free, so the benefits of registering become more and more compelling. Instead of having your program completely disable itself after your trial period, you gradually degrade its usability with additional usage. This approach can be superior to a strict 30-day trial, since it allows your program to still be used for a while, but after prolonged usage it becomes effectively unusable. However, you don’t simply shock the user by taking away all the benefits s/he has become accustomed to on a particular day. Instead, you begin with a gentle reminder that becomes harder and harder to ignore. There may be times when your 30-day trial shuts off at an inconvenient time for the user, and you may lose a sale as a result. For instance, the user may not have the money at the time, or s/he may be busy at the trial’s end and forget to register. In that case s/he may quickly replace what was lost with a competitor’s trial version. The gradual degradation approach allows the user to continue using your product, but with increasing difficulty over time. Eventually, there is a breaking point where the user either decides to buy or to stop using the program completely, but this can be done within a window of time at the user’s convenience. Hopefully this article has gotten you thinking creatively about all the overlooked ways you can entice people to buy your shareware products. The most important thing you can do is to begin seeing your products through your customers’ eyes. What additional motivation would convince you to buy? What would represent an irresistible offer to you? There is no limit to how many incentives you can add. Don’t stop at just one or two; instead, give the customer a half dozen or more reasons to buy, and you’ll see your registration rate soar. Is it worth spending a day to do this? I think so.

yesterday 4 votes
Maybe writing speed actually is a bottleneck for programming

I'm a big (neo)vim buff. My config is over 1500 lines and I regularly write new scripts. I recently ported my neovim config to a new laptop. Before then, I was using VSCode to write, and when I switched back I immediately saw a big gain in productivity. People often pooh-pooh vim (and other assistive writing technologies) by saying that writing code isn't the bottleneck in software development. Reading, understanding, and thinking through code is! Now I don't know how true this actually is in practice, because empirical studies of time spent coding are all over the place. Most of them, like this study, track time spent in the editor but don't distinguish between time spent reading code and time spent writing code. The only one I found that separates them was this study. It finds that developers spend only 5% of their time editing. It also finds they spend 14% of their time moving or resizing editor windows, so I don't know how clean their data is. But I have a bigger problem with "writing is not the bottleneck": when I think of a bottleneck, I imagine that no amount of improvement will lead to productivity gains. Like if a program is bottlenecked on the network, it isn't going to get noticeably faster with 100x more ram or compute. But being able to type code 100x faster, even with without corresponding improvements to reading and imagining code, would be huge. We'll assume the average developer writes at 80 words per minute, at five characters a word, for 400 characters a minute.What could we do if we instead wrote at 8,000 words/40k characters a minute? Writing fast Boilerplate is trivial Why do people like type inference? Because writing all of the types manually is annoying. Why don't people like boilerplate? Because it's annoying to write every damn time. Programmers like features that help them write less! That's not a problem if you can write all of the boilerplate in 0.1 seconds. You still have the problem of reading boilerplate heavy code, but you can use the remaining 0.9 seconds to churn out an extension that parses the file and presents the boilerplate in a more legible fashion. We can write more tooling This is something I've noticed with LLMs: when I can churn out crappy code as a free action, I use that to write lots of tools that assist me in writing good code. Even if I'm bottlenecked on a large program, I can still quickly write a script that helps me with something. Most of these aren't things I would have written because they'd take too long to write! Again, not the best comparison, because LLMs also shortcut learning the relevant APIs, so also optimize the "understanding code" part. Then again, if I could type real fast I could more quickly whip up experiments on new apis to learn them faster. We can do practices that slow us down in the short-term Something like test-driven development significantly slows down how fast you write production code, because you have to spend a lot more time writing test code. Pair programming trades speed of writing code for speed of understanding code. A two-order-of-magnitude writing speedup makes both of them effectively free. Or, if you're not an eXtreme Programming fan, you can more easily follow the The Power of Ten Rules and blanket your code with contracts and assertions. We could do more speculative editing This is probably the biggest difference in how we'd work if we could write 100x faster: it'd be much easier to try changes to the code to see if they're good ideas in the first place. How often have I tried optimizing something, only to find out it didn't make a difference? How often have I done a refactoring only to end up with lower-quality code overall? Too often. Over time it makes me prefer to try things that I know will work, and only "speculatively edit" when I think it be a fast change. If I could code 100x faster it would absolutely lead to me trying more speculative edits. This is especially big because I believe that lots of speculative edits are high-risk, high-reward: given 50 things we could do to the code, 49 won't make a difference and one will be a major improvement. If I only have time to try five things, I have a 10% chance of hitting the jackpot. If I can try 500 things I will get that reward every single time. Processes are built off constraints There are just a few ideas I came up with; there are probably others. Most of them, I suspect, will share the same property in common: they change the process of writing code to leverage the speedup. I can totally believe that a large speedup would not remove a bottleneck in the processes we currently use to write code. But that's because those processes are developed work within our existing constraints. Remove a constraint and new processes become possible. The way I see it, if our current process produces 1 Utils of Software / day, a 100x writing speedup might lead to only 1.5 UoS/day. But there are other processes that produce only 0.5 UoS/d because they are bottlenecked on writing speed. A 100x speedup would lead to 10 UoS/day. The problem with all of this that 100x speedup isn't realistic, and it's not obvious whether a 2x improvement would lead to better processes. Then again, one of the first custom vim function scripts I wrote was an aid to writing unit tests in a particular codebase, and it lead to me writing a lot more tests. So maybe even a 2x speedup is going to be speed things up, too. Patreon Stuff I wrote a couple of TLA+ specs to show how to model fork-join algorithms. I'm planning on eventually writing them up for my blog/learntla but it'll be a while, so if you want to see them in the meantime I put them up on Patreon.

2 days ago 6 votes
Occupation and Preoccupation

Here’s Jony Ive in his Stripe interview: What we make stands testament to who we are. What we make describes our values. It describes our preoccupations. It describes beautiful succinctly our preoccupation. I’d never really noticed the connection between these two words: occupation and preoccupation. What comes before occupation? Pre-occupation. What comes before what you do for a living? What you think about. What you’re preoccupied with. What you think about will drive you towards what you work on. So when you’re asking yourself, “What comes next? What should I work on?” Another way of asking that question is, “What occupies my thinking right now?” And if what you’re occupied with doesn’t align with what you’re preoccupied with, perhaps it's time for a change. Email · Mastodon · Bluesky

2 days ago 3 votes
American hype

There's no country on earth that does hype better than America. It's one of the most appealing aspects about being here. People are genuinely excited about the future and never stop searching for better ways to work, live, entertain, and profit. There's a unique critical mass in the US accelerating and celebrating tomorrow. The contrast to Europe couldn't be greater. Most Europeans are allergic to anything that even smells like a commercial promise of a better tomorrow. "Hype" is universally used as a term to ridicule anyone who dares to be excited about something new, something different. Only a fool would believe that real progress is possible! This is cultural bedrock. The fault lines have been settling for generations. It'll take an earthquake to move them. You see this in AI, you saw it in the Internet. Europeans are just as smart, just as inventive as their American brethren, but they don't do hype, so they're rarely the ones able to sell the sizzle that public opinion requires to shift its vision for tomorrow.  To say I have a complicated relationship with venture capital is putting it mildly. I've spent a career proving the counter narrative. Proving that you can build and bootstrap an incredible business without investor money, still leave a dent in the universe, while enjoying the spoils of capitalism. And yet... I must admit that the excesses of venture capital are integral to this uniquely American advantage on hype. The lavish overspending during the dot-com boom led directly to a spectacular bust, but it also built the foundation of the internet we all enjoy today. Pets.com and Webvan flamed out such that Amazon and Shopify could transform ecommerce out of the ashes. We're in the thick of peak hype on AI right now. Fantastical sums are chasing AGI along with every dumb derivative mirage along the way. The most outrageous claims are being put forth on the daily. It's easy to look at that spectacle with European eyes and roll them. Some of it is pretty cringe! But I think that would be a mistake. You don't have to throw away your critical reasoning to accept that in the face of unknown potential, optimism beats pessimism. We all have to believe in something, and you're much better off believing that things can get better than not.  Americans fundamentally believe this. They believe the hype, so they make it come to fruition. Not every time, not all of them, but more of them, more of the time than any other country in the world. That really is exceptional.

2 days ago 3 votes
File sync is very slow

I’m working on a Go library appendstore for append-only store of lots of things in a single file. To make things as robust as possible I was calling os.File.Sync() after each append. Sync() is waiting until the data is acknowledged as truly, really written to disk (as opposed to maybe floating somewhere in disk drive’s write buffer). Oh boy, is it slow. A test of appending 1000 records would take over 5 seconds. After removing the Sync() it would drop to 5 milliseconds. 1000x faster. I made sync optional - it’s now up to the user of the library to pick it, defaults to non-sync. Is it unsafe now? Well, the reality is that it probably doesn’t matter. I don’t think lots of software does the sync due to slowness and the world still runs.

2 days ago 2 votes